tasks
This commit is contained in:
@@ -2,8 +2,23 @@
|
||||
|
||||
**Feature Branch**: `017-llm-analysis-plugin`
|
||||
**Created**: 2026-01-28
|
||||
**Status**: Implemented (plugin, service, scheduler, provider config; ~1714 LOC in core files)
|
||||
**Input**: User description: "LLM Dashboard Validation Plugin для интеграции LLM в ss-tools. Плагин должен поддерживать анализ корректности работы дашбордов через мультимодальную LLM (скриншот + логи). Второй плагин должен заниматься документированием датасетов и дашбордов, используя LLM. Возможно включение в создание текста коммитов в плагине Git Поддержка провайдеров: OpenRouter, Kilo Provider, OpenAI API. Интеграция с существующей PluginBase архитектурой, Task Manager, WebSocket логами."
|
||||
**Updated**: 2026-05-31 — v2: task-based flow, dual-path execution (screenshot + text-only), URL-based dashboards, dataset health checking
|
||||
**Status**: Specification updated; implementation pending for v2 changes
|
||||
|
||||
## v2 Design Decisions
|
||||
|
||||
| # | Decision | Rationale |
|
||||
|---|----------|-----------|
|
||||
| D1 | **Remove one-click «Validate» button** — replace with «Create Validation Task» flow | Ad-hoc validation doesn't scale; users need scheduled, repeatable validation with full configuration control |
|
||||
| D2 | **Dual execution paths**: Path A (Playwright screenshot) + Path B (text-only API) | Screenshot costs image tokens but catches visual bugs; text-only is lightweight, catches data/KXD errors via dataset health checks |
|
||||
| D3 | **Screenshot is optional** (`screenshot_enabled` flag per task) | Lets users trade off visual bug detection vs cost/speed; text-only validates KXD→dataset→chart data chain |
|
||||
| D4 | **Dataset health checking in Path B** — execute `GET /api/v1/dataset/{id}` for every unique dataset, optionally `POST /api/v1/chart/data` | Core goal: verify dashboard opens correctly with KXD data. Without dataset checks, silent KXD failures are invisible |
|
||||
| D5 | **URL-based dashboard selection** — paste Superset URL, parse via `SupersetContextExtractor` (same code as dataset review) | Preserves native_filters, activeTabs, anchor — LLM validates dashboard in exact user-visible state |
|
||||
| D6 | **Provider selection moves from settings to task creation form** | Each task gets explicit provider; no implicit default. Provider bindings for `dashboard_validation` removed from LLM settings |
|
||||
| D7 | **Prompt template moves from settings to task creation form** | Users customize prompts per task, not globally. Default from `DEFAULT_LLM_PROMPTS` |
|
||||
| D8 | **LLM settings retain only**: documentation, git_commit, assistant_planner provider bindings | Dashboard validation provider is per-task, not global |
|
||||
| D9 | **Multi-chunk analysis** — dashboard sliced into logical blocks (tabs + charts) | Multiple screenshots in single LLM request; each block at high resolution instead of one compressed full-page |
|
||||
| D10 | **Image pipeline: PNG → JPEG (LLM) + WebP (archive)** — CDP captures PNG, converted to JPEG for LLM, archived as WebP lossy q=80 for storage efficiency, PNG and JPEG deleted after | JPEG universally supported by all LLM providers; WebP saves ~93% disk space vs PNG. Silent WebP failure risk on unknown providers eliminated by JPEG-only LLM path |
|
||||
|
||||
## Clarifications
|
||||
|
||||
@@ -14,126 +29,187 @@
|
||||
- Q: Git Commit Message Context → A: **Diff + Recent History**: Send diff, file names, and the last 3 commit messages to match style.
|
||||
- Q: LLM Failure Handling → A: **Retry then Fail**: Automatically retry 3 times with exponential backoff before failing.
|
||||
|
||||
## User Scenarios & Testing *(mandatory)*
|
||||
### Session 2026-05-31
|
||||
- Q: Concurrent execution — manual trigger while task is running? → Same task: toast «Already running», no queue. Different task: FIFO queue, respecting concurrency_limit.
|
||||
- Q: API key rotation — impact on existing tasks? → Tasks fetch current key at execution time from LLMProviderService. No key caching in task config.
|
||||
- Q: Multi-chunk token limit — 15+ tabs exceeding provider limit? → Estimate payload → progressive quality reduction (30% JPEG, 800px) → Path B fallback. Molecular-cot logging: REASON before each reduction, REFLECT after.
|
||||
- Q: Backward compat — old POST /api/tasks with llm_dashboard_validation? → Hard cut: returns 400. All consumers must migrate to POST /api/validation-tasks.
|
||||
- Q: URL re-parse — dashboard structure changed between runs? → Re-parse every execution. URL failure → warning «Paste new URL». Dashboard ID change → WARN log, use new ID.
|
||||
- Q: Provider deletion — what if used by active tasks? → Forbidden. Returns list of blocking task names. User must reassign or deactivate tasks first.
|
||||
- Q: Screenshot format — PNG vs WebP vs JPEG? → **Pipeline**: CDP captures PNG → converted to JPEG (LLM, universal compatibility) + WebP lossy q=80 (archive, 93% disk savings). PNG and JPEG deleted after WebP saved. LLM path and archive path are decoupled.
|
||||
|
||||
### User Story 1 - Dashboard Health Analysis (Priority: P1)
|
||||
## User Scenarios & Testing
|
||||
|
||||
As a Data Engineer, I want to automatically analyze a dashboard's status using visual and log data directly from the Environments interface so that I can identify rendering issues or data errors without manual inspection.
|
||||
### User Story 1 — Create & Schedule Dashboard Validation Task (Priority: P1)
|
||||
|
||||
**Why this priority**: Core value proposition of the feature. Enables automated quality assurance.
|
||||
As a Data Engineer, I want to create a validation task that checks multiple dashboards for health issues on a schedule, so that I can proactively detect KXD data problems, rendering failures, and chart errors without manual inspection.
|
||||
|
||||
**Independent Test**: Can be tested by selecting a dashboard in the Environment list and clicking "Validate", or by scheduling a validation task.
|
||||
**Why this priority**: Core value proposition. Replaces unreliable one-off manual checks with automated, scheduled validation covering the full KXD→dataset→chart→dashboard chain.
|
||||
|
||||
**Independent Test**: Create a validation task with 2 dashboards (one via ID, one via URL), set cron schedule `0 8 * * 1-5`, wait for first execution, verify report shows dataset health status per dashboard.
|
||||
|
||||
**Acceptance Scenarios**:
|
||||
|
||||
1. **Given** I am on the Environments page (Dashboard list), **When** I select a dashboard and click "Validate", **Then** the system triggers a validation task with the dashboard's context.
|
||||
2. **Given** the validation task is running, **When** it completes, **Then** I see the analysis report (visual + logs) in the task output history.
|
||||
3. **Given** I want regular checks, **When** I configure a schedule for the validation task (similar to Backup plugin), **Then** the system runs the check automatically at the specified interval.
|
||||
4. **Given** a validation issue is found, **When** the task completes, **Then** the system sends a notification (Email/Pulse) if configured.
|
||||
1. **Given** I am on the Validation Tasks page, **When** I click «+ New Task», **Then** the task creation form opens with fields: name, environment, sources (dashboard IDs + URLs), provider, prompt, screenshot toggle, schedule.
|
||||
2. **Given** I paste a Superset dashboard URL with native filters, **When** the form parses it, **Then** the system extracts dashboard_id, native_filters, activeTabs, and anchor — and validates the URL is accessible.
|
||||
3. **Given** I enable `screenshot_enabled`, **When** the task executes, **Then** the system uses Path A: Playwright browser → CDP screenshot → multimodal LLM analysis.
|
||||
4. **Given** I disable `screenshot_enabled`, **When** the task executes, **Then** the system uses Path B: API calls → dashboard structure + chart params + dataset schema (+ optional chart data execution) → text-only LLM analysis.
|
||||
5. **Given** the task has a cron schedule, **When** the schedule triggers, **Then** the system runs validation on all configured dashboards and persists results.
|
||||
6. **Given** a validation finds issues, **When** the task completes, **Then** the system sends a notification (Email/Pulse) if configured.
|
||||
|
||||
---
|
||||
### User Story 2 — Validation Task List & History (Priority: P1)
|
||||
|
||||
### User Story 2 - Automated Dataset Documentation (Priority: P1)
|
||||
As a Data Engineer, I want to see all my validation tasks, their last run status, and drill into historical results per dashboard, so that I can track health trends over time.
|
||||
|
||||
As a Data Steward, I want to generate documentation for datasets and dashboards using LLMs so that I can maintain up-to-date metadata with minimal manual effort.
|
||||
**Why this priority**: Without a list view and history, the task-based flow is invisible — users can't monitor what's running.
|
||||
|
||||
**Why this priority**: significantly reduces maintenance overhead for data governance.
|
||||
|
||||
**Independent Test**: Can be tested by selecting a dataset and triggering the documentation task, then checking if a description is generated.
|
||||
**Independent Test**: View the task list, see status per task (active/inactive), last run time, aggregated status across dashboards.
|
||||
|
||||
**Acceptance Scenarios**:
|
||||
|
||||
1. **Given** a dataset identifier, **When** I run the Documentation task, **Then** the system fetches the dataset's schema and metadata.
|
||||
2. **Given** the metadata is fetched, **When** sent to the LLM, **Then** a structured description/documentation text is returned.
|
||||
3. **Given** the documentation is generated, **When** the task completes, **Then** the result is available for review (e.g., in the task log or saved to a file/db).
|
||||
1. **Given** I am on the Validation Tasks page, **When** the page loads, **Then** I see all created tasks with: name, schedule, last run status, dashboard count.
|
||||
2. **Given** I click a task, **When** the detail opens, **Then** I see per-dashboard breakdown: status (PASS/WARN/FAIL), detected issues, last run timestamp, link to full report.
|
||||
3. **Given** I click a dashboard in the task detail, **When** the report opens, **Then** I see: check result, text summary, issues table, dataset health status (Path B), screenshot thumbnails (Path A), source logs, execution logs.
|
||||
|
||||
---
|
||||
### User Story 3 — Dual-Path Dashboard Health Analysis (Priority: P1)
|
||||
|
||||
### User Story 3 - LLM Provider Configuration (Priority: P1)
|
||||
As a Data Engineer, I want the validation to check both visual rendering AND data availability, so that I catch KXD backend failures that produce no visual error but return empty/incomplete data.
|
||||
|
||||
As an Administrator, I want to configure different LLM providers (OpenAI, OpenRouter, Kilo) so that I can switch between models based on cost or capability.
|
||||
**Why this priority**: The primary purpose of validation is verifying the KXD→dataset→chart data chain works. Screenshot-only validation misses silent data failures (empty results, partial data, wrong time range).
|
||||
|
||||
**Why this priority**: Prerequisite for any LLM functionality.
|
||||
|
||||
**Independent Test**: Can be tested by entering API keys in settings and verifying a connection/test call.
|
||||
**Independent Test**: Create a task with `screenshot_enabled=false`, run against a dashboard with a known broken dataset, verify the LLM reports the dataset health issue.
|
||||
|
||||
**Acceptance Scenarios**:
|
||||
|
||||
1. **Given** the Settings page, **When** I select an LLM provider (e.g., OpenAI) and enter an API key, **Then** the system saves the configuration.
|
||||
2. **Given** a configured provider, **When** I run an analysis task, **Then** the system uses the selected provider for API calls.
|
||||
1. **Given** Path B execution, **When** a dataset `GET /api/v1/dataset/{id}` returns KXD connection error, **Then** the LLM report flags the dataset as unhealthy and lists affected charts.
|
||||
2. **Given** Path B execution with chart data execution enabled, **When** `POST /api/v1/chart/data` times out, **Then** the LLM report includes the timeout with affected chart info.
|
||||
3. **Given** Path B execution, **When** all datasets are healthy and logs show no errors, **Then** the LLM report returns `status: PASS` with a summary of verified data sources.
|
||||
4. **Given** Path A (screenshot), **When** LLM analyzes the multi-chunk screenshots, **Then** visual issues (empty charts, rendering errors, overlap) are detected and reported with tab/chart location.
|
||||
|
||||
---
|
||||
### User Story 4 — Automated Dataset Documentation (Priority: P2)
|
||||
|
||||
### User Story 4 - Git Commit Message Suggestion (Priority: P3)
|
||||
*(unchanged from v1)*
|
||||
|
||||
As a Developer, I want the system to suggest commit messages based on changes directly within the Git plugin interface so that I can maintain consistent history with minimal effort.
|
||||
As a Data Steward, I want to generate documentation for datasets and dashboards using LLMs.
|
||||
|
||||
**Why this priority**: Enhances the existing Git workflow and improves commit quality.
|
||||
### User Story 5 — LLM Provider Configuration (Priority: P2)
|
||||
|
||||
**Independent Test**: Can be tested by staging files in the Git plugin and clicking the "Generate Message" button.
|
||||
*(unchanged from v1 — but provider bindings for dashboard_validation removed)*
|
||||
|
||||
**Acceptance Scenarios**:
|
||||
As an Administrator, I want to configure LLM providers for documentation, git commits, and chat — with dashboard validation provider configured per-task.
|
||||
|
||||
1. **Given** staged changes in the Git plugin, **When** I click "Generate Message", **Then** the system analyzes the diff using the configured LLM and populates the commit message field with a suggested summary.
|
||||
### User Story 6 — Git Commit Message Suggestion (Priority: P3)
|
||||
|
||||
### Edge Cases
|
||||
*(unchanged from v1)*
|
||||
|
||||
- What happens when the LLM provider API is down or times out? (System should retry or fail gracefully with a clear error message).
|
||||
- What happens if the dashboard screenshot cannot be generated? (System should proceed with logs only or fail depending on configuration).
|
||||
- What happens if the context (logs/metadata) exceeds the LLM's token limit? (System should truncate or summarize input).
|
||||
- How does the system handle missing API keys? (Task should fail immediately with a configuration error).
|
||||
- What happens if the dashboard has multiple tabs with lazy-loaded charts? (System must switch through all tabs recursively to trigger chart rendering before capture).
|
||||
- What happens if Playwright encounters font loading timeouts in headless mode? (System must use CDP `Page.captureScreenshot` to bypass Playwright's internal timeout mechanism).
|
||||
## Requirements
|
||||
|
||||
## Requirements *(mandatory)*
|
||||
### Functional Requirements — v2 Additions & Changes
|
||||
|
||||
### Functional Requirements
|
||||
#### Task Management
|
||||
- **FR-032**: System MUST replace the one-click «Validate» button with a «Create Validation Task» flow accessible from a new `/validation-tasks` page.
|
||||
- **FR-033**: Task creation form MUST include: task name, environment, dashboard sources (IDs + URLs), LLM provider selector, prompt template textarea (pre-filled with default), screenshot toggle, logs toggle, schedule (cron or one-off).
|
||||
- **FR-034**: System MUST support two source types per task: `dashboard_id` (numeric ID selected from list) and `dashboard_url` (full Superset URL parsed via `SupersetContextExtractor`).
|
||||
- **FR-035**: URL parsing MUST extract: dashboard_id (numeric or slug), native_filters, activeTabs, anchor — using the same `SupersetContextExtractor` code path as dataset review.
|
||||
- **FR-036**: System MUST validate that every URL-resolved dashboard belongs to the selected environment before task creation.
|
||||
- **FR-037**: Task list MUST show: task name, status (active/inactive), schedule, last run timestamp, aggregated validation status across dashboards.
|
||||
- **FR-038**: Task detail MUST show per-dashboard breakdown with: dashboard title, validation status (PASS/WARN/FAIL), detected issue count, last run timestamp, link to full report.
|
||||
|
||||
- **FR-001**: System MUST allow configuration of multiple LLM providers, specifically supporting OpenAI API, OpenRouter, and Kilo Provider.
|
||||
- **FR-002**: System MUST securely store API keys for these providers using AES-256 encryption. [Security]
|
||||
- **FR-028**: The system MUST mask all API keys in the UI and logs, displaying only the last 4 characters (e.g., `sk-...1234`). [Security]
|
||||
- **FR-003**: System MUST implement a `DashboardValidationPlugin` that integrates with the existing `PluginBase` architecture.
|
||||
- **FR-004**: `DashboardValidationPlugin` MUST accept a dashboard identifier as input.
|
||||
- **FR-005**: `DashboardValidationPlugin` MUST be capable of retrieving a visual representation (screenshot) of the dashboard. The visual representation MUST be a PNG image with a resolution of 1920px width and full page height to ensure all dashboard content is captured. [Clarity]
|
||||
- **FR-016**: System MUST support configurable screenshot strategies: 'Headless Browser' (default, high accuracy) and 'API Thumbnail' (fallback/fast).
|
||||
- **FR-030**: The screenshot capture MUST use Playwright with Chrome DevTools Protocol (CDP) to avoid font loading timeouts in headless mode.
|
||||
- **FR-031**: The screenshot capture MUST implement recursive tab switching to trigger lazy-loaded chart rendering on multi-tab dashboards before capturing.
|
||||
- **FR-006**: `DashboardValidationPlugin` MUST retrieve recent execution logs associated with the dashboard from the Environment API (e.g., `/api/v1/log/`), limited to the last 100 lines or 24 hours (whichever is smaller) to prevent token overflow. [Reliability]
|
||||
- **FR-007**: `DashboardValidationPlugin` MUST combine visual and text data to prompt a Multimodal LLM for analysis. The analysis output MUST be structured as a JSON object containing `status` (Pass/Fail), `issues` (list of strings), and `summary` (text) to enable structured UI presentation. [Clarity]
|
||||
- **FR-008**: System MUST implement a `DocumentationPlugin` (or similar) for documenting datasets and dashboards.
|
||||
- **FR-009**: `DocumentationPlugin` MUST retrieve schema and metadata for the target asset.
|
||||
- **FR-017**: `DocumentationPlugin` MUST apply generated descriptions directly to the target object's metadata fields (dataset description, column descriptions). It MUST handle schema changes by only updating fields that exist in the current schema and ignoring hallucinated columns. [Data Integrity]
|
||||
- **FR-023**: The system MUST use optimistic locking or version checks to prevent overwrites during concurrent documentation updates. [Data Integrity]
|
||||
- **FR-024**: Generated documentation MUST be plain text or Markdown, strictly avoiding executable code blocks or active HTML. [Security]
|
||||
- **FR-025**: The system MUST validate that generated commit messages follow the conventional commits format (e.g., `feat:`, `fix:`) and do not exceed 72 characters in the subject line. [Data Integrity]
|
||||
- **FR-026**: If a metadata update fails partially, the system MUST rollback all changes to preserve data consistency. [Data Integrity]
|
||||
- **FR-027**: The system MUST reject documentation requests for empty or null schemas with a clear error message. [Edge Case]
|
||||
- **FR-010**: All LLM interactions MUST be executed as asynchronous tasks via the Task Manager.
|
||||
- **FR-018**: System MUST implement automatic retry logic (3 attempts with exponential backoff: 2s, 4s, 8s) for failed LLM API calls. [Reliability]
|
||||
- **FR-029**: The system MUST filter sensitive data (PII, credentials) from logs and screenshots before sending them to external LLM providers. [Privacy]
|
||||
- **FR-011**: Task execution logs and results MUST be streamed via the existing WebSocket infrastructure.
|
||||
- **FR-012**: System SHOULD expose an interface to generate text summaries for Git diffs, utilizing the diff, file list, and recent commit history as context. If the generated message is empty or invalid, the system MUST display a user-friendly error toast and retain the manual entry capability. [Edge Case]
|
||||
- **FR-013**: System MUST support scheduling of validation tasks for dashboards (leveraging existing scheduler architecture).
|
||||
- **FR-014**: System SHOULD support notification dispatch (Email, Pulse) upon validation failure or completion.
|
||||
- **FR-015**: Notifications MUST contain a summary of results (Status, Issue Count) and a direct link to the full report, avoiding sensitive full details in the message body.
|
||||
- **FR-019**: The "Validate" button MUST display a loading spinner and be disabled during active execution to prevent multiple triggers. [UX]
|
||||
- **FR-020**: The system MUST provide immediate visual feedback (Toast notifications) for successful or failed connection tests in LLM Settings. [UX]
|
||||
- **FR-021**: New LLM actions (Validate, Generate Docs) MUST use standard system icons (e.g., `heroicons:beaker` for validate, `heroicons:document-text` for docs) and follow existing button placement patterns. [Consistency]
|
||||
- **FR-022**: Error messages for LLM failures MUST follow the standard `ss-tools` error format, including a clear title, descriptive message, and troubleshooting link if applicable. [Consistency]
|
||||
#### Dual-Path Execution
|
||||
- **FR-039**: System MUST support two execution paths: Path A — Playwright screenshot (screenshot_enabled=true) and Path B — text-only API (screenshot_enabled=false).
|
||||
- **FR-040**: Path A MUST: launch Playwright → login → navigate to URL (with parsed filters/tabs) → recursive tab switching → multi-chunk CDP screenshots → compress images → multimodal LLM call.
|
||||
- **FR-041**: Multi-chunk screenshots MUST capture each dashboard tab as a separate high-resolution image (max 1920×1200 per chunk) instead of one compressed full-page.
|
||||
- **FR-059**: Multi-chunk screenshots MUST follow the image pipeline: CDP capture → PNG → convert to JPEG (quality 60, 1024px) for LLM + convert to WebP (lossy, quality 80) for archive → delete PNG and JPEG after WebP saved. LLM receives ONLY JPEG for universal provider compatibility.
|
||||
- **FR-042**: Path B MUST: call `GET /api/v1/dashboard/{id}` for structure → `GET /api/v1/chart/{id}` for each chart → `GET /api/v1/dataset/{id}` for each unique dataset → `GET /api/v1/log/` for execution logs → construct text description of dashboard topology → text-only LLM call.
|
||||
- **FR-043**: Path B MUST optionally execute `POST /api/v1/chart/data` for each chart to verify query execution (controlled by `execute_chart_data` toggle, default false to avoid load).
|
||||
- **FR-044**: Path B dataset health check MUST verify four levels: (1) **metadata_accessible** — HTTP 200 from `GET /api/v1/dataset/{id}`, database name, backend, kind; (2) **datasource_resolvable** — chart datasource_id maps to a valid dataset; (3) **query_executable** — only when `execute_chart_data=true`: `POST /api/v1/chart/data` succeeds; (4) **data_returned** — only when `execute_chart_data=true`: row_count > 0 or no error. Levels 1-2 are always checked; levels 3-4 require explicit opt-in.
|
||||
- **FR-045**: System MUST use different LLM prompt templates for Path A (multimodal — image + text, one dashboard per call) vs Path B (text-only — dashboard topology + dataset health + logs, batched per FR-045a).
|
||||
- **FR-045a**: Path B execution MAY support batching: multiple dashboards validated in a single LLM call up to `llm_batch_size`. Default `batch_size=1` (full isolation — one LLM call per dashboard). Batching is experimental: values >1 reduce cost at risk of cross-dashboard context contamination. Each batch call MUST use a structured prompt enforcing per-dashboard isolation: labeled sections + `{dashboards: [{dashboard_id, status, summary, issues}]}` JSON schema. Missing/parse-error dashboards marked UNKNOWN individually — other dashboards in batch preserved. See research decision 15 for rationale and risks.
|
||||
- **FR-045b**: Path A execution does NOT support batching — each dashboard requires a separate multimodal LLM call due to image token constraints. Multiple Path A dashboards within a run are parallelized via `policy_dashboard_concurrency_limit`.
|
||||
- **FR-046**: Path B prompt MUST include: dashboard title, owner, tab hierarchy, chart names/types/params/datasource, dataset health status per datasource, execution logs — all as structured text.
|
||||
|
||||
### Key Entities
|
||||
#### Provider & Prompt Configuration
|
||||
- **FR-047**: Dashboard validation provider MUST be configured per-task in the task creation form — NOT as a global binding in LLM settings.
|
||||
- **FR-048**: Dashboard validation prompt template MUST be configurable per-task as a single `prompt_template` field. When `null`, the system MUST use the path-specific default: `dashboard_validation_prompt_multimodal` for Path A (`screenshot_enabled=true`) and `dashboard_validation_prompt_text` for Path B (`screenshot_enabled=false`). The legacy v1 key `dashboard_validation_prompt` MUST NOT be used as default.
|
||||
- **FR-048a**: When the user toggles `screenshot_enabled` on an existing task with a custom prompt, the system MUST show a warning: «Prompt may no longer match the selected execution path. Review your prompt template.»
|
||||
- **FR-049**: LLM settings page (`/admin/settings/llm`) MUST retain provider bindings only for: documentation, git_commit, assistant_planner. Dashboard validation binding MUST be removed.
|
||||
- **FR-050**: The provider selector in task creation form MUST filter to show only active, multimodal-compatible providers (for Path A) or active providers of any type (for Path B).
|
||||
|
||||
- **LLMProviderConfig**: Stores provider type (OpenAI, etc.), base URL, model name, and API key.
|
||||
- **ValidationResult**: Stores the analysis output, timestamp, and reference to the dashboard.
|
||||
- **AutoDocumentation**: Stores the generated documentation text for an asset.
|
||||
#### Dashboard Hub Changes
|
||||
- **FR-051**: Dashboard hub (`/dashboards`) MUST remove the per-row «Validate» action from the action dropdown.
|
||||
- **FR-051a**: Dashboard hub MUST add a lightweight validation status indicator (colored dot: 🟢 PASS / 🟡 WARN / 🔴 FAIL / ⚪ UNKNOWN) next to each dashboard title, reflecting the most recent validation result across all tasks that include this dashboard. Clicking the indicator MUST open a dropdown with the last 3 validation runs (date, status, task name) and a «View all runs» link.
|
||||
- **FR-052**: Dashboard hub MUST remove the LLM validation status column (PASS/WARN/FAIL badges from `lastTask.validationStatus`). Replaced by the indicator in FR-051a.
|
||||
- **FR-053**: Dashboard hub MUST retain a «Create Validation Task» button in the toolbar that pre-fills selected dashboard IDs into a new task form.
|
||||
- **FR-053a**: Dashboard detail page (`/dashboards/{id}`) MUST include a «Validation History» section showing the last 10 validation runs for this dashboard across all tasks, with links to canonical run detail pages. This section replaces the removed Validate button and LLM status column.
|
||||
|
||||
## Success Criteria *(mandatory)*
|
||||
#### Reporting & Routing
|
||||
- **FR-055b**: System MUST serve validation reports at the canonical URL `/validation-tasks/{policy_id}/runs/{run_id}`. The page MUST render:
|
||||
- **Run header**: policy name, run timestamp, trigger type (manual/scheduled), total duration, aggregate status (N PASS / M WARN / K FAIL across all dashboards).
|
||||
- **Dashboard list**: each dashboard as a collapsible row showing: title, status icon (PASS/WARN/FAIL/UNKNOWN), issue count, mini-summary. Expanded row shows: full issues list, execution path (Path A/B), dataset health (Path B), screenshot thumbnails with tab selector (Path A), logs sent to LLM, task execution logs.
|
||||
- **Mixed paths**: a single run may contain both Path A and Path B dashboard records ONLY when a Path A dashboard falls back to Path B due to payload limit (FR-056) or when Path A execution fails and retries as Path B. Mixed paths do NOT occur from per-source path override — `screenshot_enabled` is a policy-level setting. The run report handles mixed records by rendering screenshot sections for `execution_path=screenshot` records and dataset health sections for `execution_path=text_only` records.
|
||||
- **FR-055c**: The legacy `/reports/llm/{taskId}` route MUST redirect (302) to `/validation-tasks/{policy_id}/runs/{run_id}` when `taskId` matches a `ValidationRun.task_id`. Returns 404 if no match.
|
||||
- **FR-054**: System MUST prevent duplicate execution of the same task: if a task is already running, manual «Run Now» triggers return a toast «Task is already running» without queuing.
|
||||
- **FR-055**: System MUST queue manual execution requests for different tasks in a FIFO queue, respecting `global_validation_worker_limit` (max concurrent policy runs across all tasks). If the queue is full, the system MUST show a toast with estimated wait time. Same-policy lock: if a policy is already running (manual or scheduled), additional triggers are rejected with status `SKIPPED_ALREADY_RUNNING`.
|
||||
- **FR-055a**: Within a single `ValidationRun`, dashboards MUST be validated with parallelism up to `policy_dashboard_concurrency_limit` (default 3). Dashboards are processed in waves: first 3 start → any finishes → next starts, until all done or exhausted. Each dashboard produces one `ValidationRecord` with FK to the same `run_id`.
|
||||
- **FR-055d**: System MUST create a `ValidationRun` record when a task execution starts (manual or scheduled). The run groups all per-dashboard `ValidationRecord`s. Upon completion, the run MUST compute aggregate counts: `dashboard_count`, `pass_count`, `warn_count`, `fail_count`, `unknown_count`. Run status = `completed` (all done), `partial` (some sources skipped/invalid), or `failed` (execution error before any dashboard).
|
||||
- **FR-056**: System MUST calculate estimated LLM payload size before sending multi-chunk screenshots. If >80% of model context window: progressively reduce JPEG quality to 30%, then width to 800px. If still exceeds — fall back to Path B text-only with a warning. All reduction decisions logged via molecular-cot: REASON before each step, REFLECT after outcome.
|
||||
- **FR-057**: System MUST re-parse URL sources on every task execution. If URL parsing fails — source marked with warning «Dashboard structure changed. Paste a new URL». If dashboard_id changed — logged as WARN, new ID used.
|
||||
- **FR-058**: System MUST forbid deletion of an LLM provider if it is referenced by any active `ValidationPolicy`. The deletion endpoint MUST return a list of blocking task names. Users MUST reassign or deactivate affected tasks before deletion.
|
||||
|
||||
### Measurable Outcomes
|
||||
#### Existing Requirements (v1, preserved)
|
||||
- **FR-001**: System MUST allow configuration of multiple LLM providers (OpenAI, OpenRouter, Kilo, LiteLLM).
|
||||
- **FR-002**: System MUST securely store API keys using AES-256 encryption.
|
||||
- **FR-005**: Screenshot capture (when enabled) MUST be PNG with 1920px width.
|
||||
- **FR-006**: Log retrieval limited to last 100 lines or 24 hours.
|
||||
- **FR-007**: Analysis output structured as JSON `{status, summary, issues: [{severity, message, location}]}`.
|
||||
- **FR-010**: All LLM interactions as asynchronous tasks via Task Manager.
|
||||
- **FR-013**: Scheduling support for validation tasks via cron.
|
||||
- **FR-014**: Notification dispatch (Email, Pulse) on validation failure/completion.
|
||||
- **FR-015**: Notifications MUST contain a summary (Status, Issue Count) and a direct link to the canonical run detail: `/validation-tasks/{policy_id}/runs/{run_id}`. Per-dashboard deep link via query parameter: `?dashboard_id={id}`. The legacy `/reports/llm/{taskId}` route is a compatibility redirect. Sensitive details MUST NOT appear in the notification body.
|
||||
- **FR-018**: Retry logic: 3 attempts (now 5) with exponential backoff.
|
||||
- **FR-022**: Error messages in standard ss-tools format.
|
||||
- **FR-028**: API key masking in UI and logs (last 4 chars only).
|
||||
- **FR-029**: System MUST redact PII, credentials, query text, tokens in URLs, and native filter values from text/log payloads before sending to external LLM providers and before persistence. Redaction is deterministic (regex/pattern-based). Default retention: screenshots 30 days, raw LLM responses 30 days (debug-only, nullable column). Redacted summaries/issues retained indefinitely. Auditable via structured decision audit logs (REASON/REFLECT markers — deterministic metadata only, no free-form reasoning text).
|
||||
- **FR-029c**: Screenshot redaction MUST mask known DOM regions (URL bar, header with user info) and CSS selectors where available before capture. Full OCR-based screenshot PII redaction is out of scope for v2.
|
||||
|
||||
- **SC-001**: Users can successfully configure and validate a connection to at least one LLM provider.
|
||||
- **SC-002**: A dashboard validation task completes within 90 seconds (assuming standard LLM latency).
|
||||
- **SC-003**: The system successfully processes a multimodal prompt (image + text) and returns a structured analysis.
|
||||
- **SC-004**: Generated documentation for a standard dataset contains descriptions for at least 80% of the columns (based on LLM capability, but pipeline must support it).
|
||||
- **SC-005**: Screenshots capture full dashboard content including all tabs (1920px width, full height) without font loading timeouts.
|
||||
- **SC-006**: Analysis results are displayed in task logs with clear `[ANALYSIS_SUMMARY]` and `[ANALYSIS_ISSUE]` markers for easy parsing.
|
||||
### Key Entities — v2
|
||||
|
||||
- **ValidationPolicy** (updated): Name, environment_id, sources (list of `ValidationSource`), provider_id, prompt_template, screenshot_enabled, logs_enabled, execute_chart_data, cron_expression, is_active, notification_channels.
|
||||
- **ValidationSource** (new): `type` (dashboard_id | dashboard_url), `value` (ID or URL), `parsed_context` (ParsedNativeFilters — only for URL type).
|
||||
- **ValidationRecord** (unchanged): Per-execution result: dashboard_id, status, summary, issues, screenshot_paths, dataset_health, raw_response.
|
||||
- **LLMProviderConfig** (unchanged): Provider type, base URL, model, encrypted API key, is_multimodal flag.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
### Measurable Outcomes — v2 Additions
|
||||
|
||||
- **SC-007**: A Path B (text-only) validation of a 5-tab dashboard with 15 charts completes within 30 seconds (excluding LLM latency).
|
||||
- **SC-008**: A Path A (screenshot) multi-chunk validation of a 5-tab dashboard completes within 120 seconds.
|
||||
- **SC-009**: URL parsing correctly extracts dashboard_id, native_filters, and activeTabs from 100% of standard Superset dashboard URLs (numeric ID, slug, permalink formats).
|
||||
- **SC-010**: Dataset metadata errors surfaced by Superset API (levels 1-2) are detected with 100% accuracy. Query-level KXD failures (levels 3-4) require `execute_chart_data=true` and depend on Superset query execution behavior.
|
||||
- **SC-011**: User can create a validation task (form fill + URL parse + provider select) in under 60 seconds.
|
||||
|
||||
### Existing (v1, preserved)
|
||||
- **SC-001**: Users can configure and test connection to at least one LLM provider.
|
||||
- **SC-003**: Multimodal prompt (image + text) returns structured JSON analysis.
|
||||
- **SC-005**: Screenshots capture full dashboard content including all tabs.
|
||||
|
||||
## Edge Cases — v2
|
||||
|
||||
- What happens when a URL-parsed dashboard has been deleted before task execution? (System marks source as invalid, skips it, reports in task log.)
|
||||
- What happens when `screenshot_enabled=false` but no dataset info is retrievable? (System proceeds with logs-only analysis, flags missing data sources.)
|
||||
- What happens when a task has 20 dashboards and each takes 30s? (System executes sequentially with configurable concurrency limit, default 3.)
|
||||
- What happens when user clicks «Run Now» while the same task is already executing? (System shows toast «Task is already running», does NOT queue a duplicate run.)
|
||||
- What happens when user clicks «Run Now» while another task is executing? (System places the manual trigger in a FIFO queue, respecting concurrency_limit. If queue is full — shows toast with estimated wait time.)
|
||||
- What happens when the API key for a provider used by existing tasks is changed? (Tasks always fetch the current key from `LLMProviderService` at execution time — no key caching in task configuration. Old key is never reused.)
|
||||
- What happens when multi-chunk screenshots (15+ tabs) exceed the LLM provider's token limit? (System calculates estimated payload size before sending. If >80% of model context window: reduces JPEG quality progressively to 30%, then width to 800px. If still exceeds — falls back to Path B text-only with a warning. All decisions logged via molecular-cot: REASON before each reduction, REFLECT after outcome.)
|
||||
- What happens if WebP encoding fails during archive step? (System falls back to PNG for archive with a WARN log. JPEG for LLM is unaffected — the pipeline is independent: LLM path and archive path are decoupled.)
|
||||
- What happens when user toggles `screenshot_enabled` on a task with a custom prompt? (System shows warning: «Prompt may no longer match execution path» per FR-048a. Task saves successfully with user confirmation.)
|
||||
- What happens when old API consumers send `POST /api/tasks` with `plugin_id: llm_dashboard_validation` after v2 migration? (Returns `400 Bad Request` with message: «llm_dashboard_validation is deprecated. Use POST /api/validation-tasks instead.» All consumers — assistant tool, external scripts — must be updated to v2 API during migration.)
|
||||
- What happens when a URL-parsed dashboard changes structure between task executions? (System re-parses the URL on every execution. If the URL fails — dashboard structure has changed. System marks the source with a warning: «Dashboard structure changed. Paste a new URL to update filters.» If dashboard_id changed — logs WARN and uses the new ID.)
|
||||
- What happens when a user tries to delete an LLM provider that is used by active validation tasks? (System forbids deletion and returns a list of blocking task names. User must reassign the provider in each listed task or deactivate the tasks before deleting the provider.)
|
||||
- What happens when Path B `POST /api/v1/chart/data` overloads Superset? (Toggle is off by default; user explicitly opts in with a warning about load.)
|
||||
- What happens when `SupersetContextExtractor.parse_superset_link()` returns partial recovery? (System flags the source with `partial_recovery`, shows warning in task detail, attempts best-effort validation.)
|
||||
- What happens when a dashboard URL contains a permalink key that has expired? (System marks the source as stale, prompts user to paste a fresh URL.)
|
||||
|
||||
(End of file — 214 lines)
|
||||
|
||||
Reference in New Issue
Block a user