ss-tools/specs/017-llm-analysis-plugin/spec.md

Feature Specification: LLM Analysis & Documentation Plugins

Feature Branch: 017-llm-analysis-plugin Created: 2026-01-28 Updated: 2026-06-07 — v2: task-based flow, dual-path execution, fully implemented Status: Implemented ✅ (~30 files, ~7 880 строк backend + ~4 558 строк frontend)

v2 Design Decisions

#	Decision	Rationale
D1	Remove one-click «Validate» button — replace with «Create Validation Task» flow	Ad-hoc validation doesn't scale; users need scheduled, repeatable validation with full configuration control
D2	Dual execution paths: Path A (Playwright screenshot) + Path B (text-only API)	Screenshot costs image tokens but catches visual bugs; text-only is lightweight, catches data/KXD errors via dataset health checks
D3	Screenshot is optional (screenshot_enabled flag per task)	Lets users trade off visual bug detection vs cost/speed; text-only validates KXD→dataset→chart data chain
D4	Dataset health checking in Path B — execute GET /api/v1/dataset/{id} for every unique dataset, optionally POST /api/v1/chart/data	Core goal: verify dashboard opens correctly with KXD data. Without dataset checks, silent KXD failures are invisible
D5	URL-based dashboard selection — paste Superset URL, parse via SupersetContextExtractor (same code as dataset review)	Preserves native_filters, activeTabs, anchor — LLM validates dashboard in exact user-visible state
D6	Provider selection moves from settings to task creation form	Each task gets explicit provider; no implicit default. Provider bindings for dashboard_validation removed from LLM settings
D7	Prompt template moves from settings to task creation form	Users customize prompts per task, not globally. Default from DEFAULT_LLM_PROMPTS
D8	LLM settings retain only: documentation, git_commit, assistant_planner provider bindings	Dashboard validation provider is per-task, not global
D9	Multi-chunk analysis — dashboard sliced into logical blocks (tabs + charts)	Multiple screenshots in single LLM request; each block at high resolution instead of one compressed full-page
D10	Image pipeline: PNG → JPEG (LLM) + WebP (archive) — CDP captures PNG, converted to JPEG for LLM, archived as WebP lossy q=80 for storage efficiency, PNG and JPEG deleted after	JPEG universally supported by all LLM providers; WebP saves ~93% disk space vs PNG. Silent WebP failure risk on unknown providers eliminated by JPEG-only LLM path

Clarifications

Session 2026-01-28

• Q: Notification Content Strategy → A: Summary with Link: Include status (Pass/Fail), key issues count, and a direct link to the full report in the UI.
• Q: Dashboard Screenshot Source → A: Hybrid (Configurable): Support both Headless Browser (accurate) and API/Thumbnail (fast) methods, allowing admin configuration.
• Q: Dataset Documentation Output Format → A: Direct Object Update: Update the description fields of the dataset and its columns directly within the dataset object (persisted to backend/metadata).
• Q: Git Commit Message Context → A: Diff + Recent History: Send diff, file names, and the last 3 commit messages to match style.
• Q: LLM Failure Handling → A: Retry then Fail: Automatically retry 3 times with exponential backoff before failing.

Session 2026-05-31

• Q: Concurrent execution — manual trigger while task is running? → Same task: toast «Already running», no queue. Different task: FIFO queue, respecting concurrency_limit.
• Q: API key rotation — impact on existing tasks? → Tasks fetch current key at execution time from LLMProviderService. No key caching in task config.
• Q: Multi-chunk token limit — 15+ tabs exceeding provider limit? → Estimate payload → progressive quality reduction (30% JPEG, 800px) → Path B fallback. Molecular-cot logging: REASON before each reduction, REFLECT after.
• Q: Backward compat — old POST /api/tasks with llm_dashboard_validation? → Hard cut: returns 400. All consumers must migrate to POST /api/validation-tasks.
• Q: URL re-parse — dashboard structure changed between runs? → Re-parse every execution. URL failure → warning «Paste new URL». Dashboard ID change → WARN log, use new ID.
• Q: Provider deletion — what if used by active tasks? → Forbidden. Returns list of blocking task names. User must reassign or deactivate tasks first.
• Q: Screenshot format — PNG vs WebP vs JPEG? → Pipeline: CDP captures PNG → converted to JPEG (LLM, universal compatibility) + WebP lossy q=80 (archive, 93% disk savings). PNG and JPEG deleted after WebP saved. LLM path and archive path are decoupled.

User Scenarios & Testing

User Story 1 — Create & Schedule Dashboard Validation Task (Priority: P1)

As a Data Engineer, I want to create a validation task that checks multiple dashboards for health issues on a schedule, so that I can proactively detect KXD data problems, rendering failures, and chart errors without manual inspection.

Why this priority: Core value proposition. Replaces unreliable one-off manual checks with automated, scheduled validation covering the full KXD→dataset→chart→dashboard chain.

Independent Test: Create a validation task with 2 dashboards (one via ID, one via URL), set cron schedule 0 8 * * 1-5, wait for first execution, verify report shows dataset health status per dashboard.

Acceptance Scenarios:

1. Given I am on the Validation Tasks page, When I click «+ New Task», Then the task creation form opens with fields: name, environment, sources (dashboard IDs + URLs), provider, prompt, screenshot toggle, schedule.
2. Given I paste a Superset dashboard URL with native filters, When the form parses it, Then the system extracts dashboard_id, native_filters, activeTabs, and anchor — and validates the URL is accessible.
3. Given I enable screenshot_enabled, When the task executes, Then the system uses Path A: Playwright browser → CDP screenshot → multimodal LLM analysis.
4. Given I disable screenshot_enabled, When the task executes, Then the system uses Path B: API calls → dashboard structure + chart params + dataset schema (+ optional chart data execution) → text-only LLM analysis.
5. Given the task has a cron schedule, When the schedule triggers, Then the system runs validation on all configured dashboards and persists results.
6. Given a validation finds issues, When the task completes, Then the system sends a notification (Email/Pulse) if configured.

User Story 2 — Validation Task List & History (Priority: P1)

As a Data Engineer, I want to see all my validation tasks, their last run status, and drill into historical results per dashboard, so that I can track health trends over time.

Why this priority: Without a list view and history, the task-based flow is invisible — users can't monitor what's running.

Independent Test: View the task list, see status per task (active/inactive), last run time, aggregated status across dashboards.

Acceptance Scenarios:

1. Given I am on the Validation Tasks page, When the page loads, Then I see all created tasks with: name, schedule, last run status, dashboard count.
2. Given I click a task, When the detail opens, Then I see per-dashboard breakdown: status (PASS/WARN/FAIL), detected issues, last run timestamp, link to full report.
3. Given I click a dashboard in the task detail, When the report opens, Then I see: check result, text summary, issues table, dataset health status (Path B), screenshot thumbnails (Path A), source logs, execution logs.

User Story 3 — Dual-Path Dashboard Health Analysis (Priority: P1)

As a Data Engineer, I want the validation to check both visual rendering AND data availability, so that I catch KXD backend failures that produce no visual error but return empty/incomplete data.

Why this priority: The primary purpose of validation is verifying the KXD→dataset→chart data chain works. Screenshot-only validation misses silent data failures (empty results, partial data, wrong time range).

Independent Test: Create a task with screenshot_enabled=false, run against a dashboard with a known broken dataset, verify the LLM reports the dataset health issue.

Acceptance Scenarios:

1. Given Path B execution, When a dataset GET /api/v1/dataset/{id} returns KXD connection error, Then the LLM report flags the dataset as unhealthy and lists affected charts.
2. Given Path B execution with chart data execution enabled, When POST /api/v1/chart/data times out, Then the LLM report includes the timeout with affected chart info.
3. Given Path B execution, When all datasets are healthy and logs show no errors, Then the LLM report returns status: PASS with a summary of verified data sources.
4. Given Path A (screenshot), When LLM analyzes the multi-chunk screenshots, Then visual issues (empty charts, rendering errors, overlap) are detected and reported with tab/chart location.

User Story 4 — Automated Dataset Documentation (Priority: P2)

(unchanged from v1)

As a Data Steward, I want to generate documentation for datasets and dashboards using LLMs.

User Story 5 — LLM Provider Configuration (Priority: P2)

(unchanged from v1 — but provider bindings for dashboard_validation removed)

As an Administrator, I want to configure LLM providers for documentation, git commits, and chat — with dashboard validation provider configured per-task.

User Story 6 — Git Commit Message Suggestion (Priority: P3)

(unchanged from v1)

Requirements

Functional Requirements — v2 Additions & Changes

Task Management

• FR-032: System MUST replace the one-click «Validate» button with a «Create Validation Task» flow accessible from a new /validation-tasks page.
• FR-033: Task creation form MUST include: task name, environment, dashboard sources (IDs + URLs), LLM provider selector, prompt template textarea (pre-filled with default), screenshot toggle, logs toggle, schedule (cron or one-off).
• FR-034: System MUST support two source types per task: dashboard_id (numeric ID selected from list) and dashboard_url (full Superset URL parsed via SupersetContextExtractor).
• FR-035: URL parsing MUST extract: dashboard_id (numeric or slug), native_filters, activeTabs, anchor — using the same SupersetContextExtractor code path as dataset review.
• FR-036: System MUST validate that every URL-resolved dashboard belongs to the selected environment before task creation.
• FR-037: Task list MUST show: task name, status (active/inactive), schedule, last run timestamp, aggregated validation status across dashboards.
• FR-038: Task detail MUST show per-dashboard breakdown with: dashboard title, validation status (PASS/WARN/FAIL), detected issue count, last run timestamp, link to full report.

Dual-Path Execution

• FR-039: System MUST support two execution paths: Path A — Playwright screenshot (screenshot_enabled=true) and Path B — text-only API (screenshot_enabled=false).
• FR-040: Path A MUST: launch Playwright → login → navigate to URL (with parsed filters/tabs) → recursive tab switching → multi-chunk CDP screenshots → compress images → multimodal LLM call.
• FR-041: Multi-chunk screenshots MUST capture each dashboard tab as a separate high-resolution image (max 1920×1200 per chunk) instead of one compressed full-page.
• FR-059: Multi-chunk screenshots MUST follow the image pipeline: CDP capture → PNG → convert to JPEG (quality 60, 1024px) for LLM + convert to WebP (lossy, quality 80) for archive → delete PNG and JPEG after WebP saved. LLM receives ONLY JPEG for universal provider compatibility.
• FR-042: Path B MUST: call GET /api/v1/dashboard/{id} for structure → GET /api/v1/chart/{id} for each chart → GET /api/v1/dataset/{id} for each unique dataset → GET /api/v1/log/ for execution logs → construct text description of dashboard topology → text-only LLM call.
• FR-043: Path B MUST optionally execute POST /api/v1/chart/data for each chart to verify query execution (controlled by execute_chart_data toggle, default false to avoid load).
• FR-044: Path B dataset health check MUST verify four levels: (1) metadata_accessible — HTTP 200 from GET /api/v1/dataset/{id}, database name, backend, kind; (2) datasource_resolvable — chart datasource_id maps to a valid dataset; (3) query_executable — only when execute_chart_data=true: POST /api/v1/chart/data succeeds; (4) data_returned — only when execute_chart_data=true: row_count > 0 or no error. Levels 1-2 are always checked; levels 3-4 require explicit opt-in.
• FR-045: System MUST use different LLM prompt templates for Path A (multimodal — image + text, one dashboard per call) vs Path B (text-only — dashboard topology + dataset health + logs, batched per FR-045a).
• FR-045a: Path B execution MAY support batching: multiple dashboards validated in a single LLM call up to llm_batch_size. Default batch_size=1 (full isolation — one LLM call per dashboard). Batching is experimental: values >1 reduce cost at risk of cross-dashboard context contamination. Each batch call MUST use a structured prompt enforcing per-dashboard isolation: labeled sections + {dashboards: [{dashboard_id, status, summary, issues}]} JSON schema. Missing/parse-error dashboards marked UNKNOWN individually — other dashboards in batch preserved. See research decision 15 for rationale and risks.
• FR-045b: Path A execution does NOT support batching — each dashboard requires a separate multimodal LLM call due to image token constraints. Multiple Path A dashboards within a run are parallelized via policy_dashboard_concurrency_limit.
• FR-046: Path B prompt MUST include: dashboard title, owner, tab hierarchy, chart names/types/params/datasource, dataset health status per datasource, execution logs — all as structured text.

Provider & Prompt Configuration

• FR-047: Dashboard validation provider MUST be configured per-task in the task creation form — NOT as a global binding in LLM settings.
• FR-048: Dashboard validation prompt template MUST be configurable per-task as a single prompt_template field. When null, the system MUST use the path-specific default: dashboard_validation_prompt_multimodal for Path A (screenshot_enabled=true) and dashboard_validation_prompt_text for Path B (screenshot_enabled=false). The legacy v1 key dashboard_validation_prompt MUST NOT be used as default.
• FR-048a: When the user toggles screenshot_enabled on an existing task with a custom prompt, the system MUST show a warning: «Prompt may no longer match the selected execution path. Review your prompt template.»
• FR-049: LLM settings page (/admin/settings/llm) MUST retain provider bindings only for: documentation, git_commit, assistant_planner. Dashboard validation binding MUST be removed.
• FR-050: The provider selector in task creation form MUST filter to show only active, multimodal-compatible providers (for Path A) or active providers of any type (for Path B).

Dashboard Hub Changes

• FR-051: Dashboard hub (/dashboards) MUST remove the per-row «Validate» action from the action dropdown.
• FR-051a: Dashboard hub MUST add a lightweight validation status indicator (colored dot: 🟢 PASS / 🟡 WARN / 🔴 FAIL / ⚪ UNKNOWN) next to each dashboard title, reflecting the most recent validation result across all tasks that include this dashboard. Clicking the indicator MUST open a dropdown with the last 3 validation runs (date, status, task name) and a «View all runs» link.
• FR-052: Dashboard hub MUST remove the LLM validation status column (PASS/WARN/FAIL badges from lastTask.validationStatus). Replaced by the indicator in FR-051a.
• FR-053: Dashboard hub MUST retain a «Create Validation Task» button in the toolbar that pre-fills selected dashboard IDs into a new task form.
• FR-053a: Dashboard detail page (/dashboards/{id}) MUST include a «Validation History» section showing the last 10 validation runs for this dashboard across all tasks, with links to canonical run detail pages. This section replaces the removed Validate button and LLM status column.

Reporting & Routing

• FR-055b: System MUST serve validation reports at the canonical URL /validation-tasks/{policy_id}/runs/{run_id}. The page MUST render:
  • Run header: policy name, run timestamp, trigger type (manual/scheduled), total duration, aggregate status (N PASS / M WARN / K FAIL across all dashboards).
  • Dashboard list: each dashboard as a collapsible row showing: title, status icon (PASS/WARN/FAIL/UNKNOWN), issue count, mini-summary. Expanded row shows: full issues list, execution path (Path A/B), dataset health (Path B), screenshot thumbnails with tab selector (Path A), logs sent to LLM, task execution logs.
  • Mixed paths: a single run may contain both Path A and Path B dashboard records ONLY when a Path A dashboard falls back to Path B due to payload limit (FR-056) or when Path A execution fails and retries as Path B. Mixed paths do NOT occur from per-source path override — screenshot_enabled is a policy-level setting. The run report handles mixed records by rendering screenshot sections for execution_path=screenshot records and dataset health sections for execution_path=text_only records.
• FR-055c: The legacy /reports/llm/{taskId} route MUST redirect (302) to /validation-tasks/{policy_id}/runs/{run_id} when taskId matches a ValidationRun.task_id. Returns 404 if no match.
• FR-054: System MUST prevent duplicate execution of the same task: if a task is already running, manual «Run Now» triggers return a toast «Task is already running» without queuing.
• FR-055: System MUST queue manual execution requests for different tasks in a FIFO queue, respecting global_validation_worker_limit (max concurrent policy runs across all tasks). If the queue is full, the system MUST show a toast with estimated wait time. Same-policy lock: if a policy is already running (manual or scheduled), additional triggers are rejected with status SKIPPED_ALREADY_RUNNING.
• FR-055a: Within a single ValidationRun, dashboards MUST be validated with parallelism up to policy_dashboard_concurrency_limit (default 3). Dashboards are processed in waves: first 3 start → any finishes → next starts, until all done or exhausted. Each dashboard produces one ValidationRecord with FK to the same run_id.
• FR-055d: System MUST create a ValidationRun record when a task execution starts (manual or scheduled). The run groups all per-dashboard ValidationRecords. Upon completion, the run MUST compute aggregate counts: dashboard_count, pass_count, warn_count, fail_count, unknown_count. Run status = completed (all done), partial (some sources skipped/invalid), or failed (execution error before any dashboard).
• FR-056: System MUST calculate estimated LLM payload size before sending multi-chunk screenshots. If >80% of model context window: progressively reduce JPEG quality to 30%, then width to 800px. If still exceeds — fall back to Path B text-only with a warning. All reduction decisions logged via molecular-cot: REASON before each step, REFLECT after outcome.
• FR-057: System MUST re-parse URL sources on every task execution. If URL parsing fails — source marked with warning «Dashboard structure changed. Paste a new URL». If dashboard_id changed — logged as WARN, new ID used.
• FR-058: System MUST forbid deletion of an LLM provider if it is referenced by any active ValidationPolicy. The deletion endpoint MUST return a list of blocking task names. Users MUST reassign or deactivate affected tasks before deletion.

Existing Requirements (v1, preserved)

• FR-001: System MUST allow configuration of multiple LLM providers (OpenAI, OpenRouter, Kilo, LiteLLM).
• FR-002: System MUST securely store API keys using AES-256 encryption.
• FR-005: Screenshot capture (when enabled) MUST be PNG with 1920px width.
• FR-006: Log retrieval limited to last 100 lines or 24 hours.
• FR-007: Analysis output structured as JSON {status, summary, issues: [{severity, message, location}]}.
• FR-010: All LLM interactions as asynchronous tasks via Task Manager.
• FR-013: Scheduling support for validation tasks via cron.
• FR-014: Notification dispatch (Email, Pulse) on validation failure/completion.
• FR-015: Notifications MUST contain a summary (Status, Issue Count) and a direct link to the canonical run detail: /validation-tasks/{policy_id}/runs/{run_id}. Per-dashboard deep link via query parameter: ?dashboard_id={id}. The legacy /reports/llm/{taskId} route is a compatibility redirect. Sensitive details MUST NOT appear in the notification body.
• FR-018: Retry logic: 3 attempts (now 5) with exponential backoff.
• FR-022: Error messages in standard ss-tools format.
• FR-028: API key masking in UI and logs (last 4 chars only).
• FR-029: System MUST redact PII, credentials, query text, tokens in URLs, and native filter values from text/log payloads before sending to external LLM providers and before persistence. Redaction is deterministic (regex/pattern-based). Default retention: screenshots 30 days, raw LLM responses 30 days (debug-only, nullable column). Redacted summaries/issues retained indefinitely. Auditable via structured decision audit logs (REASON/REFLECT markers — deterministic metadata only, no free-form reasoning text).
• FR-029c: Screenshot redaction MUST mask known DOM regions (URL bar, header with user info) and CSS selectors where available before capture. Full OCR-based screenshot PII redaction is out of scope for v2.

Key Entities — v2

• ValidationPolicy (updated): Name, environment_id, sources (list of ValidationSource), provider_id, prompt_template, screenshot_enabled, logs_enabled, execute_chart_data, cron_expression, is_active, notification_channels.
• ValidationSource (new): type (dashboard_id | dashboard_url), value (ID or URL), parsed_context (ParsedNativeFilters — only for URL type).
• ValidationRecord (unchanged): Per-execution result: dashboard_id, status, summary, issues, screenshot_paths, dataset_health, raw_response.
• LLMProviderConfig (unchanged): Provider type, base URL, model, encrypted API key, is_multimodal flag.

Success Criteria

Measurable Outcomes — v2 Additions

• SC-007: A Path B (text-only) validation of a 5-tab dashboard with 15 charts completes within 30 seconds (excluding LLM latency).
• SC-008: A Path A (screenshot) multi-chunk validation of a 5-tab dashboard completes within 120 seconds.
• SC-009: URL parsing correctly extracts dashboard_id, native_filters, and activeTabs from 100% of standard Superset dashboard URLs (numeric ID, slug, permalink formats).
• SC-010: Dataset metadata errors surfaced by Superset API (levels 1-2) are detected with 100% accuracy. Query-level KXD failures (levels 3-4) require execute_chart_data=true and depend on Superset query execution behavior.
• SC-011: User can create a validation task (form fill + URL parse + provider select) in under 60 seconds.

Existing (v1, preserved)

• SC-001: Users can configure and test connection to at least one LLM provider.
• SC-003: Multimodal prompt (image + text) returns structured JSON analysis.
• SC-005: Screenshots capture full dashboard content including all tabs.

Edge Cases — v2

• What happens when a URL-parsed dashboard has been deleted before task execution? (System marks source as invalid, skips it, reports in task log.)
• What happens when screenshot_enabled=false but no dataset info is retrievable? (System proceeds with logs-only analysis, flags missing data sources.)
• What happens when a task has 20 dashboards and each takes 30s? (System executes sequentially with configurable concurrency limit, default 3.)
• What happens when user clicks «Run Now» while the same task is already executing? (System shows toast «Task is already running», does NOT queue a duplicate run.)
• What happens when user clicks «Run Now» while another task is executing? (System places the manual trigger in a FIFO queue, respecting concurrency_limit. If queue is full — shows toast with estimated wait time.)
• What happens when the API key for a provider used by existing tasks is changed? (Tasks always fetch the current key from LLMProviderService at execution time — no key caching in task configuration. Old key is never reused.)
• What happens when multi-chunk screenshots (15+ tabs) exceed the LLM provider's token limit? (System calculates estimated payload size before sending. If >80% of model context window: reduces JPEG quality progressively to 30%, then width to 800px. If still exceeds — falls back to Path B text-only with a warning. All decisions logged via molecular-cot: REASON before each reduction, REFLECT after outcome.)
• What happens if WebP encoding fails during archive step? (System falls back to PNG for archive with a WARN log. JPEG for LLM is unaffected — the pipeline is independent: LLM path and archive path are decoupled.)
• What happens when user toggles screenshot_enabled on a task with a custom prompt? (System shows warning: «Prompt may no longer match execution path» per FR-048a. Task saves successfully with user confirmation.)
• What happens when old API consumers send POST /api/tasks with plugin_id: llm_dashboard_validation after v2 migration? (Returns 400 Bad Request with message: «llm_dashboard_validation is deprecated. Use POST /api/validation-tasks instead.» All consumers — assistant tool, external scripts — must be updated to v2 API during migration.)
• What happens when a URL-parsed dashboard changes structure between task executions? (System re-parses the URL on every execution. If the URL fails — dashboard structure has changed. System marks the source with a warning: «Dashboard structure changed. Paste a new URL to update filters.» If dashboard_id changed — logs WARN and uses the new ID.)
• What happens when a user tries to delete an LLM provider that is used by active validation tasks? (System forbids deletion and returns a list of blocking task names. User must reassign the provider in each listed task or deactivate the tasks before deleting the provider.)
• What happens when Path B POST /api/v1/chart/data overloads Superset? (Toggle is off by default; user explicitly opts in with a warning about load.)
• What happens when SupersetContextExtractor.parse_superset_link() returns partial recovery? (System flags the source with partial_recovery, shows warning in task detail, attempts best-effort validation.)
• What happens when a dashboard URL contains a permalink key that has expired? (System marks the source as stale, prompts user to paste a fresh URL.)

(End of file — 214 lines)