Files

busya 652de471d2 tasks

2026-05-31 17:38:43 +03:00

32 KiB

Raw Blame History

Tasks: LLM Analysis & Documentation Plugins v2

Feature: 017-llm-analysis-plugin Status: v1 Completed; v2 Planned Spec: spec.md | Plan: plan.md | Data Model: data-model.md

User Stories & Priorities

ID	Story	Priority	Phase
US1	Create & Schedule Validation Task	P1	3
US2	Validation Task List & History	P1	3
US3	Dual-Path Dashboard Health Analysis	P1	4, 5
US4	URL Parsing Integration	P1	6
US5	Provider/Prompt Per-Task + Settings Cleanup	P2	7
US6	Dashboard Hub Cleanup	P2	8
US7	Dataset Documentation (unchanged)	P2	—
US8	Git Commit Suggestion (unchanged)	P3	—

Dependencies

Phase 1 (Setup)
    └─→ Phase 2 (Foundational)
            ├─→ Phase 3 (US1 + US2) — Task CRUD
            │       └─→ Phase 7 (US5) — Provider/Prompt
            ├─→ Phase 4 (US3 Path B) — Text-only execution
            ├─→ Phase 5 (US3 Path A) — Multi-chunk screenshots
            ├─→ Phase 6 (US4) — URL parsing
            └─→ Phase 8 (US6) — Hub cleanup (independent)
                    └─→ Phase 9 (Polish)

Parallel: Phase 4 + 5 + 6 can run concurrently after Phase 2. Phase 7 is tight with Phase 3. Phase 8 is independent.

Phase 1: Setup — Database Migrations

Goal: Schema changes for v2 models. Blocks everything.

T001 [P] Add sources ORM relationship + source_snapshot JSON (audit only, nullable) to ValidationPolicy in backend/src/models/llm.py. No JSON denormalization column for authoritative state — relational ValidationSource table is the single source of truth.
T002 [P] Create ValidationSource SQLAlchemy model in backend/src/models/llm.py (id, policy_id FK, type, value, parsed_context JSON, is_valid, title, last_error)
T003 [P] Add new columns to ValidationPolicy model: provider_id (String), prompt_template (Text, nullable), screenshot_enabled (Boolean, default true), logs_enabled (Boolean, default true), execute_chart_data (Boolean, default false), llm_batch_size (Integer, default 1 — full isolation), policy_dashboard_concurrency_limit (Integer, default 3) in backend/src/models/llm.py
T003a [P] Add global_validation_worker_limit config setting (default: 3) — max concurrent policy runs across all tasks. Stored in app config, not per-policy. in backend/src/core/config_models.py
T004 [P] Add new columns to ValidationRecord model: execution_path (String), source_id (FK → ValidationSource, nullable), dataset_health (JSON, nullable), chart_data_results (JSON, nullable), tab_screenshots (JSON, nullable), token_usage (JSON, nullable), dataset_key (String, nullable) in backend/src/models/llm.py
T004a [P] Create ValidationRun SQLAlchemy model in backend/src/models/llm.py: id, policy_id (FK), task_id (nullable), started_at, finished_at, trigger (manual/scheduled), status (completed/partial/failed), dashboard_count, pass_count, warn_count, fail_count, unknown_count
T005 Create Alembic migration for all T001–T004a schema changes in backend/alembic/versions/
T006 Write data migration script: convert existing dashboard_ids: list[str] → sources: [{type: "dashboard_id", value: id, is_valid: true}] in backend/src/plugins/llm_analysis/migrations/
T007 [P] Add DEFAULT_LLM_PROMPTS["dashboard_validation_prompt_multimodal"] (Path A, image-focused) and DEFAULT_LLM_PROMPTS["dashboard_validation_prompt_text"] (Path B, topology-focused) to backend/src/services/llm_prompt_templates.py

Verification: pytest backend/tests/models/test_llm_v2.py -v — all models create/query correctly. Migration runs without data loss.

Phase 2: Foundational — Services & Refactored Plugin Core

Goal: ValidationTaskService, DatasetHealthChecker, refactored ScreenshotService (multi-chunk), LLMClient dual-mode. Blocks US1–US6.

2.1 ValidationTaskService

T008 [P] Implement ValidationTaskService.create_task() — creates ValidationPolicy + ValidationSource[] rows, validates provider_id against execution path, in backend/src/services/validation_service.py
T009 [P] Implement ValidationTaskService.get_task(id) — returns policy + sources with current status, in backend/src/services/validation_service.py
T010 [P] Implement ValidationTaskService.list_tasks(env_id) — returns all policies with aggregated last-run status, in backend/src/services/validation_service.py
T011 [P] Implement ValidationTaskService.update_task(id, data) — updates policy fields + sources, in backend/src/services/validation_service.py
T012 [P] Implement ValidationTaskService.delete_task(id) — deletes policy + cascades to sources + records, in backend/src/services/validation_service.py
T013 [P] Implement ValidationTaskService.trigger_run(policy_id) — checks FR-054 (same task not already running via ValidationRun with status=running), creates a ValidationRun record (FR-055d), enqueues dashboard validation tasks per FR-055 (FIFO with global_validation_worker_limit), in backend/src/services/validation_service.py
T014 [P] Implement ValidationTaskService.get_run_history(policy_id, page) — paginated ValidationRun list with aggregate counts, in backend/src/services/validation_service.py
T015 [P] Implement ValidationTaskService.get_run_detail(run_id) — full run with all dashboard records, aggregate counts, in backend/src/services/validation_service.py
T015a [P] Implement ValidationTaskService._finalize_run(run_id) — computes aggregate counts from all ValidationRecords with this run_id, sets finished_at and status, in backend/src/services/validation_service.py

2.2 DatasetHealthChecker (Path B)

T016 [P] Implement DatasetHealthChecker class in backend/src/plugins/llm_analysis/service.py:
- check_dataset_health(dataset_id: int, client: SupersetClient) → DatasetHealth — calls GET /api/v1/dataset/{id}, extracts database name, backend, kind, verifies accessibility
- check_dashboard_datasets(dashboard_id: int, chart_list: list[dict], client: SupersetClient, execute_data: bool) → tuple[list[DatasetHealth], list[ChartDataResult]]

2.3 ScreenshotService Refactor (multi-chunk for Path A)

T017 [P] Implement ScreenshotService.capture_dashboard_chunks(dashboard_id, parsed_filters, output_dir) → list[{tab_name, path}] in backend/src/plugins/llm_analysis/service.py:
- Login → navigate with filters/tabs → iterate tabs → per-tab viewport (1920×1200) → CDP screenshot → collect PNG paths

2.3b Image Pipeline (PNG → JPEG + WebP)

T017a [P] Implement ScreenshotService._convert_screenshots_for_llm(png_paths: list[str]) → list[str] in backend/src/plugins/llm_analysis/service.py:
- PNG → Pillow convert → JPEG quality=60, max 1024px → save as temp JPEG → return paths
- JPEG files deleted after LLM call succeeds
T017b [P] Implement ScreenshotService._archive_screenshots_as_webp(png_paths: list[str], output_dir: str) → list[{tab_name, webp_path}] in backend/src/plugins/llm_analysis/service.py:
- PNG → Pillow convert → WebP quality=80, lossy → save to {output_dir}/{tab_name}_{timestamp}.webp
- Delete PNG after WebP saved
- Fallback: if WebP encode fails → keep PNG, log WARN (FR-059 edge case)
T017c [P] Implement ScreenshotService._cleanup_temp_files(paths: list[str]) in backend/src/plugins/llm_analysis/service.py:
- Delete PNG and JPEG intermediates after successful archive + LLM call

2.4 LLMClient Dual-Mode

T018 [P] Implement LLMClient.analyze_dashboard_multimodal(screenshot_paths: list[str], logs: list[str], prompt: str) → dict in backend/src/plugins/llm_analysis/service.py:
- Sends all images in single content[] array → multimodal LLM → JSON result
T019 [P] Implement LLMClient.analyze_dashboard_text(topology_text: str, dataset_health: list[dict], logs: list[str], prompt: str) → dict in backend/src/plugins/llm_analysis/service.py:
- Text-only LLM call with dashboard topology → JSON result
T020 [P] Implement LLMClient._estimate_payload_size() and LLMClient._reduce_image_quality() in backend/src/plugins/llm_analysis/service.py:
- FR-056: estimate → progressive reduction (JPEG quality 60→30, width 1024→800) → Path B fallback
- Belief-runtime: belief_scope("payload_reduction") + reason() before each reduction + reflect() after outcome

2.5 Plugin Refactor (dispatch to Path A / Path B)

T021 Implement DashboardValidationPlugin._build_dashboard_topology(dashboard_data: dict, charts_data: list[dict]) → str in backend/src/plugins/llm_analysis/plugin.py:
- Recursive position_json parsing → tab → row → chart hierarchy → text description
T022 Refactor DashboardValidationPlugin.execute() to dispatch based on screenshot_enabled:
- True → _execute_path_a(): call ScreenshotService → LLMClient.analyze_dashboard_multimodal()
- False → _execute_path_b(): call API chain → DatasetHealthChecker → LLMClient.analyze_dashboard_text()
- Belief-runtime: belief_scope("execute_path_a") / belief_scope("execute_path_b") + reason() at entry + reflect() at exit

2.7 Security & Retention (new)

T024a [P] Implement RedactionService in backend/src/plugins/llm_analysis/service.py:
- redact_logs(logs: list[str]) → list[str] — strip PII, credentials, query text, tokens from URLs, native filter values
- redact_raw_response(raw: str) → str — strip sensitive data before persistence
- Applied at capture boundary (before LLM send + before DB save) per FR-029/FR-029a
T024b [P] Add retention config to app settings: LLM_SCREENSHOT_RETENTION_DAYS=30, LLM_RAW_RESPONSE_RETENTION_DAYS=30 in backend/src/core/config_models.py
T024c [P] Rename all «molecular-cot» references in logs to «decision audit logging»: REASON/REFLECT markers use structured JSON with deterministic metadata only (no free-form reasoning text, no PII, no screenshot content) per FR-029b. Update docstrings in backend/src/core/logger.py and skill files.
T023 [P] Add dashboard_validation_prompt_multimodal (Path A) template to DEFAULT_LLM_PROMPTS in backend/src/services/llm_prompt_templates.py:
- «Analyze the attached {N} dashboard tab screenshots and execution logs for visual and data health issues...»
T024 [P] Add dashboard_validation_prompt_text (Path B) template to DEFAULT_LLM_PROMPTS in backend/src/services/llm_prompt_templates.py:
- «Analyze the following dashboard topology, dataset health, and execution logs for data consistency and KXD connectivity issues...»

Verification: pytest backend/tests/services/test_validation_task_service.py backend/tests/plugins/llm_analysis/test_service.py -v

Phase 3: US1 + US2 — Task CRUD API + Task List Page

Goal: REST API for validation tasks + frontend task list and create/edit form.

3.1 API Routes

T025 [US1] Implement POST /api/validation-tasks — creates ValidationPolicy + ValidationSource[] in backend/src/api/routes/validation_tasks.py
T026 [US1] Implement GET /api/validation-tasks — list all tasks with aggregated last-run status in backend/src/api/routes/validation_tasks.py
T027 [US2] Implement GET /api/validation-tasks/{policy_id} — task detail + per-source status in backend/src/api/routes/validation_tasks.py
T028 [US1] Implement PUT /api/validation-tasks/{policy_id} — update task in backend/src/api/routes/validation_tasks.py
T029 [US1] Implement DELETE /api/validation-tasks/{policy_id} — delete task (cascade sources + records), in backend/src/api/routes/validation_tasks.py
T030 [US1] Implement POST /api/validation-tasks/{policy_id}/run — manual trigger (checks FR-054 + FR-055) in backend/src/api/routes/validation_tasks.py
T031 [US2] Implement GET /api/validation-tasks/{policy_id}/runs — paginated run history (ValidationRun list, not individual records) in backend/src/api/routes/validation_tasks.py
T032 [US2] Implement GET /api/validation-tasks/{policy_id}/runs/{run_id} — full run detail: run metadata + all per-dashboard ValidationRecords with dataset_health, screenshots, logs, timings, in backend/src/api/routes/validation_tasks.py
T032b Implement legacy redirect: GET /reports/llm/{taskId} → 302 to /validation-tasks/{policy_id}/runs/{run_id} when taskId maps to ValidationRun.task_id, in backend/src/api/routes/validation_tasks.py

3.2 Task List Page

T035 [US2] Create frontend/src/routes/validation-tasks/+page.svelte — task list page with columns: name, schedule, last run status (PASS/WARN/FAIL/UNKNOWN), dashboard count, status (active/inactive), actions (edit/delete/run now)
T036 [US2] Create frontend/src/routes/validation-tasks/+page.js — load function calling GET /api/validation-tasks

3.3 Task Create/Edit Form

T037 [US1] Create frontend/src/routes/validation-tasks/new/+page.svelte — task creation page
T038 [US1] Create frontend/src/components/llm/ValidationTaskForm.svelte — multi-step form component:
- Step 1: Name, description, environment selector
- Step 2: Sources — dashboard ID multi-select + URL paste tabs
- Step 3: Analysis options — provider selector (filtered per FR-050), prompt textarea (pre-filled with path-specific default: multimodal when screenshot_enabled=true, text when false), screenshot toggle, logs toggle, execute_chart_data toggle (Path B only). When toggling screenshot_enabled on a task with custom prompt → show warning «Prompt may no longer match execution path» (FR-048a)
- Step 4: Schedule — cron input + preview next runs
T039 [US1] Create frontend/src/routes/validation-tasks/[policyId]/edit/+page.svelte — task edit page (reuses ValidationTaskForm)

3.5 Task Detail Page + Report Page

T040 [US2] Create frontend/src/routes/validation-tasks/[policyId]/+page.svelte — task detail page:
- Task info (name, environment, schedule, provider)
- Per-source breakdown table: dashboard title, last status, issues count, last run timestamp, link to latest run
- Run history table: timestamp, trigger type, aggregate status (N PASS / M WARN / K FAIL), duration, link to run detail
- Actions: Run Now, Edit, Delete
T041 [US2] Create frontend/src/routes/validation-tasks/[policyId]/+page.js — load function calling GET /api/validation-tasks/{id} + GET /api/validation-tasks/{id}/runs
T041a [US2] Create frontend/src/routes/validation-tasks/[policyId]/runs/[runId]/+page.svelte — multi-dashboard run report page (FR-055b):
- Run header: policy name, timestamp, trigger, duration, aggregate badge (e.g. «3 PASS • 1 WARN • 0 FAIL»)
- Collapsible dashboard list: each row = one dashboard, shows status icon + issue count + mini-summary
- Expanded dashboard row: full issues list + execution path badge (Path A/B) + dataset health table (Path B) + screenshot thumbnails with tab selector (Path A) + logs sent to LLM + task execution logs
- Handles mixed runs: some dashboards Path A, some Path B — renders appropriate sections per dashboard
T041b Implement GET /api/validation-tasks/{policy_id}/runs/{run_id} — returns full ValidationRun with all dashboard records, each containing dataset_health, chart_data_results, tab_screenshots, token_usage, timings. Also resolves policy name, provider name, and dashboard titles for display.

3.6 Shared Report Component

T042 [US1] Create frontend/src/components/llm/ValidationTaskReport.svelte — shared report component:
- Check result badge (PASS/WARN/FAIL) with getCheckResultClasses()
- Summary text
- Issues table (severity, message, location)
- Execution path indicator (Path A icon / Path B icon)
- Placeholder slots for Path A (screenshots) and Path B (dataset health)

Verification: Create task via UI → list shows task → detail shows sources → run now → report renders.

Phase 4: US3 — Path B Execution (Text-Only + Batching)

Goal: Text-only validation with dataset health checking, optional chart data execution, and LLM batching.

T043 [US3] Implement DashboardValidationPlugin._execute_path_b(run_id, dashboard_sources, params, context) in backend/src/plugins/llm_analysis/plugin.py:
1. For each dashboard: collect topology via _build_dashboard_topology() + dataset health via DatasetHealthChecker + logs
2. Default: llm_batch_size=1 — one LLM call per dashboard (full isolation). Values >1 group dashboards into batches per FR-045a (experimental, documented accuracy risk in research decision 15). Path A dashboards excluded from batching
3. For each batch: build structured prompt with per-dashboard labeled sections + strict JSON response schema {dashboards: [{dashboard_id, status, summary, issues}]}
4. Call LLMClient.analyze_dashboard_text_batch(dashboard_payloads, prompt) → parse response
5. Missing/parse-error dashboards → mark UNKNOWN individually, preserve other dashboards in batch
6. Persist individual ValidationRecord per dashboard, linked to run_id
- Belief-runtime: belief_scope("execute_path_b_batch") + reason(f"batch {i}/{total}: {n} dashboards") + reflect() with token_usage per batch
T043a [US3] Implement LLMClient.analyze_dashboard_text_batch(payloads: list[dict], prompt: str) → dict in backend/src/plugins/llm_analysis/service.py:
- Builds prompt: per-dashboard labeled sections, enforced {dashboards: [{dashboard_id, ...}]} JSON schema
- Validates response completeness: every input dashboard_id must appear; missing → UNKNOWN fallback per-dashboard (not whole batch)
T044 [US3] Create frontend/src/components/llm/DatasetHealthTable.svelte
T045 [US3] Create frontend/src/components/llm/ChartDataResults.svelte

Rejected-path regression test:

T047 [US3] Add test: Path B batch of 5 dashboards, 1 with KXD error → only that dashboard UNKNOWN, other 4 preserved with valid results, in backend/tests/plugins/llm_analysis/test_path_b_batch.py
T047a [US3] Add test: batch_size=1 (full isolation) → each dashboard gets individual LLM call with no cross-contamination, in backend/tests/plugins/llm_analysis/test_path_b_isolated.py

Verification: Create task with 5 Path B dashboards, llm_batch_size=1 (default) → 5 individual LLM calls, full isolation. Opt-in llm_batch_size=3 → 2 LLM calls, structured prompt with per-dashboard isolation. Dashboard with broken dataset → only its record UNKNOWN, others preserved.

Phase 5: US3 — Path A Execution (Multi-Chunk Screenshots)

Goal: Per-tab screenshots + multimodal LLM + payload estimation.

T048 [US3] Implement DashboardValidationPlugin._execute_path_a(params, context) in backend/src/plugins/llm_analysis/plugin.py:
1. Parse URL filters if source is URL type (re-parse per FR-057)
2. ScreenshotService.capture_dashboard_chunks() → per-tab screenshots
3. GET /api/v1/log/ → execution logs
4. LLMClient._estimate_payload_size() → check token budget per FR-056
5. Quality reduction loop if needed (FR-056)
6. LLMClient.analyze_dashboard_multimodal() → JSON result
7. Fallback to _execute_path_b() if payload still exceeds limit
8. Persist ValidationRecord with execution_path="screenshot", tab_screenshots, token_usage
- Belief-runtime: belief_scope("execute_path_a") + reason() before payload reduction + reflect() after each reduction + reason() on Path B fallback
T049 [US3] Update frontend/src/routes/validation-tasks/[policyId]/runs/[runId]/+page.svelte — add Path A sections: tab selector (from tab_screenshots), per-tab image viewer with zoom, issues table with tab/chart location. Legacy /reports/llm/[taskId] is a redirect-only passthrough — NOT updated.

Rejected-path regression test:

T050 [US3] Add test: multi-chunk screenshots exceed token limit → quality reduction triggered → Path B fallback when still exceeded, logged via molecular-cot, in backend/tests/plugins/llm_analysis/test_payload_reduction.py

Verification: Create task with screenshot_enabled=true, 5-tab dashboard → report shows per-tab screenshots + issues.

Phase 6: US4 — URL Parsing Integration

Goal: Paste Superset URL → parse → use in task creation + execution.

T051 [US4] Implement POST /api/validation-tasks/parse-url in backend/src/api/routes/validation_tasks.py:
- Receives {url, environment_id}
- Calls SupersetContextExtractor.parse_superset_link(url)
- Returns {dashboard_id, title, native_filters, activeTabs, anchor, partial_recovery, warnings}
T052 [US4] Create frontend/src/components/llm/UrlParser.svelte:
- URL paste input with debounced parse
- Loading state during API call
- Success: green banner with dashboard title, native_filters count, activeTabs list
- Error: red banner with partial_recovery warnings or parse failure message
- Copy button: «Dashboard structure changed? Paste new URL»
T053 [US4] Integrate UrlParser into ValidationTaskForm.svelte — add «Paste URL» tab in source selector (Step 2)
T054 [US4] Implement URL re-parse on execution per FR-057 in DashboardValidationPlugin:
- For URL-type sources: call SupersetContextExtractor.parse_superset_link() on every run
- If fails → mark source is_valid=false, set last_error, log WARN
- If dashboard_id changed → log WARN, use new ID

Rejected-path regression test:

T055 [US4] Add test: URL source with deleted dashboard → re-parse fails → source marked invalid, error logged, task skips source and continues with other valid ones, in backend/tests/plugins/llm_analysis/test_url_reparse.py

Verification: Paste Superset URL with filters → form shows dashboard title + 3 native filters + 2 active tabs → create task → execution re-parses URL successfully.

Phase 7: US5 — Provider/Prompt Per-Task + Settings Cleanup

Goal: Remove dashboard_validation binding from LLM settings. Provider + prompt configured per-task.

T056 [US5] Remove "dashboard_validation": "" from DEFAULT_LLM_PROVIDER_BINDINGS in backend/src/services/llm_prompt_templates.py
T057 [US5] Remove dashboard_validation binding UI row from frontend/src/routes/admin/settings/llm/+page.svelte
T058 [US5] Update LLM settings API schema to exclude dashboard_validation from provider_bindings payload in backend/src/schemas/llm.py
T059 [US5] Implement provider selector filter logic in ValidationTaskForm.svelte (Step 3):
- screenshot_enabled=true → show only multimodal providers (is_multimodal=true)
- screenshot_enabled=false → show all active providers
T060 [US5] Add ValidationTaskService.create_task() validation: reject if screenshot_enabled=true but provider is_multimodal=false

Rejected-path regression test:

T061 [US5] Add test: create task with screenshot_enabled=true and non-multimodal provider → HTTP 400, in backend/tests/api/test_validation_tasks_api.py
T061a [US5] Implement DELETE /api/llm/providers/{id} enforcement: query active ValidationPolicy references, return 409 with {error, blocking_tasks: [{id, name}]} (FR-058), in backend/src/api/routes/llm.py
T062 [US5] Add test: delete provider with active tasks → HTTP 409 with blocking task list (FR-058), in backend/tests/api/test_llm_provider_delete.py
T063 [US5] Add test: POST /api/tasks with plugin_id=llm_dashboard_validation → HTTP 400 with migration message (FR hard cut), in backend/tests/api/test_tasks_deprecated_llm.py

Verification: LLM settings page shows only 3 bindings. Create task with multimodal provider → works. Create with non-multimodal + screenshot → rejected.

Phase 8: US6 — Dashboard Hub Cleanup + Validation Indicator

Goal: Remove Validate button + LLM status column. Add lightweight validation indicator per dashboard row + detail page history.

T064 [US6] Remove handleValidate() function from frontend/src/routes/dashboards/+page.svelte
T065 [US6] Remove LLM validation status column from dashboard hub grid (column header + getLlmSummaryLabel() + getValidationBadgeClass() rendering) in frontend/src/routes/dashboards/+page.svelte
T065a [US6] Implement GET /api/dashboards/{env_id}/validation-status in backend/src/api/routes/dashboards.py — batch endpoint accepting dashboard_ids list, returns {dashboard_id: {status, last_run_at, task_name, run_id}} per dashboard
T065b [US6] Add validation indicator dot (PASS/WARN/FAIL/UNKNOWN) next to dashboard title in hub rows. Click opens dropdown with last 3 runs (date, aggregate status across all dashboards in that run, task name) + «View all» link → /validation-tasks/{policyId}/runs/{runId}, in frontend/src/routes/dashboards/+page.svelte
T066 [US6] Remove «Validate» action from action dropdown menu in frontend/src/routes/dashboards/+page.svelte
T067 [US6] Deprecate (comment out) normalizeValidationStatus() + getLlmSummaryLabel() — replaced by indicator + batch endpoint, in frontend/src/routes/dashboards/+page.svelte
T068 [US6] Remove LLM validation status loading from frontend/src/routes/dashboards/[id]/+page.svelte (old runLlmValidationTask() + openLlmReport())
T068a [US6] Add «Validation History» section to frontend/src/routes/dashboards/[id]/+page.svelte — last 10 runs for this dashboard across all tasks, links to canonical run detail pages
T069 [US6] Add «Create Validation Task» button to dashboard hub toolbar that navigates to /validation-tasks/new?ids=1,2,3 (pre-fills selected dashboard IDs) in frontend/src/routes/dashboards/+page.svelte
T070 [US6] Update frontend/src/lib/components/layout/TaskDrawer.svelte — ensure LLM validation tasks still show correct status after refactor

Verification: Dashboard hub has no Validate button, no LLM column. Colored dot indicator per row with dropdown. Detail page has Validation History section. «Create Validation Task» button pre-fills IDs.

Phase 9: Polish & Cross-Cutting Verification

Goal: Migration testing, backward compat, ADR updates, final verification.

9.1 Migration & Backward Compat

T071 Run data migration on test DB: verify all existing dashboard_ids → sources[] conversion in backend/tests/migrations/test_v2_data_migration.py
T072 Update assistant tool _tool_llm_validation.py — replace ad-hoc run_llm_validation dispatch with task creation via POST /api/validation-tasks, in backend/src/api/routes/assistant/_tool_llm_validation.py

9.2 ADR Updates

T073 Update docs/adr/ADR-0008-assistant-tool-registry.md — note that _tool_llm_validation.py now creates tasks instead of ad-hoc validation
T074 Update docs/adr/ADR-0004-plugin-architecture.md — mention v2 dual-path + task-based flow as plugin evolution example

9.3 Final Verification

T075 Run full end-to-end: create task → URL source (with filters) → Path A run → view report with per-tab screenshots
T076 Run full end-to-end: create task → dashboard ID sources → Path B run (execute_chart_data=true) → view report with dataset health + chart data
T077 Run full end-to-end: create task → Path B run against dashboard with broken dataset → verify LLM reports UNHEALTHY
T078 Run full end-to-end: delete provider used by active task → verify HTTP 409 with blocking task names
T079 Run full test suite: cd backend && source .venv/bin/activate && python -m pytest -v
T080 Run frontend tests: cd frontend && npm run test
T081 Run backend lint: cd backend && python -m ruff check .
T082 Run frontend lint: cd frontend && npm run lint
T083 Run semantic audit: axiom_semantic_validation audit_belief_protocol on all new C4/C5 contracts
T084 Run semantic audit: axiom_semantic_validation audit_contracts on backend/src/plugins/llm_analysis/ and backend/src/services/validation_service.py

Verification Criteria per User Story

Story	Independent Verification
US1	Create task with 2 dashboards (ID + URL), Path A, cron `0 8 * * *` → task appears in list → scheduled execution succeeds → report shows per-tab screenshots
US2	Task list shows 3 tasks with different statuses → click task → detail shows per-dashboard breakdown → click dashboard → full report renders
US3	Path B against broken dataset → report shows dataset UNHEALTHY + affected charts. Path A against dashboard with visual error → report shows issue with tab/chart location
US4	Paste Superset URL with native_filters → parse returns dashboard title + 3 filters + 2 tabs → saved in ValidationSource → re-parsed on execution
US5	LLM settings shows 3 bindings (no dashboard_validation). Create task with multimodal provider → works. Create with non-multimodal + screenshot → rejected
US6	Dashboard hub: no Validate button, no LLM column. «Create Validation Task» toolbar button pre-fills selected IDs

Rejected-Path Regression Coverage

Rejected Path	Test Task	Expected Behavior
Old `POST /api/tasks` with `llm_dashboard_validation`	T063	HTTP 400: «deprecated, use /api/validation-tasks»
Provider deletion when used by active tasks	T062	HTTP 409: returns blocking task names
Duplicate manual trigger of same task	T013 (FR-054 check)	Toast «Already running», no queue
Non-multimodal provider + screenshot task	T061	HTTP 400: «provider must be multimodal»
URL re-parse fails (dashboard deleted)	T055	Source marked invalid, task skips, continues with other sources
Multi-chunk token limit exceeded	T050	Quality reduction → Path B fallback, logged via molecular-cot
Path B dataset health check fails	T047	LLM report flags dataset UNHEALTHY + affected charts

Belief-Runtime Instrumentation Tasks

Contract	Tier	Task	Instrumentation
`LLMClient._estimate_payload_size` + `_reduce_image_quality`	C4	T020	`belief_scope("payload_reduction")` + `reason()` per reduction + `reflect()` on outcome
`DashboardValidationPlugin._execute_path_a`	C4	T048	`belief_scope("execute_path_a")` + `reason()` at navigation/tab/resize + `reflect()` with timings
`DashboardValidationPlugin._execute_path_b`	C4	T043	`belief_scope("execute_path_b")` + `reason()` at each API call + `reflect()` with token_usage
`ValidationTaskService.trigger_run`	C4	T013	`belief_scope("trigger_run")` + `reason()` on queue decision + `reflect()` after enqueue
`ScreenshotService.capture_dashboard_chunks`	C4	T017	`belief_scope("capture_chunks")` + `reason()` per tab switch + `reflect()` with screenshot count

Parallel Execution Opportunities

Group	Tasks	Dependencies
Phase 1	T001–T007	None (all parallel)
Phase 2	T008–T015 (service), T016 (health checker), T017 (screenshot), T018–T020 (LLM client)	After Phase 1
Phase 3	T025–T034 (API), T035–T042 (frontend)	After Phase 2
Phase 4+5+6	T043–T047 (Path B), T048–T050 (Path A), T051–T055 (URL)	After Phase 2, all three parallel
Phase 7+8	T056–T063 (provider/prompt), T064–T070 (hub)	After Phase 3, both parallel
Phase 9	T071–T084	After all phases

Total: 98 tasks across 9 phases

Phase	Tasks	User Stories
1 — Setup	8	—
2 — Foundational	23	—
3 — Task CRUD + Reports	22	US1, US2
4 — Path B	5	US3
5 — Path A	3	US3
6 — URL Parsing	5	US4
7 — Provider/Prompt	8	US5
8 — Hub Cleanup	10	US6
9 — Polish	14	All

32 KiB Raw Blame History Unescape Escape

Tasks: LLM Analysis & Documentation Plugins v2

User Stories & Priorities

Dependencies

Phase 1: Setup — Database Migrations

Phase 2: Foundational — Services & Refactored Plugin Core

2.1 ValidationTaskService

2.2 DatasetHealthChecker (Path B)

2.3 ScreenshotService Refactor (multi-chunk for Path A)

2.3b Image Pipeline (PNG → JPEG + WebP)

2.4 LLMClient Dual-Mode

2.5 Plugin Refactor (dispatch to Path A / Path B)

2.7 Security & Retention (new)

Phase 3: US1 + US2 — Task CRUD API + Task List Page

3.1 API Routes

3.2 Task List Page

3.3 Task Create/Edit Form

3.5 Task Detail Page + Report Page

3.6 Shared Report Component

Phase 4: US3 — Path B Execution (Text-Only + Batching)

Phase 5: US3 — Path A Execution (Multi-Chunk Screenshots)

Phase 6: US4 — URL Parsing Integration

Phase 7: US5 — Provider/Prompt Per-Task + Settings Cleanup

Phase 8: US6 — Dashboard Hub Cleanup + Validation Indicator

Phase 9: Polish & Cross-Cutting Verification

9.1 Migration & Backward Compat

9.2 ADR Updates

9.3 Final Verification

Verification Criteria per User Story

Rejected-Path Regression Coverage

Belief-Runtime Instrumentation Tasks

Parallel Execution Opportunities

Total: 98 tasks across 9 phases

32 KiB

Raw Blame History