ss-tools/specs/017-llm-analysis-plugin/plan.md

Implementation Plan: LLM Analysis & Documentation Plugins v2

Branch: 017-llm-analysis-plugin | Date: 2026-01-28 | Updated: 2026-06-07 — v2 implemented Spec: spec.md

Summary

v2 redesign replaces the ad-hoc «Validate» button with a task-based validation flow. Users create ValidationPolicy tasks with dashboard sources (IDs + URLs), select an LLM provider and prompt per-task, choose between two execution paths (Playwright screenshot or text-only API with dataset health checking), and set a schedule. The dashboard hub is simplified — LLM validation status moves to the Validation Tasks page.

Key architectural changes:

• Task-based flow: New /validation-tasks page + CRUD API + ValidationSource model for dashboard IDs/URLs
• Dual-path execution: DashboardValidationPlugin.execute() dispatches to _execute_path_a() (screenshot) or _execute_path_b() (text-only) based on screenshot_enabled
• Multi-chunk screenshots: ScreenshotService.capture_dashboard_chunks() captures per-tab images instead of one full-page
• Dataset health checker (new): DatasetHealthChecker verifies KXD connectivity for every unique dataset
• URL parsing (reused): SupersetContextExtractor.parse_superset_link() for URL-based dashboard sources
• Provider/prompt per-task: Removed from global LLM settings; configured in task creation form

Technical Context

Language/Version: Python 3.13+ (Backend, per ADR-0001), Node.js 18+ (Frontend) Primary Dependencies (real versions from repository):

• Backend: fastapi==0.126.0, pydantic==2.12.5, httpx==0.28.1, openai (latest), playwright (latest), tenacity (latest)
• Frontend: svelte==^5.43.8, @sveltejs/kit==^2.49.2, tailwindcss==^3.0.0, vite==^7.2.4 Storage: SQLite (tasks.db, auth.db, plugins.db) via SQLAlchemy Testing: pytest (Backend), vitest (Frontend) Deployment: Docker Compose (backend + frontend + nginx per docker/) Performance Goals:
• Path B (text-only, no chart data): < 30s per dashboard (excluding LLM latency)
• Path A (screenshot): < 120s per dashboard
• Task creation (form fill + URL parse): < 60s for user Constraints: Must integrate with existing PluginBase and TaskManager. Secure storage for API keys (AES-256). Modules < 400 lines per ADR-0001.

v2 Implementation Notes

Path B — Dashboard Topology Builder

The text-only path builds a structured description of the dashboard from API responses:

1. GET /api/v1/dashboard/{id}
   → dashboard_title, position_json (layout tree), slices[], json_metadata

2. Parse position_json recursively:
   - TABS → children[] → ROW → children[] → CHART
   - Build tab → chart hierarchy with slice_id references

3. For each unique chart: GET /api/v1/chart/{slice_id}
   → slice_name, viz_type, params (metrics, groupby, filters, time_range), datasource_id

4. For each unique dataset: GET /api/v1/dataset/{datasource_id}
   → database (name, backend), sql (virtual datasets), kind, is_managed_externally

5. Optionally: POST /api/v1/chart/data with form_data from chart params
   → Verify query execution, row count, duration

6. GET /api/v1/log/ (filter: dashboard_id, 24h)
   → Execution logs

7. Build text prompt:
   Dashboard: "{title}" (id: {id})
   ─── Tab: "{tab_name}" ({N} charts) ───
     Chart "{chart_name}" ({viz_type})
       Dataset: {dataset_name} ({kind}, backend: {backend})
       Params: {metrics}, grouped by {groupby}, filters: {filters}
       Health: ✓ accessible / ✗ {error}
     ...
   Logs (last 24h, {N} records):
     [{timestamp}] {action}: {details}

Path A — Multi-Chunk Screenshot

The screenshot service is refactored to capture per-tab screenshots:

1. Login (unchanged)
2. Navigate to dashboard URL (with parsed filters/tabs from URL source)
3. Iterate tabs recursively (unchanged logic, but now save a screenshot per tab)
4. For each tab: resize viewport to 1920×1200, wait for charts, CDP screenshot
5. Return list of {tab_name, screenshot_path} to LLMClient
6. LLMClient sends all chunks as separate image_url blocks in a single content[] array

URL Parsing (from dataset review)

The SupersetContextExtractor is already implemented and tested for dataset review. For validation tasks, we reuse its parse_superset_link() method:

1. User pastes URL in task creation form
2. Frontend calls POST /api/validation-tasks/parse-url with {url, environment_id}
3. Backend calls extractor.parse_superset_link(url) → SupersetParsedContext
4. Returns {dashboard_id, title, native_filters, activeTabs, anchor, partial_recovery, warnings}
5. Frontend shows preview: dashboard title + filter names + tab names
6. On task save: ValidationSource.value = url, ValidationSource.parsed_context = {...}

Prompt Templates for Dual Paths

Two path-specific defaults in DEFAULT_LLM_PROMPTS:

"dashboard_validation_prompt_multimodal": """..."""  # Path A: image-focused
"dashboard_validation_prompt_text": """..."""         # Path B: topology-focused

The v1 legacy key dashboard_validation_prompt is retained for backward reference but MUST NOT be used as default for new tasks. Each task has a single prompt_template field:

• null → system auto-selects path-specific default based on screenshot_enabled
• non-null → custom prompt used regardless of path (with FR-048a warning on path toggle)

Contract: DashboardValidationPlugin.execute() reads prompt_template from params. If null, resolves: screenshot_enabled ? multimodal_prompt : text_prompt.

Constitution Check

GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.

Source: .specify/memory/constitution.md v1.0.0

• I. Semantic Contract First — All v2 contracts use #region/#endregion anchors with [C:N] complexity tiers. @RELATION edges trace dependencies. @RATIONALE/@REJECTED document architectural choices. No naked code. → Verified: 6 contract modules defined in contracts/modules.md with proper anchoring.
• II. Decision Memory — D1–D9 design decisions in spec.md with rationale. D8–D13 research decisions in research.md with Decision/Rationale/Alternatives/Impact format. No @REJECTED path silently reintroduced. → Verified: 9 spec decisions + 6 research decisions all documented.
• III. External Orchestrator — No change to orchestrator pattern (ADR-0003). DashboardValidationPlugin remains an external plugin, not embedded in Superset. DatasetHealthChecker is a service within superset-tools, not a Superset modification. → PASS
• IV. Module Discipline — New files: validation_tasks.py (API routes, ~150 LOC), DatasetHealthChecker class (~120 LOC), ValidationTaskForm.svelte (~300 LOC), UrlParser.svelte (~150 LOC). All under 400-line limit. Function CC ≤ 10. No cyclic imports (routes → services → models direction enforced per ADR-0001). → PASS
• V. RBAC Enforcement — Validation task CRUD requires existing RBAC roles (plugin:llm:validate, plugin:llm:configure per ADR-0005). Assistant tool _tool_llm_validation.py inherits permission checks from @assistant_tool decorator (ADR-0008). → PASS
• VI. Frontend — Svelte 5 Runes Only — New components: ValidationTaskForm.svelte, UrlParser.svelte, DatasetHealthTable.svelte, ValidationTaskReport.svelte. All use $state, $derived, $props, $effect (no Svelte 4 syntax). No fromStore + $derived anti-pattern (ADR-0007). → PASS
• VII. Test-Driven for C3+ Contracts — ValidationTaskService [C:4] → test_validation_task_service.py. DatasetHealthChecker [C:3] → test_dataset_health_checker.py. DashboardValidationPlugin._execute_path_b [C:3] → test_path_b_execution.py. Each includes @REJECTED path regression test. → Planned in T108-T156.

GATE RESULT: CONDITIONAL PASS (AMBER) — No blocking constitutional conflicts. 7 of 7 principles pass. Spec completeness is AMBER per checklists/requirements.md: 12 UX/data/reliability checklist items remain as implementation-readiness markers (defined in FRs, to be validated during Phase 3+ implementation). Gate upgrades to GREEN after Phase 3 Task CRUD implementation validates form flows, report layout, and migration against checklist.

ADR Continuity

ADR	Impact of v2	Action Required
ADR-0001 (module layout)	New files: backend/src/api/routes/validation_tasks.py, frontend/src/routes/validation-tasks/. All fit within canonical boundaries (API routes → api/routes/, pages → routes/, components → lib/components/).	None — compliant
ADR-0002 (semantic protocol)	v2 contracts use GRACE-Poly v2.6 #region anchors with [C:N]. @RELATION edges trace DEPENDS_ON / CALLS / IMPLEMENTS.	None — compliant
ADR-0003 (orchestrator pattern)	No changes. superset-tools remains external orchestrator over Superset.	None
ADR-0004 (plugin architecture)	DashboardValidationPlugin stays a PluginBase implementation. DatasetHealthChecker is a service (not a plugin) — correct per ADR-0004 §Isolation Guarantees: plugins get no DB access; services do.	None — compliant
ADR-0005 (RBAC)	Existing plugin:llm:validate permission covers new task-triggered validation. Task CRUD requires plugin:llm:configure.	None — reused
ADR-0006 (frontend architecture)	New pages use SvelteKit file-based routing (+page.svelte). Components in lib/components/. API clients via $lib/api.js.	None — compliant
ADR-0007 (fromStore+derived)	No new uses of the rejected fromStore + $derived pattern.	None
ADR-0008 (assistant tool registry)	_tool_llm_validation.py must be updated: replace ad-hoc run_llm_validation dispatch with task creation via ValidationTaskService.	T153: Update assistant tool to create task instead of triggering ad-hoc validation
ADR-0009 (SSL)	No changes. LLM client SSL verification applies to both Path A and Path B.	None

Project Structure — v2 Changes

backend/src/
├── plugins/llm_analysis/
│   ├── plugin.py             # DashboardValidationPlugin (1 004 строки) — dispatch to Path A/B
│   ├── service.py            # ScreenshotService + LLMClient + DatasetHealthChecker (1 691 строка)
│   ├── _topology.py          # DashboardTopologyBuilder (290 строк)
│   ├── scheduler.py          # APScheduler integration (57 строк)
│   ├── models.py             # Plugin models (73 строки)
│   ├── migrations/           # v1_to_v2 migration (196 строк)
│   └── __tests__/            # 3 test files (655 строк)
├── services/
│   ├── validation_service.py # ValidationTaskService CRUD (767 строк)
│   └── llm_prompt_templates.py # Dual-path prompt defaults (259 строк)
├── api/routes/
│   ├── validation_tasks.py   # CRUD routes for /api/validation-tasks (481 строка)
│   └── llm.py                # LLM provider/settings routes (608 строк)
├── models/
│   └── llm.py               # LLM ORM models: ValidationPolicy, ValidationSource, ValidationRecord, ValidationRun (156 строк)

frontend/src/
├── routes/
│   ├── validation-tasks/               # Task list + detail + history + runs (6 страниц)
│   │   ├── +page.svelte                # Task list (204 строки)
│   │   ├── new/+page.svelte            # Task creation (95 строк)
│   │   ├── [policyId]/+page.svelte     # Task detail (398 строк)
│   │   ├── [policyId]/edit/+page.svelte # Task edit (102 строки)
│   │   ├── [policyId]/runs/[runId]/+page.svelte  # Run detail report (200 строк)
│   │   └── history/+page.svelte        # Run history (183 строки)
│   ├── dashboards/[id]/validation/+page.svelte  # Dashboard validation history (325 строк)
│   ├── reports/llm/[taskId]/+page.svelte        # Legacy report redirect (219 строк)
│   └── admin/settings/llm/+page.svelte          # LLM provider settings (258 строк)
├── lib/components/llm/
│   ├── ValidationTaskForm.svelte        # Multi-step task creation form (1 096 строк)
│   ├── ValidationTaskReport.svelte      # Shared report component (39 строк)
│   ├── UrlParser.svelte                # URL paste + parse preview (207 строк)
│   ├── ProviderConfig.svelte           # Provider configuration form (745 строк)
│   └── DocPreview.svelte               # Documentation preview (87 строк)
├── lib/i18n/locales/
│   ├── en/validation.json + llm.json   # English locale
│   └── ru/validation.json + llm.json   # Russian locale

Complexity Tracking

Violation	Why Needed	Simpler Alternative Rejected Because
Dual execution paths	Covers visual + data-layer failure modes	Single path leaves blind spots: screenshot-only misses KXD errors, text-only misses visual bugs
Multi-chunk screenshots	Long dashboards lose readability in single compressed image	Single full-page at higher resolution costs excessive image tokens
Reused URL parser from different feature	Avoids duplicating complex parsing logic for native_filters/permalink	Custom parser would duplicate 300+ lines of already-tested code