# Implementation Plan: LLM Analysis & Documentation Plugins v2

**Branch**: `017-llm-analysis-plugin` | **Date**: 2026-01-28 | **Updated**: 2026-06-07 — v2 implemented
**Spec**: [spec.md](spec.md)

## Summary

v2 redesign replaces the ad-hoc «Validate» button with a task-based validation flow. Users create `ValidationPolicy` tasks with dashboard sources (IDs + URLs), select an LLM provider and prompt per-task, choose between two execution paths (Playwright screenshot or text-only API with dataset health checking), and set a schedule. The dashboard hub is simplified — LLM validation status moves to the Validation Tasks page.

Key architectural changes:
- **Task-based flow**: New `/validation-tasks` page + CRUD API + `ValidationSource` model for dashboard IDs/URLs
- **Dual-path execution**: `DashboardValidationPlugin.execute()` dispatches to `_execute_path_a()` (screenshot) or `_execute_path_b()` (text-only) based on `screenshot_enabled`
- **Multi-chunk screenshots**: `ScreenshotService.capture_dashboard_chunks()` captures per-tab images instead of one full-page
- **Dataset health checker** (new): `DatasetHealthChecker` verifies KXD connectivity for every unique dataset
- **URL parsing** (reused): `SupersetContextExtractor.parse_superset_link()` for URL-based dashboard sources
- **Provider/prompt per-task**: Removed from global LLM settings; configured in task creation form

## Technical Context

**Language/Version**: Python 3.13+ (Backend, per ADR-0001), Node.js 18+ (Frontend)
**Primary Dependencies** (real versions from repository):
- Backend: `fastapi==0.126.0`, `pydantic==2.12.5`, `httpx==0.28.1`, `openai` (latest), `playwright` (latest), `tenacity` (latest)
- Frontend: `svelte==^5.43.8`, `@sveltejs/kit==^2.49.2`, `tailwindcss==^3.0.0`, `vite==^7.2.4`
**Storage**: SQLite (`tasks.db`, `auth.db`, `plugins.db`) via SQLAlchemy
**Testing**: `pytest` (Backend), `vitest` (Frontend)
**Deployment**: Docker Compose (backend + frontend + nginx per `docker/`)
**Performance Goals**:
- Path B (text-only, no chart data): < 30s per dashboard (excluding LLM latency)
- Path A (screenshot): < 120s per dashboard
- Task creation (form fill + URL parse): < 60s for user
**Constraints**: Must integrate with existing `PluginBase` and `TaskManager`. Secure storage for API keys (AES-256). Modules < 400 lines per ADR-0001.

## v2 Implementation Notes

### Path B — Dashboard Topology Builder

The text-only path builds a structured description of the dashboard from API responses:

```
1. GET /api/v1/dashboard/{id}
   → dashboard_title, position_json (layout tree), slices[], json_metadata

2. Parse position_json recursively:
   - TABS → children[] → ROW → children[] → CHART
   - Build tab → chart hierarchy with slice_id references

3. For each unique chart: GET /api/v1/chart/{slice_id}
   → slice_name, viz_type, params (metrics, groupby, filters, time_range), datasource_id

4. For each unique dataset: GET /api/v1/dataset/{datasource_id}
   → database (name, backend), sql (virtual datasets), kind, is_managed_externally

5. Optionally: POST /api/v1/chart/data with form_data from chart params
   → Verify query execution, row count, duration

6. GET /api/v1/log/ (filter: dashboard_id, 24h)
   → Execution logs

7. Build text prompt:
   Dashboard: "{title}" (id: {id})
   ─── Tab: "{tab_name}" ({N} charts) ───
     Chart "{chart_name}" ({viz_type})
       Dataset: {dataset_name} ({kind}, backend: {backend})
       Params: {metrics}, grouped by {groupby}, filters: {filters}
       Health: ✓ accessible / ✗ {error}
     ...
   Logs (last 24h, {N} records):
     [{timestamp}] {action}: {details}
```

### Path A — Multi-Chunk Screenshot

The screenshot service is refactored to capture per-tab screenshots:

1. Login (unchanged)
2. Navigate to dashboard URL (with parsed filters/tabs from URL source)
3. Iterate tabs recursively (unchanged logic, but now save a screenshot per tab)
4. For each tab: resize viewport to 1920×1200, wait for charts, CDP screenshot
5. Return list of `{tab_name, screenshot_path}` to LLMClient
6. LLMClient sends all chunks as separate `image_url` blocks in a single `content[]` array

### URL Parsing (from dataset review)

The `SupersetContextExtractor` is already implemented and tested for dataset review. For validation tasks, we reuse its `parse_superset_link()` method:

1. User pastes URL in task creation form
2. Frontend calls `POST /api/validation-tasks/parse-url` with `{url, environment_id}`
3. Backend calls `extractor.parse_superset_link(url)` → `SupersetParsedContext`
4. Returns `{dashboard_id, title, native_filters, activeTabs, anchor, partial_recovery, warnings}`
5. Frontend shows preview: dashboard title + filter names + tab names
6. On task save: `ValidationSource.value = url`, `ValidationSource.parsed_context = {...}`

### Prompt Templates for Dual Paths

Two path-specific defaults in `DEFAULT_LLM_PROMPTS`:

```python
"dashboard_validation_prompt_multimodal": """..."""  # Path A: image-focused
"dashboard_validation_prompt_text": """..."""         # Path B: topology-focused
```

The v1 legacy key `dashboard_validation_prompt` is retained for backward reference but **MUST NOT** be used as default for new tasks. Each task has a single `prompt_template` field:
- `null` → system auto-selects path-specific default based on `screenshot_enabled`
- `non-null` → custom prompt used regardless of path (with FR-048a warning on path toggle)

**Contract**: `DashboardValidationPlugin.execute()` reads `prompt_template` from params. If null, resolves: `screenshot_enabled ? multimodal_prompt : text_prompt`.

## Constitution Check

*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*

Source: `.specify/memory/constitution.md` v1.0.0

- [x] **I. Semantic Contract First** — All v2 contracts use `#region`/`#endregion` anchors with `[C:N]` complexity tiers. `@RELATION` edges trace dependencies. `@RATIONALE`/`@REJECTED` document architectural choices. No naked code. → Verified: 6 contract modules defined in `contracts/modules.md` with proper anchoring.
- [x] **II. Decision Memory** — D1–D9 design decisions in `spec.md` with rationale. D8–D13 research decisions in `research.md` with Decision/Rationale/Alternatives/Impact format. No `@REJECTED` path silently reintroduced. → Verified: 9 spec decisions + 6 research decisions all documented.
- [x] **III. External Orchestrator** — No change to orchestrator pattern (ADR-0003). `DashboardValidationPlugin` remains an external plugin, not embedded in Superset. `DatasetHealthChecker` is a service within superset-tools, not a Superset modification. → PASS
- [x] **IV. Module Discipline** — New files: `validation_tasks.py` (API routes, ~150 LOC), `DatasetHealthChecker` class (~120 LOC), `ValidationTaskForm.svelte` (~300 LOC), `UrlParser.svelte` (~150 LOC). All under 400-line limit. Function CC ≤ 10. No cyclic imports (routes → services → models direction enforced per ADR-0001). → PASS
- [x] **V. RBAC Enforcement** — Validation task CRUD requires existing RBAC roles (`plugin:llm:validate`, `plugin:llm:configure` per ADR-0005). Assistant tool `_tool_llm_validation.py` inherits permission checks from `@assistant_tool` decorator (ADR-0008). → PASS
- [x] **VI. Frontend — Svelte 5 Runes Only** — New components: `ValidationTaskForm.svelte`, `UrlParser.svelte`, `DatasetHealthTable.svelte`, `ValidationTaskReport.svelte`. All use `$state`, `$derived`, `$props`, `$effect` (no Svelte 4 syntax). No `fromStore` + `$derived` anti-pattern (ADR-0007). → PASS
- [x] **VII. Test-Driven for C3+ Contracts** — `ValidationTaskService` [C:4] → `test_validation_task_service.py`. `DatasetHealthChecker` [C:3] → `test_dataset_health_checker.py`. `DashboardValidationPlugin._execute_path_b` [C:3] → `test_path_b_execution.py`. Each includes `@REJECTED` path regression test. → Planned in T108-T156.

**GATE RESULT: CONDITIONAL PASS (AMBER)** — No blocking constitutional conflicts. 7 of 7 principles pass. Spec completeness is AMBER per `checklists/requirements.md`: 12 UX/data/reliability checklist items remain as implementation-readiness markers (defined in FRs, to be validated during Phase 3+ implementation). Gate upgrades to GREEN after Phase 3 Task CRUD implementation validates form flows, report layout, and migration against checklist.

## ADR Continuity

| ADR | Impact of v2 | Action Required |
|-----|-------------|-----------------|
| **ADR-0001** (module layout) | New files: `backend/src/api/routes/validation_tasks.py`, `frontend/src/routes/validation-tasks/`. All fit within canonical boundaries (API routes → `api/routes/`, pages → `routes/`, components → `lib/components/`). | None — compliant |
| **ADR-0002** (semantic protocol) | v2 contracts use GRACE-Poly v2.6 `#region` anchors with `[C:N]`. `@RELATION` edges trace `DEPENDS_ON` / `CALLS` / `IMPLEMENTS`. | None — compliant |
| **ADR-0003** (orchestrator pattern) | No changes. superset-tools remains external orchestrator over Superset. | None |
| **ADR-0004** (plugin architecture) | `DashboardValidationPlugin` stays a `PluginBase` implementation. `DatasetHealthChecker` is a service (not a plugin) — correct per ADR-0004 §Isolation Guarantees: plugins get no DB access; services do. | None — compliant |
| **ADR-0005** (RBAC) | Existing `plugin:llm:validate` permission covers new task-triggered validation. Task CRUD requires `plugin:llm:configure`. | None — reused |
| **ADR-0006** (frontend architecture) | New pages use SvelteKit file-based routing (`+page.svelte`). Components in `lib/components/`. API clients via `$lib/api.js`. | None — compliant |
| **ADR-0007** (fromStore+derived) | No new uses of the rejected `fromStore` + `$derived` pattern. | None |
| **ADR-0008** (assistant tool registry) | `_tool_llm_validation.py` must be updated: replace ad-hoc `run_llm_validation` dispatch with task creation via `ValidationTaskService`. | **T153**: Update assistant tool to create task instead of triggering ad-hoc validation |
| **ADR-0009** (SSL) | No changes. LLM client SSL verification applies to both Path A and Path B. | None |

## Project Structure — v2 Changes

```text
backend/src/
├── plugins/llm_analysis/
│   ├── plugin.py             # DashboardValidationPlugin (1 004 строки) — dispatch to Path A/B
│   ├── service.py            # ScreenshotService + LLMClient + DatasetHealthChecker (1 691 строка)
│   ├── _topology.py          # DashboardTopologyBuilder (290 строк)
│   ├── scheduler.py          # APScheduler integration (57 строк)
│   ├── models.py             # Plugin models (73 строки)
│   ├── migrations/           # v1_to_v2 migration (196 строк)
│   └── __tests__/            # 3 test files (655 строк)
├── services/
│   ├── validation_service.py # ValidationTaskService CRUD (767 строк)
│   └── llm_prompt_templates.py # Dual-path prompt defaults (259 строк)
├── api/routes/
│   ├── validation_tasks.py   # CRUD routes for /api/validation-tasks (481 строка)
│   └── llm.py                # LLM provider/settings routes (608 строк)
├── models/
│   └── llm.py               # LLM ORM models: ValidationPolicy, ValidationSource, ValidationRecord, ValidationRun (156 строк)

frontend/src/
├── routes/
│   ├── validation-tasks/               # Task list + detail + history + runs (6 страниц)
│   │   ├── +page.svelte                # Task list (204 строки)
│   │   ├── new/+page.svelte            # Task creation (95 строк)
│   │   ├── [policyId]/+page.svelte     # Task detail (398 строк)
│   │   ├── [policyId]/edit/+page.svelte # Task edit (102 строки)
│   │   ├── [policyId]/runs/[runId]/+page.svelte  # Run detail report (200 строк)
│   │   └── history/+page.svelte        # Run history (183 строки)
│   ├── dashboards/[id]/validation/+page.svelte  # Dashboard validation history (325 строк)
│   ├── reports/llm/[taskId]/+page.svelte        # Legacy report redirect (219 строк)
│   └── admin/settings/llm/+page.svelte          # LLM provider settings (258 строк)
├── lib/components/llm/
│   ├── ValidationTaskForm.svelte        # Multi-step task creation form (1 096 строк)
│   ├── ValidationTaskReport.svelte      # Shared report component (39 строк)
│   ├── UrlParser.svelte                # URL paste + parse preview (207 строк)
│   ├── ProviderConfig.svelte           # Provider configuration form (745 строк)
│   └── DocPreview.svelte               # Documentation preview (87 строк)
├── lib/i18n/locales/
│   ├── en/validation.json + llm.json   # English locale
│   └── ru/validation.json + llm.json   # Russian locale
```

## Complexity Tracking

| Violation | Why Needed | Simpler Alternative Rejected Because |
|-----------|------------|-------------------------------------|
| Dual execution paths | Covers visual + data-layer failure modes | Single path leaves blind spots: screenshot-only misses KXD errors, text-only misses visual bugs |
| Multi-chunk screenshots | Long dashboards lose readability in single compressed image | Single full-page at higher resolution costs excessive image tokens |
| Reused URL parser from different feature | Avoids duplicating complex parsing logic for native_filters/permalink | Custom parser would duplicate 300+ lines of already-tested code |