# Implementation Plan: LLM Analysis & Documentation Plugins v2

**Branch**: `017-llm-analysis-plugin` | **Date**: 2026-01-28 | **Updated**: 2026-05-31 — v2
**Spec**: [spec.md](spec.md)

## Summary

v2 redesign replaces the ad-hoc «Validate» button with a task-based validation flow. Users create `ValidationPolicy` tasks with dashboard sources (IDs + URLs), select an LLM provider and prompt per-task, choose between two execution paths (Playwright screenshot or text-only API with dataset health checking), and set a schedule. The dashboard hub is simplified — LLM validation status moves to the Validation Tasks page.

Key architectural changes:
- **Task-based flow**: New `/validation-tasks` page + CRUD API + `ValidationSource` model for dashboard IDs/URLs
- **Dual-path execution**: `DashboardValidationPlugin.execute()` dispatches to `_execute_path_a()` (screenshot) or `_execute_path_b()` (text-only) based on `screenshot_enabled`
- **Multi-chunk screenshots**: `ScreenshotService.capture_dashboard_chunks()` captures per-tab images instead of one full-page
- **Dataset health checker** (new): `DatasetHealthChecker` verifies KXD connectivity for every unique dataset
- **URL parsing** (reused): `SupersetContextExtractor.parse_superset_link()` for URL-based dashboard sources
- **Provider/prompt per-task**: Removed from global LLM settings; configured in task creation form

## Technical Context

**Language/Version**: Python 3.13+ (Backend, per ADR-0001), Node.js 18+ (Frontend)
**Primary Dependencies** (real versions from repository):
- Backend: `fastapi==0.126.0`, `pydantic==2.12.5`, `httpx==0.28.1`, `openai` (latest), `playwright` (latest), `tenacity` (latest)
- Frontend: `svelte==^5.43.8`, `@sveltejs/kit==^2.49.2`, `tailwindcss==^3.0.0`, `vite==^7.2.4`
**Storage**: SQLite (`tasks.db`, `auth.db`, `plugins.db`) via SQLAlchemy
**Testing**: `pytest` (Backend), `vitest` (Frontend)
**Deployment**: Docker Compose (backend + frontend + nginx per `docker/`)
**Performance Goals**:
- Path B (text-only, no chart data): < 30s per dashboard (excluding LLM latency)
- Path A (screenshot): < 120s per dashboard
- Task creation (form fill + URL parse): < 60s for user
**Constraints**: Must integrate with existing `PluginBase` and `TaskManager`. Secure storage for API keys (AES-256). Modules < 400 lines per ADR-0001.

## v2 Implementation Notes

### Path B — Dashboard Topology Builder

The text-only path builds a structured description of the dashboard from API responses:

```
1. GET /api/v1/dashboard/{id}
   → dashboard_title, position_json (layout tree), slices[], json_metadata

2. Parse position_json recursively:
   - TABS → children[] → ROW → children[] → CHART
   - Build tab → chart hierarchy with slice_id references

3. For each unique chart: GET /api/v1/chart/{slice_id}
   → slice_name, viz_type, params (metrics, groupby, filters, time_range), datasource_id

4. For each unique dataset: GET /api/v1/dataset/{datasource_id}
   → database (name, backend), sql (virtual datasets), kind, is_managed_externally

5. Optionally: POST /api/v1/chart/data with form_data from chart params
   → Verify query execution, row count, duration

6. GET /api/v1/log/ (filter: dashboard_id, 24h)
   → Execution logs

7. Build text prompt:
   Dashboard: "{title}" (id: {id})
   ─── Tab: "{tab_name}" ({N} charts) ───
     Chart "{chart_name}" ({viz_type})
       Dataset: {dataset_name} ({kind}, backend: {backend})
       Params: {metrics}, grouped by {groupby}, filters: {filters}
       Health: ✓ accessible / ✗ {error}
     ...
   Logs (last 24h, {N} records):
     [{timestamp}] {action}: {details}
```

### Path A — Multi-Chunk Screenshot

The screenshot service is refactored to capture per-tab screenshots:

1. Login (unchanged)
2. Navigate to dashboard URL (with parsed filters/tabs from URL source)
3. Iterate tabs recursively (unchanged logic, but now save a screenshot per tab)
4. For each tab: resize viewport to 1920×1200, wait for charts, CDP screenshot
5. Return list of `{tab_name, screenshot_path}` to LLMClient
6. LLMClient sends all chunks as separate `image_url` blocks in a single `content[]` array

### URL Parsing (from dataset review)

The `SupersetContextExtractor` is already implemented and tested for dataset review. For validation tasks, we reuse its `parse_superset_link()` method:

1. User pastes URL in task creation form
2. Frontend calls `POST /api/validation-tasks/parse-url` with `{url, environment_id}`
3. Backend calls `extractor.parse_superset_link(url)` → `SupersetParsedContext`
4. Returns `{dashboard_id, title, native_filters, activeTabs, anchor, partial_recovery, warnings}`
5. Frontend shows preview: dashboard title + filter names + tab names
6. On task save: `ValidationSource.value = url`, `ValidationSource.parsed_context = {...}`

### Prompt Templates for Dual Paths

Two path-specific defaults in `DEFAULT_LLM_PROMPTS`:

```python
"dashboard_validation_prompt_multimodal": """..."""  # Path A: image-focused
"dashboard_validation_prompt_text": """..."""         # Path B: topology-focused
```

The v1 legacy key `dashboard_validation_prompt` is retained for backward reference but **MUST NOT** be used as default for new tasks. Each task has a single `prompt_template` field:
- `null` → system auto-selects path-specific default based on `screenshot_enabled`
- `non-null` → custom prompt used regardless of path (with FR-048a warning on path toggle)

**Contract**: `DashboardValidationPlugin.execute()` reads `prompt_template` from params. If null, resolves: `screenshot_enabled ? multimodal_prompt : text_prompt`.

## Constitution Check

*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*

Source: `.specify/memory/constitution.md` v1.0.0

- [x] **I. Semantic Contract First** — All v2 contracts use `#region`/`#endregion` anchors with `[C:N]` complexity tiers. `@RELATION` edges trace dependencies. `@RATIONALE`/`@REJECTED` document architectural choices. No naked code. → Verified: 6 contract modules defined in `contracts/modules.md` with proper anchoring.
- [x] **II. Decision Memory** — D1–D9 design decisions in `spec.md` with rationale. D8–D13 research decisions in `research.md` with Decision/Rationale/Alternatives/Impact format. No `@REJECTED` path silently reintroduced. → Verified: 9 spec decisions + 6 research decisions all documented.
- [x] **III. External Orchestrator** — No change to orchestrator pattern (ADR-0003). `DashboardValidationPlugin` remains an external plugin, not embedded in Superset. `DatasetHealthChecker` is a service within ss-tools, not a Superset modification. → PASS
- [x] **IV. Module Discipline** — New files: `validation_tasks.py` (API routes, ~150 LOC), `DatasetHealthChecker` class (~120 LOC), `ValidationTaskForm.svelte` (~300 LOC), `UrlParser.svelte` (~150 LOC). All under 400-line limit. Function CC ≤ 10. No cyclic imports (routes → services → models direction enforced per ADR-0001). → PASS
- [x] **V. RBAC Enforcement** — Validation task CRUD requires existing RBAC roles (`plugin:llm:validate`, `plugin:llm:configure` per ADR-0005). Assistant tool `_tool_llm_validation.py` inherits permission checks from `@assistant_tool` decorator (ADR-0008). → PASS
- [x] **VI. Frontend — Svelte 5 Runes Only** — New components: `ValidationTaskForm.svelte`, `UrlParser.svelte`, `DatasetHealthTable.svelte`, `ValidationTaskReport.svelte`. All use `$state`, `$derived`, `$props`, `$effect` (no Svelte 4 syntax). No `fromStore` + `$derived` anti-pattern (ADR-0007). → PASS
- [x] **VII. Test-Driven for C3+ Contracts** — `ValidationTaskService` [C:4] → `test_validation_task_service.py`. `DatasetHealthChecker` [C:3] → `test_dataset_health_checker.py`. `DashboardValidationPlugin._execute_path_b` [C:3] → `test_path_b_execution.py`. Each includes `@REJECTED` path regression test. → Planned in T108-T156.

**GATE RESULT: CONDITIONAL PASS (AMBER)** — No blocking constitutional conflicts. 7 of 7 principles pass. Spec completeness is AMBER per `checklists/requirements.md`: 12 UX/data/reliability checklist items remain as implementation-readiness markers (defined in FRs, to be validated during Phase 3+ implementation). Gate upgrades to GREEN after Phase 3 Task CRUD implementation validates form flows, report layout, and migration against checklist.

## ADR Continuity

| ADR | Impact of v2 | Action Required |
|-----|-------------|-----------------|
| **ADR-0001** (module layout) | New files: `backend/src/api/routes/validation_tasks.py`, `frontend/src/routes/validation-tasks/`. All fit within canonical boundaries (API routes → `api/routes/`, pages → `routes/`, components → `lib/components/`). | None — compliant |
| **ADR-0002** (semantic protocol) | v2 contracts use GRACE-Poly v2.6 `#region` anchors with `[C:N]`. `@RELATION` edges trace `DEPENDS_ON` / `CALLS` / `IMPLEMENTS`. | None — compliant |
| **ADR-0003** (orchestrator pattern) | No changes. ss-tools remains external orchestrator over Superset. | None |
| **ADR-0004** (plugin architecture) | `DashboardValidationPlugin` stays a `PluginBase` implementation. `DatasetHealthChecker` is a service (not a plugin) — correct per ADR-0004 §Isolation Guarantees: plugins get no DB access; services do. | None — compliant |
| **ADR-0005** (RBAC) | Existing `plugin:llm:validate` permission covers new task-triggered validation. Task CRUD requires `plugin:llm:configure`. | None — reused |
| **ADR-0006** (frontend architecture) | New pages use SvelteKit file-based routing (`+page.svelte`). Components in `lib/components/`. API clients via `$lib/api.js`. | None — compliant |
| **ADR-0007** (fromStore+derived) | No new uses of the rejected `fromStore` + `$derived` pattern. | None |
| **ADR-0008** (assistant tool registry) | `_tool_llm_validation.py` must be updated: replace ad-hoc `run_llm_validation` dispatch with task creation via `ValidationTaskService`. | **T153**: Update assistant tool to create task instead of triggering ad-hoc validation |
| **ADR-0009** (SSL) | No changes. LLM client SSL verification applies to both Path A and Path B. | None |

## Project Structure — v2 Changes

```text
backend/src/
├── plugins/llm_analysis/
│   ├── plugin.py             # REFACTOR: dispatch to _execute_path_a / _execute_path_b
│   ├── service.py            # REFACTOR: ScreenshotService.capture_dashboard_chunks()
│   │                         #          LLMClient.analyze_dashboard_multimodal()
│   │                         #          LLMClient.analyze_dashboard_text()
│   │                         # NEW:     DatasetHealthChecker class
│   ├── scheduler.py          # UPDATE: use ValidationPolicy model
│   └── models.py             # UPDATE: add ValidationSource, expand ValidationPolicy
├── services/
│   ├── validation_service.py # REFACTOR: ValidationTaskService (renamed, expanded CRUD)
│   ├── llm_prompt_templates.py # UPDATE: add dashboard_validation_prompt_text + multimodal keys
│   └── llm_provider.py       # (unchanged)
├── api/routes/
│   ├── llm.py                # UPDATE: remove dashboard_validation from provider_bindings schemas
│   └── validation_tasks.py   # NEW: CRUD routes for /api/validation-tasks
└── core/utils/
    └── superset_context_extractor/  # (unchanged — reused for URL parsing)

frontend/src/
├── routes/
│   ├── validation-tasks/     # NEW: task list page
│   │   ├── +page.svelte
│   │   ├── new/+page.svelte  # NEW: task creation form
│   │   └── [policyId]/+page.svelte  # NEW: task detail + history
│   ├── dashboards/+page.svelte  # UPDATE: remove Validate button, remove LLM status column
│   ├── reports/llm/[taskId]/+page.svelte  # UPDATE: multi-chunk screenshots, dataset health display
│   └── admin/settings/llm/+page.svelte  # UPDATE: remove dashboard_validation binding
└── components/
    └── llm/
        ├── ValidationTaskForm.svelte    # NEW: multi-step task creation form
        ├── ValidationTaskReport.svelte  # NEW: shared report component
        ├── UrlParser.svelte             # NEW: URL paste + parse preview
        └── DatasetHealthTable.svelte    # NEW: dataset health status table
```

## Complexity Tracking

| Violation | Why Needed | Simpler Alternative Rejected Because |
|-----------|------------|-------------------------------------|
| Dual execution paths | Covers visual + data-layer failure modes | Single path leaves blind spots: screenshot-only misses KXD errors, text-only misses visual bugs |
| Multi-chunk screenshots | Long dashboards lose readability in single compressed image | Single full-page at higher resolution costs excessive image tokens |
| Reused URL parser from different feature | Avoids duplicating complex parsing logic for native_filters/permalink | Custom parser would duplicate 300+ lines of already-tested code |
