189 lines
13 KiB
Markdown
189 lines
13 KiB
Markdown
# Implementation Plan: LLM Analysis & Documentation Plugins v2
|
||
|
||
**Branch**: `017-llm-analysis-plugin` | **Date**: 2026-01-28 | **Updated**: 2026-06-07 — v2 implemented
|
||
**Spec**: [spec.md](spec.md)
|
||
|
||
## Summary
|
||
|
||
v2 redesign replaces the ad-hoc «Validate» button with a task-based validation flow. Users create `ValidationPolicy` tasks with dashboard sources (IDs + URLs), select an LLM provider and prompt per-task, choose between two execution paths (Playwright screenshot or text-only API with dataset health checking), and set a schedule. The dashboard hub is simplified — LLM validation status moves to the Validation Tasks page.
|
||
|
||
Key architectural changes:
|
||
- **Task-based flow**: New `/validation-tasks` page + CRUD API + `ValidationSource` model for dashboard IDs/URLs
|
||
- **Dual-path execution**: `DashboardValidationPlugin.execute()` dispatches to `_execute_path_a()` (screenshot) or `_execute_path_b()` (text-only) based on `screenshot_enabled`
|
||
- **Multi-chunk screenshots**: `ScreenshotService.capture_dashboard_chunks()` captures per-tab images instead of one full-page
|
||
- **Dataset health checker** (new): `DatasetHealthChecker` verifies KXD connectivity for every unique dataset
|
||
- **URL parsing** (reused): `SupersetContextExtractor.parse_superset_link()` for URL-based dashboard sources
|
||
- **Provider/prompt per-task**: Removed from global LLM settings; configured in task creation form
|
||
|
||
## Technical Context
|
||
|
||
**Language/Version**: Python 3.13+ (Backend, per ADR-0001), Node.js 18+ (Frontend)
|
||
**Primary Dependencies** (real versions from repository):
|
||
- Backend: `fastapi==0.126.0`, `pydantic==2.12.5`, `httpx==0.28.1`, `openai` (latest), `playwright` (latest), `tenacity` (latest)
|
||
- Frontend: `svelte==^5.43.8`, `@sveltejs/kit==^2.49.2`, `tailwindcss==^3.0.0`, `vite==^7.2.4`
|
||
**Storage**: SQLite (`tasks.db`, `auth.db`, `plugins.db`) via SQLAlchemy
|
||
**Testing**: `pytest` (Backend), `vitest` (Frontend)
|
||
**Deployment**: Docker Compose (backend + frontend + nginx per `docker/`)
|
||
**Performance Goals**:
|
||
- Path B (text-only, no chart data): < 30s per dashboard (excluding LLM latency)
|
||
- Path A (screenshot): < 120s per dashboard
|
||
- Task creation (form fill + URL parse): < 60s for user
|
||
**Constraints**: Must integrate with existing `PluginBase` and `TaskManager`. Secure storage for API keys (AES-256). Modules < 400 lines per ADR-0001.
|
||
|
||
## v2 Implementation Notes
|
||
|
||
### Path B — Dashboard Topology Builder
|
||
|
||
The text-only path builds a structured description of the dashboard from API responses:
|
||
|
||
```
|
||
1. GET /api/v1/dashboard/{id}
|
||
→ dashboard_title, position_json (layout tree), slices[], json_metadata
|
||
|
||
2. Parse position_json recursively:
|
||
- TABS → children[] → ROW → children[] → CHART
|
||
- Build tab → chart hierarchy with slice_id references
|
||
|
||
3. For each unique chart: GET /api/v1/chart/{slice_id}
|
||
→ slice_name, viz_type, params (metrics, groupby, filters, time_range), datasource_id
|
||
|
||
4. For each unique dataset: GET /api/v1/dataset/{datasource_id}
|
||
→ database (name, backend), sql (virtual datasets), kind, is_managed_externally
|
||
|
||
5. Optionally: POST /api/v1/chart/data with form_data from chart params
|
||
→ Verify query execution, row count, duration
|
||
|
||
6. GET /api/v1/log/ (filter: dashboard_id, 24h)
|
||
→ Execution logs
|
||
|
||
7. Build text prompt:
|
||
Dashboard: "{title}" (id: {id})
|
||
─── Tab: "{tab_name}" ({N} charts) ───
|
||
Chart "{chart_name}" ({viz_type})
|
||
Dataset: {dataset_name} ({kind}, backend: {backend})
|
||
Params: {metrics}, grouped by {groupby}, filters: {filters}
|
||
Health: ✓ accessible / ✗ {error}
|
||
...
|
||
Logs (last 24h, {N} records):
|
||
[{timestamp}] {action}: {details}
|
||
```
|
||
|
||
### Path A — Multi-Chunk Screenshot
|
||
|
||
The screenshot service is refactored to capture per-tab screenshots:
|
||
|
||
1. Login (unchanged)
|
||
2. Navigate to dashboard URL (with parsed filters/tabs from URL source)
|
||
3. Iterate tabs recursively (unchanged logic, but now save a screenshot per tab)
|
||
4. For each tab: resize viewport to 1920×1200, wait for charts, CDP screenshot
|
||
5. Return list of `{tab_name, screenshot_path}` to LLMClient
|
||
6. LLMClient sends all chunks as separate `image_url` blocks in a single `content[]` array
|
||
|
||
### URL Parsing (from dataset review)
|
||
|
||
The `SupersetContextExtractor` is already implemented and tested for dataset review. For validation tasks, we reuse its `parse_superset_link()` method:
|
||
|
||
1. User pastes URL in task creation form
|
||
2. Frontend calls `POST /api/validation-tasks/parse-url` with `{url, environment_id}`
|
||
3. Backend calls `extractor.parse_superset_link(url)` → `SupersetParsedContext`
|
||
4. Returns `{dashboard_id, title, native_filters, activeTabs, anchor, partial_recovery, warnings}`
|
||
5. Frontend shows preview: dashboard title + filter names + tab names
|
||
6. On task save: `ValidationSource.value = url`, `ValidationSource.parsed_context = {...}`
|
||
|
||
### Prompt Templates for Dual Paths
|
||
|
||
Two path-specific defaults in `DEFAULT_LLM_PROMPTS`:
|
||
|
||
```python
|
||
"dashboard_validation_prompt_multimodal": """...""" # Path A: image-focused
|
||
"dashboard_validation_prompt_text": """...""" # Path B: topology-focused
|
||
```
|
||
|
||
The v1 legacy key `dashboard_validation_prompt` is retained for backward reference but **MUST NOT** be used as default for new tasks. Each task has a single `prompt_template` field:
|
||
- `null` → system auto-selects path-specific default based on `screenshot_enabled`
|
||
- `non-null` → custom prompt used regardless of path (with FR-048a warning on path toggle)
|
||
|
||
**Contract**: `DashboardValidationPlugin.execute()` reads `prompt_template` from params. If null, resolves: `screenshot_enabled ? multimodal_prompt : text_prompt`.
|
||
|
||
## Constitution Check
|
||
|
||
*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
|
||
|
||
Source: `.specify/memory/constitution.md` v1.0.0
|
||
|
||
- [x] **I. Semantic Contract First** — All v2 contracts use `#region`/`#endregion` anchors with `[C:N]` complexity tiers. `@RELATION` edges trace dependencies. `@RATIONALE`/`@REJECTED` document architectural choices. No naked code. → Verified: 6 contract modules defined in `contracts/modules.md` with proper anchoring.
|
||
- [x] **II. Decision Memory** — D1–D9 design decisions in `spec.md` with rationale. D8–D13 research decisions in `research.md` with Decision/Rationale/Alternatives/Impact format. No `@REJECTED` path silently reintroduced. → Verified: 9 spec decisions + 6 research decisions all documented.
|
||
- [x] **III. External Orchestrator** — No change to orchestrator pattern (ADR-0003). `DashboardValidationPlugin` remains an external plugin, not embedded in Superset. `DatasetHealthChecker` is a service within ss-tools, not a Superset modification. → PASS
|
||
- [x] **IV. Module Discipline** — New files: `validation_tasks.py` (API routes, ~150 LOC), `DatasetHealthChecker` class (~120 LOC), `ValidationTaskForm.svelte` (~300 LOC), `UrlParser.svelte` (~150 LOC). All under 400-line limit. Function CC ≤ 10. No cyclic imports (routes → services → models direction enforced per ADR-0001). → PASS
|
||
- [x] **V. RBAC Enforcement** — Validation task CRUD requires existing RBAC roles (`plugin:llm:validate`, `plugin:llm:configure` per ADR-0005). Assistant tool `_tool_llm_validation.py` inherits permission checks from `@assistant_tool` decorator (ADR-0008). → PASS
|
||
- [x] **VI. Frontend — Svelte 5 Runes Only** — New components: `ValidationTaskForm.svelte`, `UrlParser.svelte`, `DatasetHealthTable.svelte`, `ValidationTaskReport.svelte`. All use `$state`, `$derived`, `$props`, `$effect` (no Svelte 4 syntax). No `fromStore` + `$derived` anti-pattern (ADR-0007). → PASS
|
||
- [x] **VII. Test-Driven for C3+ Contracts** — `ValidationTaskService` [C:4] → `test_validation_task_service.py`. `DatasetHealthChecker` [C:3] → `test_dataset_health_checker.py`. `DashboardValidationPlugin._execute_path_b` [C:3] → `test_path_b_execution.py`. Each includes `@REJECTED` path regression test. → Planned in T108-T156.
|
||
|
||
**GATE RESULT: CONDITIONAL PASS (AMBER)** — No blocking constitutional conflicts. 7 of 7 principles pass. Spec completeness is AMBER per `checklists/requirements.md`: 12 UX/data/reliability checklist items remain as implementation-readiness markers (defined in FRs, to be validated during Phase 3+ implementation). Gate upgrades to GREEN after Phase 3 Task CRUD implementation validates form flows, report layout, and migration against checklist.
|
||
|
||
## ADR Continuity
|
||
|
||
| ADR | Impact of v2 | Action Required |
|
||
|-----|-------------|-----------------|
|
||
| **ADR-0001** (module layout) | New files: `backend/src/api/routes/validation_tasks.py`, `frontend/src/routes/validation-tasks/`. All fit within canonical boundaries (API routes → `api/routes/`, pages → `routes/`, components → `lib/components/`). | None — compliant |
|
||
| **ADR-0002** (semantic protocol) | v2 contracts use GRACE-Poly v2.6 `#region` anchors with `[C:N]`. `@RELATION` edges trace `DEPENDS_ON` / `CALLS` / `IMPLEMENTS`. | None — compliant |
|
||
| **ADR-0003** (orchestrator pattern) | No changes. ss-tools remains external orchestrator over Superset. | None |
|
||
| **ADR-0004** (plugin architecture) | `DashboardValidationPlugin` stays a `PluginBase` implementation. `DatasetHealthChecker` is a service (not a plugin) — correct per ADR-0004 §Isolation Guarantees: plugins get no DB access; services do. | None — compliant |
|
||
| **ADR-0005** (RBAC) | Existing `plugin:llm:validate` permission covers new task-triggered validation. Task CRUD requires `plugin:llm:configure`. | None — reused |
|
||
| **ADR-0006** (frontend architecture) | New pages use SvelteKit file-based routing (`+page.svelte`). Components in `lib/components/`. API clients via `$lib/api.js`. | None — compliant |
|
||
| **ADR-0007** (fromStore+derived) | No new uses of the rejected `fromStore` + `$derived` pattern. | None |
|
||
| **ADR-0008** (assistant tool registry) | `_tool_llm_validation.py` must be updated: replace ad-hoc `run_llm_validation` dispatch with task creation via `ValidationTaskService`. | **T153**: Update assistant tool to create task instead of triggering ad-hoc validation |
|
||
| **ADR-0009** (SSL) | No changes. LLM client SSL verification applies to both Path A and Path B. | None |
|
||
|
||
## Project Structure — v2 Changes
|
||
|
||
```text
|
||
backend/src/
|
||
├── plugins/llm_analysis/
|
||
│ ├── plugin.py # DashboardValidationPlugin (1 004 строки) — dispatch to Path A/B
|
||
│ ├── service.py # ScreenshotService + LLMClient + DatasetHealthChecker (1 691 строка)
|
||
│ ├── _topology.py # DashboardTopologyBuilder (290 строк)
|
||
│ ├── scheduler.py # APScheduler integration (57 строк)
|
||
│ ├── models.py # Plugin models (73 строки)
|
||
│ ├── migrations/ # v1_to_v2 migration (196 строк)
|
||
│ └── __tests__/ # 3 test files (655 строк)
|
||
├── services/
|
||
│ ├── validation_service.py # ValidationTaskService CRUD (767 строк)
|
||
│ └── llm_prompt_templates.py # Dual-path prompt defaults (259 строк)
|
||
├── api/routes/
|
||
│ ├── validation_tasks.py # CRUD routes for /api/validation-tasks (481 строка)
|
||
│ └── llm.py # LLM provider/settings routes (608 строк)
|
||
├── models/
|
||
│ └── llm.py # LLM ORM models: ValidationPolicy, ValidationSource, ValidationRecord, ValidationRun (156 строк)
|
||
|
||
frontend/src/
|
||
├── routes/
|
||
│ ├── validation-tasks/ # Task list + detail + history + runs (6 страниц)
|
||
│ │ ├── +page.svelte # Task list (204 строки)
|
||
│ │ ├── new/+page.svelte # Task creation (95 строк)
|
||
│ │ ├── [policyId]/+page.svelte # Task detail (398 строк)
|
||
│ │ ├── [policyId]/edit/+page.svelte # Task edit (102 строки)
|
||
│ │ ├── [policyId]/runs/[runId]/+page.svelte # Run detail report (200 строк)
|
||
│ │ └── history/+page.svelte # Run history (183 строки)
|
||
│ ├── dashboards/[id]/validation/+page.svelte # Dashboard validation history (325 строк)
|
||
│ ├── reports/llm/[taskId]/+page.svelte # Legacy report redirect (219 строк)
|
||
│ └── admin/settings/llm/+page.svelte # LLM provider settings (258 строк)
|
||
├── lib/components/llm/
|
||
│ ├── ValidationTaskForm.svelte # Multi-step task creation form (1 096 строк)
|
||
│ ├── ValidationTaskReport.svelte # Shared report component (39 строк)
|
||
│ ├── UrlParser.svelte # URL paste + parse preview (207 строк)
|
||
│ ├── ProviderConfig.svelte # Provider configuration form (745 строк)
|
||
│ └── DocPreview.svelte # Documentation preview (87 строк)
|
||
├── lib/i18n/locales/
|
||
│ ├── en/validation.json + llm.json # English locale
|
||
│ └── ru/validation.json + llm.json # Russian locale
|
||
```
|
||
|
||
## Complexity Tracking
|
||
|
||
| Violation | Why Needed | Simpler Alternative Rejected Because |
|
||
|-----------|------------|-------------------------------------|
|
||
| Dual execution paths | Covers visual + data-layer failure modes | Single path leaves blind spots: screenshot-only misses KXD errors, text-only misses visual bugs |
|
||
| Multi-chunk screenshots | Long dashboards lose readability in single compressed image | Single full-page at higher resolution costs excessive image tokens |
|
||
| Reused URL parser from different feature | Avoids duplicating complex parsing logic for native_filters/permalink | Custom parser would duplicate 300+ lines of already-tested code |
|