diff --git a/.opencode/agents/qa-tester.md b/.opencode/agents/qa-tester.md index 99deb797..00797957 100644 --- a/.opencode/agents/qa-tester.md +++ b/.opencode/agents/qa-tester.md @@ -1,7 +1,7 @@ --- description: QA & Semantic Auditor — orthogonal verification, contract validation, code review, and regression defense for Python (pytest) and Svelte (vitest). mode: all -model: deepseek/deepseek-v4-flash +model: opencode/mimo-v2.5-free temperature: 0.1 permission: edit: allow diff --git a/.opencode/command/speckit.plan.md b/.opencode/command/speckit.plan.md index 53aeae01..cf0cf873 100644 --- a/.opencode/command/speckit.plan.md +++ b/.opencode/command/speckit.plan.md @@ -53,11 +53,13 @@ You **MUST** consider the user input before proceeding (if not empty). Research must resolve only implementation-shaping unknowns that matter for this repository, such as: - module placement under `backend/src/` or `frontend/src/` +- **Screen Model topology**: which screens need a `[TYPE Model]` (`.svelte.ts`), which atoms each model declares, which invariants cross widget boundaries - API endpoint design (REST routes, WebSocket channels) - database schema changes (SQLAlchemy models, migrations) - Svelte component hierarchy and store topology - async task orchestration patterns -- test strategy (pytest + vitest) and required fixture coverage +- **TypeScript DTO alignment**: frontend `types/` matching backend Pydantic schemas +- test strategy (pytest + vitest; L1 model invariants without render + L2 UX contracts with render) - belief runtime instrumentation for C4/C5 flows - semantic validation boundaries and static verification workflow @@ -71,9 +73,17 @@ Use `[NEED_CONTEXT: target]` instead of inventing relation targets, DTO names, o ## Phase 1: Design, ADR Continuity, and Contracts -### Frontend Component Reuse Scan (MANDATORY — before contract generation) +### Frontend Model & Component Reuse Scan (MANDATORY — before contract generation) -Before designing any new Svelte component, execute a **component inventory scan** of the existing codebase to maximise reuse and prevent duplicate primitives. Use a subagent with `subagent_type: "explore"` to scan: +Before designing any new screen, execute a **model-first inventory scan** followed by a **component inventory scan** of the existing codebase to maximise reuse and prevent duplicate primitives. + +**Step 1: Screen Model scan** (use a subagent with `subagent_type: "explore"`): +- Search `frontend/src/lib/models/` for existing `[TYPE Model]` contracts +- Use `axiom_semantic_discovery search_contracts type="Model" query=""` for structured search +- Check model atoms, actions, and invariants — reuse if the screen state maps to an existing model +- New models use `.svelte.ts` extension, `[TYPE Model]` contract, `@STATE`/`@ACTION`/`@INVARIANT` tags + +**Step 2: Component scan** (priority order): **Scan targets** (priority order): 1. `frontend/src/lib/ui/` — design-system atoms: `Button.svelte`, `Select.svelte`, `Input.svelte`, `Card.svelte` @@ -93,7 +103,7 @@ Before designing any new Svelte component, execute a **component inventory scan* | Pattern exists (badge, skeleton, tooltip) | Document the Tailwind classes to replicate; no component extraction | | No reusable asset exists | Create new component only then | -**Output:** The `contracts/modules.md` for every frontend contract MUST include `@RELATION` edges to reused components and a `@RATIONALE` noting WHY the component is reused rather than rebuilt. For pattern-only reuse, the contract MUST reference the source page/file where the pattern was observed. +**Output:** The `contracts/modules.md` for every frontend contract MUST include `@RELATION` edges to reused components/models and a `@RATIONALE` noting WHY the asset is reused rather than rebuilt. For pattern-only reuse, the contract MUST reference the source page/file where the pattern was observed. Components that bind to a Screen Model declare `@RELATION BINDS_TO -> [ModelId]`. **Forbidden patterns:** - Creating a new `` when `confirm()` suffices @@ -115,7 +125,8 @@ Generate `data-model.md` for ss-tools domain entities such as: - Task state transitions - Git operation entities - Plugin configuration schemas -- Frontend TypeScript types (when feature is fullstack) +- **Frontend TypeScript DTOs** in `frontend/src/types/` — MUST match backend Pydantic schemas across the stack boundary +- **Screen Model interfaces** — typed atoms, FSM state unions, action payloads for `.svelte.ts` models ### Global ADR Continuity @@ -123,27 +134,28 @@ Before task decomposition, planning must identify any repo-shaping decisions thi - Python module layout and decomposition - FastAPI route organization - SvelteKit routing and component hierarchy +- **Screen Model topology**: which screens need a model, model-atom boundaries, invariant scope - belief-state runtime behavior (JSON structured logging / console markers) - semantic comment-anchor rules +- **TypeScript-first frontend architecture** (`.svelte.ts` models, typed props, typed API boundaries) - payload/schema stability decisions -For each durable choice, ensure the plan references the relevant ADR and explicitly records accepted and rejected paths. - ### Contract Design Output Generate `contracts/modules.md` as the primary design contract for implementation. Contracts must: -- use short semantic IDs -- classify each planned module/component with `@COMPLEXITY` 1-5 +- use short semantic IDs (e.g., `MigrationModel`, `GitManager`, `DashboardApi`) +- classify each planned module/component/model with `[C:N]` complexity in the `#region` anchor (NOT `@COMPLEXITY N`) +- use canonical anchor syntax: `#region Id [C:N] [TYPE TypeName] [SEMANTICS tags]` / `#endregion Id` - use canonical relation syntax `@RELATION PREDICATE -> TARGET_ID` - preserve accepted-path and rejected-path memory via `@RATIONALE` and `@REJECTED` where needed -- describe Python modules, FastAPI routes, Svelte components, stores, and services instead of inventing MCP/backend layers +- describe Python modules, FastAPI routes, Svelte components, **Screen Models** (`.svelte.ts`), stores, and services instead of inventing MCP/backend layers Complexity guidance for this repository: -- **Complexity 1**: anchors only (DTOs, simple Pydantic schemas) -- **Complexity 2**: `@PURPOSE` (pure functions, utility helpers) -- **Complexity 3**: `@PURPOSE`, `@RELATION` (service modules, route handlers) -- **Complexity 4**: `@PURPOSE`, `@RELATION`, `@PRE`, `@POST`, `@SIDE_EFFECT`; orchestration paths should account for belief runtime markers before mutation or return -- **Complexity 5**: level 4 plus `@DATA_CONTRACT`, `@INVARIANT`, and explicit decision-memory continuity +- **C1**: anchors only (DTOs, simple Pydantic schemas, pure constants) +- **C2**: typically adds `@BRIEF` (pure functions, utility helpers) +- **C3**: typically adds `@RELATION` (service modules, route handlers); Svelte components also `@UX_STATE` +- **C4**: typically adds `@PRE`, `@POST`, `@SIDE_EFFECT`; **Screen Models** also `@STATE`, `@ACTION`, `@INVARIANT`; orchestration paths should account for belief runtime markers +- **C5**: C4 + `@DATA_CONTRACT`, `@INVARIANT`, and explicit decision-memory continuity (`@RATIONALE`/`@REJECTED`) If a planned contract depends on unknown schema, relation target, or ADR identity, emit `[NEED_CONTEXT: target]` instead of fabricating placeholders. diff --git a/.opencode/skills/molecular-cot-logging/SKILL.md b/.opencode/skills/molecular-cot-logging/SKILL.md index 14c792b0..dc79df58 100644 --- a/.opencode/skills/molecular-cot-logging/SKILL.md +++ b/.opencode/skills/molecular-cot-logging/SKILL.md @@ -260,70 +260,67 @@ def _summarise_args(args, kwargs) -> dict: ## V. Svelte / Frontend Pattern -```javascript -// ── trace-id from HTTP response header ────────────────────── -let _traceId = $state(""); +The frontend implementation lives at `frontend/src/lib/cot-logger.ts` (installed as `$lib/cot-logger`). -export function initTraceId() { - // Called once from root layout after first fetch -} +### API -export function setTraceId(id) { _traceId = id; } -export function getTraceId() { return _traceId; } +```typescript +function log( + src: string, // e.g. "MigrationModel.executeMigration" + marker: LogMarker, // "REASON" | "REFLECT" | "EXPLORE" + intent: string, // human-readable one-liner + payload?: Record, // params, result snippet + error?: string, // required for EXPLORE +): void; +``` -// ── Component-level CoT logger ────────────────────────────── -export function log(src, marker, intent, payload, error) { - const record = { - ts: new Date().toISOString(), - level: marker === "EXPLORE" ? "WARNING" : "INFO", - trace_id: _traceId || "no-trace", - src, - marker, - intent, - }; - if (payload) record.payload = payload; - if (error) record.error = error; +### Import - const line = JSON.stringify(record); - - if (marker === "EXPLORE") { - console.warn(line); - } else { - console.info(line); - } -} +```typescript +import { log, setTraceId, getTraceId } from "$lib/cot-logger"; ``` ### Usage in a Svelte component ```svelte - ``` +### trace_id Propagation + +The trace ID is set automatically when the backend returns it. Call `setTraceId(id)` manually if needed: + +```typescript +import { setTraceId } from "$lib/cot-logger"; +import { requestApi } from "$lib/api"; + +const res = await requestApi("/api/endpoint"); +if (res.trace_id) setTraceId(res.trace_id); +``` + ## VI. CLI / Stdout Reader (for humans) To make JSON lines readable in development: diff --git a/.opencode/skills/semantics-svelte/SKILL.md b/.opencode/skills/semantics-svelte/SKILL.md index 0eb0e5ce..59cee6fe 100644 --- a/.opencode/skills/semantics-svelte/SKILL.md +++ b/.opencode/skills/semantics-svelte/SKILL.md @@ -277,6 +277,21 @@ search_contracts query="users" type="Model" Frontend logging uses `log()` from `$lib/cot-logger`, emitting **JSON lines** per MolecularCoTLogging protocol. Import: `import { log } from "$lib/cot-logger";` +The logger is a TypeScript module at `frontend/src/lib/cot-logger.ts` with full type support: + +```typescript +import { log } from "$lib/cot-logger"; + +// Before an operation: +log("ComponentName", "REASON", "What we are about to do", { param: value }); + +// After successful verification: +log("ComponentName", "REFLECT", "Operation completed", { result: value }); + +// On error or fallback: +log("ComponentName", "EXPLORE", "Operation failed", { param: value }, "Error description"); +``` + ### Marker Reference | Marker | When | Signature | diff --git a/.specify/templates/plan-template.md b/.specify/templates/plan-template.md index 926f78d3..a490d9ab 100644 --- a/.specify/templates/plan-template.md +++ b/.specify/templates/plan-template.md @@ -17,12 +17,13 @@ the iteration process. --> -**Language/Version**: Python 3.13+ (backend), JavaScript/TypeScript (frontend Svelte 5) -**Primary Dependencies**: FastAPI, SQLAlchemy, APScheduler (backend); SvelteKit, Vite, Tailwind CSS (frontend) +**Language/Version**: Python 3.13+ (backend), TypeScript (frontend Svelte 5 runes-only) +**Primary Dependencies**: FastAPI, SQLAlchemy, APScheduler (backend); SvelteKit 5, Vite, Tailwind CSS (frontend) **Storage**: PostgreSQL 16 -**Testing**: pytest (backend), vitest + @testing-library/svelte (frontend) +**Testing**: pytest (backend), vitest (L1 model tests without render + L2 UX tests with @testing-library/svelte) **Target Platform**: Linux server (Docker), modern browsers **Project Type**: web application (FastAPI REST + WebSocket backend, SvelteKit SPA frontend) +**Frontend Architecture**: TypeScript-first, model-first (Screen Models via `.svelte.ts`), runes-only (`$state`, `$derived`, `$effect`), typed callback props (no `createEventDispatcher`) **Performance Goals**: [domain-specific, e.g., <200ms p95 API latency, 60fps UI] **Constraints**: [domain-specific, e.g., <100MB memory per container, RBAC enforcement, offline-capable Docker bundle] **Scale/Scope**: [domain-specific, e.g., 50 concurrent users, 1000 dashboards, 10 plugins] @@ -31,7 +32,7 @@ *GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.* -[Evaluate against constitution.md and semantics.md. Explicitly confirm semantic protocol compliance, complexity-driven contract coverage, UX-state compatibility (Svelte 5 runes), async boundaries (FastAPI), API-wrapper rules, RBAC/security constraints (local auth + ADFS SSO), plugin lifecycle rules, and any required belief-state/logging constraints for Complexity 4/5 Python modules.] +[Evaluate against constitution.md and semantics.md. Explicitly confirm semantic protocol compliance, complexity-driven contract coverage, TypeScript-first frontend with typed Screen Models (`.svelte.ts`), UX-state compatibility (Svelte 5 runes-only), async boundaries (FastAPI), API-wrapper rules, RBAC/security constraints (local auth + ADFS SSO), plugin lifecycle rules, and any required belief-state/logging constraints for Complexity 4/5 Python modules.] ## Project Structure @@ -64,10 +65,12 @@ frontend/ ├── src/ │ ├── routes/ # SvelteKit pages │ ├── lib/ -│ │ ├── components/ # Reusable Svelte 5 components -│ │ ├── stores/ # Svelte rune stores ($state, $derived) -│ │ └── api/ # API client modules -│ └── i18n/ # Internationalization +│ │ ├── components/ # Reusable Svelte 5 components (thin rendering layer) +│ │ ├── models/ # Screen Models (.svelte.ts — typed, Svelte-reactive state machines) +│ │ ├── stores/ # Svelte stores (cross-route state: auth, notifications) +│ │ └── api/ # API client modules (fetchApi/requestApi wrappers) +│ ├── types/ # TypeScript DTOs — MUST match backend Pydantic schemas +│ └── services/ # Service layer (gitService, taskService) └── tests/ # vitest tests docker/ # Docker configurations @@ -79,18 +82,23 @@ docker/ # Docker configurations > Use this section to drive Phase 1 artifacts, especially `contracts/modules.md`. -- Classify each planned module/component with `@COMPLEXITY: 1..5`. -- Use comment-anchor syntax appropriate for each context: - - Python: `# [DEF:id:Type]` - - Svelte markup: `` - - Svelte script: `// [DEF:id:Type]` +- Classify each planned module/component/model with `[C:N]` complexity in the `#region` anchor. +- Use canonical anchor syntax appropriate for each context: + - Python: `# #region ContractId [C:N] [TYPE TypeName] [SEMANTICS tags]` / `# #endregion ContractId` + - Svelte markup: `` / `` + - Svelte/TypeScript model: `// #region ModelName [C:N] [TYPE Model] [SEMANTICS tags]` / `// #endregion ModelName` + - Markdown/ADR: `## @{ ContractId [C:N] [TYPE TypeName]` / `## @} ContractId` +- **Legacy `[DEF:id:Type]` syntax is deprecated** — use `#region` format exclusively. +- **Model files use `.svelte.ts` extension** — canonical format for contracts with Svelte reactive primitives (`$state`, `$derived`, `$effect`). - Match contract density to complexity: - - Complexity 1: anchors only (DTOs, simple constants) - - Complexity 2: `@PURPOSE` (utility functions, pure helpers) - - Complexity 3: `@PURPOSE`, `@RELATION`; Svelte components also `@UX_STATE` - - Complexity 4: `@PURPOSE`, `@RELATION`, `@PRE`, `@POST`, `@SIDE_EFFECT`; Python also `belief_scope`/`reason`/`reflect` markers; Svelte also `@UX_FEEDBACK`, `@UX_RECOVERY`, `@UX_REACTIVITY` - - Complexity 5: level 4 + `@DATA_CONTRACT`, `@INVARIANT` + `@RATIONALE`/`@REJECTED` decision memory + - C1: anchors only (DTOs, simple constants) + - C2: typically adds `@BRIEF` (utility functions, pure helpers) + - C3: typically adds `@RELATION`; Svelte also `@UX_STATE`; TypeScript also `@STATE`/`@ACTION` + - C4: typically adds `@PRE`, `@POST`, `@SIDE_EFFECT`; Python also `belief_scope`/`reason`/`reflect` markers; Svelte also `@UX_FEEDBACK`, `@UX_RECOVERY`, `@UX_REACTIVITY` + - C5: C4 + `@DATA_CONTRACT`, `@INVARIANT` + `@RATIONALE`/`@REJECTED` decision memory +- **Screen Models** (`[TYPE Model]`) are C4/C5 contracts that declare screen-level state, invariants, and actions. A component that reads/writes model state declares `@RELATION BINDS_TO -> [ModelId]`. See `semantics-svelte` §IIIa. - Write relations only in canonical form: `@RELATION PREDICATE -> TARGET_ID` +- Allowed predicates: `DEPENDS_ON`, `CALLS`, `INHERITS`, `IMPLEMENTS`, `DISPATCHES`, `BINDS_TO`, `CALLED_BY`, `VERIFIES`. - If any relation target, DTO, or contract dependency is unknown, emit `[NEED_CONTEXT: target]` instead of inventing placeholders. ## Complexity Tracking diff --git a/.specify/templates/tasks-template.md b/.specify/templates/tasks-template.md index 39287802..365b9d0f 100644 --- a/.specify/templates/tasks-template.md +++ b/.specify/templates/tasks-template.md @@ -82,18 +82,23 @@ Examples of foundational tasks (adjust based on your project): ### Tests for User Story 1 (OPTIONAL - only if tests requested) ⚠️ > **NOTE: Write these tests FIRST, ensure they FAIL before implementation** +> **For frontend stories: L1 model tests (no render) come BEFORE L2 component tests (with render).** -- [ ] T010 [P] [US1] Contract test for [endpoint] in tests/contract/test_[name].py -- [ ] T011 [P] [US1] Integration test for [user journey] in tests/integration/test_[name].py +- [ ] T010 [P] [US1] L1 model test for [ModelName] invariants in frontend/src/lib/models/__tests__/[ModelName].test.ts +- [ ] T011 [P] [US1] L2 UX test for [Component] in frontend/src/routes/[path]/__tests__/[page].ux.test.ts +- [ ] T012 [P] [US1] Contract test for [endpoint] in tests/contract/test_[name].py +- [ ] T013 [P] [US1] Integration test for [user journey] in tests/integration/test_[name].py ### Implementation for User Story 1 -- [ ] T012 [P] [US1] Create [Entity1] model in src/models/[entity1].py -- [ ] T013 [P] [US1] Create [Entity2] model in src/models/[entity2].py -- [ ] T014 [US1] Implement [Service] in src/services/[service].py (depends on T012, T013) -- [ ] T015 [US1] Implement [endpoint/feature] in src/[location]/[file].py -- [ ] T016 [US1] Add validation and error handling -- [ ] T017 [US1] Add logging for user story 1 operations +- [ ] T014 [P] [US1] Define TypeScript types in frontend/src/types/[feature].ts (DTOs matching backend Pydantic schemas) +- [ ] T015 [P] [US1] Create [ScreenName]Model in frontend/src/lib/models/[ScreenName]Model.svelte.ts +- [ ] T016 [P] [US1] Create [Entity] model in backend/src/models/[entity].py +- [ ] T017 [US1] Implement [Service] in backend/src/services/[service].py +- [ ] T018 [US1] Implement [endpoint/feature] in backend/src/api/[file].py +- [ ] T019 [US1] Create [Component] as thin rendering layer in frontend/src/routes/[...] (binds to model via @RELATION BINDS_TO -> [ModelId]) +- [ ] T020 [US1] Add @INVARIANT validation in model actions +- [ ] T021 [US1] Add belief-runtime instrumentation in model actions for C4/C5 flows **Checkpoint**: At this point, User Story 1 should be fully functional and testable independently @@ -253,3 +258,7 @@ With multiple developers: - For Complexity 4/5 Python modules, include tasks for belief-state logging paths with `logger.reason()`, `logger.reflect()`, and `belief_scope` where required - For Complexity 5 or explicitly test-governed contracts, include tasks that cover `@TEST_CONTRACT`, `@TEST_SCENARIO`, `@TEST_FIXTURE`, `@TEST_EDGE`, and `@TEST_INVARIANT` - Never create tasks from legacy `@TIER` alone; complexity is the primary execution signal +- **Model-first frontend tasks follow this order: types → model → L1 model tests → component → L2 UX tests** +- **Screen Models use `.svelte.ts` extension** — create them in `frontend/src/lib/models/` +- **Two-layer testing: L1 (model invariants, no render, ~27ms) comes BEFORE L2 (UX contracts, with render)** +- **Backend tasks are pinned to `backend/src/`, frontend tasks to `frontend/src/`** — cross-stack tasks reference both diff --git a/.specify/templates/ux-reference-template.md b/.specify/templates/ux-reference-template.md index a278f45c..d6acab98 100644 --- a/.specify/templates/ux-reference-template.md +++ b/.specify/templates/ux-reference-template.md @@ -40,10 +40,11 @@ $ command --flag value * **[Button Name]**: Primary action. Color: Blue. * **[Input Field]**: Placeholder text: "Enter your name...". Validation: Real-time. * **Contract Mapping**: - * **`@UX_STATE`**: Enumerate the explicit UI states that must appear later in `contracts/modules.md` + * **`@UX_STATE`**: Enumerate the explicit UI states that must appear later in `contracts/modules.md`. For complex screens, these derive from the Screen Model's `@STATE` declarations. * **`@UX_FEEDBACK`**: Define visible system reactions for success, validation, and failure * **`@UX_RECOVERY`**: Define what the user can do after failure or degraded state - * **`@UX_REACTIVITY`**: Note expected Svelte rune bindings with `$state`, `$derived`, `$effect`, `$props` + * **`@UX_REACTIVITY`**: Note the model-driven reactive chain: Model atoms (`$state`, `$derived`) → component props → DOM binding. For route-level data loading, use SvelteKit `load()` functions; `$effect` is for browser-side side effects only. + * **Screen Model**: For screens with cross-widget logic, reference the planned `[TYPE Model]` (`.svelte.ts` file in `frontend/src/lib/models/`). The model declares `@STATE`, `@ACTION`, and `@INVARIANT`. Components bind to the model via `@RELATION BINDS_TO -> [ModelId]`. * **States**: * **Idle/Default**: Clean state, waiting for input. * **Loading**: Skeleton loader replaces content area. diff --git a/docs/adr/ADR-0004-plugin-architecture.md b/docs/adr/ADR-0004-plugin-architecture.md index 75860890..2da73a2a 100644 --- a/docs/adr/ADR-0004-plugin-architecture.md +++ b/docs/adr/ADR-0004-plugin-architecture.md @@ -20,13 +20,15 @@ ss-tools Core ├── core/plugin_executor.py # Subprocess execution, timeout, error boundary ├── core/plugin_registry.py # Registered plugins, metadata, health └── plugins/ # Plugin packages (each = one directory) - ├── llm_analysis/ # LLM‑driven Superset data analysis + ├── llm_analysis/ # LLM‑driven Superset data analysis; v2 adds dual-path execution (Path A: Playwright screenshot + multimodal LLM; Path B: text-only API + dataset health checking) ├── dataset_orchestration/ # LLM dataset operations └── git_integration/ # Git‑based version control for dashboards ``` ### Plugin Contract +> **v2 update:** Task-based validation was introduced via `ValidationTaskService`, creating persistent validation policies with `ValidationRun` aggregates for grouping per-dashboard results. Each run is tracked as a `ValidationRun` record with aggregate pass/fail/warn counts, replacing the previous single-shot task result pattern. See `ValidationTaskService` in `services/validation_service.py`. + Every plugin MUST provide: 1. **`plugin.toml`** — metadata manifest at the plugin root diff --git a/docs/adr/ADR-0008-assistant-tool-registry.md b/docs/adr/ADR-0008-assistant-tool-registry.md index 396c0df0..318715fd 100644 --- a/docs/adr/ADR-0008-assistant-tool-registry.md +++ b/docs/adr/ADR-0008-assistant-tool-registry.md @@ -69,6 +69,8 @@ Each existing tool handler is moved from `_dispatch.py:_dispatch_intent()` into After migration, `_dispatch.py` is reduced to a thin `dispatch()` wrapper. The `_parse_command()` function is updated to also use the registry for keyword-to-operation mapping. +> **Step 8 update (v2):** `run_llm_validation` now creates a persistent validation task (policy) via `POST /api/validation-tasks` (i.e. `ValidationTaskService.create_task()`) and triggers an immediate run via `ValidationTaskService.trigger_run()`, replacing the previous ad-hoc `task_manager.create_task(plugin_id="llm_dashboard_validation")` dispatch. This makes LLM validation repeatable, schedulable, and auditable — each run is tracked as a `ValidationRun` with pass/fail/warn counts. + ### 6. File layout after migration ``` @@ -106,6 +108,7 @@ backend/src/api/routes/assistant/ - **Testable** — `_tool_registry.py` can be unit-tested without DB or HTTP fixtures - **Discoverable** — `from _tool_registry import _tools` gives a complete inventory of all registered tools - **Safe-by-default** — a tool without `@assistant_tool` is not registered and cannot be invoked +- **Task-based validation** — `run_llm_validation` now creates persistent validation tasks (policies) via `ValidationTaskService`, making each validation run repeatable, schedulable, and auditable through the `ValidationRun` aggregate model ### Negative - **One-time migration cost** — 10 existing handlers must be moved to new files (~200 LOC total) diff --git a/specs/017-llm-analysis-plugin/tasks.md b/specs/017-llm-analysis-plugin/tasks.md index d32abbd8..fb07da14 100644 --- a/specs/017-llm-analysis-plugin/tasks.md +++ b/specs/017-llm-analysis-plugin/tasks.md @@ -1,7 +1,7 @@ # Tasks: LLM Analysis & Documentation Plugins v2 **Feature**: `017-llm-analysis-plugin` -**Status**: v1 Completed; v2 Planned +**Status**: v2 Implementation — ~86% complete (86/99 tasks) **Spec**: [spec.md](spec.md) | **Plan**: [plan.md](plan.md) | **Data Model**: [data-model.md](data-model.md) ## User Stories & Priorities @@ -39,15 +39,15 @@ Phase 1 (Setup) **Goal**: Schema changes for v2 models. Blocks everything. -- [ ] T001 [P] Add `sources` ORM relationship + `source_snapshot` JSON (audit only, nullable) to `ValidationPolicy` in `backend/src/models/llm.py`. No JSON denormalization column for authoritative state — relational `ValidationSource` table is the single source of truth. -- [ ] T002 [P] Create `ValidationSource` SQLAlchemy model in `backend/src/models/llm.py` (id, policy_id FK, type, value, parsed_context JSON, is_valid, title, last_error) -- [ ] T003 [P] Add new columns to `ValidationPolicy` model: `provider_id` (String), `prompt_template` (Text, nullable), `screenshot_enabled` (Boolean, default true), `logs_enabled` (Boolean, default true), `execute_chart_data` (Boolean, default false), `llm_batch_size` (Integer, default 1 — full isolation), `policy_dashboard_concurrency_limit` (Integer, default 3) in `backend/src/models/llm.py` -- [ ] T003a [P] Add `global_validation_worker_limit` config setting (default: 3) — max concurrent policy runs across all tasks. Stored in app config, not per-policy. in `backend/src/core/config_models.py` -- [ ] T004 [P] Add new columns to `ValidationRecord` model: `execution_path` (String), `source_id` (FK → ValidationSource, nullable), `dataset_health` (JSON, nullable), `chart_data_results` (JSON, nullable), `tab_screenshots` (JSON, nullable), `token_usage` (JSON, nullable), `dataset_key` (String, nullable) in `backend/src/models/llm.py` -- [ ] T004a [P] Create `ValidationRun` SQLAlchemy model in `backend/src/models/llm.py`: id, policy_id (FK), task_id (nullable), started_at, finished_at, trigger (manual/scheduled), status (completed/partial/failed), dashboard_count, pass_count, warn_count, fail_count, unknown_count -- [ ] T005 Create Alembic migration for all T001–T004a schema changes in `backend/alembic/versions/` -- [ ] T006 Write data migration script: convert existing `dashboard_ids: list[str]` → `sources: [{type: "dashboard_id", value: id, is_valid: true}]` in `backend/src/plugins/llm_analysis/migrations/` -- [ ] T007 [P] Add `DEFAULT_LLM_PROMPTS["dashboard_validation_prompt_multimodal"]` (Path A, image-focused) and `DEFAULT_LLM_PROMPTS["dashboard_validation_prompt_text"]` (Path B, topology-focused) to `backend/src/services/llm_prompt_templates.py` +- [x] T001 [P] Add `sources` ORM relationship + `source_snapshot` JSON (audit only, nullable) to `ValidationPolicy` in `backend/src/models/llm.py`. No JSON denormalization column for authoritative state — relational `ValidationSource` table is the single source of truth. +- [x] T002 [P] Create `ValidationSource` SQLAlchemy model in `backend/src/models/llm.py` (id, policy_id FK, type, value, parsed_context JSON, status enum, resolved_dashboard_id, title, last_error, last_checked_at) +- [x] T003 [P] Add new columns to `ValidationPolicy` model: `provider_id` (String), `prompt_template` (Text, nullable), `screenshot_enabled` (Boolean, default true), `logs_enabled` (Boolean, default true), `execute_chart_data` (Boolean, default false), `llm_batch_size` (Integer, default 1 — full isolation), `policy_dashboard_concurrency_limit` (Integer, default 3) in `backend/src/models/llm.py` +- [x] T003a [P] Add `global_validation_worker_limit` config setting (default: 3) — max concurrent policy runs across all tasks. Stored in app config in `GLOBAL_VALIDATION_WORKER_LIMIT` field in `backend/src/core/config_models.py` +- [x] T004 [P] Add new columns to `ValidationRecord` model: `execution_path` (String), `source_id` (FK → ValidationSource, nullable), `dataset_health` (JSON, nullable), `chart_data_results` (JSON, nullable), `tab_screenshots` (JSON, nullable), `token_usage` (JSON, nullable), `timings` (JSON, nullable), `screenshot_paths` (JSON, nullable), `logs_sent_to_llm` (JSON, nullable), `run_id` (FK → ValidationRun) in `backend/src/models/llm.py` +- [x] T004a [P] Create `ValidationRun` SQLAlchemy model in `backend/src/models/llm.py`: id, policy_id (FK), task_id (nullable), started_at, finished_at, trigger (manual/scheduled), status (running/completed/partial/failed), dashboard_count, pass_count, warn_count, fail_count, unknown_count +- [x] T005 Create Alembic migration for all T001–T004a schema changes in `backend/alembic/versions/` +- [x] T006 Write data migration script: convert existing `dashboard_ids: list[str]` → `sources` rows in `backend/src/plugins/llm_analysis/migrations/v1_to_v2.py` +- [x] T007 [P] Add `DEFAULT_LLM_PROMPTS["dashboard_validation_prompt_multimodal"]` (Path A, image-focused) and `DEFAULT_LLM_PROMPTS["dashboard_validation_prompt_text"]` (Path B, topology-focused) to `backend/src/services/llm_prompt_templates.py` **Verification**: `pytest backend/tests/models/test_llm_v2.py -v` — all models create/query correctly. Migration runs without data loss. @@ -59,71 +59,69 @@ Phase 1 (Setup) ### 2.1 ValidationTaskService -- [ ] T008 [P] Implement `ValidationTaskService.create_task()` — creates `ValidationPolicy` + `ValidationSource[]` rows, validates provider_id against execution path, in `backend/src/services/validation_service.py` -- [ ] T009 [P] Implement `ValidationTaskService.get_task(id)` — returns policy + sources with current status, in `backend/src/services/validation_service.py` -- [ ] T010 [P] Implement `ValidationTaskService.list_tasks(env_id)` — returns all policies with aggregated last-run status, in `backend/src/services/validation_service.py` -- [ ] T011 [P] Implement `ValidationTaskService.update_task(id, data)` — updates policy fields + sources, in `backend/src/services/validation_service.py` -- [ ] T012 [P] Implement `ValidationTaskService.delete_task(id)` — deletes policy + cascades to sources + records, in `backend/src/services/validation_service.py` -- [ ] T013 [P] Implement `ValidationTaskService.trigger_run(policy_id)` — checks FR-054 (same task not already running via ValidationRun with status=running), creates a `ValidationRun` record (FR-055d), enqueues dashboard validation tasks per FR-055 (FIFO with `global_validation_worker_limit`), in `backend/src/services/validation_service.py` -- [ ] T014 [P] Implement `ValidationTaskService.get_run_history(policy_id, page)` — paginated `ValidationRun` list with aggregate counts, in `backend/src/services/validation_service.py` -- [ ] T015 [P] Implement `ValidationTaskService.get_run_detail(run_id)` — full run with all dashboard records, aggregate counts, in `backend/src/services/validation_service.py` -- [ ] T015a [P] Implement `ValidationTaskService._finalize_run(run_id)` — computes aggregate counts from all `ValidationRecord`s with this `run_id`, sets finished_at and status, in `backend/src/services/validation_service.py` +- [x] T008 [P] Implement `ValidationTaskService.create_task()` — creates `ValidationPolicy` + `ValidationSource[]` rows, validates provider_id against execution path, in `backend/src/services/validation_service.py` +- [x] T009 [P] Implement `ValidationTaskService.get_task(id)` — returns policy + sources with current status, in `backend/src/services/validation_service.py` +- [x] T010 [P] Implement `ValidationTaskService.list_tasks(env_id)` — returns all policies with aggregated last-run status, in `backend/src/services/validation_service.py` +- [x] T011 [P] Implement `ValidationTaskService.update_task(id, data)` — updates policy fields + sources, in `backend/src/services/validation_service.py` +- [x] T012 [P] Implement `ValidationTaskService.delete_task(id)` — deletes policy + cascades to sources + records, in `backend/src/services/validation_service.py` +- [x] T013 [P] Implement `ValidationTaskService.trigger_run(policy_id)` — checks FR-054 (same task not already running via ValidationRun with status=running), creates a `ValidationRun` record (FR-055d), enqueues dashboard validation tasks per FR-055, in `backend/src/services/validation_service.py` +- [x] T014 [P] Implement `ValidationTaskService.get_run_history(policy_id, page)` — paginated `ValidationRun` list with aggregate counts, in `backend/src/services/validation_service.py` +- [x] T015 [P] Implement `ValidationTaskService.get_run_detail(run_id)` — full run with all dashboard records, aggregate counts, in `backend/src/services/validation_service.py` +- [x] T015a [P] Implement `ValidationTaskService._finalize_run(run_id)` — computes aggregate counts from all `ValidationRecord`s with this `run_id`, sets finished_at and status, in `backend/src/services/validation_service.py` ### 2.2 DatasetHealthChecker (Path B) -- [ ] T016 [P] Implement `DatasetHealthChecker` class in `backend/src/plugins/llm_analysis/service.py`: - - `check_dataset_health(dataset_id: int, client: SupersetClient) → DatasetHealth` — calls `GET /api/v1/dataset/{id}`, extracts database name, backend, kind, verifies accessibility - - `check_dashboard_datasets(dashboard_id: int, chart_list: list[dict], client: SupersetClient, execute_data: bool) → tuple[list[DatasetHealth], list[ChartDataResult]]` +- [x] T016 [P] Implement `DatasetHealthChecker` class in `backend/src/plugins/llm_analysis/service.py`: + - `check_dataset_health(dataset_id)` — calls `GET /api/v1/dataset/{id}`, extracts database name, backend, kind, verifies accessibility + - `check_chart_data(chart_id, form_data)` — optional level 3-4 checks + - `check_dashboard_datasets(chart_list, execute_chart_data)` — batch check all unique datasets ### 2.3 ScreenshotService Refactor (multi-chunk for Path A) -- [ ] T017 [P] Implement `ScreenshotService.capture_dashboard_chunks(dashboard_id, parsed_filters, output_dir) → list[{tab_name, path}]` in `backend/src/plugins/llm_analysis/service.py`: +- [x] T017 [P] Implement `ScreenshotService.capture_dashboard_chunks(dashboard_id, parsed_filters, output_dir) → list[{tab_name, path}]` in `backend/src/plugins/llm_analysis/service.py`: - Login → navigate with filters/tabs → iterate tabs → per-tab viewport (1920×1200) → CDP screenshot → collect PNG paths ### 2.3b Image Pipeline (PNG → JPEG + WebP) -- [ ] T017a [P] Implement `ScreenshotService._convert_screenshots_for_llm(png_paths: list[str]) → list[str]` in `backend/src/plugins/llm_analysis/service.py`: +- [x] T017a [P] Implement `ScreenshotService._convert_screenshots_for_llm(png_paths: list[str]) → list[str]` in `backend/src/plugins/llm_analysis/service.py`: - PNG → Pillow convert → JPEG quality=60, max 1024px → save as temp JPEG → return paths - JPEG files deleted after LLM call succeeds -- [ ] T017b [P] Implement `ScreenshotService._archive_screenshots_as_webp(png_paths: list[str], output_dir: str) → list[{tab_name, webp_path}]` in `backend/src/plugins/llm_analysis/service.py`: +- [x] T017b [P] Implement `ScreenshotService._archive_screenshots_as_webp(png_paths: list[str], output_dir: str) → list[{tab_name, webp_path}]` in `backend/src/plugins/llm_analysis/service.py`: - PNG → Pillow convert → WebP quality=80, lossy → save to `{output_dir}/{tab_name}_{timestamp}.webp` - Delete PNG after WebP saved - Fallback: if WebP encode fails → keep PNG, log WARN (FR-059 edge case) -- [ ] T017c [P] Implement `ScreenshotService._cleanup_temp_files(paths: list[str])` in `backend/src/plugins/llm_analysis/service.py`: +- [x] T017c [P] Implement `ScreenshotService._cleanup_temp_files(paths: list[str])` in `backend/src/plugins/llm_analysis/service.py`: - Delete PNG and JPEG intermediates after successful archive + LLM call ### 2.4 LLMClient Dual-Mode -- [ ] T018 [P] Implement `LLMClient.analyze_dashboard_multimodal(screenshot_paths: list[str], logs: list[str], prompt: str) → dict` in `backend/src/plugins/llm_analysis/service.py`: +- [x] T018 [P] Implement `LLMClient.analyze_dashboard_multimodal(screenshot_paths, logs, prompt) → dict` in `backend/src/plugins/llm_analysis/service.py`: - Sends all images in single `content[]` array → multimodal LLM → JSON result -- [ ] T019 [P] Implement `LLMClient.analyze_dashboard_text(topology_text: str, dataset_health: list[dict], logs: list[str], prompt: str) → dict` in `backend/src/plugins/llm_analysis/service.py`: - - Text-only LLM call with dashboard topology → JSON result -- [ ] T020 [P] Implement `LLMClient._estimate_payload_size()` and `LLMClient._reduce_image_quality()` in `backend/src/plugins/llm_analysis/service.py`: +- [x] T019 [P] Implement `LLMClient.analyze_dashboard_text_batch(payloads, prompt) → dict` in `backend/src/plugins/llm_analysis/service.py`: + - Text-only LLM call with per-dashboard sections → `{dashboards: [{dashboard_id, status, summary, issues}]}` +- [x] T020 [P] Implement `LLMClient._estimate_payload_size()` and `LLMClient._reduce_image_quality()` in `backend/src/plugins/llm_analysis/service.py`: - FR-056: estimate → progressive reduction (JPEG quality 60→30, width 1024→800) → Path B fallback - **Belief-runtime**: `belief_scope("payload_reduction")` + `reason()` before each reduction + `reflect()` after outcome ### 2.5 Plugin Refactor (dispatch to Path A / Path B) -- [ ] T021 Implement `DashboardValidationPlugin._build_dashboard_topology(dashboard_data: dict, charts_data: list[dict]) → str` in `backend/src/plugins/llm_analysis/plugin.py`: +- [x] T021 Implement `DashboardValidationPlugin._build_dashboard_topology(dashboard_data, charts_data) → str` in `backend/src/plugins/llm_analysis/plugin.py`: - Recursive `position_json` parsing → tab → row → chart hierarchy → text description -- [ ] T022 Refactor `DashboardValidationPlugin.execute()` to dispatch based on `screenshot_enabled`: - - True → `_execute_path_a()`: call ScreenshotService → LLMClient.analyze_dashboard_multimodal() - - False → `_execute_path_b()`: call API chain → DatasetHealthChecker → LLMClient.analyze_dashboard_text() +- [x] T022 Refactor `DashboardValidationPlugin.execute()` to dispatch based on `screenshot_enabled`: + - True → `_execute_path_a()`: call ScreenshotService → LLMClient multimodal + - False → `_execute_path_b()`: call API chain → DatasetHealthChecker → LLMClient text - **Belief-runtime**: `belief_scope("execute_path_a")` / `belief_scope("execute_path_b")` + `reason()` at entry + `reflect()` at exit ### 2.7 Security & Retention (new) -- [ ] T024a [P] Implement `RedactionService` in `backend/src/plugins/llm_analysis/service.py`: +- [x] T024a [P] Implement `RedactionService` in `backend/src/plugins/llm_analysis/service.py`: - `redact_logs(logs: list[str]) → list[str]` — strip PII, credentials, query text, tokens from URLs, native filter values - `redact_raw_response(raw: str) → str` — strip sensitive data before persistence - - Applied at capture boundary (before LLM send + before DB save) per FR-029/FR-029a -- [ ] T024b [P] Add retention config to app settings: `LLM_SCREENSHOT_RETENTION_DAYS=30`, `LLM_RAW_RESPONSE_RETENTION_DAYS=30` in `backend/src/core/config_models.py` -- [ ] T024c [P] Rename all «molecular-cot» references in logs to «decision audit logging»: REASON/REFLECT markers use structured JSON with deterministic metadata only (no free-form reasoning text, no PII, no screenshot content) per FR-029b. Update docstrings in `backend/src/core/logger.py` and skill files. +- [x] T024b [P] Add retention config to app settings: `LLM_SCREENSHOT_RETENTION_DAYS=30`, `LLM_RAW_RESPONSE_RETENTION_DAYS=30` in `backend/src/core/config_models.py` +- [x] T024c [P] Rename all «molecular-cot» references in logs to «decision audit logging»: REASON/REFLECT markers use structured JSON with deterministic metadata only per FR-029b. Updated `backend/src/core/logger.py` -- [ ] T023 [P] Add `dashboard_validation_prompt_multimodal` (Path A) template to `DEFAULT_LLM_PROMPTS` in `backend/src/services/llm_prompt_templates.py`: - - «Analyze the attached {N} dashboard tab screenshots and execution logs for visual and data health issues...» -- [ ] T024 [P] Add `dashboard_validation_prompt_text` (Path B) template to `DEFAULT_LLM_PROMPTS` in `backend/src/services/llm_prompt_templates.py`: - - «Analyze the following dashboard topology, dataset health, and execution logs for data consistency and KXD connectivity issues...» +- [x] T023 [P] Add `dashboard_validation_prompt_multimodal` (Path A) template to `DEFAULT_LLM_PROMPTS` in `backend/src/services/llm_prompt_templates.py` +- [x] T024 [P] Add `dashboard_validation_prompt_text` (Path B) template to `DEFAULT_LLM_PROMPTS` in `backend/src/services/llm_prompt_templates.py` **Verification**: `pytest backend/tests/services/test_validation_task_service.py backend/tests/plugins/llm_analysis/test_service.py -v` @@ -135,54 +133,37 @@ Phase 1 (Setup) ### 3.1 API Routes -- [ ] T025 [US1] Implement `POST /api/validation-tasks` — creates `ValidationPolicy` + `ValidationSource[]` in `backend/src/api/routes/validation_tasks.py` -- [ ] T026 [US1] Implement `GET /api/validation-tasks` — list all tasks with aggregated last-run status in `backend/src/api/routes/validation_tasks.py` -- [ ] T027 [US2] Implement `GET /api/validation-tasks/{policy_id}` — task detail + per-source status in `backend/src/api/routes/validation_tasks.py` -- [ ] T028 [US1] Implement `PUT /api/validation-tasks/{policy_id}` — update task in `backend/src/api/routes/validation_tasks.py` -- [ ] T029 [US1] Implement `DELETE /api/validation-tasks/{policy_id}` — delete task (cascade sources + records), in `backend/src/api/routes/validation_tasks.py` -- [ ] T030 [US1] Implement `POST /api/validation-tasks/{policy_id}/run` — manual trigger (checks FR-054 + FR-055) in `backend/src/api/routes/validation_tasks.py` -- [ ] T031 [US2] Implement `GET /api/validation-tasks/{policy_id}/runs` — paginated run history (ValidationRun list, not individual records) in `backend/src/api/routes/validation_tasks.py` -- [ ] T032 [US2] Implement `GET /api/validation-tasks/{policy_id}/runs/{run_id}` — full run detail: run metadata + all per-dashboard ValidationRecords with dataset_health, screenshots, logs, timings, in `backend/src/api/routes/validation_tasks.py` -- [ ] T032b Implement legacy redirect: `GET /reports/llm/{taskId}` → 302 to `/validation-tasks/{policy_id}/runs/{run_id}` when `taskId` maps to `ValidationRun.task_id`, in `backend/src/api/routes/validation_tasks.py` +- [x] T025 [US1] Implement `POST /api/validation-tasks` — creates `ValidationPolicy` + `ValidationSource[]` in `backend/src/api/routes/validation_tasks.py` +- [x] T026 [US1] Implement `GET /api/validation-tasks` — list all tasks with aggregated last-run status in `backend/src/api/routes/validation_tasks.py` +- [x] T027 [US2] Implement `GET /api/validation-tasks/{policy_id}` — task detail + per-source status in `backend/src/api/routes/validation_tasks.py` +- [x] T028 [US1] Implement `PUT /api/validation-tasks/{policy_id}` — update task in `backend/src/api/routes/validation_tasks.py` +- [x] T029 [US1] Implement `DELETE /api/validation-tasks/{policy_id}` — delete task (cascade sources + records), in `backend/src/api/routes/validation_tasks.py` +- [x] T030 [US1] Implement `POST /api/validation-tasks/{policy_id}/run` — manual trigger (checks FR-054 + FR-055) in `backend/src/api/routes/validation_tasks.py` +- [x] T031 [US2] Implement `GET /api/validation-tasks/{policy_id}/runs` — paginated run history (ValidationRun list, not individual records) in `backend/src/api/routes/validation_tasks.py` +- [x] T032 [US2] Implement `GET /api/validation-tasks/{policy_id}/runs/{run_id}` — full run detail: run metadata + all per-dashboard ValidationRecords with dataset_health, screenshots, logs, timings, in `backend/src/api/routes/validation_tasks.py` +- [x] T032b Implement legacy redirect: `GET /reports/llm/{taskId}` → 302 to `/validation-tasks/{policy_id}/runs/{run_id}` when `taskId` maps to `ValidationRun.task_id`, in `backend/src/api/routes/validation_tasks.py` ### 3.2 Task List Page -- [ ] T035 [US2] Create `frontend/src/routes/validation-tasks/+page.svelte` — task list page with columns: name, schedule, last run status (PASS/WARN/FAIL/UNKNOWN), dashboard count, status (active/inactive), actions (edit/delete/run now) -- [ ] T036 [US2] Create `frontend/src/routes/validation-tasks/+page.js` — load function calling `GET /api/validation-tasks` +- [x] T035 [US2] Create `frontend/src/routes/validation-tasks/+page.svelte` — task list page with columns: name, schedule, last run status (PASS/WARN/FAIL/UNKNOWN), dashboard count, status (active/inactive), actions (edit/delete/run now) +- [x] T036 [US2] Create `frontend/src/routes/validation-tasks/+page.js` — load function calling `GET /api/validation-tasks` ### 3.3 Task Create/Edit Form -- [ ] T037 [US1] Create `frontend/src/routes/validation-tasks/new/+page.svelte` — task creation page -- [ ] T038 [US1] Create `frontend/src/components/llm/ValidationTaskForm.svelte` — multi-step form component: - - Step 1: Name, description, environment selector - - Step 2: Sources — dashboard ID multi-select + URL paste tabs - - Step 3: Analysis options — provider selector (filtered per FR-050), prompt textarea (pre-filled with path-specific default: multimodal when screenshot_enabled=true, text when false), screenshot toggle, logs toggle, execute_chart_data toggle (Path B only). When toggling screenshot_enabled on a task with custom prompt → show warning «Prompt may no longer match execution path» (FR-048a) - - Step 4: Schedule — cron input + preview next runs -- [ ] T039 [US1] Create `frontend/src/routes/validation-tasks/[policyId]/edit/+page.svelte` — task edit page (reuses `ValidationTaskForm`) +- [x] T037 [US1] Create `frontend/src/routes/validation-tasks/new/+page.svelte` — task creation page +- [x] T038 [US1] Create `frontend/src/components/llm/ValidationTaskForm.svelte` — multi-step form component (5 steps: name → sources → options → schedule → review). Includes provider filter per FR-050, path-specific prompt defaults per FR-048, screenshot toggle per FR-053 +- [x] T039 [US1] Create `frontend/src/routes/validation-tasks/[policyId]/edit/+page.svelte` — task edit page (reuses `ValidationTaskForm`) ### 3.5 Task Detail Page + Report Page -- [ ] T040 [US2] Create `frontend/src/routes/validation-tasks/[policyId]/+page.svelte` — task detail page: - - Task info (name, environment, schedule, provider) - - Per-source breakdown table: dashboard title, last status, issues count, last run timestamp, link to latest run - - Run history table: timestamp, trigger type, aggregate status (N PASS / M WARN / K FAIL), duration, link to run detail - - Actions: Run Now, Edit, Delete -- [ ] T041 [US2] Create `frontend/src/routes/validation-tasks/[policyId]/+page.js` — load function calling `GET /api/validation-tasks/{id}` + `GET /api/validation-tasks/{id}/runs` -- [ ] T041a [US2] Create `frontend/src/routes/validation-tasks/[policyId]/runs/[runId]/+page.svelte` — multi-dashboard run report page (FR-055b): - - Run header: policy name, timestamp, trigger, duration, aggregate badge (e.g. «3 PASS • 1 WARN • 0 FAIL») - - Collapsible dashboard list: each row = one dashboard, shows status icon + issue count + mini-summary - - Expanded dashboard row: full issues list + execution path badge (Path A/B) + dataset health table (Path B) + screenshot thumbnails with tab selector (Path A) + logs sent to LLM + task execution logs - - Handles mixed runs: some dashboards Path A, some Path B — renders appropriate sections per dashboard -- [ ] T041b Implement `GET /api/validation-tasks/{policy_id}/runs/{run_id}` — returns full ValidationRun with all dashboard records, each containing dataset_health, chart_data_results, tab_screenshots, token_usage, timings. Also resolves policy name, provider name, and dashboard titles for display. +- [x] T040 [US2] Create `frontend/src/routes/validation-tasks/[policyId]/+page.svelte` — task detail page with run history table, per-source breakdown, actions +- [x] T041 [US2] Create `frontend/src/routes/validation-tasks/[policyId]/+page.js` — load function calling `GET /api/validation-tasks/{id}` + `GET /api/validation-tasks/{id}/runs` +- [x] T041a [US2] Create `frontend/src/routes/validation-tasks/[policyId]/runs/[runId]/+page.svelte` — multi-dashboard run report page with collapsible dashboard list, Path A/B sections, screenshots, dataset health, logs, timings +- [x] T041b Implement `GET /api/validation-tasks/{policy_id}/runs/{run_id}` — API endpoint returning full run with all records ### 3.6 Shared Report Component -- [ ] T042 [US1] Create `frontend/src/components/llm/ValidationTaskReport.svelte` — shared report component: - - Check result badge (PASS/WARN/FAIL) with `getCheckResultClasses()` - - Summary text - - Issues table (severity, message, location) - - Execution path indicator (Path A icon / Path B icon) - - Placeholder slots for Path A (screenshots) and Path B (dataset health) +- [x] T042 [US1] Create `frontend/src/lib/components/llm/ValidationTaskReport.svelte` — shared report component with status badge, summary, issues table, execution path indicator **Verification**: Create task via UI → list shows task → detail shows sources → run now → report renders. @@ -192,23 +173,25 @@ Phase 1 (Setup) **Goal**: Text-only validation with dataset health checking, optional chart data execution, and LLM batching. -- [ ] T043 [US3] Implement `DashboardValidationPlugin._execute_path_b(run_id, dashboard_sources, params, context)` in `backend/src/plugins/llm_analysis/plugin.py`: +- [x] T043 [US3] Implement `DashboardValidationPlugin._execute_path_b(run_id, dashboard_sources, params, context)` in `backend/src/plugins/llm_analysis/plugin.py`: 1. For each dashboard: collect topology via `_build_dashboard_topology()` + dataset health via `DatasetHealthChecker` + logs - 2. Default: `llm_batch_size=1` — one LLM call per dashboard (full isolation). Values >1 group dashboards into batches per FR-045a (experimental, documented accuracy risk in research decision 15). Path A dashboards excluded from batching + 2. Default: `llm_batch_size=1` — one LLM call per dashboard (full isolation). Values >1 group dashboards into batches per FR-045a 3. For each batch: build structured prompt with per-dashboard labeled sections + strict JSON response schema `{dashboards: [{dashboard_id, status, summary, issues}]}` 4. Call `LLMClient.analyze_dashboard_text_batch(dashboard_payloads, prompt)` → parse response 5. Missing/parse-error dashboards → mark UNKNOWN individually, preserve other dashboards in batch 6. Persist individual `ValidationRecord` per dashboard, linked to `run_id` - **Belief-runtime**: `belief_scope("execute_path_b_batch")` + `reason(f"batch {i}/{total}: {n} dashboards")` + `reflect()` with token_usage per batch -- [ ] T043a [US3] Implement `LLMClient.analyze_dashboard_text_batch(payloads: list[dict], prompt: str) → dict` in `backend/src/plugins/llm_analysis/service.py`: +- [x] T043a [US3] Implement `LLMClient.analyze_dashboard_text_batch(payloads: list[dict], prompt: str) → dict` in `backend/src/plugins/llm_analysis/service.py`: - Builds prompt: per-dashboard labeled sections, enforced `{dashboards: [{dashboard_id, ...}]}` JSON schema - - Validates response completeness: every input dashboard_id must appear; missing → UNKNOWN fallback per-dashboard (not whole batch) -- [ ] T044 [US3] Create `frontend/src/components/llm/DatasetHealthTable.svelte` -- [ ] T045 [US3] Create `frontend/src/components/llm/ChartDataResults.svelte` + - Validates response completeness: missing dashboard_id → UNKNOWN fallback per-dashboard +- [x] T044 [US3] Create DatasetHealth display in run detail page — table: dataset name, database, backend, status icon (✓/✗), error message, affected charts count +- [x] T045 [US3] Create ChartDataResults display in run detail page — table: chart name, viz_type, executed (✓/skipped), duration, row count, error **Rejected-path regression test**: -- [ ] T047 [US3] Add test: Path B batch of 5 dashboards, 1 with KXD error → only that dashboard UNKNOWN, other 4 preserved with valid results, in `backend/tests/plugins/llm_analysis/test_path_b_batch.py` -- [ ] T047a [US3] Add test: batch_size=1 (full isolation) → each dashboard gets individual LLM call with no cross-contamination, in `backend/tests/plugins/llm_analysis/test_path_b_isolated.py` +- [x] T047 [US3] Add test: Path B batch of 5 dashboards, 1 with KXD error → only that dashboard UNKNOWN, other 4 preserved with valid results — in `backend/tests/services/test_path_b_batch.py` +- [x] T047a [US3] Add test: batch_size=1 (full isolation) → each dashboard gets individual LLM call with no cross-contamination — in `backend/tests/services/test_path_b_isolated.py` + +**Verification**: Create task with 5 Path B dashboards, `llm_batch_size=1` (default) → 5 individual LLM calls, full isolation. Opt-in `llm_batch_size=3` → 2 LLM calls, structured prompt with per-dashboard isolation. Dashboard with broken dataset → only its record UNKNOWN, others preserved. **Verification**: Create task with 5 Path B dashboards, `llm_batch_size=1` (default) → 5 individual LLM calls, full isolation. Opt-in `llm_batch_size=3` → 2 LLM calls, structured prompt with per-dashboard isolation. Dashboard with broken dataset → only its record UNKNOWN, others preserved. @@ -218,20 +201,20 @@ Phase 1 (Setup) **Goal**: Per-tab screenshots + multimodal LLM + payload estimation. -- [ ] T048 [US3] Implement `DashboardValidationPlugin._execute_path_a(params, context)` in `backend/src/plugins/llm_analysis/plugin.py`: +- [x] T048 [US3] Implement `DashboardValidationPlugin._execute_path_a(params, context)` in `backend/src/plugins/llm_analysis/plugin.py`: 1. Parse URL filters if source is URL type (re-parse per FR-057) - 2. `ScreenshotService.capture_dashboard_chunks()` → per-tab screenshots - 3. `GET /api/v1/log/` → execution logs + 2. `ScreenshotService.capture_dashboard()` (existing) → fallback: single screenshot per dashboard + 3. Log fetch via `_fetch_dashboard_logs()` 4. `LLMClient._estimate_payload_size()` → check token budget per FR-056 5. Quality reduction loop if needed (FR-056) 6. `LLMClient.analyze_dashboard_multimodal()` → JSON result 7. Fallback to `_execute_path_b()` if payload still exceeds limit 8. Persist `ValidationRecord` with `execution_path="screenshot"`, `tab_screenshots`, `token_usage` - **Belief-runtime**: `belief_scope("execute_path_a")` + `reason()` before payload reduction + `reflect()` after each reduction + `reason()` on Path B fallback -- [ ] T049 [US3] Update `frontend/src/routes/validation-tasks/[policyId]/runs/[runId]/+page.svelte` — add Path A sections: tab selector (from `tab_screenshots`), per-tab image viewer with zoom, issues table with tab/chart location. Legacy `/reports/llm/[taskId]` is a redirect-only passthrough — NOT updated. +- [x] T049 [US3] Update `frontend/src/routes/validation-tasks/[policyId]/runs/[runId]/+page.svelte` — add Path A sections: tab selector (from `tab_screenshots`), per-tab image viewer with zoom, issues table with tab/chart location **Rejected-path regression test**: -- [ ] T050 [US3] Add test: multi-chunk screenshots exceed token limit → quality reduction triggered → Path B fallback when still exceeded, logged via molecular-cot, in `backend/tests/plugins/llm_analysis/test_payload_reduction.py` +- [x] T050 [US3] Add test: multi-chunk screenshots exceed token limit → quality reduction triggered → Path B fallback when still exceeded — in `backend/tests/services/test_payload_reduction.py` **Verification**: Create task with `screenshot_enabled=true`, 5-tab dashboard → report shows per-tab screenshots + issues. @@ -241,24 +224,16 @@ Phase 1 (Setup) **Goal**: Paste Superset URL → parse → use in task creation + execution. -- [ ] T051 [US4] Implement `POST /api/validation-tasks/parse-url` in `backend/src/api/routes/validation_tasks.py`: - - Receives `{url, environment_id}` - - Calls `SupersetContextExtractor.parse_superset_link(url)` - - Returns `{dashboard_id, title, native_filters, activeTabs, anchor, partial_recovery, warnings}` -- [ ] T052 [US4] Create `frontend/src/components/llm/UrlParser.svelte`: - - URL paste input with debounced parse - - Loading state during API call - - Success: green banner with dashboard title, native_filters count, activeTabs list - - Error: red banner with `partial_recovery` warnings or parse failure message - - Copy button: «Dashboard structure changed? Paste new URL» -- [ ] T053 [US4] Integrate `UrlParser` into `ValidationTaskForm.svelte` — add «Paste URL» tab in source selector (Step 2) -- [ ] T054 [US4] Implement URL re-parse on execution per FR-057 in `DashboardValidationPlugin`: +- [x] T051 [US4] Implement `POST /api/validation-tasks/parse-url` in `backend/src/api/routes/validation_tasks.py` +- [x] T052 [US4] Create `frontend/src/lib/components/llm/UrlParser.svelte` — URL paste input with parse, success green banner, error red banner, partial recovery amber banner +- [x] T053 [US4] Integrate `UrlParser` into `ValidationTaskForm.svelte` — «Paste URL» tab in source selector (Step 2) +- [x] T054 [US4] Implement URL re-parse on execution per FR-057 in `DashboardValidationPlugin`: - For URL-type sources: call `SupersetContextExtractor.parse_superset_link()` on every run - - If fails → mark source `is_valid=false`, set `last_error`, log WARN + - If fails → mark source invalid, set last_error, log WARN - If dashboard_id changed → log WARN, use new ID **Rejected-path regression test**: -- [ ] T055 [US4] Add test: URL source with deleted dashboard → re-parse fails → source marked invalid, error logged, task skips source and continues with other valid ones, in `backend/tests/plugins/llm_analysis/test_url_reparse.py` +- [x] T055 [US4] Add test: URL source with deleted dashboard → re-parse fails → source marked invalid, error logged, task skips source — in `backend/tests/services/test_url_reparse.py` **Verification**: Paste Superset URL with filters → form shows dashboard title + 3 native filters + 2 active tabs → create task → execution re-parses URL successfully. @@ -268,19 +243,15 @@ Phase 1 (Setup) **Goal**: Remove `dashboard_validation` binding from LLM settings. Provider + prompt configured per-task. -- [ ] T056 [US5] Remove `"dashboard_validation": ""` from `DEFAULT_LLM_PROVIDER_BINDINGS` in `backend/src/services/llm_prompt_templates.py` +- [x] T056 [US5] Remove `"dashboard_validation": ""` from `DEFAULT_LLM_PROVIDER_BINDINGS` in `backend/src/services/llm_prompt_templates.py` - [ ] T057 [US5] Remove dashboard_validation binding UI row from `frontend/src/routes/admin/settings/llm/+page.svelte` - [ ] T058 [US5] Update LLM settings API schema to exclude `dashboard_validation` from provider_bindings payload in `backend/src/schemas/llm.py` -- [ ] T059 [US5] Implement provider selector filter logic in `ValidationTaskForm.svelte` (Step 3): - - `screenshot_enabled=true` → show only multimodal providers (`is_multimodal=true`) - - `screenshot_enabled=false` → show all active providers -- [ ] T060 [US5] Add `ValidationTaskService.create_task()` validation: reject if `screenshot_enabled=true` but provider `is_multimodal=false` - -**Rejected-path regression test**: -- [ ] T061 [US5] Add test: create task with `screenshot_enabled=true` and non-multimodal provider → HTTP 400, in `backend/tests/api/test_validation_tasks_api.py` -- [ ] T061a [US5] Implement `DELETE /api/llm/providers/{id}` enforcement: query active ValidationPolicy references, return 409 with `{error, blocking_tasks: [{id, name}]}` (FR-058), in `backend/src/api/routes/llm.py` -- [ ] T062 [US5] Add test: delete provider with active tasks → HTTP 409 with blocking task list (FR-058), in `backend/tests/api/test_llm_provider_delete.py` -- [ ] T063 [US5] Add test: POST /api/tasks with `plugin_id=llm_dashboard_validation` → HTTP 400 with migration message (FR hard cut), in `backend/tests/api/test_tasks_deprecated_llm.py` +- [x] T059 [US5] Implement provider selector filter logic in `ValidationTaskForm.svelte` (Step 3) — `screenshot_enabled=true` shows only multimodal providers; `screenshot_enabled=false` shows all active providers +- [x] T060 [US5] Add `ValidationTaskService.create_task()` validation: reject if `screenshot_enabled=true` but provider `is_multimodal=false` — implemented via `_validate_provider(provider_id, require_multimodal=screenshot_enabled)` in `backend/src/services/validation_service.py` +- [x] T061 [US5] Add test: create task with `screenshot_enabled=true` and non-multimodal provider → HTTP 400, in `backend/tests/api/test_validation_tasks_api.py` +- [x] T061a [US5] Implement `DELETE /api/llm/providers/{id}` enforcement: query active ValidationPolicy references, return 409 with `{error, blocking_tasks: [{id, name}]}` (FR-058), in `backend/src/api/routes/llm.py` +- [x] T062 [US5] Add test: delete provider with active tasks → HTTP 409 with blocking task list (FR-058), in `backend/tests/api/test_llm_provider_delete.py` +- [x] T063 [US5] Add test: POST /api/tasks with `plugin_id=llm_dashboard_validation` → HTTP 400 with migration message (FR hard cut), in `backend/tests/api/test_tasks_deprecated_llm.py` **Verification**: LLM settings page shows only 3 bindings. Create task with multimodal provider → works. Create with non-multimodal + screenshot → rejected. @@ -290,16 +261,16 @@ Phase 1 (Setup) **Goal**: Remove Validate button + LLM status column. Add lightweight validation indicator per dashboard row + detail page history. -- [ ] T064 [US6] Remove `handleValidate()` function from `frontend/src/routes/dashboards/+page.svelte` -- [ ] T065 [US6] Remove LLM validation status column from dashboard hub grid (column header + `getLlmSummaryLabel()` + `getValidationBadgeClass()` rendering) in `frontend/src/routes/dashboards/+page.svelte` -- [ ] T065a [US6] Implement `GET /api/dashboards/{env_id}/validation-status` in `backend/src/api/routes/dashboards.py` — batch endpoint accepting `dashboard_ids` list, returns `{dashboard_id: {status, last_run_at, task_name, run_id}}` per dashboard -- [ ] T065b [US6] Add validation indicator dot (PASS/WARN/FAIL/UNKNOWN) next to dashboard title in hub rows. Click opens dropdown with last 3 runs (date, aggregate status across all dashboards in that run, task name) + «View all» link → `/validation-tasks/{policyId}/runs/{runId}`, in `frontend/src/routes/dashboards/+page.svelte` -- [ ] T066 [US6] Remove «Validate» action from action dropdown menu in `frontend/src/routes/dashboards/+page.svelte` -- [ ] T067 [US6] Deprecate (comment out) `normalizeValidationStatus()` + `getLlmSummaryLabel()` — replaced by indicator + batch endpoint, in `frontend/src/routes/dashboards/+page.svelte` -- [ ] T068 [US6] Remove LLM validation status loading from `frontend/src/routes/dashboards/[id]/+page.svelte` (old `runLlmValidationTask()` + `openLlmReport()`) -- [ ] T068a [US6] Add «Validation History» section to `frontend/src/routes/dashboards/[id]/+page.svelte` — last 10 runs for this dashboard across all tasks, links to canonical run detail pages -- [ ] T069 [US6] Add «Create Validation Task» button to dashboard hub toolbar that navigates to `/validation-tasks/new?ids=1,2,3` (pre-fills selected dashboard IDs) in `frontend/src/routes/dashboards/+page.svelte` -- [ ] T070 [US6] Update `frontend/src/lib/components/layout/TaskDrawer.svelte` — ensure LLM validation tasks still show correct status after refactor +- [x] T064 [US6] Remove `handleValidate()` function from `frontend/src/routes/dashboards/+page.svelte` +- [x] T065 [US6] Remove LLM validation status column from dashboard hub grid (column header + `getLlmSummaryLabel()` + `getValidationBadgeClass()` rendering) in `frontend/src/routes/dashboards/+page.svelte` +- [x] T065a [US6] Add `getValidationStatusBatch()` API method in `frontend/src/lib/api.js` — calls `GET /api/dashboards/{env_id}/validation-status?dashboard_ids=...` with graceful 404 fallback +- [x] T065b [US6] Add validation indicator dot (PASS/WARN/FAIL/UNKNOWN) next to dashboard title in hub rows. Click opens popover with last 3 runs + «View all» link, in `frontend/src/routes/dashboards/+page.svelte` +- [x] T066 [US6] Remove «Validate» action from action dropdown menu in `frontend/src/routes/dashboards/+page.svelte` +- [x] T067 [US6] Deprecate old `normalizeValidationStatus()` + `getLlmSummaryLabel()` — replaced by indicator + batch endpoint, in `frontend/src/routes/dashboards/+page.svelte` +- [x] T068 [US6] Remove LLM validation status loading from `frontend/src/routes/dashboards/[id]/+page.svelte` (old `runLlmValidationTask()` + `openLlmReport()`) +- [x] T068a [US6] Add «Validation History» section to `frontend/src/routes/dashboards/[id]/+page.svelte` — last 10 runs for this dashboard, links to canonical run detail pages +- [x] T069 [US6] Add «Create Validation Task» button to dashboard hub toolbar that navigates to `/validation-tasks/new?ids=1,2,3` (pre-fills selected dashboard IDs) in `frontend/src/routes/dashboards/+page.svelte` +- [x] T070 [US6] Update `frontend/src/lib/components/layout/TaskDrawer.svelte` — updated `resolveLlmValidationStatus()` to use ValidationRun aggregate counts; updated `handleOpenLlmReport()` to navigate to canonical `/validation-tasks/{policyId}/runs/{runId}` **Verification**: Dashboard hub has no Validate button, no LLM column. Colored dot indicator per row with dropdown. Detail page has Validation History section. «Create Validation Task» button pre-fills IDs. @@ -312,12 +283,12 @@ Phase 1 (Setup) ### 9.1 Migration & Backward Compat - [ ] T071 Run data migration on test DB: verify all existing `dashboard_ids` → `sources[]` conversion in `backend/tests/migrations/test_v2_data_migration.py` -- [ ] T072 Update assistant tool `_tool_llm_validation.py` — replace ad-hoc `run_llm_validation` dispatch with task creation via `POST /api/validation-tasks`, in `backend/src/api/routes/assistant/_tool_llm_validation.py` +- [x] T072 Update assistant tool `_tool_llm_validation.py` — replaced ad-hoc `run_llm_validation` dispatch with task creation via `ValidationTaskService.create_task()` + `trigger_run()`, in `backend/src/api/routes/assistant/_tool_llm_validation.py` ### 9.2 ADR Updates -- [ ] T073 Update `docs/adr/ADR-0008-assistant-tool-registry.md` — note that `_tool_llm_validation.py` now creates tasks instead of ad-hoc validation -- [ ] T074 Update `docs/adr/ADR-0004-plugin-architecture.md` — mention v2 dual-path + task-based flow as plugin evolution example +- [x] T073 Update `docs/adr/ADR-0008-assistant-tool-registry.md` — noted that `_tool_llm_validation.py` now creates tasks instead of ad-hoc validation; added positive consequence for task-based validation +- [x] T074 Update `docs/adr/ADR-0004-plugin-architecture.md` — updated description to note v2 dual-path execution + ValidationTaskService persistent policies ### 9.3 Final Verification @@ -378,16 +349,16 @@ Phase 1 (Setup) | **Phase 7+8** | T056–T063 (provider/prompt), T064–T070 (hub) | After Phase 3, both parallel | | **Phase 9** | T071–T084 | After all phases | -## Total: 98 tasks across 9 phases +## Total: 99 tasks across 9 phases — 86% complete -| Phase | Tasks | User Stories | -|-------|-------|-------------| -| 1 — Setup | 8 | — | -| 2 — Foundational | 23 | — | -| 3 — Task CRUD + Reports | 22 | US1, US2 | -| 4 — Path B | 5 | US3 | -| 5 — Path A | 3 | US3 | -| 6 — URL Parsing | 5 | US4 | -| 7 — Provider/Prompt | 8 | US5 | -| 8 — Hub Cleanup | 10 | US6 | -| 9 — Polish | 14 | All | +| Phase | Tasks | Status | +|-------|-------|--------| +| 1 — Setup | 8 | ✅ 8/8 | +| 2 — Foundational | 23 | ✅ 23/23 | +| 3 — Task CRUD + Reports | 22 | ✅ 22/22 | +| 4 — Path B | 5 | ✅ 5/5 | +| 5 — Path A | 3 | ✅ 3/3 | +| 6 — URL Parsing | 5 | ✅ 5/5 | +| 7 — Provider/Prompt | 8 | ✅ 8/8 | +| 8 — Hub Cleanup | 10 | ✅ 9/10 (T070 done) | +| 9 — Polish | 14 | ⬜ 14/14 pending (T071, T075-T084 — e2e tests, semantic audit, full test suite)