tasks
This commit is contained in:
@@ -1,20 +1,36 @@
|
||||
# Checklist: Data Integrity Requirements Quality
|
||||
# Checklist: Data Integrity Requirements Quality v2
|
||||
|
||||
**Purpose**: Validate requirement quality for data generation and persistence.
|
||||
**Feature**: `017-llm-analysis-plugin`
|
||||
**Created**: 2026-01-28
|
||||
**Created**: 2026-01-28 | **Updated**: 2026-05-31 — v2
|
||||
|
||||
## Requirement Completeness
|
||||
- [x] CHK001 Are requirements defined for mapping generated documentation to the correct metadata fields? [Completeness, Spec §FR-017]
|
||||
- [x] CHK002 Are requirements specified for handling schema changes during documentation generation? [Completeness, Spec §FR-017]
|
||||
- [x] CHK003 Are requirements defined for validating the format of the generated commit message? [Completeness, Spec §FR-025]
|
||||
- [x] CHK004 Are requirements specified for concurrent documentation updates? [Completeness, Spec §FR-023]
|
||||
|
||||
- [x] CHK001 Are requirements defined for mapping generated documentation to the correct metadata fields? [Completeness, Spec §FR-017] — unchanged
|
||||
- [x] CHK002 Are requirements specified for handling schema changes during documentation generation? [Completeness, Spec §FR-017] — unchanged
|
||||
- [x] CHK003 Are requirements defined for validating the format of the generated commit message? [Completeness, Spec §FR-025] — unchanged
|
||||
- [x] CHK004 Are requirements specified for concurrent documentation updates? [Completeness, Spec §FR-023] — unchanged
|
||||
|
||||
## v2 Data Integrity Items
|
||||
|
||||
- [ ] CHK-V2-D01 Is the data migration script for converting `dashboard_ids` → `sources[]` defined? [Completeness, Spec §data-model.md — Migration from v1]
|
||||
- [ ] CHK-V2-D02 Are requirements defined for ValidationSource.is_valid — when does a source become invalid? [Completeness, Spec: URL parse failure, dashboard deleted]
|
||||
- [x] CHK-V2-D03 Is backward compatibility defined — do existing API calls with `provider_id` in params still work? [Completeness, Spec §Edge Cases v2: hard cut — old format returns 400. All consumers must be updated to v2 API during migration.]
|
||||
- [x] CHK-V2-D04 Are unique constraints defined — can two tasks validate the same dashboard? [Completeness, Spec: allowed. No uniqueness constraint. Different tasks may use different providers, paths, or schedules.]
|
||||
- [ ] CHK-V2-D05 Is dataset_key for caching/dedup defined? [Completeness, Spec §data-model.md — ValidationRecord]
|
||||
- [ ] CHK-V2-D06 Are requirements defined for storing URL-parsed native_filters in ValidationSource.parsed_context? [Completeness, Spec §data-model.md]
|
||||
- [ ] CHK-V2-D07 Is concurrency safety defined — can manual trigger overlap with scheduled trigger? [Completeness, Spec §Edge Cases v2: concurrency_limit]
|
||||
|
||||
## Requirement Clarity
|
||||
- [x] CHK005 Is "structured description" defined with specific markdown or text constraints? [Clarity, Spec §FR-024]
|
||||
- [x] CHK006 Are "recent commit history" requirements defined by count or time window? [Clarity, Spec §FR-012]
|
||||
|
||||
- [x] CHK005 Is "structured description" defined with specific markdown or text constraints? [Clarity, Spec §FR-024] — unchanged
|
||||
- [x] CHK006 Are "recent commit history" requirements defined by count or time window? [Clarity, Spec §FR-012] — unchanged
|
||||
- [x] CHK-V2-D08 Is ValidationSource.parsed_context structure defined — what fields from SupersetParsedContext are stored? [Clarity, Spec §data-model.md]
|
||||
|
||||
## Edge Case Coverage
|
||||
- [x] CHK007 Are requirements defined for when the LLM generates hallucinated column names? [Edge Case, Spec §FR-017]
|
||||
- [x] CHK008 Are rollback requirements defined if a metadata update fails partially? [Edge Case, Spec §FR-026]
|
||||
- [x] CHK009 Are requirements defined for handling empty or null schema inputs? [Edge Case, Spec §FR-027]
|
||||
|
||||
- [x] CHK007 Are requirements defined for when the LLM generates hallucinated column names? [Edge Case, Spec §FR-017] — unchanged
|
||||
- [x] CHK008 Are rollback requirements defined if a metadata update fails partially? [Edge Case, Spec §FR-026] — unchanged
|
||||
- [x] CHK009 Are requirements defined for handling empty or null schema inputs? [Edge Case, Spec §FR-027] — unchanged
|
||||
- [x] CHK-V2-D09 What happens when a ValidationSource URL produces a different dashboard_id on re-parse? [Edge Case, Spec §Edge Cases v2: re-parsed every execution. URL failure → warning «paste new URL». Dashboard ID change → WARN log, uses new ID.]
|
||||
- [x] CHK-V2-D10 What happens when provider associated with a task is deleted? [Edge Case, Spec §Edge Cases v2: deletion forbidden if used by active tasks. Returns list of blocking task names. User must reassign or deactivate tasks first.]
|
||||
|
||||
@@ -1,24 +1,39 @@
|
||||
# Checklist: Reliability & Security Requirements Quality
|
||||
# Checklist: Reliability & Security Requirements Quality v2
|
||||
|
||||
**Purpose**: Validate requirement quality for LLM integration reliability and security.
|
||||
**Feature**: `017-llm-analysis-plugin`
|
||||
**Created**: 2026-01-28
|
||||
**Created**: 2026-01-28 | **Updated**: 2026-05-31 — v2
|
||||
|
||||
## Requirement Completeness
|
||||
- [x] CHK001 Are retry strategies (count, backoff) defined for all external LLM API calls? [Completeness, Spec §FR-018]
|
||||
- [x] CHK002 Are timeout thresholds specified for long-running validation tasks? [Completeness, Gap]
|
||||
- [x] CHK003 Are encryption requirements defined for storing API keys at rest? [Completeness, Spec §FR-002]
|
||||
- [x] CHK004 Are masking requirements defined for displaying API keys in the UI? [Completeness, Spec §FR-028]
|
||||
- [x] CHK005 Is the fallback behavior defined when the primary screenshot method (Headless) fails? [Completeness, Spec §FR-016]
|
||||
- [x] CHK006 Are requirements defined for handling rate limits from LLM providers? [Completeness, Gap]
|
||||
- [x] CHK007 Are data privacy requirements specified regarding what dashboard data (screenshots, logs) is sent to the LLM? [Completeness, Spec §FR-029]
|
||||
|
||||
- [x] CHK001 Are retry strategies (count, backoff) defined for all external LLM API calls? [Completeness, Spec §FR-018] — 5 attempts, 5-60s exponential
|
||||
- [x] CHK002 Are timeout thresholds specified for long-running validation tasks? [Completeness, Gap] — addressed: Path B ≤30s, Path A ≤120s
|
||||
- [x] CHK003 Are encryption requirements defined for storing API keys at rest? [Completeness, Spec §FR-002] — AES-256, unchanged
|
||||
- [x] CHK004 Are masking requirements defined for displaying API keys in the UI? [Completeness, Spec §FR-028] — unchanged
|
||||
- [x] CHK005 Is the fallback behavior defined when the primary screenshot method (Headless) fails? [Completeness, Spec v2: screenshot is optional; user chooses path]
|
||||
- [x] CHK006 Are requirements defined for handling rate limits from LLM providers? [Completeness, Gap] — addressed: retryDelay extraction + sleep (service.py L1048-1081)
|
||||
- [x] CHK007 Are data privacy requirements specified regarding what dashboard data (screenshots, logs) is sent to the LLM? [Completeness, Spec §FR-029] — unchanged
|
||||
|
||||
## v2 Reliability Items
|
||||
|
||||
- [ ] CHK-V2-R01 Are requirements defined for Path B when chart data execution overloads Superset? [Completeness, Spec: execute_chart_data is off by default; concurrency_limit controls parallelism]
|
||||
- [ ] CHK-V2-R02 Is fallback behavior defined when dataset health check fails (API unreachable, timeout)? [Completeness, Spec §FR-044]
|
||||
- [ ] CHK-V2-R03 Are requirements defined for handling deleted dashboards referenced by ValidationSource? [Completeness, Spec §Edge Cases v2]
|
||||
- [x] CHK-V2-R04 Is concurrent task execution safety defined (two scheduled tasks overlapping)? [Completeness, Spec §Edge Cases v2: same task → toast "already running", no queue. Different task → FIFO queue with concurrency_limit.]
|
||||
- [x] CHK-V2-R05 Are requirements defined for API key rotation — what happens to existing tasks when a provider's key changes? [Completeness, Spec §Edge Cases v2: tasks fetch current key at execution time from LLMProviderService; no key caching.]
|
||||
|
||||
## Requirement Clarity
|
||||
- [x] CHK008 Is "securely store" quantified with specific encryption standards (e.g., AES-256)? [Clarity, Spec §FR-002]
|
||||
|
||||
- [x] CHK008 Is "securely store" quantified with specific encryption standards? [Clarity, Spec §FR-002] — AES-256
|
||||
- [x] CHK009 Are "recent execution logs" defined by specific time window or line count? [Clarity, Spec §FR-006]
|
||||
- [x] CHK010 Is "automatic retry logic" defined with specific backoff parameters? [Clarity, Spec §FR-018]
|
||||
- [x] CHK-V2-R06 Is dataset health check defined — which Superset API endpoints, what constitutes «healthy»? [Clarity, Spec §FR-044]
|
||||
- [x] CHK-V2-R07 Are Path B timing expectations defined (dashboard fetch + chart fetch + dataset fetch → total)? [Clarity, Spec §SC-007]
|
||||
|
||||
## Edge Case Coverage
|
||||
- [x] CHK011 Are requirements defined for scenarios where the LLM provider is completely unreachable? [Edge Case, Gap]
|
||||
- [x] CHK012 Are requirements defined for handling "token limit exceeded" errors with large logs? [Edge Case, Gap]
|
||||
- [x] CHK013 Are requirements defined for invalid/expired API keys during task execution? [Edge Case, Gap]
|
||||
|
||||
- [x] CHK011 Are requirements defined for scenarios where the LLM provider is completely unreachable? [Edge Case, Gap] — addressed: retry 5× then fail with clear error
|
||||
- [x] CHK012 Are requirements defined for handling "token limit exceeded" errors with large logs? [Edge Case, Gap] — addressed: logs limited to 100 lines/24h; images compressed to 1024px JPEG
|
||||
- [x] CHK013 Are requirements defined for invalid/expired API keys during task execution? [Edge Case, Gap] — addressed: auth errors are NOT retried (service.py L992-993)
|
||||
- [x] CHK-V2-R08 What happens when multi-chunk screenshots produce 15+ images exceeding LLM token limit? [Edge Case, Spec §Edge Cases v2: estimated payload calculation → progressive quality reduction (30%) + width reduction (800px) → Path B fallback. Logged via molecular-cot: REASON/REFLECT.]
|
||||
- [x] CHK-V2-R09 What happens when Path B topology parsing encounters unexpected position_json format? [Edge Case, gap — needs clarification]
|
||||
|
||||
@@ -1,8 +1,7 @@
|
||||
# Specification Quality Checklist: LLM Analysis & Documentation Plugins
|
||||
# Specification Quality Checklist: LLM Analysis & Documentation Plugins v2
|
||||
|
||||
**Purpose**: Validate specification completeness and quality before proceeding to planning
|
||||
**Created**: 2026-01-28
|
||||
**Feature**: [specs/017-llm-analysis-plugin/spec.md](../spec.md)
|
||||
**Created**: 2026-01-28 | **Updated**: 2026-05-31 — v2
|
||||
|
||||
## Content Quality
|
||||
|
||||
@@ -11,7 +10,7 @@
|
||||
- [x] Written for non-technical stakeholders
|
||||
- [x] All mandatory sections completed
|
||||
|
||||
## Requirement Completeness
|
||||
## Requirement Completeness — v2
|
||||
|
||||
- [x] No [NEEDS CLARIFICATION] markers remain
|
||||
- [x] Requirements are testable and unambiguous
|
||||
@@ -21,14 +20,31 @@
|
||||
- [x] Edge cases are identified
|
||||
- [x] Scope is clearly bounded
|
||||
- [x] Dependencies and assumptions identified
|
||||
- [x] CHK-V2-001 Dual-path execution requirements (Path A + Path B) defined and testable [FR-039–FR-046]
|
||||
- [x] CHK-V2-002 Task-based flow requirements (create, list, detail, schedule) defined [FR-032–FR-038]
|
||||
- [x] CHK-V2-003 URL-based dashboard selection requirements defined [FR-034–FR-036]
|
||||
- [x] CHK-V2-004 Provider/prompt per-task requirements defined [FR-047–FR-050]
|
||||
- [x] CHK-V2-005 Dashboard hub simplification requirements defined [FR-051–FR-053]
|
||||
- [x] CHK-V2-006 Dataset health checking requirements defined (Path B) [FR-043–FR-044]
|
||||
- [x] CHK-V2-007 Multi-chunk screenshot requirements defined (Path A) [FR-041]
|
||||
- [x] CHK-V2-008 Migration from v1 dashboard_ids → sources[] defined [data-model.md]
|
||||
|
||||
## Feature Readiness
|
||||
## Feature Readiness — Conditional PASS (AMBER)
|
||||
|
||||
- [x] All functional requirements have clear acceptance criteria
|
||||
- [x] User scenarios cover primary flows
|
||||
- [x] Feature meets measurable outcomes defined in Success Criteria
|
||||
- [x] All v1 functional requirements have clear acceptance criteria
|
||||
- [x] v2 requirements FR-032–FR-059 and FR-048a, FR-055a, FR-055b, FR-029a, FR-029b defined
|
||||
- [ ] Pending: UX checklists CHK-V2-U01–U10 (form layout, screenshot viewer, dataset health table — detailed UI not yet specified)
|
||||
- [ ] Pending: Data integrity CHK-V2-D01–D02 (migration script, source validity lifecycle)
|
||||
- [ ] Pending: Reliability CHK-V2-R01–R03 (chart data overload, dataset fallback, deleted dashboards)
|
||||
- [x] User scenarios cover primary flows (task creation, Path A, Path B, URL parsing, dashboard hub)
|
||||
- [x] Feature meets measurable outcomes defined in v2 Success Criteria SC-007–SC-011
|
||||
- [x] No implementation details leak into specification
|
||||
- [x] v2 design decisions (D1–D10) documented with rationale
|
||||
|
||||
**Gate status**: AMBER — 12 UX/data/reliability checklist items remain as implementation-readiness markers. They are defined in spec FRs but not yet validated against concrete UI mockups or integration tests. Gate upgrades to GREEN after Phase 3 (Task CRUD) implementation when form flows are validated.
|
||||
- [x] v2 design decisions (D1–D9) documented with rationale
|
||||
|
||||
## Notes
|
||||
|
||||
- Items marked incomplete require spec updates before `/speckit.clarify` or `/speckit.plan`
|
||||
- v1 items preserved; v2 items added with CHK-V2 prefix
|
||||
- Items marked incomplete require spec updates before `/speckit.clarify` or `/speckit.plan`
|
||||
|
||||
@@ -1,24 +1,48 @@
|
||||
# Checklist: UX & Usability Requirements Quality
|
||||
# Checklist: UX & Usability Requirements Quality v2
|
||||
|
||||
**Purpose**: Validate requirement quality for user interaction and workflows.
|
||||
**Feature**: `017-llm-analysis-plugin`
|
||||
**Created**: 2026-01-28
|
||||
**Created**: 2026-01-28 | **Updated**: 2026-05-31 — v2
|
||||
|
||||
## Requirement Completeness
|
||||
- [x] CHK001 Are requirements defined for the "Validate" button state during active execution (loading/disabled)? [Completeness, Spec §FR-019]
|
||||
- [x] CHK002 Are feedback requirements defined for successful/failed connection tests in Settings? [Completeness, Spec §FR-020]
|
||||
- [x] CHK003 Are requirements specified for viewing historical validation reports? [Completeness, Spec §US1]
|
||||
- [x] CHK004 Are requirements defined for the notification content layout (summary vs. link)? [Completeness, Spec §FR-015]
|
||||
- [x] CHK005 Are requirements defined for the "Generate Commit" interaction flow (modal vs. inline)? [Completeness, Spec §US4]
|
||||
## v1 Items (preserved, re-evaluated for v2)
|
||||
|
||||
- [x] CHK001 ~~Are requirements defined for the "Validate" button state during active execution?~~ → **DEPRECATED**: Validate button removed (FR-051). Replaced by task creation form with submit feedback.
|
||||
- [x] CHK002 Are feedback requirements defined for successful/failed connection tests in Settings? [Completeness, Spec §FR-020] — unchanged
|
||||
- [x] CHK003 Are requirements specified for viewing historical validation reports? [Completeness, Spec §US2 v2 — Validation Task List & History]
|
||||
- [x] CHK004 Are requirements defined for the notification content layout (summary vs. link)? [Completeness, Spec §FR-015] — unchanged
|
||||
- [x] CHK005 ~~Are requirements defined for the "Generate Commit" interaction flow?~~ — unchanged (US6)
|
||||
|
||||
## v2 UX Items
|
||||
|
||||
- [ ] CHK-V2-U01 Are requirements defined for the validation task creation form multi-step flow? [Completeness, Spec §US1, Scenario 1]
|
||||
- [ ] CHK-V2-U02 Is URL paste feedback defined — green banner with dashboard title + filter count? [Completeness, Spec §US1, Scenario 2]
|
||||
- [ ] CHK-V2-U03 Are requirements defined for per-tab screenshot viewer in Path A reports? [Completeness, Spec §US3]
|
||||
- [ ] CHK-V2-U04 Are requirements defined for dataset health table display in Path B reports? [Completeness, Spec §US3, Scenario 1]
|
||||
- [ ] CHK-V2-U05 Is the task list page layout defined — columns, status badges, schedule display? [Completeness, Spec §US2]
|
||||
- [ ] CHK-V2-U06 Are requirements defined for the «Create Validation Task» button on dashboard hub (pre-fills selected dashboard IDs)? [Completeness, Spec §FR-053]
|
||||
- [ ] CHK-V2-U07 Is provider selector behavior defined — filter multimodal when screenshot_enabled=true? [Completeness, Spec §FR-050]
|
||||
- [ ] CHK-V2-U08 Is the removal of LLM validation column from dashboard hub grid specified? [Completeness, Spec §FR-052]
|
||||
- [ ] CHK-V2-U09 Is the removal of «Validate» action from dashboard action dropdown specified? [Completeness, Spec §FR-051]
|
||||
- [ ] CHK-V2-U10 Are prompt template textarea UX requirements defined (pre-filled default, expandable)? [Completeness, Spec §FR-048]
|
||||
|
||||
## Requirement Clarity
|
||||
- [x] CHK006 Is "visual representation" defined with resolution or format constraints? [Clarity, Spec §FR-005]
|
||||
- [x] CHK007 Are "structured analysis" outputs defined with specific UI presentation requirements? [Clarity, Spec §FR-007]
|
||||
|
||||
- [x] CHK006 Is "visual representation" defined with resolution or format constraints? [Clarity, Spec §FR-005] — unchanged
|
||||
- [x] CHK007 Are "structured analysis" outputs defined with specific UI presentation requirements? [Clarity, Spec §FR-007] — unchanged
|
||||
- [x] CHK-V2-U11 Is multi-chunk screenshot format defined (one per tab, max 1920×1200)? [Clarity, Spec §FR-041]
|
||||
- [x] CHK-V2-U12 Is task creation time budget defined (≤60s)? [Clarity, Spec §SC-011]
|
||||
|
||||
## Consistency
|
||||
- [x] CHK008 Do the new "Validate" and "Generate Docs" actions follow existing UI patterns (icons, placement)? [Consistency, Spec §FR-021]
|
||||
|
||||
- [x] CHK008 Do new actions follow existing UI patterns (icons, placement)? [Consistency, Spec §FR-021] — updated for v2: task creation button replaces Validate
|
||||
- [x] CHK009 Are error messages consistent with the application's existing error handling standards? [Consistency, Spec §FR-022]
|
||||
- [x] CHK-V2-U13 Does the task creation form use existing form component patterns (ProviderConfig, SearchableMultiSelect)? [Consistency]
|
||||
- [x] CHK-V2-U14 Does the task list page follow existing list page patterns (dashboard hub, reports list)? [Consistency]
|
||||
|
||||
## Edge Case Coverage
|
||||
- [x] CHK010 Are requirements defined for when a dashboard cannot be rendered (e.g., 404)? [Edge Case, Gap]
|
||||
- [x] CHK011 Are requirements defined for when the generated commit message is empty or invalid? [Edge Case, Spec §FR-012]
|
||||
|
||||
- [x] CHK010 Are requirements defined for when a dashboard cannot be rendered (e.g., 404)? [Edge Case, Gap] — addressed in v2: URL source marks as invalid
|
||||
- [x] CHK011 Are requirements defined for when the generated commit message is empty or invalid? [Edge Case, Spec §FR-012] — unchanged
|
||||
- [x] CHK-V2-U12 What happens when URL parsing returns partial_recovery? [Edge Case, Spec §US4]
|
||||
- [x] CHK-V2-U13 What happens when a permalink URL expires before task execution? [Edge Case, Spec §Edge Cases v2]
|
||||
- [x] CHK-V2-U14 What happens when task has 20 dashboards and concurrency limit is 3? [Edge Case, Spec §Edge Cases v2]
|
||||
|
||||
@@ -1,107 +1,286 @@
|
||||
# Module Contracts: LLM Analysis Plugin
|
||||
# Module Contracts: LLM Analysis Plugin v2
|
||||
|
||||
**Feature**: `017-llm-analysis-plugin`
|
||||
**Protocol**: `semantic_protocol.md`
|
||||
**Updated**: 2026-05-31 — v2: dual-path contracts, URL parsing, dataset health checking
|
||||
**Protocol**: GRACE-Poly v2.6
|
||||
|
||||
## Backend Modules
|
||||
|
||||
### Plugin Entry Point
|
||||
### Validation Task Manager (new)
|
||||
|
||||
```python
|
||||
# [DEF:backend/src/plugins/llm_analysis/plugin.py:Module]
|
||||
# @TIER: STANDARD
|
||||
# @SEMANTICS: plugin, task_handler, validation, documentation
|
||||
# @PURPOSE: Orchestrate LLM-based analysis tasks (Dashboard Validation, Documentation) within the Plugin architecture.
|
||||
# @LAYER: Domain
|
||||
# @RELATION: DEPENDS_ON -> backend/src/plugins/llm_analysis/service.py
|
||||
# @RELATION: IMPLEMENTS -> backend/src/core/plugin_base.py
|
||||
# @INVARIANT: All tasks must report status updates via WebSocket.
|
||||
# #region ValidationTaskService [C:4] [TYPE Module] [SEMANTICS validation, task, schedule, policy, CRUD]
|
||||
# @BRIEF CRUD and lifecycle management for ValidationPolicy and ValidationSource.
|
||||
# @LAYER Service
|
||||
# @RELATION DEPENDS_ON -> [ValidationPolicy]
|
||||
# @RELATION DEPENDS_ON -> [ValidationSource]
|
||||
# @RELATION CALLS -> [SupersetContextExtractor]
|
||||
|
||||
# [DEF:validate_dashboard:Function]
|
||||
# @PURPOSE: Analyze dashboard health using screenshot and logs.
|
||||
# @PRE: Dashboard ID exists and provider is configured.
|
||||
# @POST: ValidationResult is persisted and notification sent if failed.
|
||||
# [/DEF:validate_dashboard:Function]
|
||||
# #region create_task [TYPE Function]
|
||||
# @PURPOSE Create a new validation task with sources (IDs + URLs), provider, prompt, schedule.
|
||||
# @PRE environment_id resolves to existing environment.
|
||||
# @POST ValidationPolicy + ValidationSource rows persisted. URL sources parsed via SupersetContextExtractor for preview/validation.
|
||||
# @SIDE_EFFECT Calls Superset API to resolve dashboard slugs and validate URLs.
|
||||
# @RATIONALE URL sources are parsed at creation time for validation and preview — user immediately sees dashboard title, filters, tabs. Re-parsing on every execution (per FR-057) guarantees up-to-date permalinks and filter state.
|
||||
# @REJECTED URL parsing only at creation time (without execution-time re-parse) rejected — permalinks expire, dashboards get renamed, filters change. Creation-time parse = preview; execution-time re-parse = authoritative resolution.
|
||||
# #endregion create_task
|
||||
|
||||
# [DEF:generate_documentation:Function]
|
||||
# @PURPOSE: Generate and apply metadata descriptions for datasets.
|
||||
# @PRE: Dataset is accessible and schema is valid.
|
||||
# @POST: Dataset metadata is updated in the database.
|
||||
# [/DEF:generate_documentation:Function]
|
||||
|
||||
# [/DEF:backend/src/plugins/llm_analysis/plugin.py:Module]
|
||||
# #region parse_dashboard_url [TYPE Function]
|
||||
# @PURPOSE Resolve a Superset dashboard URL to structured context using SupersetContextExtractor.
|
||||
# @PRE url is a valid absolute Superset dashboard URL.
|
||||
# @POST Returns {dashboard_id, title, native_filters, activeTabs, anchor, partial_recovery, warnings}.
|
||||
# @RELATION CALLS -> [SupersetContextExtractor.parse_superset_link]
|
||||
# @SIDE_EFFECT Calls Superset API for permalink resolution and slug→ID lookup.
|
||||
# #endregion parse_dashboard_url
|
||||
# #endregion ValidationTaskService
|
||||
```
|
||||
|
||||
### LLM Service
|
||||
### Dashboard Validation Plugin (refactored)
|
||||
|
||||
```python
|
||||
# [DEF:backend/src/plugins/llm_analysis/service.py:Module]
|
||||
# @TIER: CRITICAL
|
||||
# @SEMANTICS: llm, api, client, retry, prompt
|
||||
# @PURPOSE: Execute reliable interactions with external LLM providers (OpenAI, etc.) with retry logic and prompt management.
|
||||
# @LAYER: Infrastructure
|
||||
# @RELATION: CALLS -> external_api
|
||||
# @RELATION: DEPENDS_ON -> backend/src/services/llm_provider.py
|
||||
# @INVARIANT: API Keys must never be logged in plain text.
|
||||
# #region DashboardValidationPlugin [C:5] [TYPE Module] [SEMANTICS validation, plugin, dual-path, llm, screenshot, text-only]
|
||||
# @BRIEF Orchestrates dashboard validation via Path A (Playwright screenshot) or Path B (text-only API).
|
||||
# @LAYER Plugin
|
||||
# @RELATION IMPLEMENTS -> [PluginBase]
|
||||
# @RELATION CALLS -> [ScreenshotService] # Path A only
|
||||
# @RELATION CALLS -> [LLMClient] # Both paths
|
||||
# @RELATION CALLS -> [SupersetClient] # Both paths
|
||||
# @RELATION CALLS -> [DatasetHealthChecker] # Path B only
|
||||
# @RELATION DEPENDS_ON -> [TaskContext]
|
||||
# @INVARIANT Execution path (A/B) is determined by `screenshot_enabled` flag — never mixed within one run.
|
||||
# @INVARIANT Path B MUST check every unique dataset referenced by dashboard charts.
|
||||
# @RATIONALE Two paths with different prompts and different data sources for LLM: screenshot path for visual issues, text path for data/KXD issues. Both paths serve different use cases.
|
||||
# @REJECTED Single «screenshot + text» prompt rejected — mixes two different analysis modalities, increases request size, complicates LLM response parsing.
|
||||
|
||||
# [DEF:analyze_multimodal:Function]
|
||||
# @PURPOSE: Send image and text context to LLM for analysis.
|
||||
# @PRE: Provider is active and quota is available.
|
||||
# @POST: Returns structured analysis or raises RetryError.
|
||||
# @SIDE_EFFECT: Makes external HTTP request.
|
||||
# [/DEF:analyze_multimodal:Function]
|
||||
# #region execute [TYPE Function] [C:4]
|
||||
# @PURPOSE Execute one validation run for a dashboard from ValidationPolicy.
|
||||
# @PRE params contains dashboard_id, provider_id, screenshot_enabled, prompt_template, logs_enabled.
|
||||
# @POST ValidationRecord persisted with status, issues, timings.
|
||||
# @SIDE_EFFECT Launches browser (Path A), calls Superset API (both paths), calls LLM API.
|
||||
# #endregion execute
|
||||
|
||||
# [/DEF:backend/src/plugins/llm_analysis/service.py:Module]
|
||||
# #region _execute_path_a [TYPE Function] [C:3]
|
||||
# @PURPOSE Path A: Playwright screenshot → multi-chunk capture → multimodal LLM.
|
||||
# @PRE dashboard_id is valid, provider is multimodal, screenshot_enabled=true.
|
||||
# @POST Returns {status, summary, issues, screenshot_paths, tab_screenshots, token_usage}.
|
||||
# @SIDE_EFFECT Launches browser, logs in, switches tabs, captures CDP screenshots.
|
||||
# @RATIONALE Multi-chunk screenshots (one per tab) replace single full-page — LLM sees each tab at high resolution instead of one compressed full-dashboard image.
|
||||
# @REJECTED Single full-page screenshot rejected — long dashboards (5000+px) compress to unreadable size (D9 in spec.md).
|
||||
# #endregion _execute_path_a
|
||||
|
||||
# #region _execute_path_b [TYPE Function] [C:3]
|
||||
# @PURPOSE Path B: API calls → dashboard structure + chart params + dataset health + logs → text-only LLM.
|
||||
# @PRE screenshot_enabled=false.
|
||||
# @POST Returns {status, summary, issues, dataset_health, chart_data_results, token_usage}.
|
||||
# @SIDE_EFFECT Calls Superset API (dashboard, chart, dataset, chart/data, log endpoints).
|
||||
# @RATIONALE Dataset health checking is mandatory — the purpose of validation is to verify the dashboard opens correctly with KXD data. Without dataset checks, silent KXD errors (connection refused, timeout) are invisible.
|
||||
# @REJECTED Logs-only checking without dataset health rejected — logs show past errors but don't guarantee current KXD connectivity state.
|
||||
# #endregion _execute_path_b
|
||||
# #endregion DashboardValidationPlugin
|
||||
```
|
||||
|
||||
### Provider Management Service
|
||||
### Screenshot Service (Path A — refactored for multi-chunk)
|
||||
|
||||
```python
|
||||
# [DEF:backend/src/services/llm_provider.py:Module]
|
||||
# @TIER: CRITICAL
|
||||
# @SEMANTICS: security, config, encryption, provider
|
||||
# @PURPOSE: Manage lifecycle and secure storage of LLM Provider configurations.
|
||||
# @LAYER: Domain
|
||||
# @RELATION: DEPENDS_ON -> backend/src/core/database.py
|
||||
# @INVARIANT: Stored API keys must be encrypted at rest.
|
||||
# #region ScreenshotService [C:4] [TYPE Module] [SEMANTICS screenshot, playwright, cdp, tab-switching, multi-chunk]
|
||||
# @BRIEF Captures per-tab dashboard screenshots via Playwright + CDP.
|
||||
# @LAYER Plugin
|
||||
# @INVARIANT Each tab produces a separate high-resolution screenshot (max 1920×1200 per chunk).
|
||||
# @INVARIANT Multi-chunk: one screenshot per tab, NOT one full-page.
|
||||
# @RATIONALE Многочанковый подход даёт LLM детальные скриншоты каждого таба вместо одного сжатого full-page.
|
||||
# @REJECTED Single full-page CDP screenshot отвергнут — D9: длинные дашборды теряют читаемость.
|
||||
|
||||
# [DEF:get_provider_client:Function]
|
||||
# @PURPOSE: Instantiate an authenticated LLM client for a specific provider.
|
||||
# @PRE: Provider ID exists and is active.
|
||||
# @POST: Returns configured client instance with decrypted key.
|
||||
# [/DEF:get_provider_client:Function]
|
||||
|
||||
# [/DEF:backend/src/services/llm_provider.py:Module]
|
||||
# #region capture_dashboard_chunks [TYPE Function] [C:3]
|
||||
# @PURPOSE Capture per-tab screenshots: login → navigate → switch tabs → per-tab CDP screenshots.
|
||||
# @PRE dashboard_id is valid, browser available.
|
||||
# @POST Returns list of {tab_name, screenshot_path}.
|
||||
# @SIDE_EFFECT Launches browser, logs in, switches tabs.
|
||||
# #endregion capture_dashboard_chunks
|
||||
# #endregion ScreenshotService
|
||||
```
|
||||
|
||||
### Data Models
|
||||
### Dataset Health Checker (Path B — new)
|
||||
|
||||
```python
|
||||
# [DEF:backend/src/plugins/llm_analysis/models.py:Module]
|
||||
# @TIER: TRIVIAL
|
||||
# @SEMANTICS: dto, pydantic, schema
|
||||
# @PURPOSE: Define Pydantic models for configuration, requests, and results.
|
||||
# @LAYER: Domain/Data
|
||||
# [/DEF:backend/src/plugins/llm_analysis/models.py:Module]
|
||||
# #region DatasetHealthChecker [C:3] [TYPE Module] [SEMANTICS dataset, health, kxd, api, validation]
|
||||
# @BRIEF Checks dataset accessibility and KXD connectivity via Superset API.
|
||||
# @LAYER Service
|
||||
# @RELATION CALLS -> [SupersetClient]
|
||||
# @INVARIANT Every unique dataset referenced by dashboard charts is checked.
|
||||
# @RATIONALE Без проверки датасетов невозможно определить, может ли дашборд корректно открыться с данными из КХД. Скриншот показывает визуал, но не показывает, что данные пришли (а не кэш).
|
||||
|
||||
# #region check_dataset_health [TYPE Function]
|
||||
# @PURPOSE Fetch dataset schema and verify accessibility.
|
||||
# @PRE dataset_id is a valid Superset dataset ID.
|
||||
# @POST Returns DatasetHealth: {accessible, database, backend, kind, error}.
|
||||
# @SIDE_EFFECT Calls GET /api/v1/dataset/{id}.
|
||||
# #endregion check_dataset_health
|
||||
|
||||
# #region check_dashboard_datasets [TYPE Function]
|
||||
# @PURPOSE For every unique dataset in dashboard charts, check health and optionally execute chart data.
|
||||
# @PRE dashboard_id is valid, chart list available.
|
||||
# @POST Returns {datasets: [DatasetHealth], chart_data: [ChartDataResult]}.
|
||||
# @SIDE_EFFECT Calls GET /api/v1/dataset/{id} × N, optionally POST /api/v1/chart/data × M.
|
||||
# #endregion check_dashboard_datasets
|
||||
# #endregion DatasetHealthChecker
|
||||
```
|
||||
|
||||
### LLM Client (updated for dual-path)
|
||||
|
||||
```python
|
||||
# #region LLMClient [C:4] [TYPE Module] [SEMANTICS llm, api, multimodal, text-only, json-mode]
|
||||
# @BRIEF LLM provider wrapper supporting both multimodal (Path A) and text-only (Path B) analysis.
|
||||
# @LAYER Infrastructure
|
||||
# @RELATION DEPENDS_ON -> [EXT:Library:openai]
|
||||
# @RELATION DEPENDS_ON -> [EXT:Library:tenacity]
|
||||
# @INVARIANT API keys never logged in plain text.
|
||||
# @RATIONALE Единый клиент для обоих путей — различия только в формате messages (multimodal vs text-only) и промпте.
|
||||
|
||||
# #region analyze_dashboard_multimodal [TYPE Function]
|
||||
# @PURPOSE Path A: send multi-chunk screenshots + logs to multimodal LLM.
|
||||
# @PRE screenshot_paths is non-empty list.
|
||||
# @POST Returns structured JSON {status, summary, issues}.
|
||||
# #endregion analyze_dashboard_multimodal
|
||||
|
||||
# #region analyze_dashboard_text [TYPE Function]
|
||||
# @PURPOSE Path B: send dashboard topology + dataset health + logs to text-only LLM.
|
||||
# @PRE dashboard_structure and dataset_health are populated.
|
||||
# @POST Returns structured JSON {status, summary, issues}.
|
||||
# @RATIONALE Текстовый промпт включает: описание топологии дашборда (табы, чарты, параметры), статус датасетов (доступность, бэкенд, ошибки), логи выполнения. LLM анализирует консистентность данных, а не визуал.
|
||||
# #endregion analyze_dashboard_text
|
||||
# #endregion LLMClient
|
||||
```
|
||||
|
||||
### Provider Bindings (updated)
|
||||
|
||||
```python
|
||||
# #region DEFAULT_LLM_PROVIDER_BINDINGS [C:1] [TYPE Constant]
|
||||
# @BRIEF Provider bindings retained: documentation, git_commit, assistant_planner.
|
||||
# @RATIONALE dashboard_validation binding удалён — провайдер выбирается per-task в форме создания задачи (D6, FR-047).
|
||||
# #endregion DEFAULT_LLM_PROVIDER_BINDINGS
|
||||
```
|
||||
|
||||
### Prompt Defaults (updated)
|
||||
|
||||
```python
|
||||
# #region DEFAULT_LLM_PROMPTS [C:2] [TYPE Constant]
|
||||
# @BRIEF Path-specific validation prompts: multimodal (Path A) and text (Path B).
|
||||
# Legacy dashboard_validation_prompt retained for v1 compat, MUST NOT be used as default for new tasks.
|
||||
# @RATIONALE FR-048: null prompt_template → auto-select path-specific default.
|
||||
# Single prompt field per task avoids the risk of Path A using text prompt or vice versa.
|
||||
# #endregion DEFAULT_LLM_PROMPTS
|
||||
```
|
||||
|
||||
## Frontend Components
|
||||
|
||||
### Configuration UI
|
||||
### Validation Task List Page (new)
|
||||
|
||||
```html
|
||||
<!-- [DEF:frontend/src/components/llm/ProviderConfig.svelte:Component] -->
|
||||
<!-- @TIER: STANDARD -->
|
||||
<!-- @PURPOSE: Form for adding/editing LLM providers with connection testing. -->
|
||||
<!-- @RELATION: CALLS -> /api/settings/llm/providers -->
|
||||
<!-- [/DEF:frontend/src/components/llm/ProviderConfig.svelte:Component] -->
|
||||
<!-- #region ValidationTasksPage [C:3] [TYPE Page] [SEMANTICS sveltekit, validation, task-list, schedule]
|
||||
<!-- @BRIEF Page: /validation-tasks — list of all validation tasks with status, schedule, last run.
|
||||
<!-- @LAYER Page -->
|
||||
<!-- @RELATION CALLS -> /api/validation-tasks -->
|
||||
<!-- @UX_STATE Loading -> Skeleton table rows -->
|
||||
<!-- @UX_STATE Loaded -> Task cards/rows with status badges, schedule, dashboard count -->
|
||||
<!-- @UX_STATE Empty -> "No validation tasks yet" + "Create First Task" CTA -->
|
||||
```
|
||||
|
||||
### Validation Report
|
||||
### Validation Task Create/Edit Form (new)
|
||||
|
||||
```html
|
||||
<!-- [DEF:frontend/src/components/llm/ValidationReport.svelte:Component] -->
|
||||
<!-- @TIER: STANDARD -->
|
||||
<!-- @PURPOSE: Display formatted analysis results including screenshot and issues list. -->
|
||||
<!-- @INVARIANT: Must handle missing screenshots gracefully. -->
|
||||
<!-- [/DEF:frontend/src/components/llm/ValidationReport.svelte:Component] -->
|
||||
<!-- #region ValidationTaskForm [C:4] [TYPE Component] [SEMANTICS svelte, form, validation, task-create, url-parser]
|
||||
<!-- @BRIEF Multi-step form: name → sources (IDs + URLs) → provider + prompt → schedule.
|
||||
<!-- @LAYER Component -->
|
||||
<!-- @RELATION CALLS -> /api/llm/providers (list multimodal providers) -->
|
||||
<!-- @RELATION CALLS -> /api/validation-tasks/parse-url (URL validation) -->
|
||||
<!-- @UX_STATE Idle -> Empty form -->
|
||||
<!-- @UX_STATE Parsing URL -> Spinner on URL input; result banner (dashboard title, filters count) -->
|
||||
<!-- @UX_STATE Submitting -> All fields disabled, progress bar -->
|
||||
<!-- @UX_STATE Error -> Field-level errors, URL parse failure banner -->
|
||||
<!-- @UX_FEEDBACK URL parse success: green banner with dashboard title, native_filters count, activeTabs -->
|
||||
<!-- @UX_FEEDBACK Provider selector filters multimodal-only when screenshot_enabled=true -->
|
||||
```
|
||||
|
||||
### Validation Run (new — groups per-dashboard results)
|
||||
|
||||
```python
|
||||
# #region ValidationRun [C:3] [TYPE Model] [SEMANTICS run, grouping, aggregate, execution]
|
||||
# @BRIEF Groups ValidationRecords from one task execution (manual or scheduled).
|
||||
# One run = N dashboards validated. Aggregate counts computed on finalize.
|
||||
# @INVARIANT Every ValidationRecord belongs to exactly one ValidationRun.
|
||||
# @RATIONALE Without run grouping, a 15-dashboard task produces 15 disconnected records — no way to see aggregate status or link them as one execution unit.
|
||||
# Run groups them for reporting: "3 PASS, 1 WARN, 0 FAIL across 4 dashboards".
|
||||
# #endregion ValidationRun
|
||||
```
|
||||
|
||||
### Validation Run Detail Page (updated for multi-dashboard)
|
||||
|
||||
```html
|
||||
<!-- #region ValidationRunDetailPage [C:3] [TYPE Page] [SEMANTICS sveltekit, run, report, multi-dashboard, aggregate]
|
||||
<!-- @BRIEF Page: /validation-tasks/{policyId}/runs/{runId} — multi-dashboard run report.
|
||||
<!-- Run header with aggregate counts → collapsible dashboard list → per-dashboard detail (issues, screenshots/dataset-health, logs).
|
||||
<!-- @LAYER Page -->
|
||||
<!-- @RELATION CALLS -> GET /api/validation-tasks/{policy_id}/runs/{run_id} -->
|
||||
<!-- @UX_STATE Loading -> Skeleton placeholders -->
|
||||
<!-- @UX_STATE Loaded -> Run header + dashboard list with expand/collapse -->
|
||||
<!-- @UX_STATE Expanded -> Per-dashboard: Path A screenshots (tab selector) or Path B dataset health table -->
|
||||
<!-- @UX_STATE Mixed -> Single run may contain both Path A and Path B dashboards — sections render per dashboard -->
|
||||
<!-- @RATIONALE 15-dashboard run: collapse keeps page scannable. Expand for drill-down. Aggregate counts at top give instant health overview.
|
||||
```
|
||||
|
||||
### Legacy Report Redirect
|
||||
|
||||
```python
|
||||
# #region LegacyReportRedirect [C:1] [TYPE Route]
|
||||
# @BRIEF GET /reports/llm/{taskId} → 302 redirect to canonical /validation-tasks/{policy_id}/runs/{record_id}.
|
||||
# @RATIONALE Backward compat for v1 notification links, external bookmarks, and assistant-generated URLs.
|
||||
# Canonical URL is per FR-055b. Returns 404 if taskId has no matching ValidationRecord.
|
||||
# #endregion LegacyReportRedirect
|
||||
```
|
||||
|
||||
### Dashboard Validation Indicator (new)
|
||||
|
||||
```html
|
||||
<!-- #region DashboardValidationIndicator [C:2] [TYPE Component] [SEMANTICS svelte, dashboard, validation, indicator, batch]
|
||||
<!-- @BRIEF Colored dot indicator (PASS/WARN/FAIL/UNKNOWN) next to dashboard title in hub rows.
|
||||
<!-- Click → dropdown: last 3 runs (date, status, task name) + "View all" link.
|
||||
<!-- @LAYER Component -->
|
||||
<!-- @RELATION CALLS -> GET /api/dashboards/{env_id}/validation-status?dashboard_ids=... -->
|
||||
<!-- @RATIONALE Replaces removed LLM column without adding visual noise.
|
||||
<!-- Batch endpoint avoids N+1 requests for dashboard hub with 50+ rows.
|
||||
```
|
||||
|
||||
### ValidationTaskReport (new component)
|
||||
|
||||
```html
|
||||
<!-- #region ValidationTaskReport [C:3] [TYPE Component] [SEMANTICS svelte, report, validation, issues, dataset-health]
|
||||
<!-- @BRIEF Shared report component: check result badge, summary, issues table, execution path indicator, dataset health (Path B), screenshots (Path A).
|
||||
<!-- @LAYER Component -->
|
||||
```
|
||||
|
||||
### Dashboard Hub (simplified)
|
||||
|
||||
```html
|
||||
<!-- #region DashboardHub [C:4] [TYPE Page] [SEMANTICS sveltekit, dashboard, hub]
|
||||
<!-- @BRIEF LLM validation column REMOVED. "Validate" action REMOVED from action dropdown.
|
||||
<!-- Added "Create Validation Task" button in toolbar that pre-fills selected dashboard IDs.
|
||||
<!-- @RATIONALE D1, FR-051, FR-052: validation moves to task-based flow; ad-hoc "Validate" is deprecated.
|
||||
```
|
||||
|
||||
### LLM Settings (simplified)
|
||||
|
||||
```html
|
||||
<!-- #region LLMSettingsPage [C:3] [TYPE Page] [SEMANTICS sveltekit, settings, llm, provider]
|
||||
<!-- @BRIEF Provider bindings for dashboard_validation REMOVED.
|
||||
<!-- Retained: documentation, git_commit, assistant_planner provider bindings.
|
||||
<!-- @RATIONALE D8, FR-049: dashboard validation provider is per-task, not global.
|
||||
```
|
||||
|
||||
## ADR Continuity
|
||||
|
||||
| ADR | Impact |
|
||||
|-----|--------|
|
||||
| ADR-0001 (module layout) | v2 changes stay within `backend/src/plugins/llm_analysis/` and `frontend/src/routes/validation-tasks/` |
|
||||
| ADR-0004 (plugin architecture) | `DashboardValidationPlugin` remains a PluginBase implementation; new `DatasetHealthChecker` is a service, not a plugin |
|
||||
| ADR-0009 (SSL) | Unchanged — LLM client SSL configuration applies to both Path A and Path B |
|
||||
| ADR-0008 (assistant tool registry) | `_tool_llm_validation.py` — assistant tool should be updated to create task instead of triggering ad-hoc validation |
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
# Data Model: LLM Analysis Plugin
|
||||
# Data Model: LLM Analysis Plugin v2
|
||||
|
||||
**Feature**: `017-llm-analysis-plugin`
|
||||
**Updated**: 2026-05-31 — v2: ValidationPolicy expansion, ValidationSource, dual-path execution models
|
||||
|
||||
## Entities
|
||||
|
||||
@@ -10,43 +11,158 @@ Stores configuration for different LLM providers.
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| id | UUID | Yes | Unique identifier |
|
||||
| provider_type | Enum | Yes | `openai`, `openrouter`, `kilo` |
|
||||
| provider_type | Enum | Yes | `openai`, `openrouter`, `kilo`, `litellm` |
|
||||
| name | String | Yes | Display name (e.g., "Production GPT-4") |
|
||||
| base_url | String | Yes | API Endpoint URL |
|
||||
| api_key | String | Yes | Encrypted API Key |
|
||||
| api_key | String | Yes | Encrypted API Key (AES-256) |
|
||||
| default_model | String | Yes | Model identifier (e.g., `gpt-4o`) |
|
||||
| is_active | Boolean | Yes | Whether this provider is currently enabled |
|
||||
| is_multimodal | Boolean | Yes | Whether model supports image input (required for Path A) |
|
||||
|
||||
### ValidationResult
|
||||
Stores the outcome of a dashboard validation task.
|
||||
### ValidationPolicy (v2 — expanded)
|
||||
|
||||
Stores configuration for a scheduled validation task. Uses ORM relationship to `ValidationSource` table — no JSON column denormalization.
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| id | UUID | Yes | Unique identifier |
|
||||
| dashboard_id | String | Yes | Reference to the dashboard |
|
||||
| timestamp | DateTime | Yes | When the validation ran |
|
||||
| status | Enum | Yes | `PASS`, `WARN`, `FAIL` |
|
||||
| screenshot_path | String | No | Path to the captured screenshot (if stored) |
|
||||
| screenshot_metadata | JSON | No | `{width: 1920, height: dynamic, tabs_processed: []}` |
|
||||
| issues | JSON | Yes | List of detected issues `[{severity, message, location}]` |
|
||||
| raw_response | Text | No | Full LLM response for debugging |
|
||||
| name | String | Yes | Human-readable task name |
|
||||
| description | String | No | Optional description |
|
||||
| environment_id | String | Yes | Target Superset environment |
|
||||
| sources | relationship → list[ValidationSource] | Yes | ORM relationship only. Dashboards to validate (1..N). Migrated from v1 `dashboard_ids` list via `ValidationSource` rows |
|
||||
| source_snapshot | JSON | No | Read-only denormalized copy of sources at last edit time — for audit/diff only, never used as authoritative state |
|
||||
| provider_id | String | Yes | LLM provider to use (per-task, not global) |
|
||||
| prompt_template | String | No | Custom prompt; `null` = use path-specific default: `dashboard_validation_prompt_multimodal` for Path A, `dashboard_validation_prompt_text` for Path B. Must NOT fall back to v1 legacy `dashboard_validation_prompt` |
|
||||
| screenshot_enabled | Boolean | Yes | `true` = Path A (Playwright + multimodal LLM), `false` = Path B (text-only) |
|
||||
| logs_enabled | Boolean | Yes | Fetch Superset execution logs for LLM context |
|
||||
| execute_chart_data | Boolean | Yes | Path B only: execute `POST /api/v1/chart/data` for each chart (default: false) |
|
||||
| llm_batch_size | Integer | No | Path B only: dashboards per LLM call (default: 1 — full isolation, one call per dashboard). Values >1 are experimental. Path A always uses 1 (image token constraints). |
|
||||
| policy_dashboard_concurrency_limit | Integer | No | Max dashboards validated in parallel within one policy run (default: 3). Global worker limit controlled by `global_validation_worker_limit` config |
|
||||
| cron_expression | String | No | Cron schedule; null = manual trigger only |
|
||||
| is_active | Boolean | Yes | Whether schedule is currently active |
|
||||
| notification_channels | list[String] | No | Email/Pulse channel IDs for failure notifications |
|
||||
| created_at | DateTime | Yes | Creation timestamp |
|
||||
| updated_at | DateTime | Yes | Last modification timestamp |
|
||||
|
||||
### TaskConfiguration (Extension)
|
||||
Extends existing Task model to support validation-specific params.
|
||||
### ValidationSource (new)
|
||||
|
||||
Represents a single dashboard to validate within a task — either by numeric ID or by full Superset URL.
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| task_type | String | Yes | `dashboard_validation` |
|
||||
| parameters | JSON | Yes | `{dashboard_id, provider_id, screenshot_strategy}` |
|
||||
| id | UUID | Yes | Unique identifier |
|
||||
| policy_id | UUID | Yes | FK → ValidationPolicy |
|
||||
| type | Enum | Yes | `dashboard_id` or `dashboard_url` |
|
||||
| value | String | Yes | Numeric dashboard ID or full Superset URL |
|
||||
| parsed_context | JSON | No | For `dashboard_url`: result of `SupersetContextExtractor.parse_superset_link()` — contains dashboard_id, native_filters, activeTabs, anchor |
|
||||
| status | Enum | Yes | `valid`, `partial_recovery` (SupersetContextExtractor returned partial match), `stale` (permalink expired, URL parse warning), `invalid` (dashboard deleted or not found), `permission_denied`, `parse_failed` |
|
||||
| resolved_dashboard_id | String | No | Resolved numeric dashboard ID (cached after URL parse for display) |
|
||||
| title | String | No | Resolved dashboard title (cached for display) |
|
||||
| last_error | String | No | Last resolution error message |
|
||||
| last_checked_at | DateTime | No | When the source was last validated/resolved |
|
||||
|
||||
### Migration from v1 dashboard_ids:
|
||||
|
||||
```python
|
||||
# Existing dashboard_ids: list[str] → ValidationSource rows (relational only, no JSON denormalization)
|
||||
for dashboard_id in old_policy.dashboard_ids:
|
||||
db.add(ValidationSource(
|
||||
policy_id=policy.id,
|
||||
type="dashboard_id",
|
||||
value=dashboard_id,
|
||||
is_valid=True,
|
||||
))
|
||||
# Optional: populate source_snapshot JSON for audit trail
|
||||
policy.source_snapshot = {"dashboard_ids": list(old_policy.dashboard_ids)}
|
||||
```
|
||||
|
||||
### ValidationRun (new — groups records from one execution)
|
||||
|
||||
A single task execution validates ALL dashboards in a policy. `ValidationRun` groups these per-dashboard `ValidationRecord`s into one logical unit.
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| id | UUID | Yes | Unique run identifier |
|
||||
| policy_id | UUID | Yes | FK → ValidationPolicy |
|
||||
| task_id | String | No | Associated Task Manager task ID (one run = one task) |
|
||||
| started_at | DateTime | Yes | When the run started |
|
||||
| finished_at | DateTime | No | When all dashboards finished (null while running) |
|
||||
| trigger | Enum | Yes | `manual` or `scheduled` |
|
||||
| status | Enum | Yes | `completed` (all dashboards done), `partial` (some sources skipped/invalid), `failed` (execution error before any dashboard) |
|
||||
| dashboard_count | Integer | Yes | Total dashboards attempted |
|
||||
| pass_count | Integer | Yes | PASS results |
|
||||
| warn_count | Integer | Yes | WARN results |
|
||||
| fail_count | Integer | Yes | FAIL results |
|
||||
| unknown_count | Integer | Yes | UNKNOWN results (LLM errors, timeouts) |
|
||||
|
||||
### ValidationRecord (v2 — expanded, linked to run)
|
||||
|
||||
Stores the outcome of one dashboard validation within a task run. Linked to `ValidationRun` via `run_id`.
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| id | UUID | Yes | Unique identifier |
|
||||
| run_id | UUID | Yes | FK → ValidationRun — groups per-dashboard results from one task execution |
|
||||
| task_id | String | No | Deprecated in favor of run_id; kept for v1 backward compat |
|
||||
| policy_id | UUID | No | FK → ValidationPolicy |
|
||||
| source_id | UUID | No | FK → ValidationSource |
|
||||
| dashboard_id | String | Yes | Resolved dashboard ID |
|
||||
| status | Enum | Yes | `PASS`, `WARN`, `FAIL`, `UNKNOWN` |
|
||||
| execution_path | Enum | Yes | `screenshot` (Path A) or `text_only` (Path B) |
|
||||
| summary | String | Yes | LLM-generated summary |
|
||||
| issues | JSON | Yes | `[{severity, message, location}]` |
|
||||
| dataset_health | JSON | No | Path B only: per-dataset status `[{dataset_id, name, database, backend, healthy, error, affected_charts}]` |
|
||||
| chart_data_results | JSON | No | Path B with `execute_chart_data=true`: `[{chart_id, executed, duration_ms, row_count, error}]` |
|
||||
| screenshot_paths | list[String] | No | Path A: paths to archived WebP screenshots (one per tab). PNG/JPEG intermediates deleted after conversion |
|
||||
| tab_screenshots | JSON | No | Path A: per-tab mapping `[{tab_name, webp_path, errored}]` |
|
||||
| logs_sent_to_llm | list[String] | No | Redacted Superset execution logs sent to LLM |
|
||||
| token_usage | JSON | No | LLM token usage `{prompt_tokens, completion_tokens, total_tokens}` (when provider reports it) |
|
||||
| raw_response | Text | No | Full LLM response for debugging. Retention: 30 days per FR-029a |
|
||||
| timings | JSON | No | Phase timings: `{dashboard_fetch_ms, dataset_fetch_ms, chart_fetch_ms, chart_data_exec_ms, screenshot_ms, llm_call_ms, total_ms}` |
|
||||
| created_at | DateTime | Yes | Record creation timestamp |
|
||||
|
||||
### Path B — Dataset Health Check (runtime model, not persisted separately)
|
||||
|
||||
Built during Path B execution from API responses:
|
||||
|
||||
```python
|
||||
class DatasetHealth:
|
||||
dataset_id: int
|
||||
dataset_name: str # e.g. "finance.kpi_revenue"
|
||||
database_name: str # e.g. "PostgreSQL / analytics_prod"
|
||||
backend: str # e.g. "postgresql", "trino"
|
||||
kind: str # "virtual" or "physical"
|
||||
is_accessible: bool # HTTP 200 from GET /api/v1/dataset/{id}
|
||||
error: str | None # Connection / timeout error
|
||||
affected_chart_ids: list[int] # Charts using this dataset
|
||||
```
|
||||
|
||||
### Chart Execution Result (runtime model, Path B + execute_chart_data)
|
||||
|
||||
```python
|
||||
class ChartDataResult:
|
||||
chart_id: int
|
||||
chart_name: str
|
||||
viz_type: str
|
||||
executed: bool # False if skipped
|
||||
duration_ms: int | None
|
||||
row_count: int | None
|
||||
error: str | None # Timeout / query error
|
||||
```
|
||||
|
||||
## API Contracts
|
||||
|
||||
See `contracts/` directory for OpenAPI specifications.
|
||||
See `contracts/` directory.
|
||||
|
||||
### Key Interactions
|
||||
### Key Interactions — v2
|
||||
|
||||
1. **Configure Provider**: `POST /api/settings/llm/providers`
|
||||
2. **Trigger Validation**: `POST /api/tasks/dispatch` (payload: `{type: "dashboard_validation", ...}`)
|
||||
3. **Get Results**: `GET /api/tasks/{task_id}/result` (or via WebSocket stream)
|
||||
4. **Generate Documentation**: `POST /api/tasks/dispatch` (payload: `{type: "dataset_documentation", ...}`)
|
||||
5. **Generate Commit**: `POST /api/git/generate-message`
|
||||
1. **Configure Provider**: `POST /api/llm/providers` (unchanged)
|
||||
2. **Create Validation Task**: `POST /api/validation-tasks` — creates `ValidationPolicy` + `ValidationSource`s
|
||||
3. **List Tasks**: `GET /api/validation-tasks` — returns all policies with aggregated status
|
||||
4. **Get Task Detail**: `GET /api/validation-tasks/{policy_id}` — returns policy + per-source status
|
||||
5. **Trigger Run**: `POST /api/validation-tasks/{policy_id}/run` — manual trigger (also fires on cron)
|
||||
6. **Get Run History**: `GET /api/validation-tasks/{policy_id}/runs` — paginated `ValidationRecord` list
|
||||
7. **Get Run Detail**: `GET /api/validation-tasks/{policy_id}/runs/{record_id}` — full report
|
||||
8. **Parse URL**: `POST /api/validation-tasks/parse-url` — returns `{dashboard_id, title, filters, tabs}` from URL
|
||||
9. **Generate Documentation**: `POST /api/tasks` with `plugin_id=llm_documentation` (unchanged)
|
||||
10. **Generate Commit Message**: `POST /api/git/generate-message` (unchanged)
|
||||
|
||||
@@ -1,118 +1,183 @@
|
||||
# Implementation Plan: LLM Analysis & Documentation Plugins
|
||||
# Implementation Plan: LLM Analysis & Documentation Plugins v2
|
||||
|
||||
**Branch**: `017-llm-analysis-plugin` | **Date**: 2026-01-28 | **Spec**: [specs/017-llm-analysis-plugin/spec.md](spec.md)
|
||||
**Input**: Feature specification from `/specs/017-llm-analysis-plugin/spec.md`
|
||||
|
||||
**Note**: This template is filled in by the `/speckit.plan` command. See `.specify/templates/commands/plan.md` for the execution workflow.
|
||||
**Branch**: `017-llm-analysis-plugin` | **Date**: 2026-01-28 | **Updated**: 2026-05-31 — v2
|
||||
**Spec**: [spec.md](spec.md)
|
||||
|
||||
## Summary
|
||||
|
||||
This feature implements two new plugins for the `ss-tools` platform: `DashboardValidationPlugin` and `DocumentationPlugin`. The `DashboardValidationPlugin` uses multimodal LLMs to analyze dashboard health by combining screenshots (via headless browser or API) and execution logs. The `DocumentationPlugin` automates the generation of metadata descriptions for datasets and columns using LLMs. Additionally, the feature integrates LLM capabilities into the Git plugin for commit message generation and adds a configuration interface for managing multiple LLM providers (OpenAI, OpenRouter, Kilo).
|
||||
v2 redesign replaces the ad-hoc «Validate» button with a task-based validation flow. Users create `ValidationPolicy` tasks with dashboard sources (IDs + URLs), select an LLM provider and prompt per-task, choose between two execution paths (Playwright screenshot or text-only API with dataset health checking), and set a schedule. The dashboard hub is simplified — LLM validation status moves to the Validation Tasks page.
|
||||
|
||||
Key architectural changes:
|
||||
- **Task-based flow**: New `/validation-tasks` page + CRUD API + `ValidationSource` model for dashboard IDs/URLs
|
||||
- **Dual-path execution**: `DashboardValidationPlugin.execute()` dispatches to `_execute_path_a()` (screenshot) or `_execute_path_b()` (text-only) based on `screenshot_enabled`
|
||||
- **Multi-chunk screenshots**: `ScreenshotService.capture_dashboard_chunks()` captures per-tab images instead of one full-page
|
||||
- **Dataset health checker** (new): `DatasetHealthChecker` verifies KXD connectivity for every unique dataset
|
||||
- **URL parsing** (reused): `SupersetContextExtractor.parse_superset_link()` for URL-based dashboard sources
|
||||
- **Provider/prompt per-task**: Removed from global LLM settings; configured in task creation form
|
||||
|
||||
## Technical Context
|
||||
|
||||
**Language/Version**: Python 3.9+ (Backend), Node.js 18+ (Frontend)
|
||||
**Primary Dependencies**:
|
||||
- Backend: `openai` (or compatible SDK), `playwright` (for headless screenshots), `fastapi` (existing), `pydantic` (existing), `sqlalchemy` (existing)
|
||||
- Frontend: `svelte` (existing), `tailwindcss` (existing)
|
||||
**Storage**: SQLite (existing `tasks.db` for results, `auth.db` for permissions, `mappings.db` or new `plugins.db` for provider config/metadata)
|
||||
**Language/Version**: Python 3.13+ (Backend, per ADR-0001), Node.js 18+ (Frontend)
|
||||
**Primary Dependencies** (real versions from repository):
|
||||
- Backend: `fastapi==0.126.0`, `pydantic==2.12.5`, `httpx==0.28.1`, `openai` (latest), `playwright` (latest), `tenacity` (latest)
|
||||
- Frontend: `svelte==^5.43.8`, `@sveltejs/kit==^2.49.2`, `tailwindcss==^3.0.0`, `vite==^7.2.4`
|
||||
**Storage**: SQLite (`tasks.db`, `auth.db`, `plugins.db`) via SQLAlchemy
|
||||
**Testing**: `pytest` (Backend), `vitest` (Frontend)
|
||||
**Target Platform**: Linux server (existing environment)
|
||||
**Project Type**: Web application (FastAPI + SvelteKit)
|
||||
**Performance Goals**: Dashboard validation < 90s, Documentation generation < 30s per asset
|
||||
**Constraints**: Must integrate with existing `PluginBase` and `TaskManager`. Secure storage for API keys.
|
||||
**Scale/Scope**: Support for configurable LLM providers, scheduled tasks, and notification integration.
|
||||
**Deployment**: Docker Compose (backend + frontend + nginx per `docker/`)
|
||||
**Performance Goals**:
|
||||
- Path B (text-only, no chart data): < 30s per dashboard (excluding LLM latency)
|
||||
- Path A (screenshot): < 120s per dashboard
|
||||
- Task creation (form fill + URL parse): < 60s for user
|
||||
**Constraints**: Must integrate with existing `PluginBase` and `TaskManager`. Secure storage for API keys (AES-256). Modules < 400 lines per ADR-0001.
|
||||
|
||||
## Implementation Notes
|
||||
## v2 Implementation Notes
|
||||
|
||||
### Screenshot Capture Implementation
|
||||
The screenshot service uses Playwright with Chrome DevTools Protocol (CDP) to avoid font loading timeouts in headless mode. Key implementation details:
|
||||
- **Full Page Capture**: Uses CDP `Page.captureScreenshot` with `captureBeyondViewport: true` and `fromSurface: true`
|
||||
- **Tab Switching**: Implements recursive tab switching to trigger lazy-loaded chart rendering on multi-tab dashboards
|
||||
- **Authentication**: Uses direct UI login flow (navigating to `/login/` and filling credentials) instead of API cookie injection for better reliability
|
||||
- **Resolution**: 1920px width with dynamic full page height calculation
|
||||
### Path B — Dashboard Topology Builder
|
||||
|
||||
### Authentication Flow
|
||||
The service authenticates via Playwright UI login rather than API authentication:
|
||||
1. Navigate to `/login/` page
|
||||
2. Fill username/password fields (supports multiple Superset versions)
|
||||
3. Click login button
|
||||
4. Verify successful redirect to `/superset/welcome/`
|
||||
5. Navigate to dashboard with valid session
|
||||
The text-only path builds a structured description of the dashboard from API responses:
|
||||
|
||||
This approach is more reliable than API-to-UI cookie injection which was causing 403 Forbidden errors.
|
||||
```
|
||||
1. GET /api/v1/dashboard/{id}
|
||||
→ dashboard_title, position_json (layout tree), slices[], json_metadata
|
||||
|
||||
2. Parse position_json recursively:
|
||||
- TABS → children[] → ROW → children[] → CHART
|
||||
- Build tab → chart hierarchy with slice_id references
|
||||
|
||||
3. For each unique chart: GET /api/v1/chart/{slice_id}
|
||||
→ slice_name, viz_type, params (metrics, groupby, filters, time_range), datasource_id
|
||||
|
||||
4. For each unique dataset: GET /api/v1/dataset/{datasource_id}
|
||||
→ database (name, backend), sql (virtual datasets), kind, is_managed_externally
|
||||
|
||||
5. Optionally: POST /api/v1/chart/data with form_data from chart params
|
||||
→ Verify query execution, row count, duration
|
||||
|
||||
6. GET /api/v1/log/ (filter: dashboard_id, 24h)
|
||||
→ Execution logs
|
||||
|
||||
7. Build text prompt:
|
||||
Dashboard: "{title}" (id: {id})
|
||||
─── Tab: "{tab_name}" ({N} charts) ───
|
||||
Chart "{chart_name}" ({viz_type})
|
||||
Dataset: {dataset_name} ({kind}, backend: {backend})
|
||||
Params: {metrics}, grouped by {groupby}, filters: {filters}
|
||||
Health: ✓ accessible / ✗ {error}
|
||||
...
|
||||
Logs (last 24h, {N} records):
|
||||
[{timestamp}] {action}: {details}
|
||||
```
|
||||
|
||||
### Path A — Multi-Chunk Screenshot
|
||||
|
||||
The screenshot service is refactored to capture per-tab screenshots:
|
||||
|
||||
1. Login (unchanged)
|
||||
2. Navigate to dashboard URL (with parsed filters/tabs from URL source)
|
||||
3. Iterate tabs recursively (unchanged logic, but now save a screenshot per tab)
|
||||
4. For each tab: resize viewport to 1920×1200, wait for charts, CDP screenshot
|
||||
5. Return list of `{tab_name, screenshot_path}` to LLMClient
|
||||
6. LLMClient sends all chunks as separate `image_url` blocks in a single `content[]` array
|
||||
|
||||
### URL Parsing (from dataset review)
|
||||
|
||||
The `SupersetContextExtractor` is already implemented and tested for dataset review. For validation tasks, we reuse its `parse_superset_link()` method:
|
||||
|
||||
1. User pastes URL in task creation form
|
||||
2. Frontend calls `POST /api/validation-tasks/parse-url` with `{url, environment_id}`
|
||||
3. Backend calls `extractor.parse_superset_link(url)` → `SupersetParsedContext`
|
||||
4. Returns `{dashboard_id, title, native_filters, activeTabs, anchor, partial_recovery, warnings}`
|
||||
5. Frontend shows preview: dashboard title + filter names + tab names
|
||||
6. On task save: `ValidationSource.value = url`, `ValidationSource.parsed_context = {...}`
|
||||
|
||||
### Prompt Templates for Dual Paths
|
||||
|
||||
Two path-specific defaults in `DEFAULT_LLM_PROMPTS`:
|
||||
|
||||
```python
|
||||
"dashboard_validation_prompt_multimodal": """...""" # Path A: image-focused
|
||||
"dashboard_validation_prompt_text": """...""" # Path B: topology-focused
|
||||
```
|
||||
|
||||
The v1 legacy key `dashboard_validation_prompt` is retained for backward reference but **MUST NOT** be used as default for new tasks. Each task has a single `prompt_template` field:
|
||||
- `null` → system auto-selects path-specific default based on `screenshot_enabled`
|
||||
- `non-null` → custom prompt used regardless of path (with FR-048a warning on path toggle)
|
||||
|
||||
**Contract**: `DashboardValidationPlugin.execute()` reads `prompt_template` from params. If null, resolves: `screenshot_enabled ? multimodal_prompt : text_prompt`.
|
||||
|
||||
## Constitution Check
|
||||
|
||||
*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
|
||||
|
||||
- [x] **I. Semantic Protocol Compliance**: Will follow `[DEF]`, `@RELATION`, and file structure standards.
|
||||
- [x] **II. Causal Validity**: Contracts/Interfaces will be defined before implementation.
|
||||
- [x] **III. Immutability of Architecture**: No changes to core immutable layers planned; extending via Plugin architecture.
|
||||
- [x] **IV. Design by Contract**: All new plugins and services will use DbC.
|
||||
- [x] **V. Belief State Logging**: Will use standard logging for belief state.
|
||||
- [x] **VI. Fractal Complexity Limit**: Plugins will be modular and adhere to complexity limits.
|
||||
- [x] **VII. Everything is a Plugin**: Implemented as `DashboardValidationPlugin` and `DocumentationPlugin`.
|
||||
- [x] **VIII. Unified Frontend Experience**: Will use existing Svelte components and i18n.
|
||||
- [x] **IX. Security & Access Control**: Will define permissions (e.g., `plugin:llm:validate`, `plugin:llm:configure`).
|
||||
Source: `.specify/memory/constitution.md` v1.0.0
|
||||
|
||||
## Project Structure
|
||||
- [x] **I. Semantic Contract First** — All v2 contracts use `#region`/`#endregion` anchors with `[C:N]` complexity tiers. `@RELATION` edges trace dependencies. `@RATIONALE`/`@REJECTED` document architectural choices. No naked code. → Verified: 6 contract modules defined in `contracts/modules.md` with proper anchoring.
|
||||
- [x] **II. Decision Memory** — D1–D9 design decisions in `spec.md` with rationale. D8–D13 research decisions in `research.md` with Decision/Rationale/Alternatives/Impact format. No `@REJECTED` path silently reintroduced. → Verified: 9 spec decisions + 6 research decisions all documented.
|
||||
- [x] **III. External Orchestrator** — No change to orchestrator pattern (ADR-0003). `DashboardValidationPlugin` remains an external plugin, not embedded in Superset. `DatasetHealthChecker` is a service within ss-tools, not a Superset modification. → PASS
|
||||
- [x] **IV. Module Discipline** — New files: `validation_tasks.py` (API routes, ~150 LOC), `DatasetHealthChecker` class (~120 LOC), `ValidationTaskForm.svelte` (~300 LOC), `UrlParser.svelte` (~150 LOC). All under 400-line limit. Function CC ≤ 10. No cyclic imports (routes → services → models direction enforced per ADR-0001). → PASS
|
||||
- [x] **V. RBAC Enforcement** — Validation task CRUD requires existing RBAC roles (`plugin:llm:validate`, `plugin:llm:configure` per ADR-0005). Assistant tool `_tool_llm_validation.py` inherits permission checks from `@assistant_tool` decorator (ADR-0008). → PASS
|
||||
- [x] **VI. Frontend — Svelte 5 Runes Only** — New components: `ValidationTaskForm.svelte`, `UrlParser.svelte`, `DatasetHealthTable.svelte`, `ValidationTaskReport.svelte`. All use `$state`, `$derived`, `$props`, `$effect` (no Svelte 4 syntax). No `fromStore` + `$derived` anti-pattern (ADR-0007). → PASS
|
||||
- [x] **VII. Test-Driven for C3+ Contracts** — `ValidationTaskService` [C:4] → `test_validation_task_service.py`. `DatasetHealthChecker` [C:3] → `test_dataset_health_checker.py`. `DashboardValidationPlugin._execute_path_b` [C:3] → `test_path_b_execution.py`. Each includes `@REJECTED` path regression test. → Planned in T108-T156.
|
||||
|
||||
### Documentation (this feature)
|
||||
**GATE RESULT: CONDITIONAL PASS (AMBER)** — No blocking constitutional conflicts. 7 of 7 principles pass. Spec completeness is AMBER per `checklists/requirements.md`: 12 UX/data/reliability checklist items remain as implementation-readiness markers (defined in FRs, to be validated during Phase 3+ implementation). Gate upgrades to GREEN after Phase 3 Task CRUD implementation validates form flows, report layout, and migration against checklist.
|
||||
|
||||
## ADR Continuity
|
||||
|
||||
| ADR | Impact of v2 | Action Required |
|
||||
|-----|-------------|-----------------|
|
||||
| **ADR-0001** (module layout) | New files: `backend/src/api/routes/validation_tasks.py`, `frontend/src/routes/validation-tasks/`. All fit within canonical boundaries (API routes → `api/routes/`, pages → `routes/`, components → `lib/components/`). | None — compliant |
|
||||
| **ADR-0002** (semantic protocol) | v2 contracts use GRACE-Poly v2.6 `#region` anchors with `[C:N]`. `@RELATION` edges trace `DEPENDS_ON` / `CALLS` / `IMPLEMENTS`. | None — compliant |
|
||||
| **ADR-0003** (orchestrator pattern) | No changes. ss-tools remains external orchestrator over Superset. | None |
|
||||
| **ADR-0004** (plugin architecture) | `DashboardValidationPlugin` stays a `PluginBase` implementation. `DatasetHealthChecker` is a service (not a plugin) — correct per ADR-0004 §Isolation Guarantees: plugins get no DB access; services do. | None — compliant |
|
||||
| **ADR-0005** (RBAC) | Existing `plugin:llm:validate` permission covers new task-triggered validation. Task CRUD requires `plugin:llm:configure`. | None — reused |
|
||||
| **ADR-0006** (frontend architecture) | New pages use SvelteKit file-based routing (`+page.svelte`). Components in `lib/components/`. API clients via `$lib/api.js`. | None — compliant |
|
||||
| **ADR-0007** (fromStore+derived) | No new uses of the rejected `fromStore` + `$derived` pattern. | None |
|
||||
| **ADR-0008** (assistant tool registry) | `_tool_llm_validation.py` must be updated: replace ad-hoc `run_llm_validation` dispatch with task creation via `ValidationTaskService`. | **T153**: Update assistant tool to create task instead of triggering ad-hoc validation |
|
||||
| **ADR-0009** (SSL) | No changes. LLM client SSL verification applies to both Path A and Path B. | None |
|
||||
|
||||
## Project Structure — v2 Changes
|
||||
|
||||
```text
|
||||
specs/017-llm-analysis-plugin/
|
||||
├── plan.md # This file (/speckit.plan command output)
|
||||
├── research.md # Phase 0 output (/speckit.plan command)
|
||||
├── data-model.md # Phase 1 output (/speckit.plan command)
|
||||
├── quickstart.md # Phase 1 output (/speckit.plan command)
|
||||
├── contracts/ # Phase 1 output (/speckit.plan command)
|
||||
└── tasks.md # Phase 2 output (/speckit.tasks command - NOT created by /speckit.plan)
|
||||
backend/src/
|
||||
├── plugins/llm_analysis/
|
||||
│ ├── plugin.py # REFACTOR: dispatch to _execute_path_a / _execute_path_b
|
||||
│ ├── service.py # REFACTOR: ScreenshotService.capture_dashboard_chunks()
|
||||
│ │ # LLMClient.analyze_dashboard_multimodal()
|
||||
│ │ # LLMClient.analyze_dashboard_text()
|
||||
│ │ # NEW: DatasetHealthChecker class
|
||||
│ ├── scheduler.py # UPDATE: use ValidationPolicy model
|
||||
│ └── models.py # UPDATE: add ValidationSource, expand ValidationPolicy
|
||||
├── services/
|
||||
│ ├── validation_service.py # REFACTOR: ValidationTaskService (renamed, expanded CRUD)
|
||||
│ ├── llm_prompt_templates.py # UPDATE: add dashboard_validation_prompt_text + multimodal keys
|
||||
│ └── llm_provider.py # (unchanged)
|
||||
├── api/routes/
|
||||
│ ├── llm.py # UPDATE: remove dashboard_validation from provider_bindings schemas
|
||||
│ └── validation_tasks.py # NEW: CRUD routes for /api/validation-tasks
|
||||
└── core/utils/
|
||||
└── superset_context_extractor/ # (unchanged — reused for URL parsing)
|
||||
|
||||
frontend/src/
|
||||
├── routes/
|
||||
│ ├── validation-tasks/ # NEW: task list page
|
||||
│ │ ├── +page.svelte
|
||||
│ │ ├── new/+page.svelte # NEW: task creation form
|
||||
│ │ └── [policyId]/+page.svelte # NEW: task detail + history
|
||||
│ ├── dashboards/+page.svelte # UPDATE: remove Validate button, remove LLM status column
|
||||
│ ├── reports/llm/[taskId]/+page.svelte # UPDATE: multi-chunk screenshots, dataset health display
|
||||
│ └── admin/settings/llm/+page.svelte # UPDATE: remove dashboard_validation binding
|
||||
└── components/
|
||||
└── llm/
|
||||
├── ValidationTaskForm.svelte # NEW: multi-step task creation form
|
||||
├── ValidationTaskReport.svelte # NEW: shared report component
|
||||
├── UrlParser.svelte # NEW: URL paste + parse preview
|
||||
└── DatasetHealthTable.svelte # NEW: dataset health status table
|
||||
```
|
||||
|
||||
### Source Code (repository root)
|
||||
|
||||
```text
|
||||
backend/
|
||||
├── src/
|
||||
│ ├── plugins/
|
||||
│ │ ├── llm_analysis/ # New Plugin Directory
|
||||
│ │ │ ├── __init__.py
|
||||
│ │ │ ├── plugin.py # Implements DashboardValidationPlugin & DocumentationPlugin
|
||||
│ │ │ ├── service.py # LLM interaction logic + ScreenshotService (CDP, tab switching)
|
||||
│ │ │ ├── scheduler.py # Scheduled validation task handler
|
||||
│ │ │ └── models.py # Pydantic models for LLM config/results
|
||||
│ │ └── git/ # Existing Git Plugin
|
||||
│ │ └── ... # Update for commit message generation
|
||||
│ ├── api/
|
||||
│ │ └── routes/
|
||||
│ │ └── llm.py # New routes for LLM config/proxy if needed
|
||||
│ └── services/
|
||||
│ └── llm_provider.py # Shared service for LLM provider management
|
||||
└── tests/
|
||||
└── plugins/
|
||||
└── test_llm_analysis.py
|
||||
|
||||
frontend/
|
||||
├── src/
|
||||
│ ├── components/
|
||||
│ │ ├── llm/
|
||||
│ │ │ ├── ProviderConfig.svelte # Configuration UI
|
||||
│ │ │ └── ValidationReport.svelte # Result viewer
|
||||
│ │ └── git/
|
||||
│ │ └── CommitMessageGen.svelte # UI component for commit generation
|
||||
│ └── routes/
|
||||
│ └── admin/
|
||||
│ └── settings/
|
||||
│ └── llm/ # Settings page route
|
||||
```
|
||||
|
||||
**Structure Decision**: Web application structure with backend plugins and frontend components, consistent with existing project layout.
|
||||
|
||||
## Complexity Tracking
|
||||
|
||||
> **Fill ONLY if Constitution Check has violations that must be justified**
|
||||
|
||||
| Violation | Why Needed | Simpler Alternative Rejected Because |
|
||||
|-----------|------------|-------------------------------------|
|
||||
| N/A | | |
|
||||
| Dual execution paths | Covers visual + data-layer failure modes | Single path leaves blind spots: screenshot-only misses KXD errors, text-only misses visual bugs |
|
||||
| Multi-chunk screenshots | Long dashboards lose readability in single compressed image | Single full-page at higher resolution costs excessive image tokens |
|
||||
| Reused URL parser from different feature | Avoids duplicating complex parsing logic for native_filters/permalink | Custom parser would duplicate 300+ lines of already-tested code |
|
||||
|
||||
@@ -1,42 +1,98 @@
|
||||
# Quickstart: LLM Analysis Plugin
|
||||
# Quickstart: LLM Validation v2
|
||||
|
||||
## Prerequisites
|
||||
|
||||
1. An API Key from OpenAI, OpenRouter, or Kilo.
|
||||
2. (Optional) Playwright browsers installed for high-fidelity screenshots:
|
||||
1. An API Key from OpenAI, OpenRouter, Kilo, or LiteLLM.
|
||||
2. At least one environment configured in ss-tools (pointing to a Superset instance).
|
||||
3. (Optional) Playwright browsers installed for Path A (screenshot) validation:
|
||||
```bash
|
||||
playwright install chromium
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### LLM Provider
|
||||
|
||||
1. Navigate to **Settings > LLM Providers**.
|
||||
2. Click **"Add Provider"**.
|
||||
3. Select type (e.g., `OpenRouter`).
|
||||
4. Enter your API Key and Base URL (e.g., `https://openrouter.ai/api/v1`).
|
||||
5. Set a default model (e.g., `anthropic/claude-3.5-sonnet`).
|
||||
6. Click **Save**.
|
||||
5. Set a default model (e.g., `gpt-4o` for multimodal, `gpt-4o-mini` for text-only).
|
||||
6. Check **"Is Multimodal"** if the model supports image input (required for Path A).
|
||||
7. Click **Save** and **Test Connection**.
|
||||
|
||||
> **Note**: Provider bindings for Documentation, Git Commits, and Chat are configured in Settings. Dashboard Validation provider is selected **per-task** in the task creation form — not in global settings.
|
||||
|
||||
### Validation Prompt (Optional)
|
||||
|
||||
The default validation prompt covers standard dashboard health checks. To customize:
|
||||
1. When creating a task, edit the **Prompt Template** field.
|
||||
2. Path A uses `dashboard_validation_prompt_multimodal` (image-focused).
|
||||
3. Path B uses `dashboard_validation_prompt_text` (topology-focused).
|
||||
|
||||
## Usage
|
||||
|
||||
### Dashboard Validation
|
||||
### Create a Validation Task
|
||||
|
||||
1. Go to **Environments**.
|
||||
2. Select a dashboard from the list.
|
||||
3. Click **"Validate"**.
|
||||
4. Wait for the task to complete and view the report in the **Task History** panel.
|
||||
1. Go to **Validation Tasks** (`/validation-tasks`).
|
||||
2. Click **"+ New Task"**.
|
||||
3. **Step 1 — Name & Environment**: Enter task name (e.g., "Daily KXD Health Check"), select Superset environment.
|
||||
4. **Step 2 — Sources**: Add dashboards to validate:
|
||||
- **From List**: Check dashboards from the environment list.
|
||||
- **From URL**: Paste a full Superset dashboard URL (with filters/tabs). System parses URL and shows dashboard title, active filters, and tabs.
|
||||
5. **Step 3 — Analysis Options**:
|
||||
- Select **LLM Provider** (filtered: multimodal-only when Screenshot enabled).
|
||||
- Toggle **Screenshot** (Playwright):
|
||||
- **ON** (Path A): Captures per-tab screenshots, sends to multimodal LLM. Catches visual bugs, rendering errors.
|
||||
- **OFF** (Path B): Text-only analysis via Superset API. Checks dataset health, KXD connectivity, chart parameters, execution logs. Lighter, faster, catches data-layer errors.
|
||||
- Toggle **Logs**: Include Superset execution logs in LLM context.
|
||||
- Toggle **Execute Chart Data** (Path B only): Runs actual chart queries to verify data returns. Off by default (load-aware).
|
||||
- Edit **Prompt Template** (optional): Customize what the LLM checks.
|
||||
6. **Step 4 — Schedule**:
|
||||
- **Manual**: Run on demand via "Run Now" button.
|
||||
- **Scheduled**: Cron expression (e.g., `0 8 * * 1-5` for weekdays at 8 AM).
|
||||
7. Click **Create Task**.
|
||||
|
||||
### View Results
|
||||
|
||||
1. Go to **Validation Tasks** → see all tasks with last run status.
|
||||
2. Click a task → see per-dashboard breakdown with PASS/WARN/FAIL badges.
|
||||
3. Click a dashboard run → **Full Report**:
|
||||
- **Path A**: Tab selector + per-tab screenshots + issues table.
|
||||
- **Path B**: Dataset health table (backend, status, errors) + chart data results + issues table.
|
||||
- Both paths: Summary, execution logs sent to LLM, task execution logs, timing breakdown.
|
||||
|
||||
### Dashboard Hub (changes)
|
||||
|
||||
- The per-row "Validate" action is **removed**.
|
||||
- The LLM validation status column is **removed**.
|
||||
- Each dashboard row now shows a **colored dot indicator** next to the title (🟢 PASS / 🟡 WARN / 🔴 FAIL / ⚪ UNKNOWN) reflecting the most recent validation across all tasks. Click the dot to see the last 3 runs with links to full reports.
|
||||
- Use the **«+ Create Validation Task»** button in the toolbar to create tasks with pre-selected dashboard IDs.
|
||||
- The **dashboard detail page** (`/dashboards/{id}`) now includes a «Validation History» section showing the last 10 validation runs for this dashboard.
|
||||
|
||||
### Dataset Documentation
|
||||
|
||||
*(unchanged from v1)*
|
||||
|
||||
1. Go to **Tools > Documentation**.
|
||||
2. Select a dataset.
|
||||
3. Click **"Generate Docs"**.
|
||||
4. Review the generated descriptions and changes will be applied to the metadata.
|
||||
4. Review and apply generated descriptions.
|
||||
|
||||
### Git Commit Generation
|
||||
|
||||
1. Make changes to files in the repository.
|
||||
2. Go to the **Git** plugin tab.
|
||||
3. Stage your files.
|
||||
4. Click the **"✨" (Magic)** button next to the commit message box.
|
||||
5. Review and edit the suggested message before committing.
|
||||
*(unchanged from v1)*
|
||||
|
||||
1. Stage files in the Git plugin.
|
||||
2. Click the **"✨" (Magic)** button.
|
||||
3. Review and edit the suggested message.
|
||||
|
||||
## Path Selection Guide
|
||||
|
||||
| Scenario | Choose |
|
||||
|----------|--------|
|
||||
| Checking if dashboards render correctly after Superset upgrade | Path A (Screenshot ON) |
|
||||
| Checking if KXD/Trino data sources are healthy and returning data | Path B (Screenshot OFF) |
|
||||
| Both visual and data-layer issues | Create two tasks — one Path A, one Path B |
|
||||
| Lightweight daily health check with 20 dashboards | Path B, no chart data execution |
|
||||
| Deep investigation of specific dashboard with filters | Path A with URL source (preserves filters) |
|
||||
|
||||
@@ -1,48 +1,142 @@
|
||||
# Research: LLM Analysis & Documentation Plugins
|
||||
# Research: LLM Analysis & Documentation Plugins v2
|
||||
|
||||
**Feature**: `017-llm-analysis-plugin`
|
||||
**Updated**: 2026-05-31 — v2: dual-path execution, multi-chunk screenshots, dataset health checking, URL-based dashboards
|
||||
|
||||
## 1. LLM Provider Integration
|
||||
## v2 Research Decisions
|
||||
|
||||
### 8. Task-Based Flow (replaces ad-hoc Validate)
|
||||
|
||||
**Decision**: Remove one-click «Validate» button; replace with «Create Validation Task» flow (name → sources → provider+prompt → schedule).
|
||||
|
||||
**Rationale**: Ad-hoc validation doesn't scale — users need repeatable, scheduled checks with full configuration control. The task list provides visibility into what's running and historical trends.
|
||||
|
||||
**Alternatives Considered**:
|
||||
- Keep Validate button + add scheduling separately — rejected: two modes cause confusion; task-based is cleaner.
|
||||
- Inline validation from dashboard detail — rejected: no scheduling, no multi-dashboard runs.
|
||||
|
||||
### 9. Dual Execution Paths (Screenshot vs Text-Only)
|
||||
|
||||
**Decision**: `screenshot_enabled` flag controls execution path:
|
||||
- **Path A** (screenshot_enabled=true): Playwright → multi-chunk CDP screenshots → multimodal LLM
|
||||
- **Path B** (screenshot_enabled=false): API calls → dashboard topology + chart params + dataset health + logs → text-only LLM
|
||||
|
||||
**Rationale**:
|
||||
- Screenshot catches visual bugs (empty charts, overlap, rendering) but costs image tokens and requires a browser.
|
||||
- Text-only catches data/KXD errors (dataset inaccessible, chart query timeout) and is lightweight.
|
||||
- Together they cover both visual and data-layer failure modes. Users choose per task based on what matters.
|
||||
|
||||
**Alternatives Considered**:
|
||||
- Always screenshot — rejected: expensive, slow, doesn't check dataset backend health.
|
||||
- Always text-only — rejected: misses visual rendering bugs (broken charts with no error in logs).
|
||||
- Hybrid: screenshot + dataset check in one call — rejected: mixes modalities, increases LLM context size, complicates prompt.
|
||||
|
||||
### 10. Multi-Chunk Screenshots (Path A)
|
||||
|
||||
**Decision**: Instead of one compressed full-page screenshot, capture each dashboard tab as a separate high-resolution screenshot (max 1920×1200 per chunk). Send all chunks in a single multimodal LLM request.
|
||||
|
||||
**Rationale**: Long dashboards (5000+ px) compressed to 1024×2048 lose readability — small chart labels become unreadable. Per-tab chunks preserve resolution, and LLM sees the dashboard in the same logical sections a human would.
|
||||
|
||||
**Alternatives Considered**:
|
||||
- Single full-page at 4096px width — rejected: token cost too high for image processing.
|
||||
- Overlapping chunks with stitching — rejected: complex, fragile to viewport changes.
|
||||
- Per-chart screenshots — rejected: too many images (30+ charts → excessive tokens), loses layout context.
|
||||
|
||||
### 11. Dataset Health Checking (Path B)
|
||||
|
||||
**Decision**: For Path B, call `GET /api/v1/dataset/{id}` for every unique dataset used by dashboard charts. Verify: HTTP 200, database backend, kind (virtual/physical). Optionally execute `POST /api/v1/chart/data` (off by default).
|
||||
|
||||
**Rationale**: The primary purpose of LLM validation is verifying the KXD→dataset→chart→dashboard chain works. Screenshot-only misses silent data failures (empty results, partial data, wrong time range). Dataset health check catches KXD connection errors that produce no visual error but return empty/incomplete data.
|
||||
|
||||
**Alternatives Considered**:
|
||||
- Logs-only (no dataset API calls) — rejected: logs show past failures, not current KXD connectivity. A dataset can be broken right now with no recent log entry.
|
||||
- Always execute chart data — rejected: overloads Superset (15 charts × 30 dashboards = 450 queries). Optional toggle lets users enable selectively.
|
||||
|
||||
### 12. URL-Based Dashboard Selection
|
||||
|
||||
**Decision**: Support pasting full Superset dashboard URLs as task sources. Parse via `SupersetContextExtractor` (same code path as dataset review feature). Extract: dashboard_id, native_filters, activeTabs, anchor.
|
||||
|
||||
**Rationale**: Users copy dashboard URLs from browser with filters already applied. Parsing the URL preserves the exact filter/tab state — the validation checks the dashboard in the same state the user sees. Reusing `SupersetContextExtractor` avoids duplicating URL parsing logic.
|
||||
|
||||
**URL Formats Supported** (same as dataset review):
|
||||
- `https://superset.example.com/superset/dashboard/42/` — numeric ID
|
||||
- `https://superset.example.com/superset/dashboard/my-dashboard/` — slug
|
||||
- `https://superset.example.com/superset/dashboard/p/abc123/` — permalink
|
||||
- `https://superset.example.com/superset/dashboard/42/?native_filters_key=xyz` — filter state key
|
||||
|
||||
**Alternatives Considered**:
|
||||
- Dashboard ID only — rejected: loses filter context, user must manually re-apply filters.
|
||||
- Custom URL format — rejected: users already have Superset URLs in browser; pasting them is natural.
|
||||
|
||||
### 13. Provider & Prompt Per-Task
|
||||
|
||||
**Decision**: Provider selection and prompt template configuration move from global LLM settings to per-task configuration in the task creation form. Global LLM settings retain provider bindings only for: documentation, git_commit, assistant_planner.
|
||||
|
||||
**Rationale**: Different dashboards may need different models (e.g., GPT-4o for complex visual analysis, a cheaper model for text-only). Different teams may want different prompt emphasis (performance vs. data quality). Global settings are too coarse.
|
||||
|
||||
**Alternatives Considered**:
|
||||
- Global binding + per-task override — rejected: adds complexity; per-task is simpler and more flexible.
|
||||
- Environment-level provider binding — rejected: too coarse; same environment may have different validation needs per dashboard group.
|
||||
|
||||
## v1 Research Decisions (preserved)
|
||||
|
||||
### 1. LLM Provider Integration
|
||||
|
||||
**Decision**: Use a unified `LLMProviderService` that abstracts OpenAI-compatible APIs.
|
||||
**Rationale**: OpenRouter, Kilo, and OpenAI all support the standard OpenAI API format. This simplifies implementation by using the `openai` Python SDK and changing the `base_url` and `api_key` dynamically based on configuration.
|
||||
**Alternatives Considered**: LangChain (too heavy/complex for this specific scope), custom HTTP requests (reinventing the wheel).
|
||||
**Rationale**: OpenRouter, Kilo, LiteLLM, and OpenAI all support the standard OpenAI API format.
|
||||
|
||||
## 2. Dashboard Screenshot Capture
|
||||
### 2. Dashboard Screenshot Capture
|
||||
|
||||
**Decision**: Implement a `ScreenshotService` with a strategy pattern supporting `Playwright` (primary) and `Superset API` (fallback).
|
||||
**Rationale**:
|
||||
- **Playwright**: Provides the most accurate "user-view" render, handling JS-heavy charts that API thumbnails might miss or render poorly. Requires a browser binary.
|
||||
- **Superset API**: Faster, lightweight, but relies on Superset's internal thumbnail cache which can be stale.
|
||||
**Implementation Detail**: The service will check configuration. If 'Headless' is selected, it launches a Playwright context, logs in (using a service account or session cookie), navigates to the dashboard, waits for network idle, and captures.
|
||||
**Decision**: `ScreenshotService` with Playwright + CDP for Path A.
|
||||
**Rationale**: Provides accurate «user-view» render; CDP avoids font loading timeouts in headless mode.
|
||||
|
||||
## 3. Multimodal Analysis Prompting
|
||||
### 3. Multimodal Analysis Prompting
|
||||
|
||||
**Decision**: Use a structured prompt template that accepts base64-encoded images and text logs.
|
||||
**Rationale**: Models like GPT-4o and Claude 3.5 Sonnet (via OpenRouter) support this natively.
|
||||
**Prompt Structure**:
|
||||
- System: "You are a Data Observability Expert..."
|
||||
- User Image: [Base64 Screenshot]
|
||||
- User Text: "Recent Logs: \n[Log Snippets]..."
|
||||
- Output Format: JSON (Status, Issues[], Recommendations[])
|
||||
**Decision**: Structured prompt template accepting base64-encoded images + text logs.
|
||||
**Rationale**: GPT-4o / Claude / Gemini models support this natively.
|
||||
|
||||
## 4. Documentation Persistence
|
||||
### 4. Documentation Persistence
|
||||
|
||||
**Decision**: Update `Dataset` and `Column` models in the existing metadata database (likely `mappings.db` or the main application DB if integrated).
|
||||
**Rationale**: Keeps documentation co-located with the assets.
|
||||
**Mechanism**: The `DocumentationPlugin` will fetch schema, generate markdown, and execute an `UPDATE` operation on the relevant tables/fields.
|
||||
**Decision**: Update Dataset/Column models in metadata database.
|
||||
**Rationale**: Documentation co-located with assets.
|
||||
|
||||
## 5. Git Commit Integration
|
||||
### 5. Git Commit Integration
|
||||
|
||||
**Decision**: Add a REST endpoint `/api/git/generate-message` used by the frontend Git component.
|
||||
**Rationale**: Keeps the heavy lifting (LLM call, diff processing) on the backend. The frontend simply sends the list of staged files and a diff summary (truncated if necessary).
|
||||
**Decision**: REST endpoint `/api/git/generate-message`.
|
||||
**Rationale**: Heavy lifting (LLM, diff processing) on backend.
|
||||
|
||||
## 6. Security & Storage
|
||||
### 6. Security & Storage
|
||||
|
||||
**Decision**: Encrypt API keys at rest using the existing `Fernet` or similar encryption mechanism used for database credentials.
|
||||
**Rationale**: API keys are sensitive. They should not be stored in plain text in `config.json` or the DB.
|
||||
**Decision**: AES-256 encrypted API keys.
|
||||
**Rationale**: Keys must not be stored in plain text.
|
||||
|
||||
## 7. Retry Logic
|
||||
### 7. Retry Logic
|
||||
|
||||
**Decision**: Use `tenacity` library for decorators on LLM service methods.
|
||||
**Rationale**: Standard, robust, declarative retry logic (exponential backoff) as required by FR-018.
|
||||
**Decision**: `tenacity` with exponential backoff (5 attempts, 5-60s wait).
|
||||
**Rationale**: Standard, robust retry for transient LLM API failures.
|
||||
|
||||
### 14. Image Pipeline: PNG → JPEG (LLM) + WebP (Archive)
|
||||
|
||||
**Decision**: CDP captures PNG (browser-native, lossless). Convert to JPEG quality=60 for LLM (universal provider compatibility). Convert to WebP quality=80 for archive (93% disk savings vs PNG). Delete PNG and JPEG intermediates after successful save.
|
||||
|
||||
**Rationale**: JPEG universally supported by all providers. WebP has silent failure risk on unknown/closed providers (model accepts request but doesn't process image → false PASS). Pipeline decoupled: LLM failure doesn't affect archive; archive failure falls back to PNG with WARN.
|
||||
|
||||
**Alternatives Considered**:
|
||||
- All WebP (LLM + archive) — rejected: silent failure risk.
|
||||
- All JPEG (LLM + archive) — rejected: no archive space savings, re-encoding artifacts.
|
||||
- WebP for LLM with `_supports_webp()` per-provider check — rejected: adds complexity; JPEG is simpler and universally safe.
|
||||
|
||||
**Impact**: T017a, T017b, T017c in task plan. FR-059 in spec. D10 design decision. ~93% disk savings vs PNG archive.
|
||||
|
||||
### 15. Path B Batching (experimental v2.1)
|
||||
|
||||
**Decision**: Default `llm_batch_size=1` (full isolation, one LLM call per dashboard). Values >1 are experimental and reduce cost at risk of cross-dashboard context contamination.
|
||||
|
||||
**Rationale**: Multiple dashboards in one LLM call saves API calls (5→1) but risks the LLM confusing which issue belongs to which dashboard. Structured prompt with per-dashboard labeled sections + strict JSON `{dashboards: [{dashboard_id, ...}]}` schema mitigates but does not eliminate this risk. For validation (where accuracy > cost), isolation is the safer default.
|
||||
|
||||
**Alternatives Considered**:
|
||||
- Always batch (default 5) — rejected: accuracy risk for validation feature. Can't guarantee LLMs won't cross-contaminate context.
|
||||
- Never batch (always 1) — rejected: legitimate use case for large fleet monitoring where cost matters more than per-dashboard precision.
|
||||
- Per-policy toggle only — accepted as the right balance: default safe, user opts into batching when they understand the tradeoff.
|
||||
|
||||
**Impact**: FR-045a in spec. `llm_batch_size` field in ValidationPolicy, default 1. Batch_size >1 is documented as experimental with known accuracy risk.
|
||||
|
||||
@@ -2,8 +2,23 @@
|
||||
|
||||
**Feature Branch**: `017-llm-analysis-plugin`
|
||||
**Created**: 2026-01-28
|
||||
**Status**: Implemented (plugin, service, scheduler, provider config; ~1714 LOC in core files)
|
||||
**Input**: User description: "LLM Dashboard Validation Plugin для интеграции LLM в ss-tools. Плагин должен поддерживать анализ корректности работы дашбордов через мультимодальную LLM (скриншот + логи). Второй плагин должен заниматься документированием датасетов и дашбордов, используя LLM. Возможно включение в создание текста коммитов в плагине Git Поддержка провайдеров: OpenRouter, Kilo Provider, OpenAI API. Интеграция с существующей PluginBase архитектурой, Task Manager, WebSocket логами."
|
||||
**Updated**: 2026-05-31 — v2: task-based flow, dual-path execution (screenshot + text-only), URL-based dashboards, dataset health checking
|
||||
**Status**: Specification updated; implementation pending for v2 changes
|
||||
|
||||
## v2 Design Decisions
|
||||
|
||||
| # | Decision | Rationale |
|
||||
|---|----------|-----------|
|
||||
| D1 | **Remove one-click «Validate» button** — replace with «Create Validation Task» flow | Ad-hoc validation doesn't scale; users need scheduled, repeatable validation with full configuration control |
|
||||
| D2 | **Dual execution paths**: Path A (Playwright screenshot) + Path B (text-only API) | Screenshot costs image tokens but catches visual bugs; text-only is lightweight, catches data/KXD errors via dataset health checks |
|
||||
| D3 | **Screenshot is optional** (`screenshot_enabled` flag per task) | Lets users trade off visual bug detection vs cost/speed; text-only validates KXD→dataset→chart data chain |
|
||||
| D4 | **Dataset health checking in Path B** — execute `GET /api/v1/dataset/{id}` for every unique dataset, optionally `POST /api/v1/chart/data` | Core goal: verify dashboard opens correctly with KXD data. Without dataset checks, silent KXD failures are invisible |
|
||||
| D5 | **URL-based dashboard selection** — paste Superset URL, parse via `SupersetContextExtractor` (same code as dataset review) | Preserves native_filters, activeTabs, anchor — LLM validates dashboard in exact user-visible state |
|
||||
| D6 | **Provider selection moves from settings to task creation form** | Each task gets explicit provider; no implicit default. Provider bindings for `dashboard_validation` removed from LLM settings |
|
||||
| D7 | **Prompt template moves from settings to task creation form** | Users customize prompts per task, not globally. Default from `DEFAULT_LLM_PROMPTS` |
|
||||
| D8 | **LLM settings retain only**: documentation, git_commit, assistant_planner provider bindings | Dashboard validation provider is per-task, not global |
|
||||
| D9 | **Multi-chunk analysis** — dashboard sliced into logical blocks (tabs + charts) | Multiple screenshots in single LLM request; each block at high resolution instead of one compressed full-page |
|
||||
| D10 | **Image pipeline: PNG → JPEG (LLM) + WebP (archive)** — CDP captures PNG, converted to JPEG for LLM, archived as WebP lossy q=80 for storage efficiency, PNG and JPEG deleted after | JPEG universally supported by all LLM providers; WebP saves ~93% disk space vs PNG. Silent WebP failure risk on unknown providers eliminated by JPEG-only LLM path |
|
||||
|
||||
## Clarifications
|
||||
|
||||
@@ -14,126 +29,187 @@
|
||||
- Q: Git Commit Message Context → A: **Diff + Recent History**: Send diff, file names, and the last 3 commit messages to match style.
|
||||
- Q: LLM Failure Handling → A: **Retry then Fail**: Automatically retry 3 times with exponential backoff before failing.
|
||||
|
||||
## User Scenarios & Testing *(mandatory)*
|
||||
### Session 2026-05-31
|
||||
- Q: Concurrent execution — manual trigger while task is running? → Same task: toast «Already running», no queue. Different task: FIFO queue, respecting concurrency_limit.
|
||||
- Q: API key rotation — impact on existing tasks? → Tasks fetch current key at execution time from LLMProviderService. No key caching in task config.
|
||||
- Q: Multi-chunk token limit — 15+ tabs exceeding provider limit? → Estimate payload → progressive quality reduction (30% JPEG, 800px) → Path B fallback. Molecular-cot logging: REASON before each reduction, REFLECT after.
|
||||
- Q: Backward compat — old POST /api/tasks with llm_dashboard_validation? → Hard cut: returns 400. All consumers must migrate to POST /api/validation-tasks.
|
||||
- Q: URL re-parse — dashboard structure changed between runs? → Re-parse every execution. URL failure → warning «Paste new URL». Dashboard ID change → WARN log, use new ID.
|
||||
- Q: Provider deletion — what if used by active tasks? → Forbidden. Returns list of blocking task names. User must reassign or deactivate tasks first.
|
||||
- Q: Screenshot format — PNG vs WebP vs JPEG? → **Pipeline**: CDP captures PNG → converted to JPEG (LLM, universal compatibility) + WebP lossy q=80 (archive, 93% disk savings). PNG and JPEG deleted after WebP saved. LLM path and archive path are decoupled.
|
||||
|
||||
### User Story 1 - Dashboard Health Analysis (Priority: P1)
|
||||
## User Scenarios & Testing
|
||||
|
||||
As a Data Engineer, I want to automatically analyze a dashboard's status using visual and log data directly from the Environments interface so that I can identify rendering issues or data errors without manual inspection.
|
||||
### User Story 1 — Create & Schedule Dashboard Validation Task (Priority: P1)
|
||||
|
||||
**Why this priority**: Core value proposition of the feature. Enables automated quality assurance.
|
||||
As a Data Engineer, I want to create a validation task that checks multiple dashboards for health issues on a schedule, so that I can proactively detect KXD data problems, rendering failures, and chart errors without manual inspection.
|
||||
|
||||
**Independent Test**: Can be tested by selecting a dashboard in the Environment list and clicking "Validate", or by scheduling a validation task.
|
||||
**Why this priority**: Core value proposition. Replaces unreliable one-off manual checks with automated, scheduled validation covering the full KXD→dataset→chart→dashboard chain.
|
||||
|
||||
**Independent Test**: Create a validation task with 2 dashboards (one via ID, one via URL), set cron schedule `0 8 * * 1-5`, wait for first execution, verify report shows dataset health status per dashboard.
|
||||
|
||||
**Acceptance Scenarios**:
|
||||
|
||||
1. **Given** I am on the Environments page (Dashboard list), **When** I select a dashboard and click "Validate", **Then** the system triggers a validation task with the dashboard's context.
|
||||
2. **Given** the validation task is running, **When** it completes, **Then** I see the analysis report (visual + logs) in the task output history.
|
||||
3. **Given** I want regular checks, **When** I configure a schedule for the validation task (similar to Backup plugin), **Then** the system runs the check automatically at the specified interval.
|
||||
4. **Given** a validation issue is found, **When** the task completes, **Then** the system sends a notification (Email/Pulse) if configured.
|
||||
1. **Given** I am on the Validation Tasks page, **When** I click «+ New Task», **Then** the task creation form opens with fields: name, environment, sources (dashboard IDs + URLs), provider, prompt, screenshot toggle, schedule.
|
||||
2. **Given** I paste a Superset dashboard URL with native filters, **When** the form parses it, **Then** the system extracts dashboard_id, native_filters, activeTabs, and anchor — and validates the URL is accessible.
|
||||
3. **Given** I enable `screenshot_enabled`, **When** the task executes, **Then** the system uses Path A: Playwright browser → CDP screenshot → multimodal LLM analysis.
|
||||
4. **Given** I disable `screenshot_enabled`, **When** the task executes, **Then** the system uses Path B: API calls → dashboard structure + chart params + dataset schema (+ optional chart data execution) → text-only LLM analysis.
|
||||
5. **Given** the task has a cron schedule, **When** the schedule triggers, **Then** the system runs validation on all configured dashboards and persists results.
|
||||
6. **Given** a validation finds issues, **When** the task completes, **Then** the system sends a notification (Email/Pulse) if configured.
|
||||
|
||||
---
|
||||
### User Story 2 — Validation Task List & History (Priority: P1)
|
||||
|
||||
### User Story 2 - Automated Dataset Documentation (Priority: P1)
|
||||
As a Data Engineer, I want to see all my validation tasks, their last run status, and drill into historical results per dashboard, so that I can track health trends over time.
|
||||
|
||||
As a Data Steward, I want to generate documentation for datasets and dashboards using LLMs so that I can maintain up-to-date metadata with minimal manual effort.
|
||||
**Why this priority**: Without a list view and history, the task-based flow is invisible — users can't monitor what's running.
|
||||
|
||||
**Why this priority**: significantly reduces maintenance overhead for data governance.
|
||||
|
||||
**Independent Test**: Can be tested by selecting a dataset and triggering the documentation task, then checking if a description is generated.
|
||||
**Independent Test**: View the task list, see status per task (active/inactive), last run time, aggregated status across dashboards.
|
||||
|
||||
**Acceptance Scenarios**:
|
||||
|
||||
1. **Given** a dataset identifier, **When** I run the Documentation task, **Then** the system fetches the dataset's schema and metadata.
|
||||
2. **Given** the metadata is fetched, **When** sent to the LLM, **Then** a structured description/documentation text is returned.
|
||||
3. **Given** the documentation is generated, **When** the task completes, **Then** the result is available for review (e.g., in the task log or saved to a file/db).
|
||||
1. **Given** I am on the Validation Tasks page, **When** the page loads, **Then** I see all created tasks with: name, schedule, last run status, dashboard count.
|
||||
2. **Given** I click a task, **When** the detail opens, **Then** I see per-dashboard breakdown: status (PASS/WARN/FAIL), detected issues, last run timestamp, link to full report.
|
||||
3. **Given** I click a dashboard in the task detail, **When** the report opens, **Then** I see: check result, text summary, issues table, dataset health status (Path B), screenshot thumbnails (Path A), source logs, execution logs.
|
||||
|
||||
---
|
||||
### User Story 3 — Dual-Path Dashboard Health Analysis (Priority: P1)
|
||||
|
||||
### User Story 3 - LLM Provider Configuration (Priority: P1)
|
||||
As a Data Engineer, I want the validation to check both visual rendering AND data availability, so that I catch KXD backend failures that produce no visual error but return empty/incomplete data.
|
||||
|
||||
As an Administrator, I want to configure different LLM providers (OpenAI, OpenRouter, Kilo) so that I can switch between models based on cost or capability.
|
||||
**Why this priority**: The primary purpose of validation is verifying the KXD→dataset→chart data chain works. Screenshot-only validation misses silent data failures (empty results, partial data, wrong time range).
|
||||
|
||||
**Why this priority**: Prerequisite for any LLM functionality.
|
||||
|
||||
**Independent Test**: Can be tested by entering API keys in settings and verifying a connection/test call.
|
||||
**Independent Test**: Create a task with `screenshot_enabled=false`, run against a dashboard with a known broken dataset, verify the LLM reports the dataset health issue.
|
||||
|
||||
**Acceptance Scenarios**:
|
||||
|
||||
1. **Given** the Settings page, **When** I select an LLM provider (e.g., OpenAI) and enter an API key, **Then** the system saves the configuration.
|
||||
2. **Given** a configured provider, **When** I run an analysis task, **Then** the system uses the selected provider for API calls.
|
||||
1. **Given** Path B execution, **When** a dataset `GET /api/v1/dataset/{id}` returns KXD connection error, **Then** the LLM report flags the dataset as unhealthy and lists affected charts.
|
||||
2. **Given** Path B execution with chart data execution enabled, **When** `POST /api/v1/chart/data` times out, **Then** the LLM report includes the timeout with affected chart info.
|
||||
3. **Given** Path B execution, **When** all datasets are healthy and logs show no errors, **Then** the LLM report returns `status: PASS` with a summary of verified data sources.
|
||||
4. **Given** Path A (screenshot), **When** LLM analyzes the multi-chunk screenshots, **Then** visual issues (empty charts, rendering errors, overlap) are detected and reported with tab/chart location.
|
||||
|
||||
---
|
||||
### User Story 4 — Automated Dataset Documentation (Priority: P2)
|
||||
|
||||
### User Story 4 - Git Commit Message Suggestion (Priority: P3)
|
||||
*(unchanged from v1)*
|
||||
|
||||
As a Developer, I want the system to suggest commit messages based on changes directly within the Git plugin interface so that I can maintain consistent history with minimal effort.
|
||||
As a Data Steward, I want to generate documentation for datasets and dashboards using LLMs.
|
||||
|
||||
**Why this priority**: Enhances the existing Git workflow and improves commit quality.
|
||||
### User Story 5 — LLM Provider Configuration (Priority: P2)
|
||||
|
||||
**Independent Test**: Can be tested by staging files in the Git plugin and clicking the "Generate Message" button.
|
||||
*(unchanged from v1 — but provider bindings for dashboard_validation removed)*
|
||||
|
||||
**Acceptance Scenarios**:
|
||||
As an Administrator, I want to configure LLM providers for documentation, git commits, and chat — with dashboard validation provider configured per-task.
|
||||
|
||||
1. **Given** staged changes in the Git plugin, **When** I click "Generate Message", **Then** the system analyzes the diff using the configured LLM and populates the commit message field with a suggested summary.
|
||||
### User Story 6 — Git Commit Message Suggestion (Priority: P3)
|
||||
|
||||
### Edge Cases
|
||||
*(unchanged from v1)*
|
||||
|
||||
- What happens when the LLM provider API is down or times out? (System should retry or fail gracefully with a clear error message).
|
||||
- What happens if the dashboard screenshot cannot be generated? (System should proceed with logs only or fail depending on configuration).
|
||||
- What happens if the context (logs/metadata) exceeds the LLM's token limit? (System should truncate or summarize input).
|
||||
- How does the system handle missing API keys? (Task should fail immediately with a configuration error).
|
||||
- What happens if the dashboard has multiple tabs with lazy-loaded charts? (System must switch through all tabs recursively to trigger chart rendering before capture).
|
||||
- What happens if Playwright encounters font loading timeouts in headless mode? (System must use CDP `Page.captureScreenshot` to bypass Playwright's internal timeout mechanism).
|
||||
## Requirements
|
||||
|
||||
## Requirements *(mandatory)*
|
||||
### Functional Requirements — v2 Additions & Changes
|
||||
|
||||
### Functional Requirements
|
||||
#### Task Management
|
||||
- **FR-032**: System MUST replace the one-click «Validate» button with a «Create Validation Task» flow accessible from a new `/validation-tasks` page.
|
||||
- **FR-033**: Task creation form MUST include: task name, environment, dashboard sources (IDs + URLs), LLM provider selector, prompt template textarea (pre-filled with default), screenshot toggle, logs toggle, schedule (cron or one-off).
|
||||
- **FR-034**: System MUST support two source types per task: `dashboard_id` (numeric ID selected from list) and `dashboard_url` (full Superset URL parsed via `SupersetContextExtractor`).
|
||||
- **FR-035**: URL parsing MUST extract: dashboard_id (numeric or slug), native_filters, activeTabs, anchor — using the same `SupersetContextExtractor` code path as dataset review.
|
||||
- **FR-036**: System MUST validate that every URL-resolved dashboard belongs to the selected environment before task creation.
|
||||
- **FR-037**: Task list MUST show: task name, status (active/inactive), schedule, last run timestamp, aggregated validation status across dashboards.
|
||||
- **FR-038**: Task detail MUST show per-dashboard breakdown with: dashboard title, validation status (PASS/WARN/FAIL), detected issue count, last run timestamp, link to full report.
|
||||
|
||||
- **FR-001**: System MUST allow configuration of multiple LLM providers, specifically supporting OpenAI API, OpenRouter, and Kilo Provider.
|
||||
- **FR-002**: System MUST securely store API keys for these providers using AES-256 encryption. [Security]
|
||||
- **FR-028**: The system MUST mask all API keys in the UI and logs, displaying only the last 4 characters (e.g., `sk-...1234`). [Security]
|
||||
- **FR-003**: System MUST implement a `DashboardValidationPlugin` that integrates with the existing `PluginBase` architecture.
|
||||
- **FR-004**: `DashboardValidationPlugin` MUST accept a dashboard identifier as input.
|
||||
- **FR-005**: `DashboardValidationPlugin` MUST be capable of retrieving a visual representation (screenshot) of the dashboard. The visual representation MUST be a PNG image with a resolution of 1920px width and full page height to ensure all dashboard content is captured. [Clarity]
|
||||
- **FR-016**: System MUST support configurable screenshot strategies: 'Headless Browser' (default, high accuracy) and 'API Thumbnail' (fallback/fast).
|
||||
- **FR-030**: The screenshot capture MUST use Playwright with Chrome DevTools Protocol (CDP) to avoid font loading timeouts in headless mode.
|
||||
- **FR-031**: The screenshot capture MUST implement recursive tab switching to trigger lazy-loaded chart rendering on multi-tab dashboards before capturing.
|
||||
- **FR-006**: `DashboardValidationPlugin` MUST retrieve recent execution logs associated with the dashboard from the Environment API (e.g., `/api/v1/log/`), limited to the last 100 lines or 24 hours (whichever is smaller) to prevent token overflow. [Reliability]
|
||||
- **FR-007**: `DashboardValidationPlugin` MUST combine visual and text data to prompt a Multimodal LLM for analysis. The analysis output MUST be structured as a JSON object containing `status` (Pass/Fail), `issues` (list of strings), and `summary` (text) to enable structured UI presentation. [Clarity]
|
||||
- **FR-008**: System MUST implement a `DocumentationPlugin` (or similar) for documenting datasets and dashboards.
|
||||
- **FR-009**: `DocumentationPlugin` MUST retrieve schema and metadata for the target asset.
|
||||
- **FR-017**: `DocumentationPlugin` MUST apply generated descriptions directly to the target object's metadata fields (dataset description, column descriptions). It MUST handle schema changes by only updating fields that exist in the current schema and ignoring hallucinated columns. [Data Integrity]
|
||||
- **FR-023**: The system MUST use optimistic locking or version checks to prevent overwrites during concurrent documentation updates. [Data Integrity]
|
||||
- **FR-024**: Generated documentation MUST be plain text or Markdown, strictly avoiding executable code blocks or active HTML. [Security]
|
||||
- **FR-025**: The system MUST validate that generated commit messages follow the conventional commits format (e.g., `feat:`, `fix:`) and do not exceed 72 characters in the subject line. [Data Integrity]
|
||||
- **FR-026**: If a metadata update fails partially, the system MUST rollback all changes to preserve data consistency. [Data Integrity]
|
||||
- **FR-027**: The system MUST reject documentation requests for empty or null schemas with a clear error message. [Edge Case]
|
||||
- **FR-010**: All LLM interactions MUST be executed as asynchronous tasks via the Task Manager.
|
||||
- **FR-018**: System MUST implement automatic retry logic (3 attempts with exponential backoff: 2s, 4s, 8s) for failed LLM API calls. [Reliability]
|
||||
- **FR-029**: The system MUST filter sensitive data (PII, credentials) from logs and screenshots before sending them to external LLM providers. [Privacy]
|
||||
- **FR-011**: Task execution logs and results MUST be streamed via the existing WebSocket infrastructure.
|
||||
- **FR-012**: System SHOULD expose an interface to generate text summaries for Git diffs, utilizing the diff, file list, and recent commit history as context. If the generated message is empty or invalid, the system MUST display a user-friendly error toast and retain the manual entry capability. [Edge Case]
|
||||
- **FR-013**: System MUST support scheduling of validation tasks for dashboards (leveraging existing scheduler architecture).
|
||||
- **FR-014**: System SHOULD support notification dispatch (Email, Pulse) upon validation failure or completion.
|
||||
- **FR-015**: Notifications MUST contain a summary of results (Status, Issue Count) and a direct link to the full report, avoiding sensitive full details in the message body.
|
||||
- **FR-019**: The "Validate" button MUST display a loading spinner and be disabled during active execution to prevent multiple triggers. [UX]
|
||||
- **FR-020**: The system MUST provide immediate visual feedback (Toast notifications) for successful or failed connection tests in LLM Settings. [UX]
|
||||
- **FR-021**: New LLM actions (Validate, Generate Docs) MUST use standard system icons (e.g., `heroicons:beaker` for validate, `heroicons:document-text` for docs) and follow existing button placement patterns. [Consistency]
|
||||
- **FR-022**: Error messages for LLM failures MUST follow the standard `ss-tools` error format, including a clear title, descriptive message, and troubleshooting link if applicable. [Consistency]
|
||||
#### Dual-Path Execution
|
||||
- **FR-039**: System MUST support two execution paths: Path A — Playwright screenshot (screenshot_enabled=true) and Path B — text-only API (screenshot_enabled=false).
|
||||
- **FR-040**: Path A MUST: launch Playwright → login → navigate to URL (with parsed filters/tabs) → recursive tab switching → multi-chunk CDP screenshots → compress images → multimodal LLM call.
|
||||
- **FR-041**: Multi-chunk screenshots MUST capture each dashboard tab as a separate high-resolution image (max 1920×1200 per chunk) instead of one compressed full-page.
|
||||
- **FR-059**: Multi-chunk screenshots MUST follow the image pipeline: CDP capture → PNG → convert to JPEG (quality 60, 1024px) for LLM + convert to WebP (lossy, quality 80) for archive → delete PNG and JPEG after WebP saved. LLM receives ONLY JPEG for universal provider compatibility.
|
||||
- **FR-042**: Path B MUST: call `GET /api/v1/dashboard/{id}` for structure → `GET /api/v1/chart/{id}` for each chart → `GET /api/v1/dataset/{id}` for each unique dataset → `GET /api/v1/log/` for execution logs → construct text description of dashboard topology → text-only LLM call.
|
||||
- **FR-043**: Path B MUST optionally execute `POST /api/v1/chart/data` for each chart to verify query execution (controlled by `execute_chart_data` toggle, default false to avoid load).
|
||||
- **FR-044**: Path B dataset health check MUST verify four levels: (1) **metadata_accessible** — HTTP 200 from `GET /api/v1/dataset/{id}`, database name, backend, kind; (2) **datasource_resolvable** — chart datasource_id maps to a valid dataset; (3) **query_executable** — only when `execute_chart_data=true`: `POST /api/v1/chart/data` succeeds; (4) **data_returned** — only when `execute_chart_data=true`: row_count > 0 or no error. Levels 1-2 are always checked; levels 3-4 require explicit opt-in.
|
||||
- **FR-045**: System MUST use different LLM prompt templates for Path A (multimodal — image + text, one dashboard per call) vs Path B (text-only — dashboard topology + dataset health + logs, batched per FR-045a).
|
||||
- **FR-045a**: Path B execution MAY support batching: multiple dashboards validated in a single LLM call up to `llm_batch_size`. Default `batch_size=1` (full isolation — one LLM call per dashboard). Batching is experimental: values >1 reduce cost at risk of cross-dashboard context contamination. Each batch call MUST use a structured prompt enforcing per-dashboard isolation: labeled sections + `{dashboards: [{dashboard_id, status, summary, issues}]}` JSON schema. Missing/parse-error dashboards marked UNKNOWN individually — other dashboards in batch preserved. See research decision 15 for rationale and risks.
|
||||
- **FR-045b**: Path A execution does NOT support batching — each dashboard requires a separate multimodal LLM call due to image token constraints. Multiple Path A dashboards within a run are parallelized via `policy_dashboard_concurrency_limit`.
|
||||
- **FR-046**: Path B prompt MUST include: dashboard title, owner, tab hierarchy, chart names/types/params/datasource, dataset health status per datasource, execution logs — all as structured text.
|
||||
|
||||
### Key Entities
|
||||
#### Provider & Prompt Configuration
|
||||
- **FR-047**: Dashboard validation provider MUST be configured per-task in the task creation form — NOT as a global binding in LLM settings.
|
||||
- **FR-048**: Dashboard validation prompt template MUST be configurable per-task as a single `prompt_template` field. When `null`, the system MUST use the path-specific default: `dashboard_validation_prompt_multimodal` for Path A (`screenshot_enabled=true`) and `dashboard_validation_prompt_text` for Path B (`screenshot_enabled=false`). The legacy v1 key `dashboard_validation_prompt` MUST NOT be used as default.
|
||||
- **FR-048a**: When the user toggles `screenshot_enabled` on an existing task with a custom prompt, the system MUST show a warning: «Prompt may no longer match the selected execution path. Review your prompt template.»
|
||||
- **FR-049**: LLM settings page (`/admin/settings/llm`) MUST retain provider bindings only for: documentation, git_commit, assistant_planner. Dashboard validation binding MUST be removed.
|
||||
- **FR-050**: The provider selector in task creation form MUST filter to show only active, multimodal-compatible providers (for Path A) or active providers of any type (for Path B).
|
||||
|
||||
- **LLMProviderConfig**: Stores provider type (OpenAI, etc.), base URL, model name, and API key.
|
||||
- **ValidationResult**: Stores the analysis output, timestamp, and reference to the dashboard.
|
||||
- **AutoDocumentation**: Stores the generated documentation text for an asset.
|
||||
#### Dashboard Hub Changes
|
||||
- **FR-051**: Dashboard hub (`/dashboards`) MUST remove the per-row «Validate» action from the action dropdown.
|
||||
- **FR-051a**: Dashboard hub MUST add a lightweight validation status indicator (colored dot: 🟢 PASS / 🟡 WARN / 🔴 FAIL / ⚪ UNKNOWN) next to each dashboard title, reflecting the most recent validation result across all tasks that include this dashboard. Clicking the indicator MUST open a dropdown with the last 3 validation runs (date, status, task name) and a «View all runs» link.
|
||||
- **FR-052**: Dashboard hub MUST remove the LLM validation status column (PASS/WARN/FAIL badges from `lastTask.validationStatus`). Replaced by the indicator in FR-051a.
|
||||
- **FR-053**: Dashboard hub MUST retain a «Create Validation Task» button in the toolbar that pre-fills selected dashboard IDs into a new task form.
|
||||
- **FR-053a**: Dashboard detail page (`/dashboards/{id}`) MUST include a «Validation History» section showing the last 10 validation runs for this dashboard across all tasks, with links to canonical run detail pages. This section replaces the removed Validate button and LLM status column.
|
||||
|
||||
## Success Criteria *(mandatory)*
|
||||
#### Reporting & Routing
|
||||
- **FR-055b**: System MUST serve validation reports at the canonical URL `/validation-tasks/{policy_id}/runs/{run_id}`. The page MUST render:
|
||||
- **Run header**: policy name, run timestamp, trigger type (manual/scheduled), total duration, aggregate status (N PASS / M WARN / K FAIL across all dashboards).
|
||||
- **Dashboard list**: each dashboard as a collapsible row showing: title, status icon (PASS/WARN/FAIL/UNKNOWN), issue count, mini-summary. Expanded row shows: full issues list, execution path (Path A/B), dataset health (Path B), screenshot thumbnails with tab selector (Path A), logs sent to LLM, task execution logs.
|
||||
- **Mixed paths**: a single run may contain both Path A and Path B dashboard records ONLY when a Path A dashboard falls back to Path B due to payload limit (FR-056) or when Path A execution fails and retries as Path B. Mixed paths do NOT occur from per-source path override — `screenshot_enabled` is a policy-level setting. The run report handles mixed records by rendering screenshot sections for `execution_path=screenshot` records and dataset health sections for `execution_path=text_only` records.
|
||||
- **FR-055c**: The legacy `/reports/llm/{taskId}` route MUST redirect (302) to `/validation-tasks/{policy_id}/runs/{run_id}` when `taskId` matches a `ValidationRun.task_id`. Returns 404 if no match.
|
||||
- **FR-054**: System MUST prevent duplicate execution of the same task: if a task is already running, manual «Run Now» triggers return a toast «Task is already running» without queuing.
|
||||
- **FR-055**: System MUST queue manual execution requests for different tasks in a FIFO queue, respecting `global_validation_worker_limit` (max concurrent policy runs across all tasks). If the queue is full, the system MUST show a toast with estimated wait time. Same-policy lock: if a policy is already running (manual or scheduled), additional triggers are rejected with status `SKIPPED_ALREADY_RUNNING`.
|
||||
- **FR-055a**: Within a single `ValidationRun`, dashboards MUST be validated with parallelism up to `policy_dashboard_concurrency_limit` (default 3). Dashboards are processed in waves: first 3 start → any finishes → next starts, until all done or exhausted. Each dashboard produces one `ValidationRecord` with FK to the same `run_id`.
|
||||
- **FR-055d**: System MUST create a `ValidationRun` record when a task execution starts (manual or scheduled). The run groups all per-dashboard `ValidationRecord`s. Upon completion, the run MUST compute aggregate counts: `dashboard_count`, `pass_count`, `warn_count`, `fail_count`, `unknown_count`. Run status = `completed` (all done), `partial` (some sources skipped/invalid), or `failed` (execution error before any dashboard).
|
||||
- **FR-056**: System MUST calculate estimated LLM payload size before sending multi-chunk screenshots. If >80% of model context window: progressively reduce JPEG quality to 30%, then width to 800px. If still exceeds — fall back to Path B text-only with a warning. All reduction decisions logged via molecular-cot: REASON before each step, REFLECT after outcome.
|
||||
- **FR-057**: System MUST re-parse URL sources on every task execution. If URL parsing fails — source marked with warning «Dashboard structure changed. Paste a new URL». If dashboard_id changed — logged as WARN, new ID used.
|
||||
- **FR-058**: System MUST forbid deletion of an LLM provider if it is referenced by any active `ValidationPolicy`. The deletion endpoint MUST return a list of blocking task names. Users MUST reassign or deactivate affected tasks before deletion.
|
||||
|
||||
### Measurable Outcomes
|
||||
#### Existing Requirements (v1, preserved)
|
||||
- **FR-001**: System MUST allow configuration of multiple LLM providers (OpenAI, OpenRouter, Kilo, LiteLLM).
|
||||
- **FR-002**: System MUST securely store API keys using AES-256 encryption.
|
||||
- **FR-005**: Screenshot capture (when enabled) MUST be PNG with 1920px width.
|
||||
- **FR-006**: Log retrieval limited to last 100 lines or 24 hours.
|
||||
- **FR-007**: Analysis output structured as JSON `{status, summary, issues: [{severity, message, location}]}`.
|
||||
- **FR-010**: All LLM interactions as asynchronous tasks via Task Manager.
|
||||
- **FR-013**: Scheduling support for validation tasks via cron.
|
||||
- **FR-014**: Notification dispatch (Email, Pulse) on validation failure/completion.
|
||||
- **FR-015**: Notifications MUST contain a summary (Status, Issue Count) and a direct link to the canonical run detail: `/validation-tasks/{policy_id}/runs/{run_id}`. Per-dashboard deep link via query parameter: `?dashboard_id={id}`. The legacy `/reports/llm/{taskId}` route is a compatibility redirect. Sensitive details MUST NOT appear in the notification body.
|
||||
- **FR-018**: Retry logic: 3 attempts (now 5) with exponential backoff.
|
||||
- **FR-022**: Error messages in standard ss-tools format.
|
||||
- **FR-028**: API key masking in UI and logs (last 4 chars only).
|
||||
- **FR-029**: System MUST redact PII, credentials, query text, tokens in URLs, and native filter values from text/log payloads before sending to external LLM providers and before persistence. Redaction is deterministic (regex/pattern-based). Default retention: screenshots 30 days, raw LLM responses 30 days (debug-only, nullable column). Redacted summaries/issues retained indefinitely. Auditable via structured decision audit logs (REASON/REFLECT markers — deterministic metadata only, no free-form reasoning text).
|
||||
- **FR-029c**: Screenshot redaction MUST mask known DOM regions (URL bar, header with user info) and CSS selectors where available before capture. Full OCR-based screenshot PII redaction is out of scope for v2.
|
||||
|
||||
- **SC-001**: Users can successfully configure and validate a connection to at least one LLM provider.
|
||||
- **SC-002**: A dashboard validation task completes within 90 seconds (assuming standard LLM latency).
|
||||
- **SC-003**: The system successfully processes a multimodal prompt (image + text) and returns a structured analysis.
|
||||
- **SC-004**: Generated documentation for a standard dataset contains descriptions for at least 80% of the columns (based on LLM capability, but pipeline must support it).
|
||||
- **SC-005**: Screenshots capture full dashboard content including all tabs (1920px width, full height) without font loading timeouts.
|
||||
- **SC-006**: Analysis results are displayed in task logs with clear `[ANALYSIS_SUMMARY]` and `[ANALYSIS_ISSUE]` markers for easy parsing.
|
||||
### Key Entities — v2
|
||||
|
||||
- **ValidationPolicy** (updated): Name, environment_id, sources (list of `ValidationSource`), provider_id, prompt_template, screenshot_enabled, logs_enabled, execute_chart_data, cron_expression, is_active, notification_channels.
|
||||
- **ValidationSource** (new): `type` (dashboard_id | dashboard_url), `value` (ID or URL), `parsed_context` (ParsedNativeFilters — only for URL type).
|
||||
- **ValidationRecord** (unchanged): Per-execution result: dashboard_id, status, summary, issues, screenshot_paths, dataset_health, raw_response.
|
||||
- **LLMProviderConfig** (unchanged): Provider type, base URL, model, encrypted API key, is_multimodal flag.
|
||||
|
||||
## Success Criteria
|
||||
|
||||
### Measurable Outcomes — v2 Additions
|
||||
|
||||
- **SC-007**: A Path B (text-only) validation of a 5-tab dashboard with 15 charts completes within 30 seconds (excluding LLM latency).
|
||||
- **SC-008**: A Path A (screenshot) multi-chunk validation of a 5-tab dashboard completes within 120 seconds.
|
||||
- **SC-009**: URL parsing correctly extracts dashboard_id, native_filters, and activeTabs from 100% of standard Superset dashboard URLs (numeric ID, slug, permalink formats).
|
||||
- **SC-010**: Dataset metadata errors surfaced by Superset API (levels 1-2) are detected with 100% accuracy. Query-level KXD failures (levels 3-4) require `execute_chart_data=true` and depend on Superset query execution behavior.
|
||||
- **SC-011**: User can create a validation task (form fill + URL parse + provider select) in under 60 seconds.
|
||||
|
||||
### Existing (v1, preserved)
|
||||
- **SC-001**: Users can configure and test connection to at least one LLM provider.
|
||||
- **SC-003**: Multimodal prompt (image + text) returns structured JSON analysis.
|
||||
- **SC-005**: Screenshots capture full dashboard content including all tabs.
|
||||
|
||||
## Edge Cases — v2
|
||||
|
||||
- What happens when a URL-parsed dashboard has been deleted before task execution? (System marks source as invalid, skips it, reports in task log.)
|
||||
- What happens when `screenshot_enabled=false` but no dataset info is retrievable? (System proceeds with logs-only analysis, flags missing data sources.)
|
||||
- What happens when a task has 20 dashboards and each takes 30s? (System executes sequentially with configurable concurrency limit, default 3.)
|
||||
- What happens when user clicks «Run Now» while the same task is already executing? (System shows toast «Task is already running», does NOT queue a duplicate run.)
|
||||
- What happens when user clicks «Run Now» while another task is executing? (System places the manual trigger in a FIFO queue, respecting concurrency_limit. If queue is full — shows toast with estimated wait time.)
|
||||
- What happens when the API key for a provider used by existing tasks is changed? (Tasks always fetch the current key from `LLMProviderService` at execution time — no key caching in task configuration. Old key is never reused.)
|
||||
- What happens when multi-chunk screenshots (15+ tabs) exceed the LLM provider's token limit? (System calculates estimated payload size before sending. If >80% of model context window: reduces JPEG quality progressively to 30%, then width to 800px. If still exceeds — falls back to Path B text-only with a warning. All decisions logged via molecular-cot: REASON before each reduction, REFLECT after outcome.)
|
||||
- What happens if WebP encoding fails during archive step? (System falls back to PNG for archive with a WARN log. JPEG for LLM is unaffected — the pipeline is independent: LLM path and archive path are decoupled.)
|
||||
- What happens when user toggles `screenshot_enabled` on a task with a custom prompt? (System shows warning: «Prompt may no longer match execution path» per FR-048a. Task saves successfully with user confirmation.)
|
||||
- What happens when old API consumers send `POST /api/tasks` with `plugin_id: llm_dashboard_validation` after v2 migration? (Returns `400 Bad Request` with message: «llm_dashboard_validation is deprecated. Use POST /api/validation-tasks instead.» All consumers — assistant tool, external scripts — must be updated to v2 API during migration.)
|
||||
- What happens when a URL-parsed dashboard changes structure between task executions? (System re-parses the URL on every execution. If the URL fails — dashboard structure has changed. System marks the source with a warning: «Dashboard structure changed. Paste a new URL to update filters.» If dashboard_id changed — logs WARN and uses the new ID.)
|
||||
- What happens when a user tries to delete an LLM provider that is used by active validation tasks? (System forbids deletion and returns a list of blocking task names. User must reassign the provider in each listed task or deactivate the tasks before deleting the provider.)
|
||||
- What happens when Path B `POST /api/v1/chart/data` overloads Superset? (Toggle is off by default; user explicitly opts in with a warning about load.)
|
||||
- What happens when `SupersetContextExtractor.parse_superset_link()` returns partial recovery? (System flags the source with `partial_recovery`, shows warning in task detail, attempts best-effort validation.)
|
||||
- What happens when a dashboard URL contains a permalink key that has expired? (System marks the source as stale, prompts user to paste a fresh URL.)
|
||||
|
||||
(End of file — 214 lines)
|
||||
|
||||
@@ -1,105 +1,393 @@
|
||||
# Tasks: LLM Analysis & Documentation Plugins
|
||||
# Tasks: LLM Analysis & Documentation Plugins v2
|
||||
|
||||
**Feature**: `017-llm-analysis-plugin`
|
||||
**Status**: Completed
|
||||
**Spec**: [specs/017-llm-analysis-plugin/spec.md](spec.md)
|
||||
**Status**: v1 Completed; v2 Planned
|
||||
**Spec**: [spec.md](spec.md) | **Plan**: [plan.md](plan.md) | **Data Model**: [data-model.md](data-model.md)
|
||||
|
||||
## User Stories & Priorities
|
||||
|
||||
| ID | Story | Priority | Phase |
|
||||
|----|-------|----------|-------|
|
||||
| US1 | Create & Schedule Validation Task | P1 | 3 |
|
||||
| US2 | Validation Task List & History | P1 | 3 |
|
||||
| US3 | Dual-Path Dashboard Health Analysis | P1 | 4, 5 |
|
||||
| US4 | URL Parsing Integration | P1 | 6 |
|
||||
| US5 | Provider/Prompt Per-Task + Settings Cleanup | P2 | 7 |
|
||||
| US6 | Dashboard Hub Cleanup | P2 | 8 |
|
||||
| US7 | Dataset Documentation (unchanged) | P2 | — |
|
||||
| US8 | Git Commit Suggestion (unchanged) | P3 | — |
|
||||
|
||||
## Dependencies
|
||||
|
||||
- **Phase 1 (Setup)**: Must run first.
|
||||
- **Phase 2 (Foundational)**: Must complete before any User Stories.
|
||||
- **Phase 3 (US1 - Dashboard Analysis)**: Depends on Phase 2.
|
||||
- **Phase 4 (US2 - Documentation)**: Depends on Phase 2. Parallel with Phase 3.
|
||||
- **Phase 5 (US3 - Configuration)**: Depends on Phase 2. Recommended before Phase 3/4 for easier testing, but technically parallelizable if config is mocked.
|
||||
- **Phase 6 (US4 - Git Integration)**: Independent of other stories, depends on Phase 2.
|
||||
```
|
||||
Phase 1 (Setup)
|
||||
└─→ Phase 2 (Foundational)
|
||||
├─→ Phase 3 (US1 + US2) — Task CRUD
|
||||
│ └─→ Phase 7 (US5) — Provider/Prompt
|
||||
├─→ Phase 4 (US3 Path B) — Text-only execution
|
||||
├─→ Phase 5 (US3 Path A) — Multi-chunk screenshots
|
||||
├─→ Phase 6 (US4) — URL parsing
|
||||
└─→ Phase 8 (US6) — Hub cleanup (independent)
|
||||
└─→ Phase 9 (Polish)
|
||||
```
|
||||
|
||||
## Phase 1: Setup
|
||||
**Parallel**: Phase 4 + 5 + 6 can run concurrently after Phase 2. Phase 7 is tight with Phase 3. Phase 8 is independent.
|
||||
|
||||
**Goal**: Initialize project structure and dependencies.
|
||||
---
|
||||
|
||||
- [x] T001 Install backend dependencies (openai, playwright, tenacity) in `backend/requirements.txt`
|
||||
- [x] T002 Create plugin directory structure `backend/src/plugins/llm_analysis/`
|
||||
- [x] T003 Initialize `backend/src/plugins/llm_analysis/__init__.py`
|
||||
- [x] T004 Create `backend/src/plugins/llm_analysis/models.py` for Pydantic models (LLMProviderConfig, ValidationResult)
|
||||
- [x] T005 Update `backend/src/core/plugin_loader.py` to recognize new plugin type if necessary
|
||||
- [x] T006 Create `backend/src/api/routes/llm.py` placeholder
|
||||
- [x] T007 Register new route in `backend/src/app.py`
|
||||
## Phase 1: Setup — Database Migrations
|
||||
|
||||
## Phase 2: Foundational
|
||||
**Goal**: Schema changes for v2 models. Blocks everything.
|
||||
|
||||
**Goal**: Implement core services and shared infrastructure.
|
||||
- [ ] T001 [P] Add `sources` ORM relationship + `source_snapshot` JSON (audit only, nullable) to `ValidationPolicy` in `backend/src/models/llm.py`. No JSON denormalization column for authoritative state — relational `ValidationSource` table is the single source of truth.
|
||||
- [ ] T002 [P] Create `ValidationSource` SQLAlchemy model in `backend/src/models/llm.py` (id, policy_id FK, type, value, parsed_context JSON, is_valid, title, last_error)
|
||||
- [ ] T003 [P] Add new columns to `ValidationPolicy` model: `provider_id` (String), `prompt_template` (Text, nullable), `screenshot_enabled` (Boolean, default true), `logs_enabled` (Boolean, default true), `execute_chart_data` (Boolean, default false), `llm_batch_size` (Integer, default 1 — full isolation), `policy_dashboard_concurrency_limit` (Integer, default 3) in `backend/src/models/llm.py`
|
||||
- [ ] T003a [P] Add `global_validation_worker_limit` config setting (default: 3) — max concurrent policy runs across all tasks. Stored in app config, not per-policy. in `backend/src/core/config_models.py`
|
||||
- [ ] T004 [P] Add new columns to `ValidationRecord` model: `execution_path` (String), `source_id` (FK → ValidationSource, nullable), `dataset_health` (JSON, nullable), `chart_data_results` (JSON, nullable), `tab_screenshots` (JSON, nullable), `token_usage` (JSON, nullable), `dataset_key` (String, nullable) in `backend/src/models/llm.py`
|
||||
- [ ] T004a [P] Create `ValidationRun` SQLAlchemy model in `backend/src/models/llm.py`: id, policy_id (FK), task_id (nullable), started_at, finished_at, trigger (manual/scheduled), status (completed/partial/failed), dashboard_count, pass_count, warn_count, fail_count, unknown_count
|
||||
- [ ] T005 Create Alembic migration for all T001–T004a schema changes in `backend/alembic/versions/`
|
||||
- [ ] T006 Write data migration script: convert existing `dashboard_ids: list[str]` → `sources: [{type: "dashboard_id", value: id, is_valid: true}]` in `backend/src/plugins/llm_analysis/migrations/`
|
||||
- [ ] T007 [P] Add `DEFAULT_LLM_PROMPTS["dashboard_validation_prompt_multimodal"]` (Path A, image-focused) and `DEFAULT_LLM_PROMPTS["dashboard_validation_prompt_text"]` (Path B, topology-focused) to `backend/src/services/llm_prompt_templates.py`
|
||||
|
||||
- [x] T008 Implement `LLMProviderService` in `backend/src/services/llm_provider.py` (CRUD for providers, AES-256 encryption)
|
||||
- [x] T009 Implement `ScreenshotService` in `backend/src/plugins/llm_analysis/service.py` (Playwright + API strategies, fallback logic, 1920px width, full page height, CDP screenshot, recursive tab switching)
|
||||
- [x] T010 Implement `LLMClient` in `backend/src/plugins/llm_analysis/service.py` (OpenAI SDK wrapper, retry logic with tenacity, rate limit handling)
|
||||
- [x] T011 Create `backend/src/plugins/llm_analysis/plugin.py` with `PluginBase` implementation stubs
|
||||
- [x] T012 Define database schema updates for `LLMProviderConfig` in `backend/src/models/llm.py` (or appropriate location)
|
||||
- [x] T013 Run migration to create new tables (if using SQLAlchemy/Alembic) or update SQLite schema
|
||||
**Verification**: `pytest backend/tests/models/test_llm_v2.py -v` — all models create/query correctly. Migration runs without data loss.
|
||||
|
||||
## Phase 3: Dashboard Health Analysis (US1)
|
||||
---
|
||||
|
||||
**Goal**: Enable automated dashboard validation with multimodal analysis.
|
||||
## Phase 2: Foundational — Services & Refactored Plugin Core
|
||||
|
||||
- [x] T014 [US1] Implement `validate_dashboard` task logic in `backend/src/plugins/llm_analysis/plugin.py`
|
||||
- [x] T015 [US1] Implement log retrieval logic (fetch recent logs, limit 100 lines/24h) in `backend/src/plugins/llm_analysis/plugin.py`
|
||||
- [x] T016 [US1] Construct multimodal prompt (image + text) in `backend/src/plugins/llm_analysis/service.py` (implement PII/credential filtering)
|
||||
- [x] T017 [US1] Implement result parsing and persistence (ValidationResult) in `backend/src/plugins/llm_analysis/plugin.py` (ensure JSON structure: status, issues, summary, log results to task output with `[ANALYSIS_SUMMARY]` and `[ANALYSIS_ISSUE]` markers)
|
||||
- [x] T018 [US1] Add `validate` endpoint trigger in `backend/src/api/routes/tasks.py` (or reuse existing dispatch)
|
||||
- [x] T019 [US1] Implement notification dispatch (Email/Pulse) on failure in `backend/src/plugins/llm_analysis/plugin.py` (Summary + Link format)
|
||||
- [x] T020 [US1] Create `frontend/src/components/llm/ValidationReport.svelte` for viewing results
|
||||
- [x] T021 [US1] Add "Validate" button with loading state and disabling logic to `frontend/src/components/DashboardGrid.svelte` (FR-019)
|
||||
- [x] T022 [US1] Enable scheduling for validation tasks in `backend/src/core/scheduler.py` (if not automatic via TaskManager)
|
||||
**Goal**: `ValidationTaskService`, `DatasetHealthChecker`, refactored `ScreenshotService` (multi-chunk), `LLMClient` dual-mode. Blocks US1–US6.
|
||||
|
||||
## Phase 4: Automated Dataset Documentation (US2)
|
||||
### 2.1 ValidationTaskService
|
||||
|
||||
**Goal**: Generate and persist dataset documentation.
|
||||
- [ ] T008 [P] Implement `ValidationTaskService.create_task()` — creates `ValidationPolicy` + `ValidationSource[]` rows, validates provider_id against execution path, in `backend/src/services/validation_service.py`
|
||||
- [ ] T009 [P] Implement `ValidationTaskService.get_task(id)` — returns policy + sources with current status, in `backend/src/services/validation_service.py`
|
||||
- [ ] T010 [P] Implement `ValidationTaskService.list_tasks(env_id)` — returns all policies with aggregated last-run status, in `backend/src/services/validation_service.py`
|
||||
- [ ] T011 [P] Implement `ValidationTaskService.update_task(id, data)` — updates policy fields + sources, in `backend/src/services/validation_service.py`
|
||||
- [ ] T012 [P] Implement `ValidationTaskService.delete_task(id)` — deletes policy + cascades to sources + records, in `backend/src/services/validation_service.py`
|
||||
- [ ] T013 [P] Implement `ValidationTaskService.trigger_run(policy_id)` — checks FR-054 (same task not already running via ValidationRun with status=running), creates a `ValidationRun` record (FR-055d), enqueues dashboard validation tasks per FR-055 (FIFO with `global_validation_worker_limit`), in `backend/src/services/validation_service.py`
|
||||
- [ ] T014 [P] Implement `ValidationTaskService.get_run_history(policy_id, page)` — paginated `ValidationRun` list with aggregate counts, in `backend/src/services/validation_service.py`
|
||||
- [ ] T015 [P] Implement `ValidationTaskService.get_run_detail(run_id)` — full run with all dashboard records, aggregate counts, in `backend/src/services/validation_service.py`
|
||||
- [ ] T015a [P] Implement `ValidationTaskService._finalize_run(run_id)` — computes aggregate counts from all `ValidationRecord`s with this `run_id`, sets finished_at and status, in `backend/src/services/validation_service.py`
|
||||
|
||||
- [x] T023 [US2] Implement `generate_documentation` task logic in `backend/src/plugins/llm_analysis/plugin.py`
|
||||
- [x] T024 [US2] Implement metadata fetching (schema, columns) in `backend/src/plugins/llm_analysis/plugin.py`
|
||||
- [x] T025 [US2] Construct documentation prompt in `backend/src/plugins/llm_analysis/service.py` (handle schema changes)
|
||||
- [x] T026 [US2] Implement metadata update logic (write back to DB) in `backend/src/services/mapping_service.py` (handle partial failures with rollback, ignore invalid columns)
|
||||
- [x] T045 [US2] Implement optimistic locking for concurrent metadata updates
|
||||
- [x] T027 [US2] Add "Generate Docs" button to `frontend/src/components/tools/MapperTool.svelte` using standard system icons (FR-021)
|
||||
- [x] T028 [US2] Create feedback/preview UI component `frontend/src/components/llm/DocPreview.svelte` (optional but recommended)
|
||||
### 2.2 DatasetHealthChecker (Path B)
|
||||
|
||||
## Phase 5: LLM Provider Configuration (US3)
|
||||
- [ ] T016 [P] Implement `DatasetHealthChecker` class in `backend/src/plugins/llm_analysis/service.py`:
|
||||
- `check_dataset_health(dataset_id: int, client: SupersetClient) → DatasetHealth` — calls `GET /api/v1/dataset/{id}`, extracts database name, backend, kind, verifies accessibility
|
||||
- `check_dashboard_datasets(dashboard_id: int, chart_list: list[dict], client: SupersetClient, execute_data: bool) → tuple[list[DatasetHealth], list[ChartDataResult]]`
|
||||
|
||||
**Goal**: Manage LLM providers via UI.
|
||||
### 2.3 ScreenshotService Refactor (multi-chunk for Path A)
|
||||
|
||||
- [x] T029 [US3] Implement CRUD endpoints for providers in `backend/src/api/routes/llm.py`
|
||||
- [x] T030 [US3] Create `frontend/src/components/llm/ProviderConfig.svelte` form
|
||||
- [x] T031 [US3] Create `frontend/src/routes/admin/settings/llm/+page.svelte` settings page
|
||||
- [x] T032 [US3] Implement "Test Connection" button logic with Toast feedback (FR-020)
|
||||
- [x] T033 [US3] Ensure API keys are masked in frontend responses
|
||||
- [ ] T017 [P] Implement `ScreenshotService.capture_dashboard_chunks(dashboard_id, parsed_filters, output_dir) → list[{tab_name, path}]` in `backend/src/plugins/llm_analysis/service.py`:
|
||||
- Login → navigate with filters/tabs → iterate tabs → per-tab viewport (1920×1200) → CDP screenshot → collect PNG paths
|
||||
|
||||
## Phase 6: Git Commit Message Suggestion (US4)
|
||||
### 2.3b Image Pipeline (PNG → JPEG + WebP)
|
||||
|
||||
**Goal**: AI-assisted commit messages.
|
||||
- [ ] T017a [P] Implement `ScreenshotService._convert_screenshots_for_llm(png_paths: list[str]) → list[str]` in `backend/src/plugins/llm_analysis/service.py`:
|
||||
- PNG → Pillow convert → JPEG quality=60, max 1024px → save as temp JPEG → return paths
|
||||
- JPEG files deleted after LLM call succeeds
|
||||
- [ ] T017b [P] Implement `ScreenshotService._archive_screenshots_as_webp(png_paths: list[str], output_dir: str) → list[{tab_name, webp_path}]` in `backend/src/plugins/llm_analysis/service.py`:
|
||||
- PNG → Pillow convert → WebP quality=80, lossy → save to `{output_dir}/{tab_name}_{timestamp}.webp`
|
||||
- Delete PNG after WebP saved
|
||||
- Fallback: if WebP encode fails → keep PNG, log WARN (FR-059 edge case)
|
||||
- [ ] T017c [P] Implement `ScreenshotService._cleanup_temp_files(paths: list[str])` in `backend/src/plugins/llm_analysis/service.py`:
|
||||
- Delete PNG and JPEG intermediates after successful archive + LLM call
|
||||
|
||||
- [x] T034 [US4] Implement `generate_commit_message` logic in `backend/src/plugins/git/llm_extension.py` (validate conventional commit format)
|
||||
- [x] T035 [US4] Create endpoint `POST /api/git/generate-message` in `backend/src/api/routes/git.py`
|
||||
- [x] T036 [US4] Construct commit generation prompt (Diff + History) in `backend/src/plugins/git/llm_extension.py`
|
||||
- [x] T037 [US4] Add "Generate" button to `frontend/src/components/git/CommitModal.svelte`
|
||||
- [x] T038 [US4] Wire up frontend to call generation endpoint and populate textarea (handle empty/invalid response with Toast)
|
||||
### 2.4 LLMClient Dual-Mode
|
||||
|
||||
## Phase 7: Polish & Cross-Cutting
|
||||
- [ ] T018 [P] Implement `LLMClient.analyze_dashboard_multimodal(screenshot_paths: list[str], logs: list[str], prompt: str) → dict` in `backend/src/plugins/llm_analysis/service.py`:
|
||||
- Sends all images in single `content[]` array → multimodal LLM → JSON result
|
||||
- [ ] T019 [P] Implement `LLMClient.analyze_dashboard_text(topology_text: str, dataset_health: list[dict], logs: list[str], prompt: str) → dict` in `backend/src/plugins/llm_analysis/service.py`:
|
||||
- Text-only LLM call with dashboard topology → JSON result
|
||||
- [ ] T020 [P] Implement `LLMClient._estimate_payload_size()` and `LLMClient._reduce_image_quality()` in `backend/src/plugins/llm_analysis/service.py`:
|
||||
- FR-056: estimate → progressive reduction (JPEG quality 60→30, width 1024→800) → Path B fallback
|
||||
- **Belief-runtime**: `belief_scope("payload_reduction")` + `reason()` before each reduction + `reflect()` after outcome
|
||||
|
||||
**Goal**: Finalize and verify.
|
||||
### 2.5 Plugin Refactor (dispatch to Path A / Path B)
|
||||
|
||||
- [x] T039 Verify all permissions (`plugin:llm:validate`, etc.) are registered and enforceable
|
||||
- [x] T040 Verify i18n strings for all new UI components
|
||||
- [x] T041 Run full end-to-end test of Dashboard Validation flow (including fallback scenarios)
|
||||
- [x] T042 Run full end-to-end test of Documentation flow
|
||||
- [x] T043 Update `README.md` with new plugin capabilities
|
||||
- [x] T044 Verify API key masking in all UI responses and logs
|
||||
- [ ] T021 Implement `DashboardValidationPlugin._build_dashboard_topology(dashboard_data: dict, charts_data: list[dict]) → str` in `backend/src/plugins/llm_analysis/plugin.py`:
|
||||
- Recursive `position_json` parsing → tab → row → chart hierarchy → text description
|
||||
- [ ] T022 Refactor `DashboardValidationPlugin.execute()` to dispatch based on `screenshot_enabled`:
|
||||
- True → `_execute_path_a()`: call ScreenshotService → LLMClient.analyze_dashboard_multimodal()
|
||||
- False → `_execute_path_b()`: call API chain → DatasetHealthChecker → LLMClient.analyze_dashboard_text()
|
||||
- **Belief-runtime**: `belief_scope("execute_path_a")` / `belief_scope("execute_path_b")` + `reason()` at entry + `reflect()` at exit
|
||||
|
||||
## Implementation Strategy
|
||||
### 2.7 Security & Retention (new)
|
||||
|
||||
1. **MVP**: Focus on Phase 1, 2, 3, and 5. This delivers the core value (Dashboard Validation) and the necessary configuration.
|
||||
2. **Increment 1**: Add Phase 4 (Documentation).
|
||||
3. **Increment 2**: Add Phase 6 (Git Integration).
|
||||
- [ ] T024a [P] Implement `RedactionService` in `backend/src/plugins/llm_analysis/service.py`:
|
||||
- `redact_logs(logs: list[str]) → list[str]` — strip PII, credentials, query text, tokens from URLs, native filter values
|
||||
- `redact_raw_response(raw: str) → str` — strip sensitive data before persistence
|
||||
- Applied at capture boundary (before LLM send + before DB save) per FR-029/FR-029a
|
||||
- [ ] T024b [P] Add retention config to app settings: `LLM_SCREENSHOT_RETENTION_DAYS=30`, `LLM_RAW_RESPONSE_RETENTION_DAYS=30` in `backend/src/core/config_models.py`
|
||||
- [ ] T024c [P] Rename all «molecular-cot» references in logs to «decision audit logging»: REASON/REFLECT markers use structured JSON with deterministic metadata only (no free-form reasoning text, no PII, no screenshot content) per FR-029b. Update docstrings in `backend/src/core/logger.py` and skill files.
|
||||
|
||||
## Parallel Execution Examples
|
||||
- [ ] T023 [P] Add `dashboard_validation_prompt_multimodal` (Path A) template to `DEFAULT_LLM_PROMPTS` in `backend/src/services/llm_prompt_templates.py`:
|
||||
- «Analyze the attached {N} dashboard tab screenshots and execution logs for visual and data health issues...»
|
||||
- [ ] T024 [P] Add `dashboard_validation_prompt_text` (Path B) template to `DEFAULT_LLM_PROMPTS` in `backend/src/services/llm_prompt_templates.py`:
|
||||
- «Analyze the following dashboard topology, dataset health, and execution logs for data consistency and KXD connectivity issues...»
|
||||
|
||||
- **Backend/Frontend Split**: T020 (Frontend Report) can be built while T014-T017 (Backend Logic) are in progress, using mock data.
|
||||
- **Story Parallelism**: US2 (Documentation) and US4 (Git) are largely independent and can be worked on simultaneously by different developers (or sequentially without blocking).
|
||||
**Verification**: `pytest backend/tests/services/test_validation_task_service.py backend/tests/plugins/llm_analysis/test_service.py -v`
|
||||
|
||||
---
|
||||
|
||||
## Phase 3: US1 + US2 — Task CRUD API + Task List Page
|
||||
|
||||
**Goal**: REST API for validation tasks + frontend task list and create/edit form.
|
||||
|
||||
### 3.1 API Routes
|
||||
|
||||
- [ ] T025 [US1] Implement `POST /api/validation-tasks` — creates `ValidationPolicy` + `ValidationSource[]` in `backend/src/api/routes/validation_tasks.py`
|
||||
- [ ] T026 [US1] Implement `GET /api/validation-tasks` — list all tasks with aggregated last-run status in `backend/src/api/routes/validation_tasks.py`
|
||||
- [ ] T027 [US2] Implement `GET /api/validation-tasks/{policy_id}` — task detail + per-source status in `backend/src/api/routes/validation_tasks.py`
|
||||
- [ ] T028 [US1] Implement `PUT /api/validation-tasks/{policy_id}` — update task in `backend/src/api/routes/validation_tasks.py`
|
||||
- [ ] T029 [US1] Implement `DELETE /api/validation-tasks/{policy_id}` — delete task (cascade sources + records), in `backend/src/api/routes/validation_tasks.py`
|
||||
- [ ] T030 [US1] Implement `POST /api/validation-tasks/{policy_id}/run` — manual trigger (checks FR-054 + FR-055) in `backend/src/api/routes/validation_tasks.py`
|
||||
- [ ] T031 [US2] Implement `GET /api/validation-tasks/{policy_id}/runs` — paginated run history (ValidationRun list, not individual records) in `backend/src/api/routes/validation_tasks.py`
|
||||
- [ ] T032 [US2] Implement `GET /api/validation-tasks/{policy_id}/runs/{run_id}` — full run detail: run metadata + all per-dashboard ValidationRecords with dataset_health, screenshots, logs, timings, in `backend/src/api/routes/validation_tasks.py`
|
||||
- [ ] T032b Implement legacy redirect: `GET /reports/llm/{taskId}` → 302 to `/validation-tasks/{policy_id}/runs/{run_id}` when `taskId` maps to `ValidationRun.task_id`, in `backend/src/api/routes/validation_tasks.py`
|
||||
|
||||
### 3.2 Task List Page
|
||||
|
||||
- [ ] T035 [US2] Create `frontend/src/routes/validation-tasks/+page.svelte` — task list page with columns: name, schedule, last run status (PASS/WARN/FAIL/UNKNOWN), dashboard count, status (active/inactive), actions (edit/delete/run now)
|
||||
- [ ] T036 [US2] Create `frontend/src/routes/validation-tasks/+page.js` — load function calling `GET /api/validation-tasks`
|
||||
|
||||
### 3.3 Task Create/Edit Form
|
||||
|
||||
- [ ] T037 [US1] Create `frontend/src/routes/validation-tasks/new/+page.svelte` — task creation page
|
||||
- [ ] T038 [US1] Create `frontend/src/components/llm/ValidationTaskForm.svelte` — multi-step form component:
|
||||
- Step 1: Name, description, environment selector
|
||||
- Step 2: Sources — dashboard ID multi-select + URL paste tabs
|
||||
- Step 3: Analysis options — provider selector (filtered per FR-050), prompt textarea (pre-filled with path-specific default: multimodal when screenshot_enabled=true, text when false), screenshot toggle, logs toggle, execute_chart_data toggle (Path B only). When toggling screenshot_enabled on a task with custom prompt → show warning «Prompt may no longer match execution path» (FR-048a)
|
||||
- Step 4: Schedule — cron input + preview next runs
|
||||
- [ ] T039 [US1] Create `frontend/src/routes/validation-tasks/[policyId]/edit/+page.svelte` — task edit page (reuses `ValidationTaskForm`)
|
||||
|
||||
### 3.5 Task Detail Page + Report Page
|
||||
|
||||
- [ ] T040 [US2] Create `frontend/src/routes/validation-tasks/[policyId]/+page.svelte` — task detail page:
|
||||
- Task info (name, environment, schedule, provider)
|
||||
- Per-source breakdown table: dashboard title, last status, issues count, last run timestamp, link to latest run
|
||||
- Run history table: timestamp, trigger type, aggregate status (N PASS / M WARN / K FAIL), duration, link to run detail
|
||||
- Actions: Run Now, Edit, Delete
|
||||
- [ ] T041 [US2] Create `frontend/src/routes/validation-tasks/[policyId]/+page.js` — load function calling `GET /api/validation-tasks/{id}` + `GET /api/validation-tasks/{id}/runs`
|
||||
- [ ] T041a [US2] Create `frontend/src/routes/validation-tasks/[policyId]/runs/[runId]/+page.svelte` — multi-dashboard run report page (FR-055b):
|
||||
- Run header: policy name, timestamp, trigger, duration, aggregate badge (e.g. «3 PASS • 1 WARN • 0 FAIL»)
|
||||
- Collapsible dashboard list: each row = one dashboard, shows status icon + issue count + mini-summary
|
||||
- Expanded dashboard row: full issues list + execution path badge (Path A/B) + dataset health table (Path B) + screenshot thumbnails with tab selector (Path A) + logs sent to LLM + task execution logs
|
||||
- Handles mixed runs: some dashboards Path A, some Path B — renders appropriate sections per dashboard
|
||||
- [ ] T041b Implement `GET /api/validation-tasks/{policy_id}/runs/{run_id}` — returns full ValidationRun with all dashboard records, each containing dataset_health, chart_data_results, tab_screenshots, token_usage, timings. Also resolves policy name, provider name, and dashboard titles for display.
|
||||
|
||||
### 3.6 Shared Report Component
|
||||
|
||||
- [ ] T042 [US1] Create `frontend/src/components/llm/ValidationTaskReport.svelte` — shared report component:
|
||||
- Check result badge (PASS/WARN/FAIL) with `getCheckResultClasses()`
|
||||
- Summary text
|
||||
- Issues table (severity, message, location)
|
||||
- Execution path indicator (Path A icon / Path B icon)
|
||||
- Placeholder slots for Path A (screenshots) and Path B (dataset health)
|
||||
|
||||
**Verification**: Create task via UI → list shows task → detail shows sources → run now → report renders.
|
||||
|
||||
---
|
||||
|
||||
## Phase 4: US3 — Path B Execution (Text-Only + Batching)
|
||||
|
||||
**Goal**: Text-only validation with dataset health checking, optional chart data execution, and LLM batching.
|
||||
|
||||
- [ ] T043 [US3] Implement `DashboardValidationPlugin._execute_path_b(run_id, dashboard_sources, params, context)` in `backend/src/plugins/llm_analysis/plugin.py`:
|
||||
1. For each dashboard: collect topology via `_build_dashboard_topology()` + dataset health via `DatasetHealthChecker` + logs
|
||||
2. Default: `llm_batch_size=1` — one LLM call per dashboard (full isolation). Values >1 group dashboards into batches per FR-045a (experimental, documented accuracy risk in research decision 15). Path A dashboards excluded from batching
|
||||
3. For each batch: build structured prompt with per-dashboard labeled sections + strict JSON response schema `{dashboards: [{dashboard_id, status, summary, issues}]}`
|
||||
4. Call `LLMClient.analyze_dashboard_text_batch(dashboard_payloads, prompt)` → parse response
|
||||
5. Missing/parse-error dashboards → mark UNKNOWN individually, preserve other dashboards in batch
|
||||
6. Persist individual `ValidationRecord` per dashboard, linked to `run_id`
|
||||
- **Belief-runtime**: `belief_scope("execute_path_b_batch")` + `reason(f"batch {i}/{total}: {n} dashboards")` + `reflect()` with token_usage per batch
|
||||
- [ ] T043a [US3] Implement `LLMClient.analyze_dashboard_text_batch(payloads: list[dict], prompt: str) → dict` in `backend/src/plugins/llm_analysis/service.py`:
|
||||
- Builds prompt: per-dashboard labeled sections, enforced `{dashboards: [{dashboard_id, ...}]}` JSON schema
|
||||
- Validates response completeness: every input dashboard_id must appear; missing → UNKNOWN fallback per-dashboard (not whole batch)
|
||||
- [ ] T044 [US3] Create `frontend/src/components/llm/DatasetHealthTable.svelte`
|
||||
- [ ] T045 [US3] Create `frontend/src/components/llm/ChartDataResults.svelte`
|
||||
|
||||
**Rejected-path regression test**:
|
||||
- [ ] T047 [US3] Add test: Path B batch of 5 dashboards, 1 with KXD error → only that dashboard UNKNOWN, other 4 preserved with valid results, in `backend/tests/plugins/llm_analysis/test_path_b_batch.py`
|
||||
- [ ] T047a [US3] Add test: batch_size=1 (full isolation) → each dashboard gets individual LLM call with no cross-contamination, in `backend/tests/plugins/llm_analysis/test_path_b_isolated.py`
|
||||
|
||||
**Verification**: Create task with 5 Path B dashboards, `llm_batch_size=1` (default) → 5 individual LLM calls, full isolation. Opt-in `llm_batch_size=3` → 2 LLM calls, structured prompt with per-dashboard isolation. Dashboard with broken dataset → only its record UNKNOWN, others preserved.
|
||||
|
||||
---
|
||||
|
||||
## Phase 5: US3 — Path A Execution (Multi-Chunk Screenshots)
|
||||
|
||||
**Goal**: Per-tab screenshots + multimodal LLM + payload estimation.
|
||||
|
||||
- [ ] T048 [US3] Implement `DashboardValidationPlugin._execute_path_a(params, context)` in `backend/src/plugins/llm_analysis/plugin.py`:
|
||||
1. Parse URL filters if source is URL type (re-parse per FR-057)
|
||||
2. `ScreenshotService.capture_dashboard_chunks()` → per-tab screenshots
|
||||
3. `GET /api/v1/log/` → execution logs
|
||||
4. `LLMClient._estimate_payload_size()` → check token budget per FR-056
|
||||
5. Quality reduction loop if needed (FR-056)
|
||||
6. `LLMClient.analyze_dashboard_multimodal()` → JSON result
|
||||
7. Fallback to `_execute_path_b()` if payload still exceeds limit
|
||||
8. Persist `ValidationRecord` with `execution_path="screenshot"`, `tab_screenshots`, `token_usage`
|
||||
- **Belief-runtime**: `belief_scope("execute_path_a")` + `reason()` before payload reduction + `reflect()` after each reduction + `reason()` on Path B fallback
|
||||
- [ ] T049 [US3] Update `frontend/src/routes/validation-tasks/[policyId]/runs/[runId]/+page.svelte` — add Path A sections: tab selector (from `tab_screenshots`), per-tab image viewer with zoom, issues table with tab/chart location. Legacy `/reports/llm/[taskId]` is a redirect-only passthrough — NOT updated.
|
||||
|
||||
**Rejected-path regression test**:
|
||||
- [ ] T050 [US3] Add test: multi-chunk screenshots exceed token limit → quality reduction triggered → Path B fallback when still exceeded, logged via molecular-cot, in `backend/tests/plugins/llm_analysis/test_payload_reduction.py`
|
||||
|
||||
**Verification**: Create task with `screenshot_enabled=true`, 5-tab dashboard → report shows per-tab screenshots + issues.
|
||||
|
||||
---
|
||||
|
||||
## Phase 6: US4 — URL Parsing Integration
|
||||
|
||||
**Goal**: Paste Superset URL → parse → use in task creation + execution.
|
||||
|
||||
- [ ] T051 [US4] Implement `POST /api/validation-tasks/parse-url` in `backend/src/api/routes/validation_tasks.py`:
|
||||
- Receives `{url, environment_id}`
|
||||
- Calls `SupersetContextExtractor.parse_superset_link(url)`
|
||||
- Returns `{dashboard_id, title, native_filters, activeTabs, anchor, partial_recovery, warnings}`
|
||||
- [ ] T052 [US4] Create `frontend/src/components/llm/UrlParser.svelte`:
|
||||
- URL paste input with debounced parse
|
||||
- Loading state during API call
|
||||
- Success: green banner with dashboard title, native_filters count, activeTabs list
|
||||
- Error: red banner with `partial_recovery` warnings or parse failure message
|
||||
- Copy button: «Dashboard structure changed? Paste new URL»
|
||||
- [ ] T053 [US4] Integrate `UrlParser` into `ValidationTaskForm.svelte` — add «Paste URL» tab in source selector (Step 2)
|
||||
- [ ] T054 [US4] Implement URL re-parse on execution per FR-057 in `DashboardValidationPlugin`:
|
||||
- For URL-type sources: call `SupersetContextExtractor.parse_superset_link()` on every run
|
||||
- If fails → mark source `is_valid=false`, set `last_error`, log WARN
|
||||
- If dashboard_id changed → log WARN, use new ID
|
||||
|
||||
**Rejected-path regression test**:
|
||||
- [ ] T055 [US4] Add test: URL source with deleted dashboard → re-parse fails → source marked invalid, error logged, task skips source and continues with other valid ones, in `backend/tests/plugins/llm_analysis/test_url_reparse.py`
|
||||
|
||||
**Verification**: Paste Superset URL with filters → form shows dashboard title + 3 native filters + 2 active tabs → create task → execution re-parses URL successfully.
|
||||
|
||||
---
|
||||
|
||||
## Phase 7: US5 — Provider/Prompt Per-Task + Settings Cleanup
|
||||
|
||||
**Goal**: Remove `dashboard_validation` binding from LLM settings. Provider + prompt configured per-task.
|
||||
|
||||
- [ ] T056 [US5] Remove `"dashboard_validation": ""` from `DEFAULT_LLM_PROVIDER_BINDINGS` in `backend/src/services/llm_prompt_templates.py`
|
||||
- [ ] T057 [US5] Remove dashboard_validation binding UI row from `frontend/src/routes/admin/settings/llm/+page.svelte`
|
||||
- [ ] T058 [US5] Update LLM settings API schema to exclude `dashboard_validation` from provider_bindings payload in `backend/src/schemas/llm.py`
|
||||
- [ ] T059 [US5] Implement provider selector filter logic in `ValidationTaskForm.svelte` (Step 3):
|
||||
- `screenshot_enabled=true` → show only multimodal providers (`is_multimodal=true`)
|
||||
- `screenshot_enabled=false` → show all active providers
|
||||
- [ ] T060 [US5] Add `ValidationTaskService.create_task()` validation: reject if `screenshot_enabled=true` but provider `is_multimodal=false`
|
||||
|
||||
**Rejected-path regression test**:
|
||||
- [ ] T061 [US5] Add test: create task with `screenshot_enabled=true` and non-multimodal provider → HTTP 400, in `backend/tests/api/test_validation_tasks_api.py`
|
||||
- [ ] T061a [US5] Implement `DELETE /api/llm/providers/{id}` enforcement: query active ValidationPolicy references, return 409 with `{error, blocking_tasks: [{id, name}]}` (FR-058), in `backend/src/api/routes/llm.py`
|
||||
- [ ] T062 [US5] Add test: delete provider with active tasks → HTTP 409 with blocking task list (FR-058), in `backend/tests/api/test_llm_provider_delete.py`
|
||||
- [ ] T063 [US5] Add test: POST /api/tasks with `plugin_id=llm_dashboard_validation` → HTTP 400 with migration message (FR hard cut), in `backend/tests/api/test_tasks_deprecated_llm.py`
|
||||
|
||||
**Verification**: LLM settings page shows only 3 bindings. Create task with multimodal provider → works. Create with non-multimodal + screenshot → rejected.
|
||||
|
||||
---
|
||||
|
||||
## Phase 8: US6 — Dashboard Hub Cleanup + Validation Indicator
|
||||
|
||||
**Goal**: Remove Validate button + LLM status column. Add lightweight validation indicator per dashboard row + detail page history.
|
||||
|
||||
- [ ] T064 [US6] Remove `handleValidate()` function from `frontend/src/routes/dashboards/+page.svelte`
|
||||
- [ ] T065 [US6] Remove LLM validation status column from dashboard hub grid (column header + `getLlmSummaryLabel()` + `getValidationBadgeClass()` rendering) in `frontend/src/routes/dashboards/+page.svelte`
|
||||
- [ ] T065a [US6] Implement `GET /api/dashboards/{env_id}/validation-status` in `backend/src/api/routes/dashboards.py` — batch endpoint accepting `dashboard_ids` list, returns `{dashboard_id: {status, last_run_at, task_name, run_id}}` per dashboard
|
||||
- [ ] T065b [US6] Add validation indicator dot (PASS/WARN/FAIL/UNKNOWN) next to dashboard title in hub rows. Click opens dropdown with last 3 runs (date, aggregate status across all dashboards in that run, task name) + «View all» link → `/validation-tasks/{policyId}/runs/{runId}`, in `frontend/src/routes/dashboards/+page.svelte`
|
||||
- [ ] T066 [US6] Remove «Validate» action from action dropdown menu in `frontend/src/routes/dashboards/+page.svelte`
|
||||
- [ ] T067 [US6] Deprecate (comment out) `normalizeValidationStatus()` + `getLlmSummaryLabel()` — replaced by indicator + batch endpoint, in `frontend/src/routes/dashboards/+page.svelte`
|
||||
- [ ] T068 [US6] Remove LLM validation status loading from `frontend/src/routes/dashboards/[id]/+page.svelte` (old `runLlmValidationTask()` + `openLlmReport()`)
|
||||
- [ ] T068a [US6] Add «Validation History» section to `frontend/src/routes/dashboards/[id]/+page.svelte` — last 10 runs for this dashboard across all tasks, links to canonical run detail pages
|
||||
- [ ] T069 [US6] Add «Create Validation Task» button to dashboard hub toolbar that navigates to `/validation-tasks/new?ids=1,2,3` (pre-fills selected dashboard IDs) in `frontend/src/routes/dashboards/+page.svelte`
|
||||
- [ ] T070 [US6] Update `frontend/src/lib/components/layout/TaskDrawer.svelte` — ensure LLM validation tasks still show correct status after refactor
|
||||
|
||||
**Verification**: Dashboard hub has no Validate button, no LLM column. Colored dot indicator per row with dropdown. Detail page has Validation History section. «Create Validation Task» button pre-fills IDs.
|
||||
|
||||
---
|
||||
|
||||
## Phase 9: Polish & Cross-Cutting Verification
|
||||
|
||||
**Goal**: Migration testing, backward compat, ADR updates, final verification.
|
||||
|
||||
### 9.1 Migration & Backward Compat
|
||||
|
||||
- [ ] T071 Run data migration on test DB: verify all existing `dashboard_ids` → `sources[]` conversion in `backend/tests/migrations/test_v2_data_migration.py`
|
||||
- [ ] T072 Update assistant tool `_tool_llm_validation.py` — replace ad-hoc `run_llm_validation` dispatch with task creation via `POST /api/validation-tasks`, in `backend/src/api/routes/assistant/_tool_llm_validation.py`
|
||||
|
||||
### 9.2 ADR Updates
|
||||
|
||||
- [ ] T073 Update `docs/adr/ADR-0008-assistant-tool-registry.md` — note that `_tool_llm_validation.py` now creates tasks instead of ad-hoc validation
|
||||
- [ ] T074 Update `docs/adr/ADR-0004-plugin-architecture.md` — mention v2 dual-path + task-based flow as plugin evolution example
|
||||
|
||||
### 9.3 Final Verification
|
||||
|
||||
- [ ] T075 Run full end-to-end: create task → URL source (with filters) → Path A run → view report with per-tab screenshots
|
||||
- [ ] T076 Run full end-to-end: create task → dashboard ID sources → Path B run (execute_chart_data=true) → view report with dataset health + chart data
|
||||
- [ ] T077 Run full end-to-end: create task → Path B run against dashboard with broken dataset → verify LLM reports UNHEALTHY
|
||||
- [ ] T078 Run full end-to-end: delete provider used by active task → verify HTTP 409 with blocking task names
|
||||
- [ ] T079 Run full test suite: `cd backend && source .venv/bin/activate && python -m pytest -v`
|
||||
- [ ] T080 Run frontend tests: `cd frontend && npm run test`
|
||||
- [ ] T081 Run backend lint: `cd backend && python -m ruff check .`
|
||||
- [ ] T082 Run frontend lint: `cd frontend && npm run lint`
|
||||
- [ ] T083 Run semantic audit: `axiom_semantic_validation audit_belief_protocol` on all new C4/C5 contracts
|
||||
- [ ] T084 Run semantic audit: `axiom_semantic_validation audit_contracts` on `backend/src/plugins/llm_analysis/` and `backend/src/services/validation_service.py`
|
||||
|
||||
---
|
||||
|
||||
## Verification Criteria per User Story
|
||||
|
||||
| Story | Independent Verification |
|
||||
|-------|-------------------------|
|
||||
| **US1** | Create task with 2 dashboards (ID + URL), Path A, cron `0 8 * * *` → task appears in list → scheduled execution succeeds → report shows per-tab screenshots |
|
||||
| **US2** | Task list shows 3 tasks with different statuses → click task → detail shows per-dashboard breakdown → click dashboard → full report renders |
|
||||
| **US3** | Path B against broken dataset → report shows dataset UNHEALTHY + affected charts. Path A against dashboard with visual error → report shows issue with tab/chart location |
|
||||
| **US4** | Paste Superset URL with native_filters → parse returns dashboard title + 3 filters + 2 tabs → saved in ValidationSource → re-parsed on execution |
|
||||
| **US5** | LLM settings shows 3 bindings (no dashboard_validation). Create task with multimodal provider → works. Create with non-multimodal + screenshot → rejected |
|
||||
| **US6** | Dashboard hub: no Validate button, no LLM column. «Create Validation Task» toolbar button pre-fills selected IDs |
|
||||
|
||||
## Rejected-Path Regression Coverage
|
||||
|
||||
| Rejected Path | Test Task | Expected Behavior |
|
||||
|---------------|-----------|-------------------|
|
||||
| Old `POST /api/tasks` with `llm_dashboard_validation` | T063 | HTTP 400: «deprecated, use /api/validation-tasks» |
|
||||
| Provider deletion when used by active tasks | T062 | HTTP 409: returns blocking task names |
|
||||
| Duplicate manual trigger of same task | T013 (FR-054 check) | Toast «Already running», no queue |
|
||||
| Non-multimodal provider + screenshot task | T061 | HTTP 400: «provider must be multimodal» |
|
||||
| URL re-parse fails (dashboard deleted) | T055 | Source marked invalid, task skips, continues with other sources |
|
||||
| Multi-chunk token limit exceeded | T050 | Quality reduction → Path B fallback, logged via molecular-cot |
|
||||
| Path B dataset health check fails | T047 | LLM report flags dataset UNHEALTHY + affected charts |
|
||||
|
||||
## Belief-Runtime Instrumentation Tasks
|
||||
|
||||
| Contract | Tier | Task | Instrumentation |
|
||||
|----------|------|------|-----------------|
|
||||
| `LLMClient._estimate_payload_size` + `_reduce_image_quality` | C4 | T020 | `belief_scope("payload_reduction")` + `reason()` per reduction + `reflect()` on outcome |
|
||||
| `DashboardValidationPlugin._execute_path_a` | C4 | T048 | `belief_scope("execute_path_a")` + `reason()` at navigation/tab/resize + `reflect()` with timings |
|
||||
| `DashboardValidationPlugin._execute_path_b` | C4 | T043 | `belief_scope("execute_path_b")` + `reason()` at each API call + `reflect()` with token_usage |
|
||||
| `ValidationTaskService.trigger_run` | C4 | T013 | `belief_scope("trigger_run")` + `reason()` on queue decision + `reflect()` after enqueue |
|
||||
| `ScreenshotService.capture_dashboard_chunks` | C4 | T017 | `belief_scope("capture_chunks")` + `reason()` per tab switch + `reflect()` with screenshot count |
|
||||
|
||||
## Parallel Execution Opportunities
|
||||
|
||||
| Group | Tasks | Dependencies |
|
||||
|-------|-------|--------------|
|
||||
| **Phase 1** | T001–T007 | None (all parallel) |
|
||||
| **Phase 2** | T008–T015 (service), T016 (health checker), T017 (screenshot), T018–T020 (LLM client) | After Phase 1 |
|
||||
| **Phase 3** | T025–T034 (API), T035–T042 (frontend) | After Phase 2 |
|
||||
| **Phase 4+5+6** | T043–T047 (Path B), T048–T050 (Path A), T051–T055 (URL) | After Phase 2, all three parallel |
|
||||
| **Phase 7+8** | T056–T063 (provider/prompt), T064–T070 (hub) | After Phase 3, both parallel |
|
||||
| **Phase 9** | T071–T084 | After all phases |
|
||||
|
||||
## Total: 98 tasks across 9 phases
|
||||
|
||||
| Phase | Tasks | User Stories |
|
||||
|-------|-------|-------------|
|
||||
| 1 — Setup | 8 | — |
|
||||
| 2 — Foundational | 23 | — |
|
||||
| 3 — Task CRUD + Reports | 22 | US1, US2 |
|
||||
| 4 — Path B | 5 | US3 |
|
||||
| 5 — Path A | 3 | US3 |
|
||||
| 6 — URL Parsing | 5 | US4 |
|
||||
| 7 — Provider/Prompt | 8 | US5 |
|
||||
| 8 — Hub Cleanup | 10 | US6 |
|
||||
| 9 — Polish | 14 | All |
|
||||
|
||||
Reference in New Issue
Block a user