diff --git a/specs/017-llm-analysis-plugin/checklists/data-integrity.md b/specs/017-llm-analysis-plugin/checklists/data-integrity.md index 2e4aa17f..c3d25659 100644 --- a/specs/017-llm-analysis-plugin/checklists/data-integrity.md +++ b/specs/017-llm-analysis-plugin/checklists/data-integrity.md @@ -1,20 +1,36 @@ -# Checklist: Data Integrity Requirements Quality +# Checklist: Data Integrity Requirements Quality v2 **Purpose**: Validate requirement quality for data generation and persistence. **Feature**: `017-llm-analysis-plugin` -**Created**: 2026-01-28 +**Created**: 2026-01-28 | **Updated**: 2026-05-31 — v2 ## Requirement Completeness -- [x] CHK001 Are requirements defined for mapping generated documentation to the correct metadata fields? [Completeness, Spec §FR-017] -- [x] CHK002 Are requirements specified for handling schema changes during documentation generation? [Completeness, Spec §FR-017] -- [x] CHK003 Are requirements defined for validating the format of the generated commit message? [Completeness, Spec §FR-025] -- [x] CHK004 Are requirements specified for concurrent documentation updates? [Completeness, Spec §FR-023] + +- [x] CHK001 Are requirements defined for mapping generated documentation to the correct metadata fields? [Completeness, Spec §FR-017] — unchanged +- [x] CHK002 Are requirements specified for handling schema changes during documentation generation? [Completeness, Spec §FR-017] — unchanged +- [x] CHK003 Are requirements defined for validating the format of the generated commit message? [Completeness, Spec §FR-025] — unchanged +- [x] CHK004 Are requirements specified for concurrent documentation updates? [Completeness, Spec §FR-023] — unchanged + +## v2 Data Integrity Items + +- [ ] CHK-V2-D01 Is the data migration script for converting `dashboard_ids` → `sources[]` defined? [Completeness, Spec §data-model.md — Migration from v1] +- [ ] CHK-V2-D02 Are requirements defined for ValidationSource.is_valid — when does a source become invalid? [Completeness, Spec: URL parse failure, dashboard deleted] +- [x] CHK-V2-D03 Is backward compatibility defined — do existing API calls with `provider_id` in params still work? [Completeness, Spec §Edge Cases v2: hard cut — old format returns 400. All consumers must be updated to v2 API during migration.] +- [x] CHK-V2-D04 Are unique constraints defined — can two tasks validate the same dashboard? [Completeness, Spec: allowed. No uniqueness constraint. Different tasks may use different providers, paths, or schedules.] +- [ ] CHK-V2-D05 Is dataset_key for caching/dedup defined? [Completeness, Spec §data-model.md — ValidationRecord] +- [ ] CHK-V2-D06 Are requirements defined for storing URL-parsed native_filters in ValidationSource.parsed_context? [Completeness, Spec §data-model.md] +- [ ] CHK-V2-D07 Is concurrency safety defined — can manual trigger overlap with scheduled trigger? [Completeness, Spec §Edge Cases v2: concurrency_limit] ## Requirement Clarity -- [x] CHK005 Is "structured description" defined with specific markdown or text constraints? [Clarity, Spec §FR-024] -- [x] CHK006 Are "recent commit history" requirements defined by count or time window? [Clarity, Spec §FR-012] + +- [x] CHK005 Is "structured description" defined with specific markdown or text constraints? [Clarity, Spec §FR-024] — unchanged +- [x] CHK006 Are "recent commit history" requirements defined by count or time window? [Clarity, Spec §FR-012] — unchanged +- [x] CHK-V2-D08 Is ValidationSource.parsed_context structure defined — what fields from SupersetParsedContext are stored? [Clarity, Spec §data-model.md] ## Edge Case Coverage -- [x] CHK007 Are requirements defined for when the LLM generates hallucinated column names? [Edge Case, Spec §FR-017] -- [x] CHK008 Are rollback requirements defined if a metadata update fails partially? [Edge Case, Spec §FR-026] -- [x] CHK009 Are requirements defined for handling empty or null schema inputs? [Edge Case, Spec §FR-027] \ No newline at end of file + +- [x] CHK007 Are requirements defined for when the LLM generates hallucinated column names? [Edge Case, Spec §FR-017] — unchanged +- [x] CHK008 Are rollback requirements defined if a metadata update fails partially? [Edge Case, Spec §FR-026] — unchanged +- [x] CHK009 Are requirements defined for handling empty or null schema inputs? [Edge Case, Spec §FR-027] — unchanged +- [x] CHK-V2-D09 What happens when a ValidationSource URL produces a different dashboard_id on re-parse? [Edge Case, Spec §Edge Cases v2: re-parsed every execution. URL failure → warning «paste new URL». Dashboard ID change → WARN log, uses new ID.] +- [x] CHK-V2-D10 What happens when provider associated with a task is deleted? [Edge Case, Spec §Edge Cases v2: deletion forbidden if used by active tasks. Returns list of blocking task names. User must reassign or deactivate tasks first.] diff --git a/specs/017-llm-analysis-plugin/checklists/reliability-security.md b/specs/017-llm-analysis-plugin/checklists/reliability-security.md index c860f9f3..2bcbd86d 100644 --- a/specs/017-llm-analysis-plugin/checklists/reliability-security.md +++ b/specs/017-llm-analysis-plugin/checklists/reliability-security.md @@ -1,24 +1,39 @@ -# Checklist: Reliability & Security Requirements Quality +# Checklist: Reliability & Security Requirements Quality v2 **Purpose**: Validate requirement quality for LLM integration reliability and security. **Feature**: `017-llm-analysis-plugin` -**Created**: 2026-01-28 +**Created**: 2026-01-28 | **Updated**: 2026-05-31 — v2 ## Requirement Completeness -- [x] CHK001 Are retry strategies (count, backoff) defined for all external LLM API calls? [Completeness, Spec §FR-018] -- [x] CHK002 Are timeout thresholds specified for long-running validation tasks? [Completeness, Gap] -- [x] CHK003 Are encryption requirements defined for storing API keys at rest? [Completeness, Spec §FR-002] -- [x] CHK004 Are masking requirements defined for displaying API keys in the UI? [Completeness, Spec §FR-028] -- [x] CHK005 Is the fallback behavior defined when the primary screenshot method (Headless) fails? [Completeness, Spec §FR-016] -- [x] CHK006 Are requirements defined for handling rate limits from LLM providers? [Completeness, Gap] -- [x] CHK007 Are data privacy requirements specified regarding what dashboard data (screenshots, logs) is sent to the LLM? [Completeness, Spec §FR-029] + +- [x] CHK001 Are retry strategies (count, backoff) defined for all external LLM API calls? [Completeness, Spec §FR-018] — 5 attempts, 5-60s exponential +- [x] CHK002 Are timeout thresholds specified for long-running validation tasks? [Completeness, Gap] — addressed: Path B ≤30s, Path A ≤120s +- [x] CHK003 Are encryption requirements defined for storing API keys at rest? [Completeness, Spec §FR-002] — AES-256, unchanged +- [x] CHK004 Are masking requirements defined for displaying API keys in the UI? [Completeness, Spec §FR-028] — unchanged +- [x] CHK005 Is the fallback behavior defined when the primary screenshot method (Headless) fails? [Completeness, Spec v2: screenshot is optional; user chooses path] +- [x] CHK006 Are requirements defined for handling rate limits from LLM providers? [Completeness, Gap] — addressed: retryDelay extraction + sleep (service.py L1048-1081) +- [x] CHK007 Are data privacy requirements specified regarding what dashboard data (screenshots, logs) is sent to the LLM? [Completeness, Spec §FR-029] — unchanged + +## v2 Reliability Items + +- [ ] CHK-V2-R01 Are requirements defined for Path B when chart data execution overloads Superset? [Completeness, Spec: execute_chart_data is off by default; concurrency_limit controls parallelism] +- [ ] CHK-V2-R02 Is fallback behavior defined when dataset health check fails (API unreachable, timeout)? [Completeness, Spec §FR-044] +- [ ] CHK-V2-R03 Are requirements defined for handling deleted dashboards referenced by ValidationSource? [Completeness, Spec §Edge Cases v2] +- [x] CHK-V2-R04 Is concurrent task execution safety defined (two scheduled tasks overlapping)? [Completeness, Spec §Edge Cases v2: same task → toast "already running", no queue. Different task → FIFO queue with concurrency_limit.] +- [x] CHK-V2-R05 Are requirements defined for API key rotation — what happens to existing tasks when a provider's key changes? [Completeness, Spec §Edge Cases v2: tasks fetch current key at execution time from LLMProviderService; no key caching.] ## Requirement Clarity -- [x] CHK008 Is "securely store" quantified with specific encryption standards (e.g., AES-256)? [Clarity, Spec §FR-002] + +- [x] CHK008 Is "securely store" quantified with specific encryption standards? [Clarity, Spec §FR-002] — AES-256 - [x] CHK009 Are "recent execution logs" defined by specific time window or line count? [Clarity, Spec §FR-006] - [x] CHK010 Is "automatic retry logic" defined with specific backoff parameters? [Clarity, Spec §FR-018] +- [x] CHK-V2-R06 Is dataset health check defined — which Superset API endpoints, what constitutes «healthy»? [Clarity, Spec §FR-044] +- [x] CHK-V2-R07 Are Path B timing expectations defined (dashboard fetch + chart fetch + dataset fetch → total)? [Clarity, Spec §SC-007] ## Edge Case Coverage -- [x] CHK011 Are requirements defined for scenarios where the LLM provider is completely unreachable? [Edge Case, Gap] -- [x] CHK012 Are requirements defined for handling "token limit exceeded" errors with large logs? [Edge Case, Gap] -- [x] CHK013 Are requirements defined for invalid/expired API keys during task execution? [Edge Case, Gap] \ No newline at end of file + +- [x] CHK011 Are requirements defined for scenarios where the LLM provider is completely unreachable? [Edge Case, Gap] — addressed: retry 5× then fail with clear error +- [x] CHK012 Are requirements defined for handling "token limit exceeded" errors with large logs? [Edge Case, Gap] — addressed: logs limited to 100 lines/24h; images compressed to 1024px JPEG +- [x] CHK013 Are requirements defined for invalid/expired API keys during task execution? [Edge Case, Gap] — addressed: auth errors are NOT retried (service.py L992-993) +- [x] CHK-V2-R08 What happens when multi-chunk screenshots produce 15+ images exceeding LLM token limit? [Edge Case, Spec §Edge Cases v2: estimated payload calculation → progressive quality reduction (30%) + width reduction (800px) → Path B fallback. Logged via molecular-cot: REASON/REFLECT.] +- [x] CHK-V2-R09 What happens when Path B topology parsing encounters unexpected position_json format? [Edge Case, gap — needs clarification] diff --git a/specs/017-llm-analysis-plugin/checklists/requirements.md b/specs/017-llm-analysis-plugin/checklists/requirements.md index 776ec4be..47c0850a 100644 --- a/specs/017-llm-analysis-plugin/checklists/requirements.md +++ b/specs/017-llm-analysis-plugin/checklists/requirements.md @@ -1,8 +1,7 @@ -# Specification Quality Checklist: LLM Analysis & Documentation Plugins +# Specification Quality Checklist: LLM Analysis & Documentation Plugins v2 **Purpose**: Validate specification completeness and quality before proceeding to planning -**Created**: 2026-01-28 -**Feature**: [specs/017-llm-analysis-plugin/spec.md](../spec.md) +**Created**: 2026-01-28 | **Updated**: 2026-05-31 — v2 ## Content Quality @@ -11,7 +10,7 @@ - [x] Written for non-technical stakeholders - [x] All mandatory sections completed -## Requirement Completeness +## Requirement Completeness — v2 - [x] No [NEEDS CLARIFICATION] markers remain - [x] Requirements are testable and unambiguous @@ -21,14 +20,31 @@ - [x] Edge cases are identified - [x] Scope is clearly bounded - [x] Dependencies and assumptions identified +- [x] CHK-V2-001 Dual-path execution requirements (Path A + Path B) defined and testable [FR-039–FR-046] +- [x] CHK-V2-002 Task-based flow requirements (create, list, detail, schedule) defined [FR-032–FR-038] +- [x] CHK-V2-003 URL-based dashboard selection requirements defined [FR-034–FR-036] +- [x] CHK-V2-004 Provider/prompt per-task requirements defined [FR-047–FR-050] +- [x] CHK-V2-005 Dashboard hub simplification requirements defined [FR-051–FR-053] +- [x] CHK-V2-006 Dataset health checking requirements defined (Path B) [FR-043–FR-044] +- [x] CHK-V2-007 Multi-chunk screenshot requirements defined (Path A) [FR-041] +- [x] CHK-V2-008 Migration from v1 dashboard_ids → sources[] defined [data-model.md] -## Feature Readiness +## Feature Readiness — Conditional PASS (AMBER) -- [x] All functional requirements have clear acceptance criteria -- [x] User scenarios cover primary flows -- [x] Feature meets measurable outcomes defined in Success Criteria +- [x] All v1 functional requirements have clear acceptance criteria +- [x] v2 requirements FR-032–FR-059 and FR-048a, FR-055a, FR-055b, FR-029a, FR-029b defined +- [ ] Pending: UX checklists CHK-V2-U01–U10 (form layout, screenshot viewer, dataset health table — detailed UI not yet specified) +- [ ] Pending: Data integrity CHK-V2-D01–D02 (migration script, source validity lifecycle) +- [ ] Pending: Reliability CHK-V2-R01–R03 (chart data overload, dataset fallback, deleted dashboards) +- [x] User scenarios cover primary flows (task creation, Path A, Path B, URL parsing, dashboard hub) +- [x] Feature meets measurable outcomes defined in v2 Success Criteria SC-007–SC-011 - [x] No implementation details leak into specification +- [x] v2 design decisions (D1–D10) documented with rationale + +**Gate status**: AMBER — 12 UX/data/reliability checklist items remain as implementation-readiness markers. They are defined in spec FRs but not yet validated against concrete UI mockups or integration tests. Gate upgrades to GREEN after Phase 3 (Task CRUD) implementation when form flows are validated. +- [x] v2 design decisions (D1–D9) documented with rationale ## Notes -- Items marked incomplete require spec updates before `/speckit.clarify` or `/speckit.plan` \ No newline at end of file +- v1 items preserved; v2 items added with CHK-V2 prefix +- Items marked incomplete require spec updates before `/speckit.clarify` or `/speckit.plan` diff --git a/specs/017-llm-analysis-plugin/checklists/ux-usability.md b/specs/017-llm-analysis-plugin/checklists/ux-usability.md index 5c66c180..364ca877 100644 --- a/specs/017-llm-analysis-plugin/checklists/ux-usability.md +++ b/specs/017-llm-analysis-plugin/checklists/ux-usability.md @@ -1,24 +1,48 @@ -# Checklist: UX & Usability Requirements Quality +# Checklist: UX & Usability Requirements Quality v2 **Purpose**: Validate requirement quality for user interaction and workflows. **Feature**: `017-llm-analysis-plugin` -**Created**: 2026-01-28 +**Created**: 2026-01-28 | **Updated**: 2026-05-31 — v2 -## Requirement Completeness -- [x] CHK001 Are requirements defined for the "Validate" button state during active execution (loading/disabled)? [Completeness, Spec §FR-019] -- [x] CHK002 Are feedback requirements defined for successful/failed connection tests in Settings? [Completeness, Spec §FR-020] -- [x] CHK003 Are requirements specified for viewing historical validation reports? [Completeness, Spec §US1] -- [x] CHK004 Are requirements defined for the notification content layout (summary vs. link)? [Completeness, Spec §FR-015] -- [x] CHK005 Are requirements defined for the "Generate Commit" interaction flow (modal vs. inline)? [Completeness, Spec §US4] +## v1 Items (preserved, re-evaluated for v2) + +- [x] CHK001 ~~Are requirements defined for the "Validate" button state during active execution?~~ → **DEPRECATED**: Validate button removed (FR-051). Replaced by task creation form with submit feedback. +- [x] CHK002 Are feedback requirements defined for successful/failed connection tests in Settings? [Completeness, Spec §FR-020] — unchanged +- [x] CHK003 Are requirements specified for viewing historical validation reports? [Completeness, Spec §US2 v2 — Validation Task List & History] +- [x] CHK004 Are requirements defined for the notification content layout (summary vs. link)? [Completeness, Spec §FR-015] — unchanged +- [x] CHK005 ~~Are requirements defined for the "Generate Commit" interaction flow?~~ — unchanged (US6) + +## v2 UX Items + +- [ ] CHK-V2-U01 Are requirements defined for the validation task creation form multi-step flow? [Completeness, Spec §US1, Scenario 1] +- [ ] CHK-V2-U02 Is URL paste feedback defined — green banner with dashboard title + filter count? [Completeness, Spec §US1, Scenario 2] +- [ ] CHK-V2-U03 Are requirements defined for per-tab screenshot viewer in Path A reports? [Completeness, Spec §US3] +- [ ] CHK-V2-U04 Are requirements defined for dataset health table display in Path B reports? [Completeness, Spec §US3, Scenario 1] +- [ ] CHK-V2-U05 Is the task list page layout defined — columns, status badges, schedule display? [Completeness, Spec §US2] +- [ ] CHK-V2-U06 Are requirements defined for the «Create Validation Task» button on dashboard hub (pre-fills selected dashboard IDs)? [Completeness, Spec §FR-053] +- [ ] CHK-V2-U07 Is provider selector behavior defined — filter multimodal when screenshot_enabled=true? [Completeness, Spec §FR-050] +- [ ] CHK-V2-U08 Is the removal of LLM validation column from dashboard hub grid specified? [Completeness, Spec §FR-052] +- [ ] CHK-V2-U09 Is the removal of «Validate» action from dashboard action dropdown specified? [Completeness, Spec §FR-051] +- [ ] CHK-V2-U10 Are prompt template textarea UX requirements defined (pre-filled default, expandable)? [Completeness, Spec §FR-048] ## Requirement Clarity -- [x] CHK006 Is "visual representation" defined with resolution or format constraints? [Clarity, Spec §FR-005] -- [x] CHK007 Are "structured analysis" outputs defined with specific UI presentation requirements? [Clarity, Spec §FR-007] + +- [x] CHK006 Is "visual representation" defined with resolution or format constraints? [Clarity, Spec §FR-005] — unchanged +- [x] CHK007 Are "structured analysis" outputs defined with specific UI presentation requirements? [Clarity, Spec §FR-007] — unchanged +- [x] CHK-V2-U11 Is multi-chunk screenshot format defined (one per tab, max 1920×1200)? [Clarity, Spec §FR-041] +- [x] CHK-V2-U12 Is task creation time budget defined (≤60s)? [Clarity, Spec §SC-011] ## Consistency -- [x] CHK008 Do the new "Validate" and "Generate Docs" actions follow existing UI patterns (icons, placement)? [Consistency, Spec §FR-021] + +- [x] CHK008 Do new actions follow existing UI patterns (icons, placement)? [Consistency, Spec §FR-021] — updated for v2: task creation button replaces Validate - [x] CHK009 Are error messages consistent with the application's existing error handling standards? [Consistency, Spec §FR-022] +- [x] CHK-V2-U13 Does the task creation form use existing form component patterns (ProviderConfig, SearchableMultiSelect)? [Consistency] +- [x] CHK-V2-U14 Does the task list page follow existing list page patterns (dashboard hub, reports list)? [Consistency] ## Edge Case Coverage -- [x] CHK010 Are requirements defined for when a dashboard cannot be rendered (e.g., 404)? [Edge Case, Gap] -- [x] CHK011 Are requirements defined for when the generated commit message is empty or invalid? [Edge Case, Spec §FR-012] \ No newline at end of file + +- [x] CHK010 Are requirements defined for when a dashboard cannot be rendered (e.g., 404)? [Edge Case, Gap] — addressed in v2: URL source marks as invalid +- [x] CHK011 Are requirements defined for when the generated commit message is empty or invalid? [Edge Case, Spec §FR-012] — unchanged +- [x] CHK-V2-U12 What happens when URL parsing returns partial_recovery? [Edge Case, Spec §US4] +- [x] CHK-V2-U13 What happens when a permalink URL expires before task execution? [Edge Case, Spec §Edge Cases v2] +- [x] CHK-V2-U14 What happens when task has 20 dashboards and concurrency limit is 3? [Edge Case, Spec §Edge Cases v2] diff --git a/specs/017-llm-analysis-plugin/contracts/modules.md b/specs/017-llm-analysis-plugin/contracts/modules.md index cddf6c8a..fde61072 100644 --- a/specs/017-llm-analysis-plugin/contracts/modules.md +++ b/specs/017-llm-analysis-plugin/contracts/modules.md @@ -1,107 +1,286 @@ -# Module Contracts: LLM Analysis Plugin +# Module Contracts: LLM Analysis Plugin v2 **Feature**: `017-llm-analysis-plugin` -**Protocol**: `semantic_protocol.md` +**Updated**: 2026-05-31 — v2: dual-path contracts, URL parsing, dataset health checking +**Protocol**: GRACE-Poly v2.6 ## Backend Modules -### Plugin Entry Point +### Validation Task Manager (new) ```python -# [DEF:backend/src/plugins/llm_analysis/plugin.py:Module] -# @TIER: STANDARD -# @SEMANTICS: plugin, task_handler, validation, documentation -# @PURPOSE: Orchestrate LLM-based analysis tasks (Dashboard Validation, Documentation) within the Plugin architecture. -# @LAYER: Domain -# @RELATION: DEPENDS_ON -> backend/src/plugins/llm_analysis/service.py -# @RELATION: IMPLEMENTS -> backend/src/core/plugin_base.py -# @INVARIANT: All tasks must report status updates via WebSocket. +# #region ValidationTaskService [C:4] [TYPE Module] [SEMANTICS validation, task, schedule, policy, CRUD] +# @BRIEF CRUD and lifecycle management for ValidationPolicy and ValidationSource. +# @LAYER Service +# @RELATION DEPENDS_ON -> [ValidationPolicy] +# @RELATION DEPENDS_ON -> [ValidationSource] +# @RELATION CALLS -> [SupersetContextExtractor] -# [DEF:validate_dashboard:Function] -# @PURPOSE: Analyze dashboard health using screenshot and logs. -# @PRE: Dashboard ID exists and provider is configured. -# @POST: ValidationResult is persisted and notification sent if failed. -# [/DEF:validate_dashboard:Function] +# #region create_task [TYPE Function] +# @PURPOSE Create a new validation task with sources (IDs + URLs), provider, prompt, schedule. +# @PRE environment_id resolves to existing environment. +# @POST ValidationPolicy + ValidationSource rows persisted. URL sources parsed via SupersetContextExtractor for preview/validation. +# @SIDE_EFFECT Calls Superset API to resolve dashboard slugs and validate URLs. +# @RATIONALE URL sources are parsed at creation time for validation and preview — user immediately sees dashboard title, filters, tabs. Re-parsing on every execution (per FR-057) guarantees up-to-date permalinks and filter state. +# @REJECTED URL parsing only at creation time (without execution-time re-parse) rejected — permalinks expire, dashboards get renamed, filters change. Creation-time parse = preview; execution-time re-parse = authoritative resolution. +# #endregion create_task -# [DEF:generate_documentation:Function] -# @PURPOSE: Generate and apply metadata descriptions for datasets. -# @PRE: Dataset is accessible and schema is valid. -# @POST: Dataset metadata is updated in the database. -# [/DEF:generate_documentation:Function] - -# [/DEF:backend/src/plugins/llm_analysis/plugin.py:Module] +# #region parse_dashboard_url [TYPE Function] +# @PURPOSE Resolve a Superset dashboard URL to structured context using SupersetContextExtractor. +# @PRE url is a valid absolute Superset dashboard URL. +# @POST Returns {dashboard_id, title, native_filters, activeTabs, anchor, partial_recovery, warnings}. +# @RELATION CALLS -> [SupersetContextExtractor.parse_superset_link] +# @SIDE_EFFECT Calls Superset API for permalink resolution and slug→ID lookup. +# #endregion parse_dashboard_url +# #endregion ValidationTaskService ``` -### LLM Service +### Dashboard Validation Plugin (refactored) ```python -# [DEF:backend/src/plugins/llm_analysis/service.py:Module] -# @TIER: CRITICAL -# @SEMANTICS: llm, api, client, retry, prompt -# @PURPOSE: Execute reliable interactions with external LLM providers (OpenAI, etc.) with retry logic and prompt management. -# @LAYER: Infrastructure -# @RELATION: CALLS -> external_api -# @RELATION: DEPENDS_ON -> backend/src/services/llm_provider.py -# @INVARIANT: API Keys must never be logged in plain text. +# #region DashboardValidationPlugin [C:5] [TYPE Module] [SEMANTICS validation, plugin, dual-path, llm, screenshot, text-only] +# @BRIEF Orchestrates dashboard validation via Path A (Playwright screenshot) or Path B (text-only API). +# @LAYER Plugin +# @RELATION IMPLEMENTS -> [PluginBase] +# @RELATION CALLS -> [ScreenshotService] # Path A only +# @RELATION CALLS -> [LLMClient] # Both paths +# @RELATION CALLS -> [SupersetClient] # Both paths +# @RELATION CALLS -> [DatasetHealthChecker] # Path B only +# @RELATION DEPENDS_ON -> [TaskContext] +# @INVARIANT Execution path (A/B) is determined by `screenshot_enabled` flag — never mixed within one run. +# @INVARIANT Path B MUST check every unique dataset referenced by dashboard charts. +# @RATIONALE Two paths with different prompts and different data sources for LLM: screenshot path for visual issues, text path for data/KXD issues. Both paths serve different use cases. +# @REJECTED Single «screenshot + text» prompt rejected — mixes two different analysis modalities, increases request size, complicates LLM response parsing. -# [DEF:analyze_multimodal:Function] -# @PURPOSE: Send image and text context to LLM for analysis. -# @PRE: Provider is active and quota is available. -# @POST: Returns structured analysis or raises RetryError. -# @SIDE_EFFECT: Makes external HTTP request. -# [/DEF:analyze_multimodal:Function] +# #region execute [TYPE Function] [C:4] +# @PURPOSE Execute one validation run for a dashboard from ValidationPolicy. +# @PRE params contains dashboard_id, provider_id, screenshot_enabled, prompt_template, logs_enabled. +# @POST ValidationRecord persisted with status, issues, timings. +# @SIDE_EFFECT Launches browser (Path A), calls Superset API (both paths), calls LLM API. +# #endregion execute -# [/DEF:backend/src/plugins/llm_analysis/service.py:Module] +# #region _execute_path_a [TYPE Function] [C:3] +# @PURPOSE Path A: Playwright screenshot → multi-chunk capture → multimodal LLM. +# @PRE dashboard_id is valid, provider is multimodal, screenshot_enabled=true. +# @POST Returns {status, summary, issues, screenshot_paths, tab_screenshots, token_usage}. +# @SIDE_EFFECT Launches browser, logs in, switches tabs, captures CDP screenshots. +# @RATIONALE Multi-chunk screenshots (one per tab) replace single full-page — LLM sees each tab at high resolution instead of one compressed full-dashboard image. +# @REJECTED Single full-page screenshot rejected — long dashboards (5000+px) compress to unreadable size (D9 in spec.md). +# #endregion _execute_path_a + +# #region _execute_path_b [TYPE Function] [C:3] +# @PURPOSE Path B: API calls → dashboard structure + chart params + dataset health + logs → text-only LLM. +# @PRE screenshot_enabled=false. +# @POST Returns {status, summary, issues, dataset_health, chart_data_results, token_usage}. +# @SIDE_EFFECT Calls Superset API (dashboard, chart, dataset, chart/data, log endpoints). +# @RATIONALE Dataset health checking is mandatory — the purpose of validation is to verify the dashboard opens correctly with KXD data. Without dataset checks, silent KXD errors (connection refused, timeout) are invisible. +# @REJECTED Logs-only checking without dataset health rejected — logs show past errors but don't guarantee current KXD connectivity state. +# #endregion _execute_path_b +# #endregion DashboardValidationPlugin ``` -### Provider Management Service +### Screenshot Service (Path A — refactored for multi-chunk) ```python -# [DEF:backend/src/services/llm_provider.py:Module] -# @TIER: CRITICAL -# @SEMANTICS: security, config, encryption, provider -# @PURPOSE: Manage lifecycle and secure storage of LLM Provider configurations. -# @LAYER: Domain -# @RELATION: DEPENDS_ON -> backend/src/core/database.py -# @INVARIANT: Stored API keys must be encrypted at rest. +# #region ScreenshotService [C:4] [TYPE Module] [SEMANTICS screenshot, playwright, cdp, tab-switching, multi-chunk] +# @BRIEF Captures per-tab dashboard screenshots via Playwright + CDP. +# @LAYER Plugin +# @INVARIANT Each tab produces a separate high-resolution screenshot (max 1920×1200 per chunk). +# @INVARIANT Multi-chunk: one screenshot per tab, NOT one full-page. +# @RATIONALE Многочанковый подход даёт LLM детальные скриншоты каждого таба вместо одного сжатого full-page. +# @REJECTED Single full-page CDP screenshot отвергнут — D9: длинные дашборды теряют читаемость. -# [DEF:get_provider_client:Function] -# @PURPOSE: Instantiate an authenticated LLM client for a specific provider. -# @PRE: Provider ID exists and is active. -# @POST: Returns configured client instance with decrypted key. -# [/DEF:get_provider_client:Function] - -# [/DEF:backend/src/services/llm_provider.py:Module] +# #region capture_dashboard_chunks [TYPE Function] [C:3] +# @PURPOSE Capture per-tab screenshots: login → navigate → switch tabs → per-tab CDP screenshots. +# @PRE dashboard_id is valid, browser available. +# @POST Returns list of {tab_name, screenshot_path}. +# @SIDE_EFFECT Launches browser, logs in, switches tabs. +# #endregion capture_dashboard_chunks +# #endregion ScreenshotService ``` -### Data Models +### Dataset Health Checker (Path B — new) ```python -# [DEF:backend/src/plugins/llm_analysis/models.py:Module] -# @TIER: TRIVIAL -# @SEMANTICS: dto, pydantic, schema -# @PURPOSE: Define Pydantic models for configuration, requests, and results. -# @LAYER: Domain/Data -# [/DEF:backend/src/plugins/llm_analysis/models.py:Module] +# #region DatasetHealthChecker [C:3] [TYPE Module] [SEMANTICS dataset, health, kxd, api, validation] +# @BRIEF Checks dataset accessibility and KXD connectivity via Superset API. +# @LAYER Service +# @RELATION CALLS -> [SupersetClient] +# @INVARIANT Every unique dataset referenced by dashboard charts is checked. +# @RATIONALE Без проверки датасетов невозможно определить, может ли дашборд корректно открыться с данными из КХД. Скриншот показывает визуал, но не показывает, что данные пришли (а не кэш). + +# #region check_dataset_health [TYPE Function] +# @PURPOSE Fetch dataset schema and verify accessibility. +# @PRE dataset_id is a valid Superset dataset ID. +# @POST Returns DatasetHealth: {accessible, database, backend, kind, error}. +# @SIDE_EFFECT Calls GET /api/v1/dataset/{id}. +# #endregion check_dataset_health + +# #region check_dashboard_datasets [TYPE Function] +# @PURPOSE For every unique dataset in dashboard charts, check health and optionally execute chart data. +# @PRE dashboard_id is valid, chart list available. +# @POST Returns {datasets: [DatasetHealth], chart_data: [ChartDataResult]}. +# @SIDE_EFFECT Calls GET /api/v1/dataset/{id} × N, optionally POST /api/v1/chart/data × M. +# #endregion check_dashboard_datasets +# #endregion DatasetHealthChecker +``` + +### LLM Client (updated for dual-path) + +```python +# #region LLMClient [C:4] [TYPE Module] [SEMANTICS llm, api, multimodal, text-only, json-mode] +# @BRIEF LLM provider wrapper supporting both multimodal (Path A) and text-only (Path B) analysis. +# @LAYER Infrastructure +# @RELATION DEPENDS_ON -> [EXT:Library:openai] +# @RELATION DEPENDS_ON -> [EXT:Library:tenacity] +# @INVARIANT API keys never logged in plain text. +# @RATIONALE Единый клиент для обоих путей — различия только в формате messages (multimodal vs text-only) и промпте. + +# #region analyze_dashboard_multimodal [TYPE Function] +# @PURPOSE Path A: send multi-chunk screenshots + logs to multimodal LLM. +# @PRE screenshot_paths is non-empty list. +# @POST Returns structured JSON {status, summary, issues}. +# #endregion analyze_dashboard_multimodal + +# #region analyze_dashboard_text [TYPE Function] +# @PURPOSE Path B: send dashboard topology + dataset health + logs to text-only LLM. +# @PRE dashboard_structure and dataset_health are populated. +# @POST Returns structured JSON {status, summary, issues}. +# @RATIONALE Текстовый промпт включает: описание топологии дашборда (табы, чарты, параметры), статус датасетов (доступность, бэкенд, ошибки), логи выполнения. LLM анализирует консистентность данных, а не визуал. +# #endregion analyze_dashboard_text +# #endregion LLMClient +``` + +### Provider Bindings (updated) + +```python +# #region DEFAULT_LLM_PROVIDER_BINDINGS [C:1] [TYPE Constant] +# @BRIEF Provider bindings retained: documentation, git_commit, assistant_planner. +# @RATIONALE dashboard_validation binding удалён — провайдер выбирается per-task в форме создания задачи (D6, FR-047). +# #endregion DEFAULT_LLM_PROVIDER_BINDINGS +``` + +### Prompt Defaults (updated) + +```python +# #region DEFAULT_LLM_PROMPTS [C:2] [TYPE Constant] +# @BRIEF Path-specific validation prompts: multimodal (Path A) and text (Path B). +# Legacy dashboard_validation_prompt retained for v1 compat, MUST NOT be used as default for new tasks. +# @RATIONALE FR-048: null prompt_template → auto-select path-specific default. +# Single prompt field per task avoids the risk of Path A using text prompt or vice versa. +# #endregion DEFAULT_LLM_PROMPTS ``` ## Frontend Components -### Configuration UI +### Validation Task List Page (new) ```html - - - - - + + + + + ``` -### Validation Report +### Validation Task Create/Edit Form (new) ```html - - - - - + + + + + + + + + +``` + +### Validation Run (new — groups per-dashboard results) + +```python +# #region ValidationRun [C:3] [TYPE Model] [SEMANTICS run, grouping, aggregate, execution] +# @BRIEF Groups ValidationRecords from one task execution (manual or scheduled). +# One run = N dashboards validated. Aggregate counts computed on finalize. +# @INVARIANT Every ValidationRecord belongs to exactly one ValidationRun. +# @RATIONALE Without run grouping, a 15-dashboard task produces 15 disconnected records — no way to see aggregate status or link them as one execution unit. +# Run groups them for reporting: "3 PASS, 1 WARN, 0 FAIL across 4 dashboards". +# #endregion ValidationRun +``` + +### Validation Run Detail Page (updated for multi-dashboard) + +```html + + + + + + + + + +``` + +### Dashboard Hub (simplified) + +```html +