287 lines
16 KiB
Markdown
287 lines
16 KiB
Markdown
# Module Contracts: LLM Analysis Plugin v2
|
||
|
||
**Feature**: `017-llm-analysis-plugin`
|
||
**Updated**: 2026-05-31 — v2: dual-path contracts, URL parsing, dataset health checking
|
||
**Protocol**: GRACE-Poly v2.6
|
||
|
||
## Backend Modules
|
||
|
||
### Validation Task Manager (new)
|
||
|
||
```python
|
||
# #region ValidationTaskService [C:4] [TYPE Module] [SEMANTICS validation, task, schedule, policy, CRUD]
|
||
# @BRIEF CRUD and lifecycle management for ValidationPolicy and ValidationSource.
|
||
# @LAYER Service
|
||
# @RELATION DEPENDS_ON -> [ValidationPolicy]
|
||
# @RELATION DEPENDS_ON -> [ValidationSource]
|
||
# @RELATION CALLS -> [SupersetContextExtractor]
|
||
|
||
# #region create_task [TYPE Function]
|
||
# @PURPOSE Create a new validation task with sources (IDs + URLs), provider, prompt, schedule.
|
||
# @PRE environment_id resolves to existing environment.
|
||
# @POST ValidationPolicy + ValidationSource rows persisted. URL sources parsed via SupersetContextExtractor for preview/validation.
|
||
# @SIDE_EFFECT Calls Superset API to resolve dashboard slugs and validate URLs.
|
||
# @RATIONALE URL sources are parsed at creation time for validation and preview — user immediately sees dashboard title, filters, tabs. Re-parsing on every execution (per FR-057) guarantees up-to-date permalinks and filter state.
|
||
# @REJECTED URL parsing only at creation time (without execution-time re-parse) rejected — permalinks expire, dashboards get renamed, filters change. Creation-time parse = preview; execution-time re-parse = authoritative resolution.
|
||
# #endregion create_task
|
||
|
||
# #region parse_dashboard_url [TYPE Function]
|
||
# @PURPOSE Resolve a Superset dashboard URL to structured context using SupersetContextExtractor.
|
||
# @PRE url is a valid absolute Superset dashboard URL.
|
||
# @POST Returns {dashboard_id, title, native_filters, activeTabs, anchor, partial_recovery, warnings}.
|
||
# @RELATION CALLS -> [SupersetContextExtractor.parse_superset_link]
|
||
# @SIDE_EFFECT Calls Superset API for permalink resolution and slug→ID lookup.
|
||
# #endregion parse_dashboard_url
|
||
# #endregion ValidationTaskService
|
||
```
|
||
|
||
### Dashboard Validation Plugin (refactored)
|
||
|
||
```python
|
||
# #region DashboardValidationPlugin [C:5] [TYPE Module] [SEMANTICS validation, plugin, dual-path, llm, screenshot, text-only]
|
||
# @BRIEF Orchestrates dashboard validation via Path A (Playwright screenshot) or Path B (text-only API).
|
||
# @LAYER Plugin
|
||
# @RELATION IMPLEMENTS -> [PluginBase]
|
||
# @RELATION CALLS -> [ScreenshotService] # Path A only
|
||
# @RELATION CALLS -> [LLMClient] # Both paths
|
||
# @RELATION CALLS -> [SupersetClient] # Both paths
|
||
# @RELATION CALLS -> [DatasetHealthChecker] # Path B only
|
||
# @RELATION DEPENDS_ON -> [TaskContext]
|
||
# @INVARIANT Execution path (A/B) is determined by `screenshot_enabled` flag — never mixed within one run.
|
||
# @INVARIANT Path B MUST check every unique dataset referenced by dashboard charts.
|
||
# @RATIONALE Two paths with different prompts and different data sources for LLM: screenshot path for visual issues, text path for data/KXD issues. Both paths serve different use cases.
|
||
# @REJECTED Single «screenshot + text» prompt rejected — mixes two different analysis modalities, increases request size, complicates LLM response parsing.
|
||
|
||
# #region execute [TYPE Function] [C:4]
|
||
# @PURPOSE Execute one validation run for a dashboard from ValidationPolicy.
|
||
# @PRE params contains dashboard_id, provider_id, screenshot_enabled, prompt_template, logs_enabled.
|
||
# @POST ValidationRecord persisted with status, issues, timings.
|
||
# @SIDE_EFFECT Launches browser (Path A), calls Superset API (both paths), calls LLM API.
|
||
# #endregion execute
|
||
|
||
# #region _execute_path_a [TYPE Function] [C:3]
|
||
# @PURPOSE Path A: Playwright screenshot → multi-chunk capture → multimodal LLM.
|
||
# @PRE dashboard_id is valid, provider is multimodal, screenshot_enabled=true.
|
||
# @POST Returns {status, summary, issues, screenshot_paths, tab_screenshots, token_usage}.
|
||
# @SIDE_EFFECT Launches browser, logs in, switches tabs, captures CDP screenshots.
|
||
# @RATIONALE Multi-chunk screenshots (one per tab) replace single full-page — LLM sees each tab at high resolution instead of one compressed full-dashboard image.
|
||
# @REJECTED Single full-page screenshot rejected — long dashboards (5000+px) compress to unreadable size (D9 in spec.md).
|
||
# #endregion _execute_path_a
|
||
|
||
# #region _execute_path_b [TYPE Function] [C:3]
|
||
# @PURPOSE Path B: API calls → dashboard structure + chart params + dataset health + logs → text-only LLM.
|
||
# @PRE screenshot_enabled=false.
|
||
# @POST Returns {status, summary, issues, dataset_health, chart_data_results, token_usage}.
|
||
# @SIDE_EFFECT Calls Superset API (dashboard, chart, dataset, chart/data, log endpoints).
|
||
# @RATIONALE Dataset health checking is mandatory — the purpose of validation is to verify the dashboard opens correctly with KXD data. Without dataset checks, silent KXD errors (connection refused, timeout) are invisible.
|
||
# @REJECTED Logs-only checking without dataset health rejected — logs show past errors but don't guarantee current KXD connectivity state.
|
||
# #endregion _execute_path_b
|
||
# #endregion DashboardValidationPlugin
|
||
```
|
||
|
||
### Screenshot Service (Path A — refactored for multi-chunk)
|
||
|
||
```python
|
||
# #region ScreenshotService [C:4] [TYPE Module] [SEMANTICS screenshot, playwright, cdp, tab-switching, multi-chunk]
|
||
# @BRIEF Captures per-tab dashboard screenshots via Playwright + CDP.
|
||
# @LAYER Plugin
|
||
# @INVARIANT Each tab produces a separate high-resolution screenshot (max 1920×1200 per chunk).
|
||
# @INVARIANT Multi-chunk: one screenshot per tab, NOT one full-page.
|
||
# @RATIONALE Многочанковый подход даёт LLM детальные скриншоты каждого таба вместо одного сжатого full-page.
|
||
# @REJECTED Single full-page CDP screenshot отвергнут — D9: длинные дашборды теряют читаемость.
|
||
|
||
# #region capture_dashboard_chunks [TYPE Function] [C:3]
|
||
# @PURPOSE Capture per-tab screenshots: login → navigate → switch tabs → per-tab CDP screenshots.
|
||
# @PRE dashboard_id is valid, browser available.
|
||
# @POST Returns list of {tab_name, screenshot_path}.
|
||
# @SIDE_EFFECT Launches browser, logs in, switches tabs.
|
||
# #endregion capture_dashboard_chunks
|
||
# #endregion ScreenshotService
|
||
```
|
||
|
||
### Dataset Health Checker (Path B — new)
|
||
|
||
```python
|
||
# #region DatasetHealthChecker [C:3] [TYPE Module] [SEMANTICS dataset, health, kxd, api, validation]
|
||
# @BRIEF Checks dataset accessibility and KXD connectivity via Superset API.
|
||
# @LAYER Service
|
||
# @RELATION CALLS -> [SupersetClient]
|
||
# @INVARIANT Every unique dataset referenced by dashboard charts is checked.
|
||
# @RATIONALE Без проверки датасетов невозможно определить, может ли дашборд корректно открыться с данными из КХД. Скриншот показывает визуал, но не показывает, что данные пришли (а не кэш).
|
||
|
||
# #region check_dataset_health [TYPE Function]
|
||
# @PURPOSE Fetch dataset schema and verify accessibility.
|
||
# @PRE dataset_id is a valid Superset dataset ID.
|
||
# @POST Returns DatasetHealth: {accessible, database, backend, kind, error}.
|
||
# @SIDE_EFFECT Calls GET /api/v1/dataset/{id}.
|
||
# #endregion check_dataset_health
|
||
|
||
# #region check_dashboard_datasets [TYPE Function]
|
||
# @PURPOSE For every unique dataset in dashboard charts, check health and optionally execute chart data.
|
||
# @PRE dashboard_id is valid, chart list available.
|
||
# @POST Returns {datasets: [DatasetHealth], chart_data: [ChartDataResult]}.
|
||
# @SIDE_EFFECT Calls GET /api/v1/dataset/{id} × N, optionally POST /api/v1/chart/data × M.
|
||
# #endregion check_dashboard_datasets
|
||
# #endregion DatasetHealthChecker
|
||
```
|
||
|
||
### LLM Client (updated for dual-path)
|
||
|
||
```python
|
||
# #region LLMClient [C:4] [TYPE Module] [SEMANTICS llm, api, multimodal, text-only, json-mode]
|
||
# @BRIEF LLM provider wrapper supporting both multimodal (Path A) and text-only (Path B) analysis.
|
||
# @LAYER Infrastructure
|
||
# @RELATION DEPENDS_ON -> [EXT:Library:openai]
|
||
# @RELATION DEPENDS_ON -> [EXT:Library:tenacity]
|
||
# @INVARIANT API keys never logged in plain text.
|
||
# @RATIONALE Единый клиент для обоих путей — различия только в формате messages (multimodal vs text-only) и промпте.
|
||
|
||
# #region analyze_dashboard_multimodal [TYPE Function]
|
||
# @PURPOSE Path A: send multi-chunk screenshots + logs to multimodal LLM.
|
||
# @PRE screenshot_paths is non-empty list.
|
||
# @POST Returns structured JSON {status, summary, issues}.
|
||
# #endregion analyze_dashboard_multimodal
|
||
|
||
# #region analyze_dashboard_text [TYPE Function]
|
||
# @PURPOSE Path B: send dashboard topology + dataset health + logs to text-only LLM.
|
||
# @PRE dashboard_structure and dataset_health are populated.
|
||
# @POST Returns structured JSON {status, summary, issues}.
|
||
# @RATIONALE Текстовый промпт включает: описание топологии дашборда (табы, чарты, параметры), статус датасетов (доступность, бэкенд, ошибки), логи выполнения. LLM анализирует консистентность данных, а не визуал.
|
||
# #endregion analyze_dashboard_text
|
||
# #endregion LLMClient
|
||
```
|
||
|
||
### Provider Bindings (updated)
|
||
|
||
```python
|
||
# #region DEFAULT_LLM_PROVIDER_BINDINGS [C:1] [TYPE Constant]
|
||
# @BRIEF Provider bindings retained: documentation, git_commit, assistant_planner.
|
||
# @RATIONALE dashboard_validation binding удалён — провайдер выбирается per-task в форме создания задачи (D6, FR-047).
|
||
# #endregion DEFAULT_LLM_PROVIDER_BINDINGS
|
||
```
|
||
|
||
### Prompt Defaults (updated)
|
||
|
||
```python
|
||
# #region DEFAULT_LLM_PROMPTS [C:2] [TYPE Constant]
|
||
# @BRIEF Path-specific validation prompts: multimodal (Path A) and text (Path B).
|
||
# Legacy dashboard_validation_prompt retained for v1 compat, MUST NOT be used as default for new tasks.
|
||
# @RATIONALE FR-048: null prompt_template → auto-select path-specific default.
|
||
# Single prompt field per task avoids the risk of Path A using text prompt or vice versa.
|
||
# #endregion DEFAULT_LLM_PROMPTS
|
||
```
|
||
|
||
## Frontend Components
|
||
|
||
### Validation Task List Page (new)
|
||
|
||
```html
|
||
<!-- #region ValidationTasksPage [C:3] [TYPE Page] [SEMANTICS sveltekit, validation, task-list, schedule]
|
||
<!-- @BRIEF Page: /validation-tasks — list of all validation tasks with status, schedule, last run.
|
||
<!-- @LAYER Page -->
|
||
<!-- @RELATION CALLS -> /api/validation-tasks -->
|
||
<!-- @UX_STATE Loading -> Skeleton table rows -->
|
||
<!-- @UX_STATE Loaded -> Task cards/rows with status badges, schedule, dashboard count -->
|
||
<!-- @UX_STATE Empty -> "No validation tasks yet" + "Create First Task" CTA -->
|
||
```
|
||
|
||
### Validation Task Create/Edit Form (new)
|
||
|
||
```html
|
||
<!-- #region ValidationTaskForm [C:4] [TYPE Component] [SEMANTICS svelte, form, validation, task-create, url-parser]
|
||
<!-- @BRIEF Multi-step form: name → sources (IDs + URLs) → provider + prompt → schedule.
|
||
<!-- @LAYER Component -->
|
||
<!-- @RELATION CALLS -> /api/llm/providers (list multimodal providers) -->
|
||
<!-- @RELATION CALLS -> /api/validation-tasks/parse-url (URL validation) -->
|
||
<!-- @UX_STATE Idle -> Empty form -->
|
||
<!-- @UX_STATE Parsing URL -> Spinner on URL input; result banner (dashboard title, filters count) -->
|
||
<!-- @UX_STATE Submitting -> All fields disabled, progress bar -->
|
||
<!-- @UX_STATE Error -> Field-level errors, URL parse failure banner -->
|
||
<!-- @UX_FEEDBACK URL parse success: green banner with dashboard title, native_filters count, activeTabs -->
|
||
<!-- @UX_FEEDBACK Provider selector filters multimodal-only when screenshot_enabled=true -->
|
||
```
|
||
|
||
### Validation Run (new — groups per-dashboard results)
|
||
|
||
```python
|
||
# #region ValidationRun [C:3] [TYPE Model] [SEMANTICS run, grouping, aggregate, execution]
|
||
# @BRIEF Groups ValidationRecords from one task execution (manual or scheduled).
|
||
# One run = N dashboards validated. Aggregate counts computed on finalize.
|
||
# @INVARIANT Every ValidationRecord belongs to exactly one ValidationRun.
|
||
# @RATIONALE Without run grouping, a 15-dashboard task produces 15 disconnected records — no way to see aggregate status or link them as one execution unit.
|
||
# Run groups them for reporting: "3 PASS, 1 WARN, 0 FAIL across 4 dashboards".
|
||
# #endregion ValidationRun
|
||
```
|
||
|
||
### Validation Run Detail Page (updated for multi-dashboard)
|
||
|
||
```html
|
||
<!-- #region ValidationRunDetailPage [C:3] [TYPE Page] [SEMANTICS sveltekit, run, report, multi-dashboard, aggregate]
|
||
<!-- @BRIEF Page: /validation-tasks/{policyId}/runs/{runId} — multi-dashboard run report.
|
||
<!-- Run header with aggregate counts → collapsible dashboard list → per-dashboard detail (issues, screenshots/dataset-health, logs).
|
||
<!-- @LAYER Page -->
|
||
<!-- @RELATION CALLS -> GET /api/validation-tasks/{policy_id}/runs/{run_id} -->
|
||
<!-- @UX_STATE Loading -> Skeleton placeholders -->
|
||
<!-- @UX_STATE Loaded -> Run header + dashboard list with expand/collapse -->
|
||
<!-- @UX_STATE Expanded -> Per-dashboard: Path A screenshots (tab selector) or Path B dataset health table -->
|
||
<!-- @UX_STATE Mixed -> Single run may contain both Path A and Path B dashboards — sections render per dashboard -->
|
||
<!-- @RATIONALE 15-dashboard run: collapse keeps page scannable. Expand for drill-down. Aggregate counts at top give instant health overview.
|
||
```
|
||
|
||
### Legacy Report Redirect
|
||
|
||
```python
|
||
# #region LegacyReportRedirect [C:1] [TYPE Route]
|
||
# @BRIEF GET /reports/llm/{taskId} → 302 redirect to canonical /validation-tasks/{policy_id}/runs/{record_id}.
|
||
# @RATIONALE Backward compat for v1 notification links, external bookmarks, and assistant-generated URLs.
|
||
# Canonical URL is per FR-055b. Returns 404 if taskId has no matching ValidationRecord.
|
||
# #endregion LegacyReportRedirect
|
||
```
|
||
|
||
### Dashboard Validation Indicator (new)
|
||
|
||
```html
|
||
<!-- #region DashboardValidationIndicator [C:2] [TYPE Component] [SEMANTICS svelte, dashboard, validation, indicator, batch]
|
||
<!-- @BRIEF Colored dot indicator (PASS/WARN/FAIL/UNKNOWN) next to dashboard title in hub rows.
|
||
<!-- Click → dropdown: last 3 runs (date, status, task name) + "View all" link.
|
||
<!-- @LAYER Component -->
|
||
<!-- @RELATION CALLS -> GET /api/dashboards/{env_id}/validation-status?dashboard_ids=... -->
|
||
<!-- @RATIONALE Replaces removed LLM column without adding visual noise.
|
||
<!-- Batch endpoint avoids N+1 requests for dashboard hub with 50+ rows.
|
||
```
|
||
|
||
### ValidationTaskReport (new component)
|
||
|
||
```html
|
||
<!-- #region ValidationTaskReport [C:3] [TYPE Component] [SEMANTICS svelte, report, validation, issues, dataset-health]
|
||
<!-- @BRIEF Shared report component: check result badge, summary, issues table, execution path indicator, dataset health (Path B), screenshots (Path A).
|
||
<!-- @LAYER Component -->
|
||
```
|
||
|
||
### Dashboard Hub (simplified)
|
||
|
||
```html
|
||
<!-- #region DashboardHub [C:4] [TYPE Page] [SEMANTICS sveltekit, dashboard, hub]
|
||
<!-- @BRIEF LLM validation column REMOVED. "Validate" action REMOVED from action dropdown.
|
||
<!-- Added "Create Validation Task" button in toolbar that pre-fills selected dashboard IDs.
|
||
<!-- @RATIONALE D1, FR-051, FR-052: validation moves to task-based flow; ad-hoc "Validate" is deprecated.
|
||
```
|
||
|
||
### LLM Settings (simplified)
|
||
|
||
```html
|
||
<!-- #region LLMSettingsPage [C:3] [TYPE Page] [SEMANTICS sveltekit, settings, llm, provider]
|
||
<!-- @BRIEF Provider bindings for dashboard_validation REMOVED.
|
||
<!-- Retained: documentation, git_commit, assistant_planner provider bindings.
|
||
<!-- @RATIONALE D8, FR-049: dashboard validation provider is per-task, not global.
|
||
```
|
||
|
||
## ADR Continuity
|
||
|
||
| ADR | Impact |
|
||
|-----|--------|
|
||
| ADR-0001 (module layout) | v2 changes stay within `backend/src/plugins/llm_analysis/` and `frontend/src/routes/validation-tasks/` |
|
||
| ADR-0004 (plugin architecture) | `DashboardValidationPlugin` remains a PluginBase implementation; new `DatasetHealthChecker` is a service, not a plugin |
|
||
| ADR-0009 (SSL) | Unchanged — LLM client SSL configuration applies to both Path A and Path B |
|
||
| ADR-0008 (assistant tool registry) | `_tool_llm_validation.py` — assistant tool should be updated to create task instead of triggering ad-hoc validation |
|