143 lines
8.9 KiB
Markdown
143 lines
8.9 KiB
Markdown
# Research: LLM Analysis & Documentation Plugins v2
|
||
|
||
**Feature**: `017-llm-analysis-plugin`
|
||
**Updated**: 2026-06-07 — v2 implemented: dual-path execution, multi-chunk screenshots, dataset health checking, URL-based dashboards
|
||
|
||
## v2 Research Decisions
|
||
|
||
### 8. Task-Based Flow (replaces ad-hoc Validate)
|
||
|
||
**Decision**: Remove one-click «Validate» button; replace with «Create Validation Task» flow (name → sources → provider+prompt → schedule).
|
||
|
||
**Rationale**: Ad-hoc validation doesn't scale — users need repeatable, scheduled checks with full configuration control. The task list provides visibility into what's running and historical trends.
|
||
|
||
**Alternatives Considered**:
|
||
- Keep Validate button + add scheduling separately — rejected: two modes cause confusion; task-based is cleaner.
|
||
- Inline validation from dashboard detail — rejected: no scheduling, no multi-dashboard runs.
|
||
|
||
### 9. Dual Execution Paths (Screenshot vs Text-Only)
|
||
|
||
**Decision**: `screenshot_enabled` flag controls execution path:
|
||
- **Path A** (screenshot_enabled=true): Playwright → multi-chunk CDP screenshots → multimodal LLM
|
||
- **Path B** (screenshot_enabled=false): API calls → dashboard topology + chart params + dataset health + logs → text-only LLM
|
||
|
||
**Rationale**:
|
||
- Screenshot catches visual bugs (empty charts, overlap, rendering) but costs image tokens and requires a browser.
|
||
- Text-only catches data/KXD errors (dataset inaccessible, chart query timeout) and is lightweight.
|
||
- Together they cover both visual and data-layer failure modes. Users choose per task based on what matters.
|
||
|
||
**Alternatives Considered**:
|
||
- Always screenshot — rejected: expensive, slow, doesn't check dataset backend health.
|
||
- Always text-only — rejected: misses visual rendering bugs (broken charts with no error in logs).
|
||
- Hybrid: screenshot + dataset check in one call — rejected: mixes modalities, increases LLM context size, complicates prompt.
|
||
|
||
### 10. Multi-Chunk Screenshots (Path A)
|
||
|
||
**Decision**: Instead of one compressed full-page screenshot, capture each dashboard tab as a separate high-resolution screenshot (max 1920×1200 per chunk). Send all chunks in a single multimodal LLM request.
|
||
|
||
**Rationale**: Long dashboards (5000+ px) compressed to 1024×2048 lose readability — small chart labels become unreadable. Per-tab chunks preserve resolution, and LLM sees the dashboard in the same logical sections a human would.
|
||
|
||
**Alternatives Considered**:
|
||
- Single full-page at 4096px width — rejected: token cost too high for image processing.
|
||
- Overlapping chunks with stitching — rejected: complex, fragile to viewport changes.
|
||
- Per-chart screenshots — rejected: too many images (30+ charts → excessive tokens), loses layout context.
|
||
|
||
### 11. Dataset Health Checking (Path B)
|
||
|
||
**Decision**: For Path B, call `GET /api/v1/dataset/{id}` for every unique dataset used by dashboard charts. Verify: HTTP 200, database backend, kind (virtual/physical). Optionally execute `POST /api/v1/chart/data` (off by default).
|
||
|
||
**Rationale**: The primary purpose of LLM validation is verifying the KXD→dataset→chart→dashboard chain works. Screenshot-only misses silent data failures (empty results, partial data, wrong time range). Dataset health check catches KXD connection errors that produce no visual error but return empty/incomplete data.
|
||
|
||
**Alternatives Considered**:
|
||
- Logs-only (no dataset API calls) — rejected: logs show past failures, not current KXD connectivity. A dataset can be broken right now with no recent log entry.
|
||
- Always execute chart data — rejected: overloads Superset (15 charts × 30 dashboards = 450 queries). Optional toggle lets users enable selectively.
|
||
|
||
### 12. URL-Based Dashboard Selection
|
||
|
||
**Decision**: Support pasting full Superset dashboard URLs as task sources. Parse via `SupersetContextExtractor` (same code path as dataset review feature). Extract: dashboard_id, native_filters, activeTabs, anchor.
|
||
|
||
**Rationale**: Users copy dashboard URLs from browser with filters already applied. Parsing the URL preserves the exact filter/tab state — the validation checks the dashboard in the same state the user sees. Reusing `SupersetContextExtractor` avoids duplicating URL parsing logic.
|
||
|
||
**URL Formats Supported** (same as dataset review):
|
||
- `https://superset.example.com/superset/dashboard/42/` — numeric ID
|
||
- `https://superset.example.com/superset/dashboard/my-dashboard/` — slug
|
||
- `https://superset.example.com/superset/dashboard/p/abc123/` — permalink
|
||
- `https://superset.example.com/superset/dashboard/42/?native_filters_key=xyz` — filter state key
|
||
|
||
**Alternatives Considered**:
|
||
- Dashboard ID only — rejected: loses filter context, user must manually re-apply filters.
|
||
- Custom URL format — rejected: users already have Superset URLs in browser; pasting them is natural.
|
||
|
||
### 13. Provider & Prompt Per-Task
|
||
|
||
**Decision**: Provider selection and prompt template configuration move from global LLM settings to per-task configuration in the task creation form. Global LLM settings retain provider bindings only for: documentation, git_commit, assistant_planner.
|
||
|
||
**Rationale**: Different dashboards may need different models (e.g., GPT-4o for complex visual analysis, a cheaper model for text-only). Different teams may want different prompt emphasis (performance vs. data quality). Global settings are too coarse.
|
||
|
||
**Alternatives Considered**:
|
||
- Global binding + per-task override — rejected: adds complexity; per-task is simpler and more flexible.
|
||
- Environment-level provider binding — rejected: too coarse; same environment may have different validation needs per dashboard group.
|
||
|
||
## v1 Research Decisions (preserved)
|
||
|
||
### 1. LLM Provider Integration
|
||
|
||
**Decision**: Use a unified `LLMProviderService` that abstracts OpenAI-compatible APIs.
|
||
**Rationale**: OpenRouter, Kilo, LiteLLM, and OpenAI all support the standard OpenAI API format.
|
||
|
||
### 2. Dashboard Screenshot Capture
|
||
|
||
**Decision**: `ScreenshotService` with Playwright + CDP for Path A.
|
||
**Rationale**: Provides accurate «user-view» render; CDP avoids font loading timeouts in headless mode.
|
||
|
||
### 3. Multimodal Analysis Prompting
|
||
|
||
**Decision**: Structured prompt template accepting base64-encoded images + text logs.
|
||
**Rationale**: GPT-4o / Claude / Gemini models support this natively.
|
||
|
||
### 4. Documentation Persistence
|
||
|
||
**Decision**: Update Dataset/Column models in metadata database.
|
||
**Rationale**: Documentation co-located with assets.
|
||
|
||
### 5. Git Commit Integration
|
||
|
||
**Decision**: REST endpoint `/api/git/generate-message`.
|
||
**Rationale**: Heavy lifting (LLM, diff processing) on backend.
|
||
|
||
### 6. Security & Storage
|
||
|
||
**Decision**: AES-256 encrypted API keys.
|
||
**Rationale**: Keys must not be stored in plain text.
|
||
|
||
### 7. Retry Logic
|
||
|
||
**Decision**: `tenacity` with exponential backoff (5 attempts, 5-60s wait).
|
||
**Rationale**: Standard, robust retry for transient LLM API failures.
|
||
|
||
### 14. Image Pipeline: PNG → JPEG (LLM) + WebP (Archive)
|
||
|
||
**Decision**: CDP captures PNG (browser-native, lossless). Convert to JPEG quality=60 for LLM (universal provider compatibility). Convert to WebP quality=80 for archive (93% disk savings vs PNG). Delete PNG and JPEG intermediates after successful save.
|
||
|
||
**Rationale**: JPEG universally supported by all providers. WebP has silent failure risk on unknown/closed providers (model accepts request but doesn't process image → false PASS). Pipeline decoupled: LLM failure doesn't affect archive; archive failure falls back to PNG with WARN.
|
||
|
||
**Alternatives Considered**:
|
||
- All WebP (LLM + archive) — rejected: silent failure risk.
|
||
- All JPEG (LLM + archive) — rejected: no archive space savings, re-encoding artifacts.
|
||
- WebP for LLM with `_supports_webp()` per-provider check — rejected: adds complexity; JPEG is simpler and universally safe.
|
||
|
||
**Impact**: T017a, T017b, T017c in task plan. FR-059 in spec. D10 design decision. ~93% disk savings vs PNG archive.
|
||
|
||
### 15. Path B Batching (experimental v2.1)
|
||
|
||
**Decision**: Default `llm_batch_size=1` (full isolation, one LLM call per dashboard). Values >1 are experimental and reduce cost at risk of cross-dashboard context contamination.
|
||
|
||
**Rationale**: Multiple dashboards in one LLM call saves API calls (5→1) but risks the LLM confusing which issue belongs to which dashboard. Structured prompt with per-dashboard labeled sections + strict JSON `{dashboards: [{dashboard_id, ...}]}` schema mitigates but does not eliminate this risk. For validation (where accuracy > cost), isolation is the safer default.
|
||
|
||
**Alternatives Considered**:
|
||
- Always batch (default 5) — rejected: accuracy risk for validation feature. Can't guarantee LLMs won't cross-contaminate context.
|
||
- Never batch (always 1) — rejected: legitimate use case for large fleet monitoring where cost matters more than per-dashboard precision.
|
||
- Per-policy toggle only — accepted as the right balance: default safe, user opts into batching when they understand the tradeoff.
|
||
|
||
**Impact**: FR-045a in spec. `llm_batch_size` field in ValidationPolicy, default 1. Batch_size >1 is documented as experimental with known accuracy risk.
|