# Research: LLM Analysis & Documentation Plugins v2

**Feature**: `017-llm-analysis-plugin`
**Updated**: 2026-06-07 — v2 implemented: dual-path execution, multi-chunk screenshots, dataset health checking, URL-based dashboards

## v2 Research Decisions

### 8. Task-Based Flow (replaces ad-hoc Validate)

**Decision**: Remove one-click «Validate» button; replace with «Create Validation Task» flow (name → sources → provider+prompt → schedule).

**Rationale**: Ad-hoc validation doesn't scale — users need repeatable, scheduled checks with full configuration control. The task list provides visibility into what's running and historical trends.

**Alternatives Considered**:
- Keep Validate button + add scheduling separately — rejected: two modes cause confusion; task-based is cleaner.
- Inline validation from dashboard detail — rejected: no scheduling, no multi-dashboard runs.

### 9. Dual Execution Paths (Screenshot vs Text-Only)

**Decision**: `screenshot_enabled` flag controls execution path:
- **Path A** (screenshot_enabled=true): Playwright → multi-chunk CDP screenshots → multimodal LLM
- **Path B** (screenshot_enabled=false): API calls → dashboard topology + chart params + dataset health + logs → text-only LLM

**Rationale**:
- Screenshot catches visual bugs (empty charts, overlap, rendering) but costs image tokens and requires a browser.
- Text-only catches data/KXD errors (dataset inaccessible, chart query timeout) and is lightweight.
- Together they cover both visual and data-layer failure modes. Users choose per task based on what matters.

**Alternatives Considered**:
- Always screenshot — rejected: expensive, slow, doesn't check dataset backend health.
- Always text-only — rejected: misses visual rendering bugs (broken charts with no error in logs).
- Hybrid: screenshot + dataset check in one call — rejected: mixes modalities, increases LLM context size, complicates prompt.

### 10. Multi-Chunk Screenshots (Path A)

**Decision**: Instead of one compressed full-page screenshot, capture each dashboard tab as a separate high-resolution screenshot (max 1920×1200 per chunk). Send all chunks in a single multimodal LLM request.

**Rationale**: Long dashboards (5000+ px) compressed to 1024×2048 lose readability — small chart labels become unreadable. Per-tab chunks preserve resolution, and LLM sees the dashboard in the same logical sections a human would.

**Alternatives Considered**:
- Single full-page at 4096px width — rejected: token cost too high for image processing.
- Overlapping chunks with stitching — rejected: complex, fragile to viewport changes.
- Per-chart screenshots — rejected: too many images (30+ charts → excessive tokens), loses layout context.

### 11. Dataset Health Checking (Path B)

**Decision**: For Path B, call `GET /api/v1/dataset/{id}` for every unique dataset used by dashboard charts. Verify: HTTP 200, database backend, kind (virtual/physical). Optionally execute `POST /api/v1/chart/data` (off by default).

**Rationale**: The primary purpose of LLM validation is verifying the KXD→dataset→chart→dashboard chain works. Screenshot-only misses silent data failures (empty results, partial data, wrong time range). Dataset health check catches KXD connection errors that produce no visual error but return empty/incomplete data.

**Alternatives Considered**:
- Logs-only (no dataset API calls) — rejected: logs show past failures, not current KXD connectivity. A dataset can be broken right now with no recent log entry.
- Always execute chart data — rejected: overloads Superset (15 charts × 30 dashboards = 450 queries). Optional toggle lets users enable selectively.

### 12. URL-Based Dashboard Selection

**Decision**: Support pasting full Superset dashboard URLs as task sources. Parse via `SupersetContextExtractor` (same code path as dataset review feature). Extract: dashboard_id, native_filters, activeTabs, anchor.

**Rationale**: Users copy dashboard URLs from browser with filters already applied. Parsing the URL preserves the exact filter/tab state — the validation checks the dashboard in the same state the user sees. Reusing `SupersetContextExtractor` avoids duplicating URL parsing logic.

**URL Formats Supported** (same as dataset review):
- `https://superset.example.com/superset/dashboard/42/` — numeric ID
- `https://superset.example.com/superset/dashboard/my-dashboard/` — slug
- `https://superset.example.com/superset/dashboard/p/abc123/` — permalink
- `https://superset.example.com/superset/dashboard/42/?native_filters_key=xyz` — filter state key

**Alternatives Considered**:
- Dashboard ID only — rejected: loses filter context, user must manually re-apply filters.
- Custom URL format — rejected: users already have Superset URLs in browser; pasting them is natural.

### 13. Provider & Prompt Per-Task

**Decision**: Provider selection and prompt template configuration move from global LLM settings to per-task configuration in the task creation form. Global LLM settings retain provider bindings only for: documentation, git_commit, assistant_planner.

**Rationale**: Different dashboards may need different models (e.g., GPT-4o for complex visual analysis, a cheaper model for text-only). Different teams may want different prompt emphasis (performance vs. data quality). Global settings are too coarse.

**Alternatives Considered**:
- Global binding + per-task override — rejected: adds complexity; per-task is simpler and more flexible.
- Environment-level provider binding — rejected: too coarse; same environment may have different validation needs per dashboard group.

## v1 Research Decisions (preserved)

### 1. LLM Provider Integration

**Decision**: Use a unified `LLMProviderService` that abstracts OpenAI-compatible APIs.
**Rationale**: OpenRouter, Kilo, LiteLLM, and OpenAI all support the standard OpenAI API format.

### 2. Dashboard Screenshot Capture

**Decision**: `ScreenshotService` with Playwright + CDP for Path A.
**Rationale**: Provides accurate «user-view» render; CDP avoids font loading timeouts in headless mode.

### 3. Multimodal Analysis Prompting

**Decision**: Structured prompt template accepting base64-encoded images + text logs.
**Rationale**: GPT-4o / Claude / Gemini models support this natively.

### 4. Documentation Persistence

**Decision**: Update Dataset/Column models in metadata database.
**Rationale**: Documentation co-located with assets.

### 5. Git Commit Integration

**Decision**: REST endpoint `/api/git/generate-message`.
**Rationale**: Heavy lifting (LLM, diff processing) on backend.

### 6. Security & Storage

**Decision**: AES-256 encrypted API keys.
**Rationale**: Keys must not be stored in plain text.

### 7. Retry Logic

**Decision**: `tenacity` with exponential backoff (5 attempts, 5-60s wait).
**Rationale**: Standard, robust retry for transient LLM API failures.

### 14. Image Pipeline: PNG → JPEG (LLM) + WebP (Archive)

**Decision**: CDP captures PNG (browser-native, lossless). Convert to JPEG quality=60 for LLM (universal provider compatibility). Convert to WebP quality=80 for archive (93% disk savings vs PNG). Delete PNG and JPEG intermediates after successful save.

**Rationale**: JPEG universally supported by all providers. WebP has silent failure risk on unknown/closed providers (model accepts request but doesn't process image → false PASS). Pipeline decoupled: LLM failure doesn't affect archive; archive failure falls back to PNG with WARN.

**Alternatives Considered**:
- All WebP (LLM + archive) — rejected: silent failure risk.
- All JPEG (LLM + archive) — rejected: no archive space savings, re-encoding artifacts.
- WebP for LLM with `_supports_webp()` per-provider check — rejected: adds complexity; JPEG is simpler and universally safe.

**Impact**: T017a, T017b, T017c in task plan. FR-059 in spec. D10 design decision. ~93% disk savings vs PNG archive.

### 15. Path B Batching (experimental v2.1)

**Decision**: Default `llm_batch_size=1` (full isolation, one LLM call per dashboard). Values >1 are experimental and reduce cost at risk of cross-dashboard context contamination.

**Rationale**: Multiple dashboards in one LLM call saves API calls (5→1) but risks the LLM confusing which issue belongs to which dashboard. Structured prompt with per-dashboard labeled sections + strict JSON `{dashboards: [{dashboard_id, ...}]}` schema mitigates but does not eliminate this risk. For validation (where accuracy > cost), isolation is the safer default.

**Alternatives Considered**:
- Always batch (default 5) — rejected: accuracy risk for validation feature. Can't guarantee LLMs won't cross-contaminate context.
- Never batch (always 1) — rejected: legitimate use case for large fleet monitoring where cost matters more than per-dashboard precision.
- Per-policy toggle only — accepted as the right balance: default safe, user opts into batching when they understand the tradeoff.

**Impact**: FR-045a in spec. `llm_batch_size` field in ValidationPolicy, default 1. Batch_size >1 is documented as experimental with known accuracy risk.