ss-tools/specs/017-llm-analysis-plugin/research.md

Research: LLM Analysis & Documentation Plugins v2

Feature: 017-llm-analysis-plugin Updated: 2026-05-31 — v2: dual-path execution, multi-chunk screenshots, dataset health checking, URL-based dashboards

v2 Research Decisions

8. Task-Based Flow (replaces ad-hoc Validate)

Decision: Remove one-click «Validate» button; replace with «Create Validation Task» flow (name → sources → provider+prompt → schedule).

Rationale: Ad-hoc validation doesn't scale — users need repeatable, scheduled checks with full configuration control. The task list provides visibility into what's running and historical trends.

Alternatives Considered:

• Keep Validate button + add scheduling separately — rejected: two modes cause confusion; task-based is cleaner.
• Inline validation from dashboard detail — rejected: no scheduling, no multi-dashboard runs.

9. Dual Execution Paths (Screenshot vs Text-Only)

Decision: screenshot_enabled flag controls execution path:

• Path A (screenshot_enabled=true): Playwright → multi-chunk CDP screenshots → multimodal LLM
• Path B (screenshot_enabled=false): API calls → dashboard topology + chart params + dataset health + logs → text-only LLM

Rationale:

• Screenshot catches visual bugs (empty charts, overlap, rendering) but costs image tokens and requires a browser.
• Text-only catches data/KXD errors (dataset inaccessible, chart query timeout) and is lightweight.
• Together they cover both visual and data-layer failure modes. Users choose per task based on what matters.

Alternatives Considered:

• Always screenshot — rejected: expensive, slow, doesn't check dataset backend health.
• Always text-only — rejected: misses visual rendering bugs (broken charts with no error in logs).
• Hybrid: screenshot + dataset check in one call — rejected: mixes modalities, increases LLM context size, complicates prompt.

10. Multi-Chunk Screenshots (Path A)

Decision: Instead of one compressed full-page screenshot, capture each dashboard tab as a separate high-resolution screenshot (max 1920×1200 per chunk). Send all chunks in a single multimodal LLM request.

Rationale: Long dashboards (5000+ px) compressed to 1024×2048 lose readability — small chart labels become unreadable. Per-tab chunks preserve resolution, and LLM sees the dashboard in the same logical sections a human would.

Alternatives Considered:

• Single full-page at 4096px width — rejected: token cost too high for image processing.
• Overlapping chunks with stitching — rejected: complex, fragile to viewport changes.
• Per-chart screenshots — rejected: too many images (30+ charts → excessive tokens), loses layout context.

11. Dataset Health Checking (Path B)

Decision: For Path B, call GET /api/v1/dataset/{id} for every unique dataset used by dashboard charts. Verify: HTTP 200, database backend, kind (virtual/physical). Optionally execute POST /api/v1/chart/data (off by default).

Rationale: The primary purpose of LLM validation is verifying the KXD→dataset→chart→dashboard chain works. Screenshot-only misses silent data failures (empty results, partial data, wrong time range). Dataset health check catches KXD connection errors that produce no visual error but return empty/incomplete data.

Alternatives Considered:

• Logs-only (no dataset API calls) — rejected: logs show past failures, not current KXD connectivity. A dataset can be broken right now with no recent log entry.
• Always execute chart data — rejected: overloads Superset (15 charts × 30 dashboards = 450 queries). Optional toggle lets users enable selectively.

12. URL-Based Dashboard Selection

Decision: Support pasting full Superset dashboard URLs as task sources. Parse via SupersetContextExtractor (same code path as dataset review feature). Extract: dashboard_id, native_filters, activeTabs, anchor.

Rationale: Users copy dashboard URLs from browser with filters already applied. Parsing the URL preserves the exact filter/tab state — the validation checks the dashboard in the same state the user sees. Reusing SupersetContextExtractor avoids duplicating URL parsing logic.

URL Formats Supported (same as dataset review):

• https://superset.example.com/superset/dashboard/42/ — numeric ID
• https://superset.example.com/superset/dashboard/my-dashboard/ — slug
• https://superset.example.com/superset/dashboard/p/abc123/ — permalink
• https://superset.example.com/superset/dashboard/42/?native_filters_key=xyz — filter state key

Alternatives Considered:

• Dashboard ID only — rejected: loses filter context, user must manually re-apply filters.
• Custom URL format — rejected: users already have Superset URLs in browser; pasting them is natural.

13. Provider & Prompt Per-Task

Decision: Provider selection and prompt template configuration move from global LLM settings to per-task configuration in the task creation form. Global LLM settings retain provider bindings only for: documentation, git_commit, assistant_planner.

Rationale: Different dashboards may need different models (e.g., GPT-4o for complex visual analysis, a cheaper model for text-only). Different teams may want different prompt emphasis (performance vs. data quality). Global settings are too coarse.

Alternatives Considered:

• Global binding + per-task override — rejected: adds complexity; per-task is simpler and more flexible.
• Environment-level provider binding — rejected: too coarse; same environment may have different validation needs per dashboard group.

v1 Research Decisions (preserved)

1. LLM Provider Integration

Decision: Use a unified LLMProviderService that abstracts OpenAI-compatible APIs. Rationale: OpenRouter, Kilo, LiteLLM, and OpenAI all support the standard OpenAI API format.

2. Dashboard Screenshot Capture

Decision: ScreenshotService with Playwright + CDP for Path A. Rationale: Provides accurate «user-view» render; CDP avoids font loading timeouts in headless mode.

3. Multimodal Analysis Prompting

Decision: Structured prompt template accepting base64-encoded images + text logs. Rationale: GPT-4o / Claude / Gemini models support this natively.

4. Documentation Persistence

Decision: Update Dataset/Column models in metadata database. Rationale: Documentation co-located with assets.

5. Git Commit Integration

Decision: REST endpoint /api/git/generate-message. Rationale: Heavy lifting (LLM, diff processing) on backend.

6. Security & Storage

Decision: AES-256 encrypted API keys. Rationale: Keys must not be stored in plain text.

7. Retry Logic

Decision: tenacity with exponential backoff (5 attempts, 5-60s wait). Rationale: Standard, robust retry for transient LLM API failures.

14. Image Pipeline: PNG → JPEG (LLM) + WebP (Archive)

Decision: CDP captures PNG (browser-native, lossless). Convert to JPEG quality=60 for LLM (universal provider compatibility). Convert to WebP quality=80 for archive (93% disk savings vs PNG). Delete PNG and JPEG intermediates after successful save.

Rationale: JPEG universally supported by all providers. WebP has silent failure risk on unknown/closed providers (model accepts request but doesn't process image → false PASS). Pipeline decoupled: LLM failure doesn't affect archive; archive failure falls back to PNG with WARN.

Alternatives Considered:

• All WebP (LLM + archive) — rejected: silent failure risk.
• All JPEG (LLM + archive) — rejected: no archive space savings, re-encoding artifacts.
• WebP for LLM with _supports_webp() per-provider check — rejected: adds complexity; JPEG is simpler and universally safe.

Impact: T017a, T017b, T017c in task plan. FR-059 in spec. D10 design decision. ~93% disk savings vs PNG archive.

15. Path B Batching (experimental v2.1)

Decision: Default llm_batch_size=1 (full isolation, one LLM call per dashboard). Values >1 are experimental and reduce cost at risk of cross-dashboard context contamination.

Rationale: Multiple dashboards in one LLM call saves API calls (5→1) but risks the LLM confusing which issue belongs to which dashboard. Structured prompt with per-dashboard labeled sections + strict JSON {dashboards: [{dashboard_id, ...}]} schema mitigates but does not eliminate this risk. For validation (where accuracy > cost), isolation is the safer default.

Alternatives Considered:

• Always batch (default 5) — rejected: accuracy risk for validation feature. Can't guarantee LLMs won't cross-contaminate context.
• Never batch (always 1) — rejected: legitimate use case for large fleet monitoring where cost matters more than per-dashboard precision.
• Per-policy toggle only — accepted as the right balance: default safe, user opts into batching when they understand the tradeoff.

Impact: FR-045a in spec. llm_batch_size field in ValidationPolicy, default 1. Batch_size >1 is documented as experimental with known accuracy risk.