ss-tools/specs/017-llm-analysis-plugin/quickstart.md

Quickstart: LLM Validation v2

Prerequisites

1. An API Key from OpenAI, OpenRouter, Kilo, or LiteLLM.
2. At least one environment configured in superset-tools (pointing to a Superset instance).
3. (Optional) Playwright browsers installed for Path A (screenshot) validation:

   playwright install chromium
   

Configuration

LLM Provider

1. Navigate to Settings > LLM Providers.
2. Click "Add Provider".
3. Select type (e.g., OpenRouter).
4. Enter your API Key and Base URL (e.g., https://openrouter.ai/api/v1).
5. Set a default model (e.g., gpt-4o for multimodal, gpt-4o-mini for text-only).
6. Check "Is Multimodal" if the model supports image input (required for Path A).
7. Click Save and Test Connection.

Note

: Provider bindings for Documentation, Git Commits, and Chat are configured in Settings. Dashboard Validation provider is selected per-task in the task creation form — not in global settings.

Validation Prompt (Optional)

The default validation prompt covers standard dashboard health checks. To customize:

1. When creating a task, edit the Prompt Template field.
2. Path A uses dashboard_validation_prompt_multimodal (image-focused).
3. Path B uses dashboard_validation_prompt_text (topology-focused).

Usage

Create a Validation Task

1. Go to Validation Tasks (/validation-tasks).
2. Click "+ New Task".
3. Step 1 — Name & Environment: Enter task name (e.g., "Daily KXD Health Check"), select Superset environment.
4. Step 2 — Sources: Add dashboards to validate:
   • From List: Check dashboards from the environment list.
   • From URL: Paste a full Superset dashboard URL (with filters/tabs). System parses URL and shows dashboard title, active filters, and tabs.
5. Step 3 — Analysis Options:
   • Select LLM Provider (filtered: multimodal-only when Screenshot enabled).
   • Toggle Screenshot (Playwright):
     • ON (Path A): Captures per-tab screenshots, sends to multimodal LLM. Catches visual bugs, rendering errors.
     • OFF (Path B): Text-only analysis via Superset API. Checks dataset health, KXD connectivity, chart parameters, execution logs. Lighter, faster, catches data-layer errors.
   • Toggle Logs: Include Superset execution logs in LLM context.
   • Toggle Execute Chart Data (Path B only): Runs actual chart queries to verify data returns. Off by default (load-aware).
   • Edit Prompt Template (optional): Customize what the LLM checks.
6. Step 4 — Schedule:
   • Manual: Run on demand via "Run Now" button.
   • Scheduled: Cron expression (e.g., 0 8 * * 1-5 for weekdays at 8 AM).
7. Click Create Task.

View Results

1. Go to Validation Tasks → see all tasks with last run status.
2. Click a task → see per-dashboard breakdown with PASS/WARN/FAIL badges.
3. Click a dashboard run → Full Report:
   • Path A: Tab selector + per-tab screenshots + issues table.
   • Path B: Dataset health table (backend, status, errors) + chart data results + issues table.
   • Both paths: Summary, execution logs sent to LLM, task execution logs, timing breakdown.

Dashboard Hub (changes)

• The per-row "Validate" action is removed.
• The LLM validation status column is removed.
• Each dashboard row now shows a colored dot indicator next to the title (🟢 PASS / 🟡 WARN / 🔴 FAIL / ⚪ UNKNOWN) reflecting the most recent validation across all tasks. Click the dot to see the last 3 runs with links to full reports.
• Use the «+ Create Validation Task» button in the toolbar to create tasks with pre-selected dashboard IDs.
• The dashboard detail page (/dashboards/{id}) now includes a «Validation History» section showing the last 10 validation runs for this dashboard.

Dataset Documentation

(unchanged from v1)

1. Go to Tools > Documentation.
2. Select a dataset.
3. Click "Generate Docs".
4. Review and apply generated descriptions.

Git Commit Generation

(unchanged from v1)

1. Stage files in the Git plugin.
2. Click the "✨" (Magic) button.
3. Review and edit the suggested message.

Path Selection Guide

Scenario	Choose
Checking if dashboards render correctly after Superset upgrade	Path A (Screenshot ON)
Checking if KXD/Trino data sources are healthy and returning data	Path B (Screenshot OFF)
Both visual and data-layer issues	Create two tasks — one Path A, one Path B
Lightweight daily health check with 20 dashboards	Path B, no chart data execution
Deep investigation of specific dashboard with filters	Path A with URL source (preserves filters)