busya
ec6421de35
- Replace all occurrences of 'ss-tools' with 'superset-tools' in 104 files - Rename git bundle file ss-tools.bundle → superset-tools.bundle - Update .gitignore pattern accordingly - Preserve variable names (hasSsTools etc.) and code identifiers
99 lines
4.6 KiB
Markdown
99 lines
4.6 KiB
Markdown
# Quickstart: LLM Validation v2
|
|
|
|
## Prerequisites
|
|
|
|
1. An API Key from OpenAI, OpenRouter, Kilo, or LiteLLM.
|
|
2. At least one environment configured in superset-tools (pointing to a Superset instance).
|
|
3. (Optional) Playwright browsers installed for Path A (screenshot) validation:
|
|
```bash
|
|
playwright install chromium
|
|
```
|
|
|
|
## Configuration
|
|
|
|
### LLM Provider
|
|
|
|
1. Navigate to **Settings > LLM Providers**.
|
|
2. Click **"Add Provider"**.
|
|
3. Select type (e.g., `OpenRouter`).
|
|
4. Enter your API Key and Base URL (e.g., `https://openrouter.ai/api/v1`).
|
|
5. Set a default model (e.g., `gpt-4o` for multimodal, `gpt-4o-mini` for text-only).
|
|
6. Check **"Is Multimodal"** if the model supports image input (required for Path A).
|
|
7. Click **Save** and **Test Connection**.
|
|
|
|
> **Note**: Provider bindings for Documentation, Git Commits, and Chat are configured in Settings. Dashboard Validation provider is selected **per-task** in the task creation form — not in global settings.
|
|
|
|
### Validation Prompt (Optional)
|
|
|
|
The default validation prompt covers standard dashboard health checks. To customize:
|
|
1. When creating a task, edit the **Prompt Template** field.
|
|
2. Path A uses `dashboard_validation_prompt_multimodal` (image-focused).
|
|
3. Path B uses `dashboard_validation_prompt_text` (topology-focused).
|
|
|
|
## Usage
|
|
|
|
### Create a Validation Task
|
|
|
|
1. Go to **Validation Tasks** (`/validation-tasks`).
|
|
2. Click **"+ New Task"**.
|
|
3. **Step 1 — Name & Environment**: Enter task name (e.g., "Daily KXD Health Check"), select Superset environment.
|
|
4. **Step 2 — Sources**: Add dashboards to validate:
|
|
- **From List**: Check dashboards from the environment list.
|
|
- **From URL**: Paste a full Superset dashboard URL (with filters/tabs). System parses URL and shows dashboard title, active filters, and tabs.
|
|
5. **Step 3 — Analysis Options**:
|
|
- Select **LLM Provider** (filtered: multimodal-only when Screenshot enabled).
|
|
- Toggle **Screenshot** (Playwright):
|
|
- **ON** (Path A): Captures per-tab screenshots, sends to multimodal LLM. Catches visual bugs, rendering errors.
|
|
- **OFF** (Path B): Text-only analysis via Superset API. Checks dataset health, KXD connectivity, chart parameters, execution logs. Lighter, faster, catches data-layer errors.
|
|
- Toggle **Logs**: Include Superset execution logs in LLM context.
|
|
- Toggle **Execute Chart Data** (Path B only): Runs actual chart queries to verify data returns. Off by default (load-aware).
|
|
- Edit **Prompt Template** (optional): Customize what the LLM checks.
|
|
6. **Step 4 — Schedule**:
|
|
- **Manual**: Run on demand via "Run Now" button.
|
|
- **Scheduled**: Cron expression (e.g., `0 8 * * 1-5` for weekdays at 8 AM).
|
|
7. Click **Create Task**.
|
|
|
|
### View Results
|
|
|
|
1. Go to **Validation Tasks** → see all tasks with last run status.
|
|
2. Click a task → see per-dashboard breakdown with PASS/WARN/FAIL badges.
|
|
3. Click a dashboard run → **Full Report**:
|
|
- **Path A**: Tab selector + per-tab screenshots + issues table.
|
|
- **Path B**: Dataset health table (backend, status, errors) + chart data results + issues table.
|
|
- Both paths: Summary, execution logs sent to LLM, task execution logs, timing breakdown.
|
|
|
|
### Dashboard Hub (changes)
|
|
|
|
- The per-row "Validate" action is **removed**.
|
|
- The LLM validation status column is **removed**.
|
|
- Each dashboard row now shows a **colored dot indicator** next to the title (🟢 PASS / 🟡 WARN / 🔴 FAIL / ⚪ UNKNOWN) reflecting the most recent validation across all tasks. Click the dot to see the last 3 runs with links to full reports.
|
|
- Use the **«+ Create Validation Task»** button in the toolbar to create tasks with pre-selected dashboard IDs.
|
|
- The **dashboard detail page** (`/dashboards/{id}`) now includes a «Validation History» section showing the last 10 validation runs for this dashboard.
|
|
|
|
### Dataset Documentation
|
|
|
|
*(unchanged from v1)*
|
|
|
|
1. Go to **Tools > Documentation**.
|
|
2. Select a dataset.
|
|
3. Click **"Generate Docs"**.
|
|
4. Review and apply generated descriptions.
|
|
|
|
### Git Commit Generation
|
|
|
|
*(unchanged from v1)*
|
|
|
|
1. Stage files in the Git plugin.
|
|
2. Click the **"✨" (Magic)** button.
|
|
3. Review and edit the suggested message.
|
|
|
|
## Path Selection Guide
|
|
|
|
| Scenario | Choose |
|
|
|----------|--------|
|
|
| Checking if dashboards render correctly after Superset upgrade | Path A (Screenshot ON) |
|
|
| Checking if KXD/Trino data sources are healthy and returning data | Path B (Screenshot OFF) |
|
|
| Both visual and data-layer issues | Create two tasks — one Path A, one Path B |
|
|
| Lightweight daily health check with 20 dashboards | Path B, no chart data execution |
|
|
| Deep investigation of specific dashboard with filters | Path A with URL source (preserves filters) |
|