busya/ss-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

tasks

Browse Source
This commit is contained in:
busya
2026-05-31 17:38:43 +03:00
parent e17f4f64a9
commit 652de471d2
11 changed files with 1387 additions and 442 deletions
Download Patch File Download Diff File Expand all files Collapse all files

88
specs/017-llm-analysis-plugin/quickstart.md
View File

@@ -1,42 +1,98 @@
# Quickstart: LLM Analysis Plugin
# Quickstart: LLM Validation v2
## Prerequisites
1. An API Key from OpenAI, OpenRouter, or Kilo.
2. (Optional) Playwright browsers installed for high-fidelity screenshots:
1. An API Key from OpenAI, OpenRouter, Kilo, or LiteLLM.
2. At least one environment configured in ss-tools (pointing to a Superset instance).
3. (Optional) Playwright browsers installed for Path A (screenshot) validation:
```bash
playwright install chromium
```
## Configuration
### LLM Provider
1. Navigate to **Settings > LLM Providers**.
2. Click **"Add Provider"**.
3. Select type (e.g., `OpenRouter`).
4. Enter your API Key and Base URL (e.g., `https://openrouter.ai/api/v1`).
5. Set a default model (e.g., `anthropic/claude-3.5-sonnet`).
6. Click **Save**.
5. Set a default model (e.g., `gpt-4o` for multimodal, `gpt-4o-mini` for text-only).
6. Check **"Is Multimodal"** if the model supports image input (required for Path A).
7. Click **Save** and **Test Connection**.
> **Note**: Provider bindings for Documentation, Git Commits, and Chat are configured in Settings. Dashboard Validation provider is selected **per-task** in the task creation form — not in global settings.
### Validation Prompt (Optional)
The default validation prompt covers standard dashboard health checks. To customize:
1. When creating a task, edit the **Prompt Template** field.
2. Path A uses `dashboard_validation_prompt_multimodal` (image-focused).
3. Path B uses `dashboard_validation_prompt_text` (topology-focused).
## Usage
### Dashboard Validation
### Create a Validation Task
1. Go to **Environments**.
2. Select a dashboard from the list.
3. Click **"Validate"**.
4. Wait for the task to complete and view the report in the **Task History** panel.
1. Go to **Validation Tasks** (`/validation-tasks`).
2. Click **"+ New Task"**.
3. **Step 1 — Name & Environment**: Enter task name (e.g., "Daily KXD Health Check"), select Superset environment.
4. **Step 2 — Sources**: Add dashboards to validate:
- **From List**: Check dashboards from the environment list.
- **From URL**: Paste a full Superset dashboard URL (with filters/tabs). System parses URL and shows dashboard title, active filters, and tabs.
5. **Step 3 — Analysis Options**:
- Select **LLM Provider** (filtered: multimodal-only when Screenshot enabled).
- Toggle **Screenshot** (Playwright):
- **ON** (Path A): Captures per-tab screenshots, sends to multimodal LLM. Catches visual bugs, rendering errors.
- **OFF** (Path B): Text-only analysis via Superset API. Checks dataset health, KXD connectivity, chart parameters, execution logs. Lighter, faster, catches data-layer errors.
- Toggle **Logs**: Include Superset execution logs in LLM context.
- Toggle **Execute Chart Data** (Path B only): Runs actual chart queries to verify data returns. Off by default (load-aware).
- Edit **Prompt Template** (optional): Customize what the LLM checks.
6. **Step 4 — Schedule**:
- **Manual**: Run on demand via "Run Now" button.
- **Scheduled**: Cron expression (e.g., `0 8 * * 1-5` for weekdays at 8 AM).
7. Click **Create Task**.
### View Results
1. Go to **Validation Tasks** → see all tasks with last run status.
2. Click a task → see per-dashboard breakdown with PASS/WARN/FAIL badges.
3. Click a dashboard run → **Full Report**:
- **Path A**: Tab selector + per-tab screenshots + issues table.
- **Path B**: Dataset health table (backend, status, errors) + chart data results + issues table.
- Both paths: Summary, execution logs sent to LLM, task execution logs, timing breakdown.
### Dashboard Hub (changes)
- The per-row "Validate" action is **removed**.
- The LLM validation status column is **removed**.
- Each dashboard row now shows a **colored dot indicator** next to the title (🟢 PASS / 🟡 WARN / 🔴 FAIL / ⚪ UNKNOWN) reflecting the most recent validation across all tasks. Click the dot to see the last 3 runs with links to full reports.
- Use the **«+ Create Validation Task»** button in the toolbar to create tasks with pre-selected dashboard IDs.
- The **dashboard detail page** (`/dashboards/{id}`) now includes a «Validation History» section showing the last 10 validation runs for this dashboard.
### Dataset Documentation
*(unchanged from v1)*
1. Go to **Tools > Documentation**.
2. Select a dataset.
3. Click **"Generate Docs"**.
4. Review the generated descriptions and changes will be applied to the metadata.
4. Review and apply generated descriptions.
### Git Commit Generation
1. Make changes to files in the repository.
2. Go to the **Git** plugin tab.
3. Stage your files.
4. Click the **"✨" (Magic)** button next to the commit message box.
5. Review and edit the suggested message before committing.
*(unchanged from v1)*
1. Stage files in the Git plugin.
2. Click the **"✨" (Magic)** button.
3. Review and edit the suggested message.
## Path Selection Guide
| Scenario | Choose |
|----------|--------|
| Checking if dashboards render correctly after Superset upgrade | Path A (Screenshot ON) |
| Checking if KXD/Trino data sources are healthy and returning data | Path B (Screenshot OFF) |
| Both visual and data-layer issues | Create two tasks — one Path A, one Path B |
| Lightweight daily health check with 20 dashboards | Path B, no chart data execution |
| Deep investigation of specific dashboard with filters | Path A with URL source (preserves filters) |