tasks
This commit is contained in:
@@ -1,6 +1,7 @@
|
||||
# Data Model: LLM Analysis Plugin
|
||||
# Data Model: LLM Analysis Plugin v2
|
||||
|
||||
**Feature**: `017-llm-analysis-plugin`
|
||||
**Updated**: 2026-05-31 — v2: ValidationPolicy expansion, ValidationSource, dual-path execution models
|
||||
|
||||
## Entities
|
||||
|
||||
@@ -10,43 +11,158 @@ Stores configuration for different LLM providers.
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| id | UUID | Yes | Unique identifier |
|
||||
| provider_type | Enum | Yes | `openai`, `openrouter`, `kilo` |
|
||||
| provider_type | Enum | Yes | `openai`, `openrouter`, `kilo`, `litellm` |
|
||||
| name | String | Yes | Display name (e.g., "Production GPT-4") |
|
||||
| base_url | String | Yes | API Endpoint URL |
|
||||
| api_key | String | Yes | Encrypted API Key |
|
||||
| api_key | String | Yes | Encrypted API Key (AES-256) |
|
||||
| default_model | String | Yes | Model identifier (e.g., `gpt-4o`) |
|
||||
| is_active | Boolean | Yes | Whether this provider is currently enabled |
|
||||
| is_multimodal | Boolean | Yes | Whether model supports image input (required for Path A) |
|
||||
|
||||
### ValidationResult
|
||||
Stores the outcome of a dashboard validation task.
|
||||
### ValidationPolicy (v2 — expanded)
|
||||
|
||||
Stores configuration for a scheduled validation task. Uses ORM relationship to `ValidationSource` table — no JSON column denormalization.
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| id | UUID | Yes | Unique identifier |
|
||||
| dashboard_id | String | Yes | Reference to the dashboard |
|
||||
| timestamp | DateTime | Yes | When the validation ran |
|
||||
| status | Enum | Yes | `PASS`, `WARN`, `FAIL` |
|
||||
| screenshot_path | String | No | Path to the captured screenshot (if stored) |
|
||||
| screenshot_metadata | JSON | No | `{width: 1920, height: dynamic, tabs_processed: []}` |
|
||||
| issues | JSON | Yes | List of detected issues `[{severity, message, location}]` |
|
||||
| raw_response | Text | No | Full LLM response for debugging |
|
||||
| name | String | Yes | Human-readable task name |
|
||||
| description | String | No | Optional description |
|
||||
| environment_id | String | Yes | Target Superset environment |
|
||||
| sources | relationship → list[ValidationSource] | Yes | ORM relationship only. Dashboards to validate (1..N). Migrated from v1 `dashboard_ids` list via `ValidationSource` rows |
|
||||
| source_snapshot | JSON | No | Read-only denormalized copy of sources at last edit time — for audit/diff only, never used as authoritative state |
|
||||
| provider_id | String | Yes | LLM provider to use (per-task, not global) |
|
||||
| prompt_template | String | No | Custom prompt; `null` = use path-specific default: `dashboard_validation_prompt_multimodal` for Path A, `dashboard_validation_prompt_text` for Path B. Must NOT fall back to v1 legacy `dashboard_validation_prompt` |
|
||||
| screenshot_enabled | Boolean | Yes | `true` = Path A (Playwright + multimodal LLM), `false` = Path B (text-only) |
|
||||
| logs_enabled | Boolean | Yes | Fetch Superset execution logs for LLM context |
|
||||
| execute_chart_data | Boolean | Yes | Path B only: execute `POST /api/v1/chart/data` for each chart (default: false) |
|
||||
| llm_batch_size | Integer | No | Path B only: dashboards per LLM call (default: 1 — full isolation, one call per dashboard). Values >1 are experimental. Path A always uses 1 (image token constraints). |
|
||||
| policy_dashboard_concurrency_limit | Integer | No | Max dashboards validated in parallel within one policy run (default: 3). Global worker limit controlled by `global_validation_worker_limit` config |
|
||||
| cron_expression | String | No | Cron schedule; null = manual trigger only |
|
||||
| is_active | Boolean | Yes | Whether schedule is currently active |
|
||||
| notification_channels | list[String] | No | Email/Pulse channel IDs for failure notifications |
|
||||
| created_at | DateTime | Yes | Creation timestamp |
|
||||
| updated_at | DateTime | Yes | Last modification timestamp |
|
||||
|
||||
### TaskConfiguration (Extension)
|
||||
Extends existing Task model to support validation-specific params.
|
||||
### ValidationSource (new)
|
||||
|
||||
Represents a single dashboard to validate within a task — either by numeric ID or by full Superset URL.
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| task_type | String | Yes | `dashboard_validation` |
|
||||
| parameters | JSON | Yes | `{dashboard_id, provider_id, screenshot_strategy}` |
|
||||
| id | UUID | Yes | Unique identifier |
|
||||
| policy_id | UUID | Yes | FK → ValidationPolicy |
|
||||
| type | Enum | Yes | `dashboard_id` or `dashboard_url` |
|
||||
| value | String | Yes | Numeric dashboard ID or full Superset URL |
|
||||
| parsed_context | JSON | No | For `dashboard_url`: result of `SupersetContextExtractor.parse_superset_link()` — contains dashboard_id, native_filters, activeTabs, anchor |
|
||||
| status | Enum | Yes | `valid`, `partial_recovery` (SupersetContextExtractor returned partial match), `stale` (permalink expired, URL parse warning), `invalid` (dashboard deleted or not found), `permission_denied`, `parse_failed` |
|
||||
| resolved_dashboard_id | String | No | Resolved numeric dashboard ID (cached after URL parse for display) |
|
||||
| title | String | No | Resolved dashboard title (cached for display) |
|
||||
| last_error | String | No | Last resolution error message |
|
||||
| last_checked_at | DateTime | No | When the source was last validated/resolved |
|
||||
|
||||
### Migration from v1 dashboard_ids:
|
||||
|
||||
```python
|
||||
# Existing dashboard_ids: list[str] → ValidationSource rows (relational only, no JSON denormalization)
|
||||
for dashboard_id in old_policy.dashboard_ids:
|
||||
db.add(ValidationSource(
|
||||
policy_id=policy.id,
|
||||
type="dashboard_id",
|
||||
value=dashboard_id,
|
||||
is_valid=True,
|
||||
))
|
||||
# Optional: populate source_snapshot JSON for audit trail
|
||||
policy.source_snapshot = {"dashboard_ids": list(old_policy.dashboard_ids)}
|
||||
```
|
||||
|
||||
### ValidationRun (new — groups records from one execution)
|
||||
|
||||
A single task execution validates ALL dashboards in a policy. `ValidationRun` groups these per-dashboard `ValidationRecord`s into one logical unit.
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| id | UUID | Yes | Unique run identifier |
|
||||
| policy_id | UUID | Yes | FK → ValidationPolicy |
|
||||
| task_id | String | No | Associated Task Manager task ID (one run = one task) |
|
||||
| started_at | DateTime | Yes | When the run started |
|
||||
| finished_at | DateTime | No | When all dashboards finished (null while running) |
|
||||
| trigger | Enum | Yes | `manual` or `scheduled` |
|
||||
| status | Enum | Yes | `completed` (all dashboards done), `partial` (some sources skipped/invalid), `failed` (execution error before any dashboard) |
|
||||
| dashboard_count | Integer | Yes | Total dashboards attempted |
|
||||
| pass_count | Integer | Yes | PASS results |
|
||||
| warn_count | Integer | Yes | WARN results |
|
||||
| fail_count | Integer | Yes | FAIL results |
|
||||
| unknown_count | Integer | Yes | UNKNOWN results (LLM errors, timeouts) |
|
||||
|
||||
### ValidationRecord (v2 — expanded, linked to run)
|
||||
|
||||
Stores the outcome of one dashboard validation within a task run. Linked to `ValidationRun` via `run_id`.
|
||||
|
||||
| Field | Type | Required | Description |
|
||||
|-------|------|----------|-------------|
|
||||
| id | UUID | Yes | Unique identifier |
|
||||
| run_id | UUID | Yes | FK → ValidationRun — groups per-dashboard results from one task execution |
|
||||
| task_id | String | No | Deprecated in favor of run_id; kept for v1 backward compat |
|
||||
| policy_id | UUID | No | FK → ValidationPolicy |
|
||||
| source_id | UUID | No | FK → ValidationSource |
|
||||
| dashboard_id | String | Yes | Resolved dashboard ID |
|
||||
| status | Enum | Yes | `PASS`, `WARN`, `FAIL`, `UNKNOWN` |
|
||||
| execution_path | Enum | Yes | `screenshot` (Path A) or `text_only` (Path B) |
|
||||
| summary | String | Yes | LLM-generated summary |
|
||||
| issues | JSON | Yes | `[{severity, message, location}]` |
|
||||
| dataset_health | JSON | No | Path B only: per-dataset status `[{dataset_id, name, database, backend, healthy, error, affected_charts}]` |
|
||||
| chart_data_results | JSON | No | Path B with `execute_chart_data=true`: `[{chart_id, executed, duration_ms, row_count, error}]` |
|
||||
| screenshot_paths | list[String] | No | Path A: paths to archived WebP screenshots (one per tab). PNG/JPEG intermediates deleted after conversion |
|
||||
| tab_screenshots | JSON | No | Path A: per-tab mapping `[{tab_name, webp_path, errored}]` |
|
||||
| logs_sent_to_llm | list[String] | No | Redacted Superset execution logs sent to LLM |
|
||||
| token_usage | JSON | No | LLM token usage `{prompt_tokens, completion_tokens, total_tokens}` (when provider reports it) |
|
||||
| raw_response | Text | No | Full LLM response for debugging. Retention: 30 days per FR-029a |
|
||||
| timings | JSON | No | Phase timings: `{dashboard_fetch_ms, dataset_fetch_ms, chart_fetch_ms, chart_data_exec_ms, screenshot_ms, llm_call_ms, total_ms}` |
|
||||
| created_at | DateTime | Yes | Record creation timestamp |
|
||||
|
||||
### Path B — Dataset Health Check (runtime model, not persisted separately)
|
||||
|
||||
Built during Path B execution from API responses:
|
||||
|
||||
```python
|
||||
class DatasetHealth:
|
||||
dataset_id: int
|
||||
dataset_name: str # e.g. "finance.kpi_revenue"
|
||||
database_name: str # e.g. "PostgreSQL / analytics_prod"
|
||||
backend: str # e.g. "postgresql", "trino"
|
||||
kind: str # "virtual" or "physical"
|
||||
is_accessible: bool # HTTP 200 from GET /api/v1/dataset/{id}
|
||||
error: str | None # Connection / timeout error
|
||||
affected_chart_ids: list[int] # Charts using this dataset
|
||||
```
|
||||
|
||||
### Chart Execution Result (runtime model, Path B + execute_chart_data)
|
||||
|
||||
```python
|
||||
class ChartDataResult:
|
||||
chart_id: int
|
||||
chart_name: str
|
||||
viz_type: str
|
||||
executed: bool # False if skipped
|
||||
duration_ms: int | None
|
||||
row_count: int | None
|
||||
error: str | None # Timeout / query error
|
||||
```
|
||||
|
||||
## API Contracts
|
||||
|
||||
See `contracts/` directory for OpenAPI specifications.
|
||||
See `contracts/` directory.
|
||||
|
||||
### Key Interactions
|
||||
### Key Interactions — v2
|
||||
|
||||
1. **Configure Provider**: `POST /api/settings/llm/providers`
|
||||
2. **Trigger Validation**: `POST /api/tasks/dispatch` (payload: `{type: "dashboard_validation", ...}`)
|
||||
3. **Get Results**: `GET /api/tasks/{task_id}/result` (or via WebSocket stream)
|
||||
4. **Generate Documentation**: `POST /api/tasks/dispatch` (payload: `{type: "dataset_documentation", ...}`)
|
||||
5. **Generate Commit**: `POST /api/git/generate-message`
|
||||
1. **Configure Provider**: `POST /api/llm/providers` (unchanged)
|
||||
2. **Create Validation Task**: `POST /api/validation-tasks` — creates `ValidationPolicy` + `ValidationSource`s
|
||||
3. **List Tasks**: `GET /api/validation-tasks` — returns all policies with aggregated status
|
||||
4. **Get Task Detail**: `GET /api/validation-tasks/{policy_id}` — returns policy + per-source status
|
||||
5. **Trigger Run**: `POST /api/validation-tasks/{policy_id}/run` — manual trigger (also fires on cron)
|
||||
6. **Get Run History**: `GET /api/validation-tasks/{policy_id}/runs` — paginated `ValidationRecord` list
|
||||
7. **Get Run Detail**: `GET /api/validation-tasks/{policy_id}/runs/{record_id}` — full report
|
||||
8. **Parse URL**: `POST /api/validation-tasks/parse-url` — returns `{dashboard_id, title, filters, tabs}` from URL
|
||||
9. **Generate Documentation**: `POST /api/tasks` with `plugin_id=llm_documentation` (unchanged)
|
||||
10. **Generate Commit Message**: `POST /api/git/generate-message` (unchanged)
|
||||
|
||||
Reference in New Issue
Block a user