ss-tools/specs/017-llm-analysis-plugin/data-model.md

Data Model: LLM Analysis Plugin v2

Feature: 017-llm-analysis-plugin Updated: 2026-05-31 — v2: ValidationPolicy expansion, ValidationSource, dual-path execution models

Entities

LLMProviderConfig

Stores configuration for different LLM providers.

Field	Type	Required	Description
id	UUID	Yes	Unique identifier
provider_type	Enum	Yes	openai, openrouter, kilo, litellm
name	String	Yes	Display name (e.g., "Production GPT-4")
base_url	String	Yes	API Endpoint URL
api_key	String	Yes	Encrypted API Key (AES-256)
default_model	String	Yes	Model identifier (e.g., gpt-4o)
is_active	Boolean	Yes	Whether this provider is currently enabled
is_multimodal	Boolean	Yes	Whether model supports image input (required for Path A)

ValidationPolicy (v2 — expanded)

Stores configuration for a scheduled validation task. Uses ORM relationship to ValidationSource table — no JSON column denormalization.

Field	Type	Required	Description
id	UUID	Yes	Unique identifier
name	String	Yes	Human-readable task name
description	String	No	Optional description
environment_id	String	Yes	Target Superset environment
sources	relationship → list[ValidationSource]	Yes	ORM relationship only. Dashboards to validate (1..N). Migrated from v1 dashboard_ids list via ValidationSource rows
source_snapshot	JSON	No	Read-only denormalized copy of sources at last edit time — for audit/diff only, never used as authoritative state
provider_id	String	Yes	LLM provider to use (per-task, not global)
prompt_template	String	No	Custom prompt; null = use path-specific default: dashboard_validation_prompt_multimodal for Path A, dashboard_validation_prompt_text for Path B. Must NOT fall back to v1 legacy dashboard_validation_prompt
screenshot_enabled	Boolean	Yes	true = Path A (Playwright + multimodal LLM), false = Path B (text-only)
logs_enabled	Boolean	Yes	Fetch Superset execution logs for LLM context
execute_chart_data	Boolean	Yes	Path B only: execute POST /api/v1/chart/data for each chart (default: false)
llm_batch_size	Integer	No	Path B only: dashboards per LLM call (default: 1 — full isolation, one call per dashboard). Values >1 are experimental. Path A always uses 1 (image token constraints).
policy_dashboard_concurrency_limit	Integer	No	Max dashboards validated in parallel within one policy run (default: 3). Global worker limit controlled by global_validation_worker_limit config
cron_expression	String	No	Cron schedule; null = manual trigger only
is_active	Boolean	Yes	Whether schedule is currently active
notification_channels	list[String]	No	Email/Pulse channel IDs for failure notifications
created_at	DateTime	Yes	Creation timestamp
updated_at	DateTime	Yes	Last modification timestamp

ValidationSource (new)

Represents a single dashboard to validate within a task — either by numeric ID or by full Superset URL.

Field	Type	Required	Description
id	UUID	Yes	Unique identifier
policy_id	UUID	Yes	FK → ValidationPolicy
type	Enum	Yes	dashboard_id or dashboard_url
value	String	Yes	Numeric dashboard ID or full Superset URL
parsed_context	JSON	No	For dashboard_url: result of SupersetContextExtractor.parse_superset_link() — contains dashboard_id, native_filters, activeTabs, anchor
status	Enum	Yes	valid, partial_recovery (SupersetContextExtractor returned partial match), stale (permalink expired, URL parse warning), invalid (dashboard deleted or not found), permission_denied, parse_failed
resolved_dashboard_id	String	No	Resolved numeric dashboard ID (cached after URL parse for display)
title	String	No	Resolved dashboard title (cached for display)
last_error	String	No	Last resolution error message
last_checked_at	DateTime	No	When the source was last validated/resolved

Migration from v1 dashboard_ids:

# Existing dashboard_ids: list[str] → ValidationSource rows (relational only, no JSON denormalization)
for dashboard_id in old_policy.dashboard_ids:
    db.add(ValidationSource(
        policy_id=policy.id,
        type="dashboard_id",
        value=dashboard_id,
        is_valid=True,
    ))
# Optional: populate source_snapshot JSON for audit trail
policy.source_snapshot = {"dashboard_ids": list(old_policy.dashboard_ids)}

ValidationRun (new — groups records from one execution)

A single task execution validates ALL dashboards in a policy. ValidationRun groups these per-dashboard ValidationRecords into one logical unit.

Field	Type	Required	Description
id	UUID	Yes	Unique run identifier
policy_id	UUID	Yes	FK → ValidationPolicy
task_id	String	No	Associated Task Manager task ID (one run = one task)
started_at	DateTime	Yes	When the run started
finished_at	DateTime	No	When all dashboards finished (null while running)
trigger	Enum	Yes	manual or scheduled
status	Enum	Yes	completed (all dashboards done), partial (some sources skipped/invalid), failed (execution error before any dashboard)
dashboard_count	Integer	Yes	Total dashboards attempted
pass_count	Integer	Yes	PASS results
warn_count	Integer	Yes	WARN results
fail_count	Integer	Yes	FAIL results
unknown_count	Integer	Yes	UNKNOWN results (LLM errors, timeouts)

ValidationRecord (v2 — expanded, linked to run)

Stores the outcome of one dashboard validation within a task run. Linked to ValidationRun via run_id.

Field	Type	Required	Description
id	UUID	Yes	Unique identifier
run_id	UUID	Yes	FK → ValidationRun — groups per-dashboard results from one task execution
task_id	String	No	Deprecated in favor of run_id; kept for v1 backward compat
policy_id	UUID	No	FK → ValidationPolicy
source_id	UUID	No	FK → ValidationSource
dashboard_id	String	Yes	Resolved dashboard ID
status	Enum	Yes	PASS, WARN, FAIL, UNKNOWN
execution_path	Enum	Yes	screenshot (Path A) or text_only (Path B)
summary	String	Yes	LLM-generated summary
issues	JSON	Yes	[{severity, message, location}]
dataset_health	JSON	No	Path B only: per-dataset status [{dataset_id, name, database, backend, healthy, error, affected_charts}]
chart_data_results	JSON	No	Path B with execute_chart_data=true: [{chart_id, executed, duration_ms, row_count, error}]
screenshot_paths	list[String]	No	Path A: paths to archived WebP screenshots (one per tab). PNG/JPEG intermediates deleted after conversion
tab_screenshots	JSON	No	Path A: per-tab mapping [{tab_name, webp_path, errored}]
logs_sent_to_llm	list[String]	No	Redacted Superset execution logs sent to LLM
token_usage	JSON	No	LLM token usage {prompt_tokens, completion_tokens, total_tokens} (when provider reports it)
raw_response	Text	No	Full LLM response for debugging. Retention: 30 days per FR-029a
timings	JSON	No	Phase timings: {dashboard_fetch_ms, dataset_fetch_ms, chart_fetch_ms, chart_data_exec_ms, screenshot_ms, llm_call_ms, total_ms}
created_at	DateTime	Yes	Record creation timestamp

Path B — Dataset Health Check (runtime model, not persisted separately)

Built during Path B execution from API responses:

class DatasetHealth:
    dataset_id: int
    dataset_name: str           # e.g. "finance.kpi_revenue"
    database_name: str          # e.g. "PostgreSQL / analytics_prod"
    backend: str                # e.g. "postgresql", "trino"
    kind: str                   # "virtual" or "physical"
    is_accessible: bool         # HTTP 200 from GET /api/v1/dataset/{id}
    error: str | None           # Connection / timeout error
    affected_chart_ids: list[int]  # Charts using this dataset

Chart Execution Result (runtime model, Path B + execute_chart_data)

class ChartDataResult:
    chart_id: int
    chart_name: str
    viz_type: str
    executed: bool              # False if skipped
    duration_ms: int | None
    row_count: int | None
    error: str | None           # Timeout / query error

API Contracts

See contracts/ directory.

Key Interactions — v2

1. Configure Provider: POST /api/llm/providers (unchanged)
2. Create Validation Task: POST /api/validation-tasks — creates ValidationPolicy + ValidationSources
3. List Tasks: GET /api/validation-tasks — returns all policies with aggregated status
4. Get Task Detail: GET /api/validation-tasks/{policy_id} — returns policy + per-source status
5. Trigger Run: POST /api/validation-tasks/{policy_id}/run — manual trigger (also fires on cron)
6. Get Run History: GET /api/validation-tasks/{policy_id}/runs — paginated ValidationRecord list
7. Get Run Detail: GET /api/validation-tasks/{policy_id}/runs/{record_id} — full report
8. Parse URL: POST /api/validation-tasks/parse-url — returns {dashboard_id, title, filters, tabs} from URL
9. Generate Documentation: POST /api/tasks with plugin_id=llm_documentation (unchanged)
10. Generate Commit Message: POST /api/git/generate-message (unchanged)