ss-tools/specs/037-superset-baseline-engine/spec.md

#region SupersetBaselineEngine.Spec [C:3] [TYPE ADR] [SEMANTICS spec,requirements,superset,baseline,dashboard-testing] @BRIEF Superset-native query and baseline engine for dashboard testing without direct SQL execution. @RELATION DEPENDS_ON -> [ADR-0001] @RELATION DEPENDS_ON -> [ADR-0003] @RELATION DEPENDS_ON -> [ADR-0005] @RELATION DEPENDS_ON -> [AgentTestStabilization.Spec] @RATIONALE Dashboard test assertions must validate the same Superset-side chart or dataset execution path that powers dashboards, not a parallel SQL path that can diverge from Superset filter/query semantics. @REJECTED Direct SQL execution for dashboard test truth — rejected because the target requirement is stable Superset dataset/chart execution and filter fidelity, not separate database querying.

Navigation (DSA Indexer keywords)

@SEMANTICS: spec, requirements, feature, superset, baseline, chart-data, dataset, filters, normalization, dashboard-testing

Feature Branch: 037-superset-baseline-engine
Created: 2026-07-07 | Status: Draft
Input: "Create a Superset-native query and baseline engine for dashboard testing. The system must inspect dashboard query models, map dashboard native filters into Superset chart or dataset query context, execute Superset-side chart or dataset queries without direct SQL, normalize returned metric and table values, compare them with approved baseline catalog entries, and create draft baseline candidates requiring human approval."

User Scenarios

Story 1 — Inspect Dashboard Query Model (P1)

Why P1: Scenario generation must know charts, datasets, metrics, filters, and filter scopes before it can build reliable tests.

Independent Test: Provide a dashboard fixture and verify the engine returns structured query model data for charts, datasets, metrics, native filters, and capabilities.

Acceptance:

1. Given a dashboard id and environment When the query model is inspected Then the result lists dashboard title, charts, datasets, metric labels/keys, native filters, filter targets, and export capabilities.
2. Given a native filter applies only to selected charts When inspection runs Then filter scope is represented so unrelated charts are not queried with invalid filters.
3. Given a chart or dataset is inaccessible When inspection runs Then the engine reports a structured warning or permission error without inventing missing metadata.

---

Story 2 — Execute Superset Query Context (P1)

Why P1: Baseline validation depends on executing Superset chart/dataset queries through Superset-native APIs, never through direct SQL.

Independent Test: Execute a fixture chart query with normalized dashboard filters and verify returned data is traceable to chart id, dataset id, metric, filters hash, and environment.

Acceptance:

1. Given a chart metric and normalized filter context When execution is requested Then the engine calls the Superset-native chart/dataset execution path and returns structured values.
2. Given dashboard filters are applied When query context is built Then Superset receives semantically equivalent filters to the UI native filter state.
3. Given a request would require arbitrary SQL text from the agent When execution is attempted Then the request is rejected as unsupported.

---

Story 3 — Normalize and Compare Results (P1)

Why P1: UI, Superset API, and XLSX values must be comparable despite formatting differences.

Independent Test: Feed formatted decimal, date, percent, big-number, and table-row fixtures and verify normalized values compare consistently with tolerance rules.

Acceptance:

1. Given Superset returns a metric value When normalization runs Then the value is represented with type, raw value, normalized value, label, and source metadata.
2. Given an approved baseline exists When an actual value is compared Then exact, absolute tolerance, relative tolerance, range, or row-set comparison rules are applied according to baseline policy.
3. Given actual data cannot be normalized When comparison is requested Then the result is inconclusive with an explanation, not a false pass.

---

Story 4 — Baseline Candidate Lifecycle (P2)

Why P2: The agent may discover candidate reference values but must not silently define the truth.

Independent Test: Run discovery for a metric without baseline and verify a draft candidate is produced; approving it requires HITL from feature 036.

Acceptance:

1. Given no approved baseline exists for metric+filters When discovery runs Then a draft baseline candidate is created with provenance and source values.
2. Given a dashboard/chart/dataset/filter fingerprint changes When existing baselines are loaded Then affected baselines are marked or reported as stale.
3. Given UI/API/XLSX sources disagree When a candidate is created Then candidate approval is blocked or warning-gated until a user reviews the discrepancy.

---

Edge Cases

• Superset returns empty result → compare result distinguishes expected empty, unexpected empty, and inconclusive.
• Filter value is not available in the target environment → query execution fails with recoverable validation error.
• Dashboard or chart metadata changes after baseline approval → fingerprint mismatch surfaces stale baseline warning.
• Superset API returns 403/404/422/5xx/timeout → taxonomy is preserved in comparison/report output.
• Date and number formats differ between locales → normalization uses canonical decimal/date representation.

Requirements

Functional

• AGBASE-FR-001: The engine MUST inspect dashboard query models into structured charts, datasets, metrics, native filters, filter scopes, and capabilities.
• AGBASE-FR-002: The engine MUST execute Superset-native chart/dataset query contexts without accepting arbitrary SQL text from the agent.
• AGBASE-FR-003: Dashboard native filters MUST be normalized into query filters with target columns, operators, values, scopes, and a deterministic filter hash.
• AGBASE-FR-004: Superset result normalization MUST support scalar metrics, big-number charts, table rows, dates, decimals, percentages, empty values, and raw/source metadata.
• AGBASE-FR-005: Baseline catalog entries MUST identify dashboard id, chart id or dataset id, metric/result key, normalized filters, expected value, tolerance, lifecycle status, fingerprints, and provenance.
• AGBASE-FR-006: Approved baselines MUST be stored as reviewable artifacts; observed runtime values and evidence remain separate from approved expectations.
• AGBASE-FR-007: The system MUST create draft baseline candidates only; approval is delegated to the HITL gate from 036.
• AGBASE-FR-008: Comparison output MUST include source, actual value, expected value, diff, status, tolerance rule, and stale/inconclusive warnings.
• AGBASE-FR-009: Direct SQL execution, generated SQL assertions, and SQL-based baseline provenance are explicitly out of scope for this feature.

Key Entities

• DashboardQueryModel: Structured representation of charts, datasets, metrics, native filters, scopes, and execution capabilities for a dashboard.
• NormalizedFilterContext: Canonical filter state shared by UI, Superset query execution, scenario graph, XLSX comparison, and baseline lookup.
• SupersetQueryContext: Executable Superset-native query request derived from chart/dataset metadata plus normalized filters.
• NormalizedSupersetResult: Canonical metric/table result with type metadata and provenance.
• BaselineEntry: Approved or draft expected value definition for a metric/table result under a normalized filter context.
• BaselineCandidate: Draft expected value discovered from Superset execution and awaiting human approval.
• ComparisonResult: Result of comparing actual normalized values to baseline expectations.

Success Criteria

• SC-001: Dashboard query model inspection returns complete chart/filter/dataset mappings for fixture dashboards with 100% deterministic JSON snapshots.
• SC-002: Superset-native execution validates at least scalar metric and table chart fixtures without direct SQL.
• SC-003: Decimal/date/percent normalization compares equivalent API/UI/XLSX formatted values without false diffs in fixture tests.
• SC-004: Baseline candidates are never marked approved without HITL approval metadata.
• SC-005: Fingerprint changes surface stale baseline warnings in 100% of changed chart/dataset/filter fixture cases.

#endregion SupersetBaselineEngine.Spec