ss-tools/specs/038-dashboard-scenario-model/spec.md

#region DashboardScenarioModel.Spec [C:3] [TYPE ADR] [SEMANTICS spec,requirements,scenario,dashboard-testing] @BRIEF Define a validated ScenarioGraph model for unique dashboard test flows generated by agents. @RELATION DEPENDS_ON -> [ADR-0001] @RELATION DEPENDS_ON -> [ADR-0002] @RELATION DEPENDS_ON -> [AgentTestStabilization.Spec] @RELATION DEPENDS_ON -> [SupersetBaselineEngine.Spec] @RATIONALE Unique dashboard tests need a stable intermediate model between agent reasoning and generated artifacts; direct LLM-to-code generation is not reviewable or safely composable. @REJECTED Asking users to choose low-level outputs such as Playwright vs SQL vs XLSX — rejected because each dashboard scenario is a goal-oriented flow whose steps select tools automatically. @REJECTED Direct generation of executable scripts without a validated scenario graph — rejected because it hides missing selectors, baseline refs, and unsafe steps until runtime.

Navigation (DSA Indexer keywords)

@SEMANTICS: spec, requirements, feature, scenario, graph, dashboard-testing, checklist, validation, artifacts

Feature Branch: 038-dashboard-scenario-model
Created: 2026-07-07 | Status: Draft
Input: "Define the dashboard test scenario model used by agents to represent unique dashboard test flows as a validated ScenarioGraph. The model must express ordered and dependent steps across browser automation, Superset query execution, XLSX parsing, assertions, screenshots, reports, human checkpoints, baseline references, parameters, warnings, and missing context markers without exposing users to low-level tool selection."

User Scenarios

Story 1 — Build Scenario Graph From Dashboard Goal (P1)

Why P1: The agent must express a dashboard testing goal as a reviewable graph of steps before generating any executable artifacts.

Independent Test: Provide dashboard query model and checklist fixture input and verify a deterministic DashboardTestScenario graph is produced.

Acceptance:

1. Given a dashboard context, query model, and testing objective When the agent builds a scenario Then the output contains scenario id, dashboard context, parameters, steps, dependencies, expected outputs, warnings, and risk summary.
2. Given a scenario needs browser, Superset API, XLSX parsing, screenshots, assertions, and report steps When graph is rendered Then every step declares its tool and input/output refs.
3. Given required metadata is missing When graph is generated Then missing data is represented as NEEDS_CONTEXT or NEEDS_SELECTOR, not invented.

---

Story 2 — Validate Scenario Graph Safety and Completeness (P1)

Why P1: Scenario artifacts must be generated only from a graph whose refs, baselines, tools, and risks are internally consistent.

Independent Test: Run valid and invalid scenario fixtures through the validator and verify precise errors for broken refs, missing baseline refs, and unsupported steps.

Acceptance:

1. Given a step consumes an output ref When validation runs Then the referenced producer must exist and precede or depend correctly.
2. Given an assertion compares a metric to expected truth When validation runs Then it must reference an approved baseline or a draft baseline candidate with explicit approval requirement.
3. Given a step uses an unsupported tool or unsafe action When validation runs Then the graph is rejected or marked blocked with a user-visible reason.

---

Story 3 — Map Checklist Cases to Scenario Capabilities (P2)

Why P2: The PDF checklist should guide scenario coverage without forcing all dashboards into identical scripts.

Independent Test: Normalize the research checklist into capability-tagged cases and verify generated scenarios include applicable cases and mark non-applicable ones.

Acceptance:

1. Given the normalized checklist template contains basic, complex, and technical cases When a dashboard capability model is supplied Then applicable cases are mapped to scenario steps or human checkpoints.
2. Given a checklist case requires capabilities absent from the dashboard When mapping runs Then the case is marked unsupported or manual-only with rationale.
3. Given multiple tool choices could verify a case When mapping runs Then the scenario selects the tool chain that best matches the business goal and available capabilities.

---

Story 4 — Represent Parameters and Human Checkpoints (P2)

Why P2: Unique dashboard scenarios require business inputs and sometimes cannot be fully automated.

Independent Test: Generate a scenario that requires test date, counterparty, and baseline selection; verify parameter prompts and human checkpoint steps are represented structurally.

Acceptance:

1. Given a scenario needs test data When graph is generated Then required parameters include name, type, validation rule, default/source, and affected steps.
2. Given an action cannot be automated reliably When graph is generated Then a human checkpoint step describes the manual action and expected evidence.
3. Given a user later supplies a parameter When graph is resolved Then dependent steps reference the resolved value without changing unrelated graph structure.

---

Edge Cases

• Scenario has a cycle in dependencies → validator rejects with cycle path.
• Two steps produce the same output ref → validator rejects ambiguous ref.
• Baseline is stale → assertion step remains present but blocked or warning-gated.
• XLSX export is unavailable → XLSX-dependent checklist cases become unsupported or manual checkpoints.
• UI selector is unknown → browser step uses NEEDS_SELECTOR and blocks executable generation for that step.

Requirements

Functional

• AGSCN-FR-001: The system MUST define a DashboardTestScenario model with dashboard context, objective, parameters, steps, dependencies, outputs, artifacts, risks, and warnings.
• AGSCN-FR-002: Every scenario step MUST declare tool category, action, inputs, outputs, expected result, dependencies, and automation status.
• AGSCN-FR-003: Supported tool categories MUST include browser automation, Superset API execution, XLSX parsing, assertion, screenshot/evidence, report generation, artifact generation, and human checkpoint.
• AGSCN-FR-004: Assertions against reference values MUST use baseline references or baseline candidate references; raw expected numbers MUST NOT be embedded directly in executable steps.
• AGSCN-FR-005: Scenario validation MUST detect missing refs, cycles, duplicate outputs, missing baselines, stale baselines, unknown selectors, unsupported tools, and unresolved required parameters.
• AGSCN-FR-006: Checklist mapping MUST use normalized checklist cases derived from the research PDF and capability tags, not hardcoded one-size-fits-all scripts.
• AGSCN-FR-007: The model MUST allow manual/human checkpoint steps where automation is unsafe, unavailable, or underspecified.
• AGSCN-FR-008: Scenario output MUST be deterministic for the same dashboard query model, checklist template, baseline catalog, and user parameters.
• AGSCN-FR-009: The scenario model MUST remain implementation-neutral and must not require the user to choose low-level artifacts such as Playwright, XLSX, or API output upfront.

Key Entities

• DashboardTestScenario: Reviewable graph representing a dashboard-specific testing objective and all steps required to validate it.
• ScenarioStep: One executable, generated, assertion, evidence, or human checkpoint node in the graph.
• ScenarioParameter: User- or context-provided input used by one or more scenario steps.
• ScenarioRef: Named output produced by one step and consumed by later steps.
• ChecklistCase: Normalized item from the research checklist with capability tags and expected verification semantics.
• CapabilityMapping: Decision record mapping dashboard capabilities to applicable checklist cases and step templates.
• ScenarioValidationResult: Structured validator output with errors, warnings, blockers, and graph coverage.

Success Criteria

• SC-001: Scenario validator catches 100% of invalid fixture cases for missing refs, cycles, duplicate outputs, and missing baseline refs.
• SC-002: At least 80% of normalized PDF checklist cases are classifiable as automated, human checkpoint, unsupported, or needs-context for fixture dashboards.
• SC-003: Same inputs produce byte-stable scenario JSON/YAML in deterministic snapshot tests.
• SC-004: No generated scenario fixture embeds raw baseline numbers directly in executable steps.
• SC-005: Scenario graph preview can display phase order, tools per step, parameters, warnings, and blockers without reading generated code.

#endregion DashboardScenarioModel.Spec