ss-tools/specs/038-dashboard-scenario-model/spec.md

#region DashboardScenarioModel.Spec [C:3] [TYPE ADR] [SEMANTICS spec,requirements,scenario,dashboard-testing] @BRIEF Define a validated ScenarioGraph model for unique dashboard test flows generated by agents. @RELATION DEPENDS_ON -> [Doc.Adr.ADR0001] @RELATION DEPENDS_ON -> [Doc.Adr.ADR0002] @RELATION DEPENDS_ON -> [AgentTestStabilization.Spec] @RELATION DEPENDS_ON -> [SupersetBaselineEngine.Spec] @RATIONALE Unique dashboard tests need a stable intermediate model between agent reasoning and generated artifacts; direct LLM-to-code generation is not reviewable or safely composable. @REJECTED Asking users to choose low-level outputs such as Playwright vs SQL vs XLSX — rejected because each dashboard scenario is a goal-oriented flow whose steps select tools automatically. @REJECTED Direct generation of executable scripts without a validated scenario graph — rejected because it hides missing selectors, baseline refs, and unsafe steps until runtime.

Navigation (DSA Indexer keywords)

@SEMANTICS: spec, requirements, feature, scenario, graph, dashboard-testing, checklist, validation, artifacts

Feature Branch: 038-dashboard-scenario-model Created: 2026-07-07 | Reworked: 2026-07-31 | Status: Reworked per new speckit flow Input: "Define the dashboard test scenario model used by agents to represent unique dashboard test flows as a validated ScenarioGraph. The model must express ordered and dependent steps across browser automation, Superset query execution, XLSX parsing, assertions, screenshots, reports, human checkpoints, baseline references, parameters, warnings, and missing context markers without exposing users to low-level tool selection."

Applicability

• Feature type: Fullstack (backend compiler/validator/resolver core + thin UI preview surface consumed by 039 + agent tools).
• UI surface: Yes — scenario graph preview, coverage, resolution, and pack states (rendered by 039; 038 supplies DTO contracts).
• API surface: Yes — compile, validate, resolve, draft-pack, capture, VLM, disposition endpoints.
• Prototype: Applicable — scenario preview is a real UI surface (see prototype/index.html).
• OpenAPI: Applicable — REST surface is a first-class deliverable (see contracts/openapi.yaml).

User Scenarios

Story 1 — Build Scenario Graph From Dashboard Goal (P1)

Why P1: The agent must express a dashboard testing goal as a reviewable graph of steps before generating any executable artifacts.

Independent Test: Provide dashboard query model and checklist fixture input and verify a deterministic DashboardTestScenario graph is produced.

Acceptance:

1. Given a dashboard context, query model, and testing objective When the agent builds a scenario Then the output contains scenario id, dashboard context, parameters, steps, dependencies, expected outputs, warnings, and risk summary.
2. Given a scenario needs browser, Superset API, XLSX parsing, screenshots, assertions, and report steps When graph is rendered Then every step declares its tool and input/output refs.
3. Given required metadata is missing When graph is generated Then missing data is represented as NEEDS_CONTEXT or NEEDS_SELECTOR, not invented.

---

Story 2 — Validate Scenario Graph Safety and Completeness (P1)

Why P1: Scenario artifacts must be generated only from a graph whose refs, baselines, tools, and risks are internally consistent.

Independent Test: Run valid and invalid scenario fixtures through the validator and verify precise errors for broken refs, missing baseline refs, and unsupported steps.

Acceptance:

1. Given a step consumes an output ref When validation runs Then the referenced producer must exist and precede or depend correctly.
2. Given an assertion compares a metric to expected truth When validation runs Then it must reference an approved baseline or a draft baseline candidate with explicit approval requirement.
3. Given a step uses an unsupported tool or unsafe action When validation runs Then the graph is rejected or marked blocked with a user-visible reason.

---

Story 3 — Map Checklist Cases to Scenario Capabilities (P2)

Why P2: The PDF checklist should guide scenario coverage without forcing all dashboards into identical scripts.

Independent Test: Normalize the research checklist into capability-tagged cases and verify generated scenarios include applicable cases and mark non-applicable ones.

Acceptance:

1. Given the normalized checklist template contains basic, complex, and technical cases When a dashboard capability model is supplied Then applicable cases are mapped to scenario steps or human checkpoints.
2. Given a checklist case requires capabilities absent from the dashboard When mapping runs Then the case is marked unsupported or manual-only with rationale.
3. Given multiple tool choices could verify a case When mapping runs Then the scenario selects the tool chain that best matches the business goal and available capabilities.

---

Story 4 — Represent Parameters and Human Checkpoints (P2)

Why P2: Unique dashboard scenarios require business inputs and sometimes cannot be fully automated.

Independent Test: Generate a scenario that requires test date, counterparty, and baseline selection; verify parameter prompts and human checkpoint steps are represented structurally.

Acceptance:

1. Given a scenario needs test data When graph is generated Then required parameters include name, type, validation rule, default/source, and affected steps.
2. Given an action cannot be automated reliably When graph is generated Then a human checkpoint step describes the manual action and expected evidence.
3. Given a user later supplies a parameter When graph is resolved Then dependent steps reference the resolved value without changing unrelated graph structure.

---

Story 5 — Capture, VLM Analysis, and Human Disposition (P2)

Why P2: Screenshot evidence and visual verification require typed, auditable capture/VLM/disposition semantics (AGSCN-FR-010..012).

Independent Test: Generate a scenario with screenshot capture, VLM analysis, and human disposition; verify typed findings and auditable dispositions.

Acceptance:

1. Given a screenshot step When capture executes Then a reproducible ScreenshotCaptureSpec (target, viewport, readiness, masking, max wait) is honored and artifacts are registered.
2. Given a masked screenshot When VLM analysis runs Then typed VlmFinding[] (severity, region, confidence, model/prompt provenance) are returned; raw prose is never treated as step state.
3. Given a human checkpoint references VLM finding ids When the user disposes Then confirm/dismiss/inconclusive is typed and auditable, and disposition never mutates the graph structure.

Edge & Failure Cases

#	Scenario	Category	Expected Behavior	Recovery / Test Ownership
E1	Scenario has a cycle in dependencies	data-integrity	Validator rejects with cycle path	User fixes graph; L1 validator test
E2	Two steps produce the same output ref	data-integrity	Validator rejects ambiguous ref	User fixes ref; L1 test
E3	Baseline is stale	data-quality	Assertion stays present but blocked/warning-gated	037 baseline discovery or mark pending; L1 test
E4	XLSX export unavailable	integration	XLSX-dependent checklist cases become unsupported or manual checkpoints	Rationale shown in coverage; L1 mapping test
E5	UI selector unknown	integration	Browser step uses NEEDS_SELECTOR and blocks executable generation	User provides selector hint / converts to checkpoint; L1 test
E6	409 stale base revision on resolve	concurrency	New revision rejected with 409; snapshot/recompile guidance; never silent merge	User recompiles; L1 API test
E7	422 invalid resolution/parameter type	validation	Field/step-mapped validation error	User corrects input; L1 API test
E8	403 forbidden role on scenario operations	auth	Permission denial rendered without approval gate	User contacts admin / RBAC test
E9	429 rate limit on compile/validate	throttling	Retry-After honored; UI countdown	User waits; L2 UX test
E10	5xx backend failure on compile	server-error	Error section + retry; partial graph not persisted	User retries; L2 UX test
E11	Malformed VLM response / empty findings	integration	Findings array empty; step inconclusive with reason; stale prompt blocked (422 STALE_PROMPT)	Re-run analysis; L1 VLM test
E12	Missing parameter value on pack compile	data-quality	Pack becomes preview_only with all save blockers listed	User resolves parameters; L1 pack test
E13	Unsafe path / executable code / SQL injection into pack	security	Template/path/code validation blocks before draft registration	L1 security test; injected-code fixture
E14	Duplicate submit of draft-pack	idempotency	Idempotency key / revision hash prevents double registration	L1 API test; 409 on changed revision

Requirements

Functional

• AGSCN-FR-001: The system MUST define a DashboardTestScenario model with dashboard context, objective, parameters, steps, dependencies, outputs, artifacts, risks, and warnings.
• AGSCN-FR-002: Every scenario step MUST declare tool category, action, inputs, outputs, expected result, dependencies, and automation status.
• AGSCN-FR-003: Supported tool categories MUST include browser automation, Superset API execution, XLSX parsing, assertion, screenshot/evidence, report generation, artifact generation, and human checkpoint.
• AGSCN-FR-004: Assertions against reference values MUST use baseline references or baseline candidate references; raw expected numbers MUST NOT be embedded directly in executable steps.
• AGSCN-FR-005: Scenario validation MUST detect missing refs, cycles, duplicate outputs, missing baselines, stale baselines, unknown selectors, unsupported tools, and unresolved required parameters.
• AGSCN-FR-006: Checklist mapping MUST use normalized checklist cases derived from the research PDF and capability tags, not hardcoded one-size-fits-all scripts. Mapping accepts the target release_version for baseline lookup.
• AGSCN-FR-007: The model MUST allow manual/human checkpoint steps where automation is unsafe, unavailable, or underspecified.
• AGSCN-FR-008: Scenario output MUST be deterministic for the same dashboard query model, checklist template, baseline catalog, and user parameters.
• AGSCN-FR-009: The scenario model MUST remain implementation-neutral and must not require the user to choose low-level artifacts such as Playwright, XLSX, or API output upfront.
• AGSCN-FR-010: Screenshot steps MUST carry a capture specification: target (tab/viewport), viewport dimensions, readiness strategy, masking selectors, and max wait.
• AGSCN-FR-011: VLM-based visual analysis steps MUST produce typed findings referencing the source screenshot artifact, including severity, region, confidence, model/prompt provenance, and a bounded textual description.
• AGSCN-FR-012: Human checkpoint steps MAY reference specific VLM finding IDs. Resolution options (confirm, dismiss, inconclusive) MUST be typed and auditable. Disposition changes finding status, not graph structure.

Key Entities

• DashboardTestScenario: Reviewable graph representing a dashboard-specific testing objective and all steps required to validate it.
• ScenarioStep: One executable, generated, assertion, evidence, or human checkpoint node in the graph.
• ScenarioParameter: User- or context-provided input used by one or more scenario steps.
• ScenarioRef: Named output produced by one step and consumed by later steps.
• ChecklistCase: Normalized item from the research checklist with capability tags and expected verification semantics.
• CapabilityMapping: Decision record mapping dashboard capabilities to applicable checklist cases and step templates.
• ScenarioValidationResult: Structured validator output with errors, warnings, blockers, and graph coverage.
• ScreenshotCaptureSpec: Capture configuration for a screenshot step — viewport, target tab, readiness strategy, masking selectors, and timeouts.
• VlmFinding: Typed VLM analysis output referencing a source screenshot artifact: severity, region, confidence, model/prompt provenance hash, and description.
• HumanDisposition: Typed human decision on a VLM finding: confirm, dismiss, or inconclusive, with optional comment.

Success Criteria

• SC-001: Scenario validator catches 100% of invalid fixture cases for missing refs, cycles, duplicate outputs, and missing baseline refs.
• SC-002: At least 80% of normalized PDF checklist cases are classifiable as automated, human checkpoint, unsupported, or needs-context for fixture dashboards.
• SC-003: Same inputs produce byte-stable scenario JSON/YAML in deterministic snapshot tests.
• SC-004: No generated scenario fixture embeds raw baseline numbers directly in executable steps.
• SC-005: Scenario graph preview can display phase order, tools per step, parameters, warnings, and blockers without reading generated code.
• SC-006: VLM findings are advisory only; disposition is auditable; no finding alters metric baseline truth.

Clarifications

Session 2026-07-31

• Q: Is the scenario model a backend-only library or does it expose a UI/API surface? → A: Fullstack — backend compiler/validator core plus thin UI preview (rendered by 039) plus REST/agent-tool API surface.
• Q: How are missing selectors/context handled? → A: Represented structurally as NEEDS_SELECTOR/NEEDS_CONTEXT, never invented; blocks executable generation for that step.
• Q: What is the determinism contract? → A: Byte-stable output for identical canonical inputs + compiler/template versions; stable derived ids, no random UUIDs in canonical graph; temperature=0 alone is rejected as a determinism mechanism.
• Q: What is the VLM safety boundary? → A: VLM output is typed, advisory findings for human review; raw prose is never step state and findings never alter baseline truth; stale prompts block analysis.
• Q: Is the draft-pack compiled through templates or direct code generation? → A: Versioned repository-owned templates only; LLM text may populate bounded descriptions but never executable code bodies, paths, imports, or shell commands.

#endregion DashboardScenarioModel.Spec