ss-tools/specs/038-dashboard-scenario-model/tasks.md

#region DashboardScenarioModel.Tasks [C:3] [TYPE ADR] [SEMANTICS tasks,scenario,implementation] @BRIEF Ordered TDD backlog for deterministic scenario graph, safe draft-pack compilation, capture/VLM/disposition, and full verification gates.

Prerequisites: plan.md, spec.md (required); contracts/modules.md, contracts/openapi.yaml, traceability.md, prototype/manifest.md (present). Tests: Write tests FIRST (fail before implementation) for every C3+ contract per constitution VII. Frontend tasks are N/A — 038 is DTO-only; UI rendering is owned by 039.

Format: - [ ] T### [P] [USx] Description with exact file path

Phase 1 — Setup (Shared Infrastructure)

• T001 Transcribe checklist-catalog.md into a versioned declarative resource in backend/src/services/dashboard_testing/scenario/catalog_v1.yaml
• T002 [P] Write catalog completeness tests for B01–B09, C01–C07, T01–T03 in backend/tests/services/dashboard_testing/scenario/test_catalog.py
• T003 [P] Create valid 18-step scenario plus invalid cycle/missing-ref/duplicate-output/raw-baseline/SQL fixtures under specs/038-dashboard-scenario-model/fixtures/
• T004 [P] Materialize fixtures from specs/038-dashboard-scenario-model/fixtures/ into backend/tests/fixtures/dashboard_scenarios/ (copy as-is, no adaptation)
• T005 Implement Pydantic models in backend/src/services/dashboard_testing/scenario/models.py matching contracts/dashboard-test-scenario.schema.json

Checkpoint: Catalog loads 19 cases; schema round-trips against golden fixtures.

Phase 2 — US1 Compile Scenario Graph

• T006 [US1] Write failing capability mapping tests in backend/tests/services/dashboard_testing/scenario/test_capability_mapper.py
• T007 [US1] Implement backend/src/services/dashboard_testing/scenario/checklist_catalog.py validation and backend/src/services/dashboard_testing/scenario/capability_mapper.py with complete classification
• T008 [US1] Cover unavailable XLSX, missing selector/test data, unsafe mutation context, cross-dashboard absence, and T01–T03 no-SQL fallbacks @TEST_EDGE: xlsx_unavailable→manual/unsupported, technical_without_dataset_fields→human checkpoint (no SQL)
• T009 [US1] Write failing deterministic compiler tests in backend/tests/services/dashboard_testing/scenario/test_compiler.py
• T010 [US1] Implement registered tool/action and step-template catalogs under backend/src/services/dashboard_testing/scenario/templates/
• T011 [US1] Implement backend/src/services/dashboard_testing/scenario/compiler.py with stable ids, phase order, refs, coverage, and fingerprints @PRE: intent, query model, catalog, baseline summary, parameters have valid fingerprints @POST: same canonical inputs/compiler version yield byte-identical graph and stable ids/order @DATA_CONTRACT: CompileScenarioRequest → DashboardTestScenario @TEST_EDGE: missing_selector→NEEDS_SELECTOR + save blocker, missing_baseline→NEEDS_BASELINE (no embedded numeric truth)
• T012 [US1] Prove repeated compile and shuffled input order produce identical graph bytes @INVARIANT: Deterministic_Graph → VERIFIED_BY: repeated_compile, shuffled_input_order

Checkpoint: Valid fixture compiles to stable graph and classifies all 19 cases.

Phase 3 — US2 Validate Safety and Completeness

• T013 [US2] Write failing full invalid-fixture matrix in backend/tests/services/dashboard_testing/scenario/test_validator.py
• T014 [US2] Implement schema, ref producer/consumer, duplicate, dependency, and cycle checks in backend/src/services/dashboard_testing/scenario/validator.py @POST: valid is true only with zero errors/blockers; findings stably ordered and actionable @TEST_EDGE: cycle→error contains cycle path, duplicate_output→both producer ids reported
• T015 [US2] Implement parameter, selector, baseline, tool/action, path, SQL/code, raw-expected, and coverage checks @TEST_EDGE: raw_metric_expected→forbidden baseline literal error, unreachable_step→warning/error per coverage
• T016 [US2] Return deterministic all-findings output with JSON pointers and recovery options
• T017 [US2] Add property tests generating small DAG/cycle/ref variations without mirroring validator logic in backend/tests/services/dashboard_testing/scenario/test_validator_properties.py
• T017b [P] [US2] Add belief-runtime instrumentation tests for ScenarioGraph.Validator.Validate in backend/tests/services/dashboard_testing/scenario/test_validator_belief.py @POST: REASON logged before mutation boundary; REFLECT after; belief_scope wraps validator run

Checkpoint: Invalid fixture matrix passes; no SQL/raw-baseline/cycle escapes.

Phase 4 — US3 Checklist Coverage and Serialization

• T018 [US3] Write JSON/YAML golden tests in backend/tests/services/dashboard_testing/scenario/test_serializer.py
• T019 [US3] Implement backend/src/services/dashboard_testing/scenario/serializer.py and revision hash exclusions @POST: key/order/decimal/date/newline rules stable across runs; JSON and YAML equal domain data @TEST_EDGE: shuffled_dicts→identical bytes, timestamp_display_field→excluded from revision identity
• T020 [US3] Validate JSON Schema and Pydantic round-trip for all golden fixtures
• T021 [US3] Add catalog-version and compiler-version fingerprints to scenario inputs

Checkpoint: Byte-stable snapshots across runs.

Phase 5 — US4 Parameters and Human Checkpoints

• T022 [US4] Write failing typed resolution/stale revision tests in backend/tests/services/dashboard_testing/scenario/test_resolver.py
• T023 [US4] Implement backend/src/services/dashboard_testing/scenario/resolver.py for parameter, selector, manual conversion, and remove-step operations @PRE: base revision hash matches; changes target declared unresolved items @POST: unrelated step ids/order unchanged; new parent/revision hashes link revisions @TEST_EDGE: stale_base_revision→409, invalid_parameter_type→422, unrelated_graph_change→invariant failure
• T024 [US4] Enforce immutable revisions and unchanged unrelated step ids/order
• T025 [US4] Cover safe-environment/test-data requirements for mutating PDF cases

Checkpoint: Resolution produces linked immutable revisions; unrelated structure stable.

Phase 6 — Safe Draft Pack

• T026 Write failing template registry, preview-only, path, SQL, shell, and injected-code tests in backend/tests/services/dashboard_testing/scenario/test_pack_compiler.py @TEST_INVARIANT: No_LLM_To_Code → VERIFIED_BY: injected_code_field, template_registry_only @TEST_EDGE: unknown_template→blocked, path_traversal→blocked before artifact registration
• T027 [P] Create versioned templates for scenario.yaml, runner.plan.json, report_template.md, evidence_manifest.json, and bounded browser/XLSX modules under backend/src/services/dashboard_testing/scenario/pack_templates/v1/
• T028 Implement backend/src/services/dashboard_testing/scenario/pack_compiler.py; accept only registered template ids and structured inputs @PRE: scenario validation result available; template ids registered; target paths safe @POST: outputs match ArtifactPlan, contain no LLM executable bodies, and are registered as 036 drafts @SIDE_EFFECT: renders bounded templates; calls AgentRuns.Artifacts.Register @INVARIANT: errors/unresolved required inputs make pack preview_only
• T029 [P] Register outputs through 036 AgentRuns.Artifacts.Register and emit generate/validate progress
• T030 Ensure invalid/unresolved graphs produce preview_only with repeated blockers

Checkpoint: Valid graph → save_eligible pack; invalid/injected → preview_only/blocked.

Phase 7 — API, Agent, Quality

• T031 Add backend/src/api/routes/dashboard_scenarios.py matching contracts/openapi.yaml and register router
• T032 Write RBAC/contract/revision tests in backend/tests/api/test_dashboard_scenarios.py @TEST_EDGE: 401→UNAUTHORIZED, 403→FORBIDDEN (per scope), 409 stale→STALE_REVISION, 422→VALIDATION_ERROR, 429→RATE_LIMITED
• T033 [P] Add thin compile/validate/resolve/generate tools in agent/src/ss_tools/agent/tools.py
• T034 Verify agent schemas cannot carry code, SQL, raw expected metrics, custom tools, or artifact paths
• T035 Run quickstart, JSON/OpenAPI schema validation, scoped/full backend tests, and ruff
• T036 Audit all 19 cases, direct SQL/code bans, contract anchors, ATTN_1–4, and unresolved relations

Checkpoint: API + agent tools match openapi.yaml; RBAC enforced.

Phase 8 — Screenshot Capture, VLM Analysis, and Human Disposition (AGSCN-FR-010..012)

• T037 [P] Write failing capture spec validation tests in backend/tests/services/dashboard_testing/scenario/test_capture.py
• T038 [P] Create backend/src/services/dashboard_testing/scenario/capture_profile.py — load and validate CaptureProfile from contracts/capture-profile.schema.json
• T039 [P] Implement ScenarioGraph.Capture.Dispatch: accept CaptureSpec from step, call AgentRuns.Evidence.Adapter, register screenshot artifacts, emit evidence_captured @POST: ScreenshotEvidence DraftArtifact registered; evidence_captured event emitted @TEST_EDGE: capture_timeout→step inconclusive; no artifact registered, masking_applied→original + masked artifacts
• T040 [P] Write failing VLM analysis tests in backend/tests/services/dashboard_testing/scenario/test_vlm.py for typed findings, provenance, stale prompt rejection @TEST_EDGE: stale_prompt→422 STALE_PROMPT, vlm_timeout→inconclusive, empty_response→empty findings + inconclusive
• T041 Implement backend/src/services/dashboard_testing/scenario/vlm.py: submit masked screenshot to VLM provider, parse typed VlmFinding[], validate model_provenance, persist raw response under redaction @POST: returns typed VlmFinding[] with model/prompt provenance; raw response stored under redaction @SIDE_EFFECT: external VLM API call; raw response persisted as separate DraftArtifact @INVARIANT: VLM findings advisory; never alter metric baseline truth; stale prompts block analysis @REJECTED: embedding VLM findings directly as assertion results (observations, not deterministic pass/fail)
• T042 Create registered VLM prompt template v1 under backend/src/services/dashboard_testing/scenario/prompt_templates/v1/ with versioned hash
• T043 [P] Write failing human disposition tests in backend/tests/services/dashboard_testing/scenario/test_disposition.py for confirm/dismiss/inconclusive, double-disposition rejection @TEST_EDGE: double_disposition→409, confirm_requires_comment→422
• T044 Implement backend/src/services/dashboard_testing/scenario/disposition.py: record immutable disposition per finding id, enforce confirm requires non-blank comment, emit audit event @POST: each finding disposition set exactly once; step transitions per policy @SIDE_EFFECT: audit record of disposition decision @INVARIANT: disposition never alters scenario graph structure or step ordering
• T045 Add VlmFinding and HumanDisposition DTOs to contracts/openapi.yaml response schemas (verify round-trip)
• T046 Wire capture/VLM/disposition into the scenario step execution loop: screenshot step → capture → analysis step → VLM call → human step → disposition
• T047 Audit: VLM findings are advisory, never alter metric baseline truth; disposition never changes graph structure; stale prompts block analysis

Checkpoint: Capture/VLM/disposition flow verified end-to-end; typed findings auditable.

Phase 9 — Polish & Cross-Cutting Verification

• T048 [P] Prototype validation: verify every @UX_STATE in contracts/ux/scenario-graph-ux.md reachable via specs/038-dashboard-scenario-model/prototype/index.html state switcher; responsive on mobile viewport
• T049 [P] OpenAPI drift check: verify operationId uniqueness (7), $ref resolution, example coverage, and RBAC scopes in contracts/openapi.yaml against implemented endpoints in backend/src/api/routes/dashboard_scenarios.py
• T050 [P] Belief runtime audit (C4/C5): axiom_audit({operation="audit_belief_runtime"}) + axiom_audit({operation="audit_belief_protocol"}) — confirm Compiler/Validator/Mapper/PackCompiler/Vlm/Capture/Resolver contracts have @RATIONALE/@REJECTED and REASON/REFLECT/EXPLORE markers
• T051 [P] Attention compliance audit: verify ATTN_1–4 per semantics-core §VIII across contracts/modules.md
• T052 [P] Semantic index rebuild: axiom_search({operation="rebuild", rebuild_mode="full"}) — 0 parse warnings required
• T053 [P] Orphan audit: axiom_search({operation="workspace_health"}) — confirm no new orphans from this feature
• T054 [P] Traceability coverage gate: verify traceability.md rows all map to real task IDs, contracts, and operationIds
• T055 [P] Run quickstart.md validation and make test-related F=backend/src/services/dashboard_testing/scenario/ for regression scope
• T056 Run /speckit.validate — confirm PASS before /speckit.implement

Dependencies

T001–T005 → US1 → US2; US3 follows compiler; US4 follows validator; draft pack follows validation/resolution; Phase 8 depends on 036 Phase 8 (screenshot evidence artifacts) and 037 Phase 7 (visual baseline infrastructure). 039 begins only after ScenarioResponse and DraftPack fixtures are stable. Frontend tasks N/A — 038 is DTO-only.

#endregion DashboardScenarioModel.Tasks