busya/ss-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
31b9a19a0c3f8b2931bbbb625703473d056ff8ff
ss-tools/specs/038-dashboard-scenario-model/tasks.md
busya 31b9a19a0c chore: commit remaining workspace updates 
Agent:
- lifecycle: run tracking, middleware hardening, langgraph setup
- tests: agent lifecycle + langgraph setup coverage

Backend:
- async_job_runner: resilience hardening, tests
- agent_conversations: run lifecycle integration
- translate: scheduler + orchestrator SQL adjustments
- schemas/services: agent_lifecycle model extensions

Frontend:
- TaskDrawer: UX improvements
- TaskLogPanel/Viewer: safety hardening, i18n (en/ru)
- FilterBar: report filters contract + tests
- Reports page: layout adjustments

Specs:
- 036-agent-test-stabilization: runs contract, modules, events
- 037-superset-baseline-engine: catalog schema, testing API, modules
- 038-dashboard-scenario-model: scenario schema, capture profile, modules
- 039-dashboard-scenario-ui: screen models, release verification UX, modules
- dashboard-verification-usecases: new cross-cutting spec
2026-07-17 19:11:09 +03:00

7.3 KiB
Raw Blame History

#region DashboardScenarioModel.Tasks [C:3] [TYPE ADR] [SEMANTICS tasks,scenario,implementation] @BRIEF Ordered TDD backlog for deterministic scenario graph and safe draft-pack compilation.

Phase 1 — Catalog, Schemas, Fixtures

  • T001 Transcribe checklist-catalog.md into a versioned declarative resource in backend/src/services/dashboard_testing/scenario/catalog_v1.yaml.
  • T002 Write catalog completeness tests for B01–B09, C01–C07, T01–T03 in backend/tests/services/dashboard_testing/scenario/test_catalog.py.
  • T003 Create valid 18-step scenario plus invalid cycle/missing-ref/duplicate-output/raw-baseline/SQL fixtures under specs/038-dashboard-scenario-model/fixtures/.
  • T004 Materialize fixtures to backend/tests/fixtures/dashboard_scenarios/.
  • T005 Implement Pydantic models in backend/src/services/dashboard_testing/scenario/models.py matching dashboard-test-scenario.schema.json.

Phase 2 — US1 Compile Scenario Graph

  • T006 [US1] Write failing capability mapping tests in backend/tests/services/dashboard_testing/scenario/test_capability_mapper.py.
  • T007 [US1] Implement backend/src/services/dashboard_testing/scenario/checklist_catalog.py validation and backend/src/services/dashboard_testing/scenario/capability_mapper.py with complete classification.
  • T008 [US1] Cover unavailable XLSX, missing selector/test data, unsafe mutation context, cross-dashboard absence, and T01–T03 no-SQL fallbacks.
  • T009 [US1] Write failing deterministic compiler tests in backend/tests/services/dashboard_testing/scenario/test_compiler.py.
  • T010 [US1] Implement registered tool/action and step-template catalogs under backend/src/services/dashboard_testing/scenario/templates/.
  • T011 [US1] Implement backend/src/services/dashboard_testing/scenario/compiler.py with stable ids, phase order, refs, coverage, and fingerprints.
  • T012 [US1] Prove repeated compile and shuffled input order produce identical graph bytes.

Checkpoint: Valid fixture compiles to stable graph and classifies all 19 cases.

Phase 3 — US2 Validate Safety and Completeness

  • T013 [US2] Write failing full invalid-fixture matrix in backend/tests/services/dashboard_testing/scenario/test_validator.py.
  • T014 [US2] Implement schema, ref producer/consumer, duplicate, dependency, and cycle checks in backend/src/services/dashboard_testing/scenario/validator.py.
  • T015 [US2] Implement parameter, selector, baseline, tool/action, path, SQL/code, raw-expected, and coverage checks.
  • T016 [US2] Return deterministic all-findings output with JSON pointers and recovery options.
  • T017 [US2] Add property tests generating small DAG/cycle/ref variations without mirroring validator logic in backend/tests/services/dashboard_testing/scenario/test_validator_properties.py.

Phase 4 — US3 Checklist Coverage and Serialization

  • T018 [US3] Write JSON/YAML golden tests in backend/tests/services/dashboard_testing/scenario/test_serializer.py.
  • T019 [US3] Implement backend/src/services/dashboard_testing/scenario/serializer.py and revision hash exclusions.
  • T020 [US3] Validate JSON Schema and Pydantic round-trip for all golden fixtures.
  • T021 [US3] Add catalog-version and compiler-version fingerprints to scenario inputs.

Phase 5 — US4 Parameters and Human Checkpoints

  • T022 [US4] Write failing typed resolution/stale revision tests in backend/tests/services/dashboard_testing/scenario/test_resolver.py.
  • T023 [US4] Implement backend/src/services/dashboard_testing/scenario/resolver.py for parameter, selector, manual conversion, and remove-step operations.
  • T024 [US4] Enforce immutable revisions and unchanged unrelated step ids/order.
  • T025 [US4] Cover safe-environment/test-data requirements for mutating PDF cases.

Phase 6 — Safe Draft Pack

  • T026 Write failing template registry, preview-only, path, SQL, shell, and injected-code tests in backend/tests/services/dashboard_testing/scenario/test_pack_compiler.py.
  • T027 Create versioned templates for scenario.yaml, runner.plan.json, report_template.md, evidence_manifest.json, and bounded browser/XLSX modules under backend/src/services/dashboard_testing/scenario/pack_templates/v1/.
  • T028 Implement backend/src/services/dashboard_testing/scenario/pack_compiler.py; accept only registered template ids and structured inputs.
  • T029 Register outputs through 036 AgentRuns.Artifacts.Register and emit generate/validate progress.
  • T030 Ensure invalid/unresolved graphs produce preview_only with repeated blockers.

Phase 7 — API, Agent, Quality

  • T031 Add backend/src/api/routes/dashboard_scenarios.py matching scenario-api.openapi.yaml and register router.
  • T032 Write RBAC/contract/revision tests in backend/tests/api/test_dashboard_scenarios.py.
  • T033 Add thin compile/validate/resolve/generate tools in agent/src/ss_tools/agent/tools.py.
  • T034 Verify agent schemas cannot carry code, SQL, raw expected metrics, custom tools, or artifact paths.
  • T035 Run quickstart, JSON/OpenAPI schema validation, scoped/full backend tests, and ruff.
  • T036 Audit all 19 cases, direct SQL/code bans, contract anchors, ATTN_1–4, and unresolved relations.

Phase 8 — Screenshot Capture, VLM Analysis, and Human Disposition (AGSCN-FR-010..012)

  • T037 [P] Write failing capture spec validation tests in backend/tests/services/dashboard_testing/scenario/test_capture.py.
  • T038 [P] Create backend/src/services/dashboard_testing/scenario/capture_profile.py — load and validate CaptureProfile from capture-profile.schema.json.
  • T039 [P] Implement ScenarioGraph.Capture.Dispatch: accept CaptureSpec from step, call AgentRuns.Evidence.Adapter, register screenshot artifacts, emit evidence_captured.
  • T040 [P] Write failing VLM analysis tests in backend/tests/services/dashboard_testing/scenario/test_vlm.py for typed findings, provenance, stale prompt rejection.
  • T041 Implement ScenarioGraph.Vlm.Analyze: submit masked screenshot to VLM provider, parse typed VlmFinding[], validate model_provenance, persist raw response under redaction.
  • T042 Create registered VLM prompt template v1 under backend/src/services/dashboard_testing/scenario/prompt_templates/v1/ with versioned hash.
  • T043 [P] Write failing human disposition tests in backend/tests/services/dashboard_testing/scenario/test_disposition.py for confirm/dismiss/inconclusive, double-disposition rejection.
  • T044 Implement ScenarioGraph.Human.Disposition: record immutable disposition per finding id, enforce confirm requires non-blank comment, emit audit event.
  • T045 Add VlmFinding and HumanDisposition DTOs to scenario-api.openapi.yaml response schemas (already defined; verify round-trip).
  • T046 Wire capture/VLM/disposition into the scenario step execution loop: screenshot step → capture → analysis step → VLM call → human step → disposition.
  • T047 Audit: VLM findings are advisory, never alter metric baseline truth; disposition never changes graph structure; stale prompts block analysis.

Dependencies

T001–T005 → US1 → US2; US3 can follow compiler, US4 follows validator, draft pack follows validation/resolution. 039 begins only after ScenarioResponse and DraftPack fixtures are stable. Phase 8 depends on 036 Phase 8 (screenshot evidence artifacts) and 037 Phase 7 (visual baseline infrastructure).

#endregion DashboardScenarioModel.Tasks