busya/ss-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

feat(speckit): workflow architecture upgrades + rework 038 spec per new flow

Browse Source 
Add prototype/openapi/validate/resume commands, wire edge-failure matrix into UX,
enforce traceability + validation gates, mandate C4/C5 belief-runtime verification.
Rework 038-dashboard-scenario-model artifacts: applicability, structured edge cases,
24-class UX state matrix, interactive HTML prototype, standardized OpenAPI 3.1 (7 ops),
full RTM with coverage gate, 56-task backlog, and PASS validation report.
This commit is contained in:
busya
2026-07-31 11:25:43 +03:00
parent 2136082d6d
commit d874a4dca6
30 changed files with 3327 additions and 355 deletions
Download Patch File Download Diff File Expand all files Collapse all files

64
specs/038-dashboard-scenario-model/spec.md
View File

@@ -11,10 +11,18 @@
## Navigation (DSA Indexer keywords)
@SEMANTICS: spec, requirements, feature, scenario, graph, dashboard-testing, checklist, validation, artifacts
**Feature Branch**: `038-dashboard-scenario-model`
**Created**: 2026-07-07 | **Status**: Ready for Implementation
**Feature Branch**: `038-dashboard-scenario-model`
**Created**: 2026-07-07 | **Reworked**: 2026-07-31 | **Status**: Reworked per new speckit flow
**Input**: "Define the dashboard test scenario model used by agents to represent unique dashboard test flows as a validated ScenarioGraph. The model must express ordered and dependent steps across browser automation, Superset query execution, XLSX parsing, assertions, screenshots, reports, human checkpoints, baseline references, parameters, warnings, and missing context markers without exposing users to low-level tool selection."
## Applicability
- **Feature type**: Fullstack (backend compiler/validator/resolver core + thin UI preview surface consumed by 039 + agent tools).
- **UI surface**: Yes — scenario graph preview, coverage, resolution, and pack states (rendered by 039; 038 supplies DTO contracts).
- **API surface**: Yes — compile, validate, resolve, draft-pack, capture, VLM, disposition endpoints.
- **Prototype**: Applicable — scenario preview is a real UI surface (see `prototype/index.html`).
- **OpenAPI**: Applicable — REST surface is a first-class deliverable (see `contracts/openapi.yaml`).
## User Scenarios
### Story 1 — Build Scenario Graph From Dashboard Goal (P1)
@@ -69,12 +77,35 @@
---
### Edge Cases
- Scenario has a cycle in dependencies → validator rejects with cycle path.
- Two steps produce the same output ref → validator rejects ambiguous ref.
- Baseline is stale → assertion step remains present but blocked or warning-gated.
- XLSX export is unavailable → XLSX-dependent checklist cases become unsupported or manual checkpoints.
- UI selector is unknown → browser step uses `NEEDS_SELECTOR` and blocks executable generation for that step.
### Story 5 — Capture, VLM Analysis, and Human Disposition (P2)
**Why P2**: Screenshot evidence and visual verification require typed, auditable capture/VLM/disposition semantics (AGSCN-FR-010..012).
**Independent Test**: Generate a scenario with screenshot capture, VLM analysis, and human disposition; verify typed findings and auditable dispositions.
**Acceptance**:
1. **Given** a screenshot step **When** capture executes **Then** a reproducible `ScreenshotCaptureSpec` (target, viewport, readiness, masking, max wait) is honored and artifacts are registered.
2. **Given** a masked screenshot **When** VLM analysis runs **Then** typed `VlmFinding[]` (severity, region, confidence, model/prompt provenance) are returned; raw prose is never treated as step state.
3. **Given** a human checkpoint references VLM finding ids **When** the user disposes **Then** confirm/dismiss/inconclusive is typed and auditable, and disposition never mutates the graph structure.
## Edge & Failure Cases
| # | Scenario | Category | Expected Behavior | Recovery / Test Ownership |
|---|----------|----------|-------------------|---------------------------|
| E1 | Scenario has a cycle in dependencies | data-integrity | Validator rejects with cycle path | User fixes graph; L1 validator test |
| E2 | Two steps produce the same output ref | data-integrity | Validator rejects ambiguous ref | User fixes ref; L1 test |
| E3 | Baseline is stale | data-quality | Assertion stays present but blocked/warning-gated | 037 baseline discovery or mark pending; L1 test |
| E4 | XLSX export unavailable | integration | XLSX-dependent checklist cases become unsupported or manual checkpoints | Rationale shown in coverage; L1 mapping test |
| E5 | UI selector unknown | integration | Browser step uses `NEEDS_SELECTOR` and blocks executable generation | User provides selector hint / converts to checkpoint; L1 test |
| E6 | 409 stale base revision on resolve | concurrency | New revision rejected with 409; snapshot/recompile guidance; never silent merge | User recompiles; L1 API test |
| E7 | 422 invalid resolution/parameter type | validation | Field/step-mapped validation error | User corrects input; L1 API test |
| E8 | 403 forbidden role on scenario operations | auth | Permission denial rendered without approval gate | User contacts admin / RBAC test |
| E9 | 429 rate limit on compile/validate | throttling | Retry-After honored; UI countdown | User waits; L2 UX test |
| E10 | 5xx backend failure on compile | server-error | Error section + retry; partial graph not persisted | User retries; L2 UX test |
| E11 | Malformed VLM response / empty findings | integration | Findings array empty; step inconclusive with reason; stale prompt blocked (422 STALE_PROMPT) | Re-run analysis; L1 VLM test |
| E12 | Missing parameter value on pack compile | data-quality | Pack becomes `preview_only` with all save blockers listed | User resolves parameters; L1 pack test |
| E13 | Unsafe path / executable code / SQL injection into pack | security | Template/path/code validation blocks before draft registration | L1 security test; injected-code fixture |
| E14 | Duplicate submit of draft-pack | idempotency | Idempotency key / revision hash prevents double registration | L1 API test; 409 on changed revision |
## Requirements
@@ -89,9 +120,9 @@
- **AGSCN-FR-007**: The model MUST allow manual/human checkpoint steps where automation is unsafe, unavailable, or underspecified.
- **AGSCN-FR-008**: Scenario output MUST be deterministic for the same dashboard query model, checklist template, baseline catalog, and user parameters.
- **AGSCN-FR-009**: The scenario model MUST remain implementation-neutral and must not require the user to choose low-level artifacts such as Playwright, XLSX, or API output upfront.
- **AGSCN-FR-010**: Screenshot steps MUST carry a capture specification: target (tab/viewport), viewport dimensions, readiness strategy, masking selectors, and max wait. The specification ensures reproducible captures across runs.
- **AGSCN-FR-011**: VLM-based visual analysis steps MUST produce typed findings referencing the source screenshot artifact, including severity, region, confidence, model/prompt provenance, and a bounded textual description. Raw VLM prose MUST NOT be interpreted as step state.
- **AGSCN-FR-012**: Human checkpoint steps MAY reference specific VLM finding IDs. The resolution options (confirm, dismiss, inconclusive) MUST be typed and auditable. Disposition changes the finding status, not the underlying scenario graph structure.
- **AGSCN-FR-010**: Screenshot steps MUST carry a capture specification: target (tab/viewport), viewport dimensions, readiness strategy, masking selectors, and max wait.
- **AGSCN-FR-011**: VLM-based visual analysis steps MUST produce typed findings referencing the source screenshot artifact, including severity, region, confidence, model/prompt provenance, and a bounded textual description.
- **AGSCN-FR-012**: Human checkpoint steps MAY reference specific VLM finding IDs. Resolution options (confirm, dismiss, inconclusive) MUST be typed and auditable. Disposition changes finding status, not graph structure.
### Key Entities
@@ -113,5 +144,16 @@
- **SC-003**: Same inputs produce byte-stable scenario JSON/YAML in deterministic snapshot tests.
- **SC-004**: No generated scenario fixture embeds raw baseline numbers directly in executable steps.
- **SC-005**: Scenario graph preview can display phase order, tools per step, parameters, warnings, and blockers without reading generated code.
- **SC-006**: VLM findings are advisory only; disposition is auditable; no finding alters metric baseline truth.
## Clarifications
### Session 2026-07-31
- Q: Is the scenario model a backend-only library or does it expose a UI/API surface? → A: Fullstack — backend compiler/validator core plus thin UI preview (rendered by 039) plus REST/agent-tool API surface.
- Q: How are missing selectors/context handled? → A: Represented structurally as `NEEDS_SELECTOR`/`NEEDS_CONTEXT`, never invented; blocks executable generation for that step.
- Q: What is the determinism contract? → A: Byte-stable output for identical canonical inputs + compiler/template versions; stable derived ids, no random UUIDs in canonical graph; temperature=0 alone is rejected as a determinism mechanism.
- Q: What is the VLM safety boundary? → A: VLM output is typed, advisory findings for human review; raw prose is never step state and findings never alter baseline truth; stale prompts block analysis.
- Q: Is the draft-pack compiled through templates or direct code generation? → A: Versioned repository-owned templates only; LLM text may populate bounded descriptions but never executable code bodies, paths, imports, or shell commands.
#endregion DashboardScenarioModel.Spec