busya
d874a4dca6
Add prototype/openapi/validate/resume commands, wire edge-failure matrix into UX, enforce traceability + validation gates, mandate C4/C5 belief-runtime verification. Rework 038-dashboard-scenario-model artifacts: applicability, structured edge cases, 24-class UX state matrix, interactive HTML prototype, standardized OpenAPI 3.1 (7 ops), full RTM with coverage gate, 56-task backlog, and PASS validation report.
181 lines
13 KiB
Markdown
181 lines
13 KiB
Markdown
#region DashboardScenarioModel.Modules [C:5] [TYPE ADR] [SEMANTICS contracts,scenario,graph,validator,compiler]
|
||
@BRIEF C3+ contracts for checklist mapping, deterministic graph compilation, validation, resolution, serialization, and safe draft packs.
|
||
@RATIONALE The scenario graph is the reviewable intermediate boundary between agent intent and generated artifacts; it centralizes safety, determinism, and capability coverage.
|
||
@REJECTED Direct agent-to-script generation — rejected because missing refs, unsafe actions, and baseline truth would be discovered only after artifact generation or runtime.
|
||
@RELATION DEPENDS_ON -> [DashboardScenarioModel.DataModel]
|
||
@RELATION DEPENDS_ON -> [DashboardScenarioModel.ChecklistCatalog]
|
||
@RELATION DEPENDS_ON -> [SupersetBaselineEngine.Modules]
|
||
@RELATION DEPENDS_ON -> [AgentRuns.Artifacts.Register]
|
||
|
||
# #region ScenarioGraph.Api [C:4] [TYPE Module] [SEMANTICS scenario,api,compile,validate]
|
||
# @defgroup ScenarioGraph REST surface for compile, validate, resolve, and draft-pack operations.
|
||
# @LAYER API
|
||
# @RELATION DEPENDS_ON -> [ScenarioGraph.Compiler]
|
||
# @RELATION DEPENDS_ON -> [ScenarioGraph.Validator]
|
||
# @RELATION DEPENDS_ON -> [ScenarioGraph.Resolver]
|
||
# @RELATION DEPENDS_ON -> [ScenarioGraph.PackCompiler]
|
||
# @RATIONALE A thin REST surface exposes the deterministic scenario operations to 039 and agent tools without leaking compiler internals.
|
||
# @REJECTED Exposing raw Pydantic models over the API — request schemas must forbid code/SQL/paths to keep the boundary safe.
|
||
# @INVARIANT Request schemas forbid executable code, SQL, raw baseline values, and local paths.
|
||
# #endregion ScenarioGraph.Api
|
||
|
||
# #region ScenarioGraph.Catalog.Load [C:4] [TYPE Function] [SEMANTICS scenario,checklist,catalog,version]
|
||
# @ingroup ScenarioGraph
|
||
# @BRIEF Load the versioned 19-case declarative catalog and validate ids/capability/template references.
|
||
# @PRE Bundled catalog version is supported.
|
||
# @POST Returns exactly B01–B09, C01–C07, T01–T03 in stable order.
|
||
# @SIDE_EFFECT Bounded package-resource read.
|
||
# @DATA_CONTRACT CatalogResource -> ChecklistCase[19]
|
||
# @INVARIANT Historic PDF outcomes are source notes, not expected values.
|
||
# @RATIONALE Versioned declarative catalog keeps the 19 PDF cases reusable and auditable across dashboards.
|
||
# @REJECTED Embedding the checklist as Python conditionals — mixed intent/data makes coverage unverifiable.
|
||
# @TEST_EDGE missing_case -> startup/catalog validation failure.
|
||
# @TEST_EDGE sql_template_for_technical_case -> rejected.
|
||
# #endregion ScenarioGraph.Catalog.Load
|
||
|
||
# #region ScenarioGraph.CapabilityMapper.Map [C:5] [TYPE Function] [SEMANTICS scenario,capability,mapping,coverage]
|
||
# @ingroup ScenarioGraph
|
||
# @BRIEF Classify every checklist case for a dashboard and select one allowed step template.
|
||
# @PRE Query model, baseline summary, and capability registry validate.
|
||
# @POST Every catalog case is automated, human_checkpoint, unsupported, or needs_context with rationale.
|
||
# @SIDE_EFFECT None.
|
||
# @DATA_CONTRACT ChecklistCase[] + DashboardCapabilities -> CapabilityMapping[]
|
||
# @RATIONALE Capability mapping keeps checklist intent reusable while allowing each dashboard to receive only safe, applicable step templates.
|
||
# @REJECTED One-size-fits-all scripts and user-facing low-level tool selection — rejected because capabilities, safety, and available evidence vary per dashboard.
|
||
# @INVARIANT No case is dropped and no tool is selected outside its registered capabilities.
|
||
# @TEST_EDGE xlsx_unavailable -> C04–C06 manual/unsupported with rationale.
|
||
# @TEST_EDGE technical_without_dataset_fields -> human checkpoint, no SQL.
|
||
# #endregion ScenarioGraph.CapabilityMapper.Map
|
||
|
||
# #region ScenarioGraph.Compiler.Compile [C:5] [TYPE Function] [SEMANTICS scenario,compiler,deterministic,dag]
|
||
# @ingroup ScenarioGraph
|
||
# @BRIEF Compile canonical inputs and mappings into a stable dashboard-specific DAG.
|
||
# @PRE Intent, query model, catalog, baseline summary, and parameters have valid fingerprints.
|
||
# @POST Same canonical inputs/compiler version yield byte-identical graph and stable ids/order.
|
||
# @SIDE_EFFECT None.
|
||
# @DATA_CONTRACT CompileScenarioRequest -> DashboardTestScenario
|
||
# @INVARIANT Steps consume only context/parameter/baseline/earlier-step refs.
|
||
# @TEST_INVARIANT Deterministic_Graph -> VERIFIED_BY: repeated_compile, shuffled_input_order.
|
||
# @TEST_EDGE missing_selector -> NEEDS_SELECTOR step and save blocker.
|
||
# @TEST_EDGE missing_baseline -> NEEDS_BASELINE; no embedded numeric truth.
|
||
# @RATIONALE Rule/template compilation makes the agent a planner/explainer, not an executable-code generator.
|
||
# @REJECTED LLM-generated ids/dependencies/code — non-deterministic and unsafe.
|
||
# #endregion ScenarioGraph.Compiler.Compile
|
||
|
||
# #region ScenarioGraph.Validator.Validate [C:5] [TYPE Function] [SEMANTICS scenario,validator,graph,safety]
|
||
# @ingroup ScenarioGraph
|
||
# @BRIEF Return complete deterministic findings for schema, DAG, refs, parameters, baselines, tools, safety, and coverage.
|
||
# @PRE Candidate graph parses against supported schema version.
|
||
# @POST Valid is true only with zero errors/blockers; findings are stably ordered and actionable.
|
||
# @SIDE_EFFECT None.
|
||
# @DATA_CONTRACT DashboardTestScenario -> ScenarioValidationResult
|
||
# @RATIONALE Validation is a hard safety boundary between agent-produced intent and artifact generation; deterministic findings give the user a recoverable explanation instead of a runtime surprise.
|
||
# @REJECTED Silent graph repair or best-effort artifact generation — rejected because auto-fixing refs, cycles, or unsafe actions can change business intent without review.
|
||
# @INVARIANT Cycles, missing/duplicate refs, unregistered tools, SQL, raw metric truth, and path traversal block compilation.
|
||
# @TEST_EDGE cycle -> error contains cycle path.
|
||
# @TEST_EDGE duplicate_output -> both producer ids reported.
|
||
# @TEST_EDGE raw_metric_expected -> forbidden baseline literal error.
|
||
# @TEST_EDGE unreachable_step -> warning/error according to required coverage.
|
||
# #endregion ScenarioGraph.Validator.Validate
|
||
|
||
# #region ScenarioGraph.Resolver.Resolve [C:4] [TYPE Function] [SEMANTICS scenario,resolve,parameter,revision]
|
||
# @ingroup ScenarioGraph
|
||
# @BRIEF Apply typed parameter/selector/manual resolutions and emit an immutable scenario revision.
|
||
# @PRE Base revision hash matches; changes target declared unresolved items.
|
||
# @POST Unrelated step ids/order remain unchanged; new parent/revision hashes link revisions.
|
||
# @SIDE_EFFECT None.
|
||
# @SIDE_EFFECT Logging (REASON/REFLECT markers required around revision emission).
|
||
# @RATIONALE Immutable revisions preserve auditability and byte-stable determinism for downstream pack compilation.
|
||
# @REJECTED In-place graph mutation — destroys revision history and breaks parent_revision_hash linkage.
|
||
# @DATA_CONTRACT ResolveScenarioRequest + BaseScenario -> DashboardTestScenario
|
||
# @TEST_EDGE stale_base_revision -> 409.
|
||
# @TEST_EDGE invalid_parameter_type -> 422.
|
||
# @TEST_EDGE unrelated_graph_change -> invariant failure.
|
||
# #endregion ScenarioGraph.Resolver.Resolve
|
||
|
||
# #region ScenarioGraph.Serializer.Canonical [C:4] [TYPE Function] [SEMANTICS scenario,serialize,json,yaml]
|
||
# @ingroup ScenarioGraph
|
||
# @BRIEF Serialize graph to canonical JSON/YAML and compute revision hash.
|
||
# @POST Key/order/decimal/date/newline rules are stable across runs; JSON and YAML represent equal domain data.
|
||
# @SIDE_EFFECT None.
|
||
# @SIDE_EFFECT Logging (REASON marker before canonicalization; REFLECT with hash after).
|
||
# @RATIONALE Canonical serialization is the revision-identity boundary; volatile display fields must never enter the hash.
|
||
# @REJECTED Pretty-printed human-first serialization — non-deterministic key ordering breaks byte-stable snapshots.
|
||
# @DATA_CONTRACT DashboardTestScenario -> CanonicalBytes + SHA256
|
||
# @TEST_EDGE shuffled_dicts -> identical bytes.
|
||
# @TEST_EDGE timestamp_display_field -> excluded from revision identity.
|
||
# #endregion ScenarioGraph.Serializer.Canonical
|
||
|
||
# #region ScenarioGraph.PackCompiler.Generate [C:5] [TYPE Function] [SEMANTICS scenario,artifact,template,draft]
|
||
# @ingroup ScenarioGraph
|
||
# @BRIEF Generate a preview-only or save-eligible draft pack using registered versioned templates.
|
||
# @PRE Scenario validation result available; template ids registered; target paths safe.
|
||
# @POST Outputs match ArtifactPlan, contain no LLM executable bodies, and are registered as 036 drafts.
|
||
# @SIDE_EFFECT Renders bounded templates and calls AgentRuns.Artifacts.Register.
|
||
# @DATA_CONTRACT DashboardTestScenario + ValidationResult -> DraftPack
|
||
# @INVARIANT Errors/unresolved required inputs make pack preview_only; direct code/path input is impossible.
|
||
# @TEST_INVARIANT No_LLM_To_Code -> VERIFIED_BY: injected_code_field, template_registry_only.
|
||
# @TEST_EDGE unknown_template -> blocked.
|
||
# @TEST_EDGE path_traversal -> blocked before artifact registration.
|
||
# @RATIONALE Versioned templates make generated behavior reviewable and reproducible.
|
||
# @REJECTED Generate arbitrary Playwright/Python code then scan it — scanners cannot prove semantic safety.
|
||
# #endregion ScenarioGraph.PackCompiler.Generate
|
||
|
||
# #region ScenarioGraph.Capture.Dispatch [C:4] [TYPE Function] [SEMANTICS scenario,capture,screenshot,browser]
|
||
# @ingroup ScenarioGraph
|
||
# @BRIEF Execute screenshot capture steps through the 036 Evidence.Adapter bridge.
|
||
# @PRE CaptureSpec is valid; dashboard and environment are accessible.
|
||
# @POST ScreenshotEvidence DraftArtifact registered; evidence_captured event emitted.
|
||
# @SIDE_EFFECT Browser capture via existing ScreenshotService; draft storage via AgentRuns.Artifacts.Register.
|
||
# @SIDE_EFFECT Logging (REASON before capture; REFLECT with artifact ids after).
|
||
# @RATIONALE Reusing the 036 Evidence bridge keeps capture consistent with the rest of agent evidence lifecycle.
|
||
# @REJECTED Direct Playwright screenshot calls inside the compiler — couples compile to browser availability and bypasses artifact registry.
|
||
# @DATA_CONTRACT ScreenshotCaptureSpec + AgentRun -> DraftArtifactRef[]
|
||
# @RELATION DEPENDS_ON -> [AgentRuns.Evidence.Adapter]
|
||
# @INVARIANT Step output refs link to artifact ids, not filesystem paths.
|
||
# @TEST_EDGE capture_timeout -> step marked inconclusive; no artifact registered.
|
||
# @TEST_EDGE masking_applied -> two artifacts: original + masked; step output refs masked.
|
||
# #endregion ScenarioGraph.Capture.Dispatch
|
||
|
||
# #region ScenarioGraph.Vlm.Analyze [C:4] [TYPE Function] [SEMANTICS scenario,vlm,analysis,finding]
|
||
# @ingroup ScenarioGraph
|
||
# @BRIEF Submit a masked screenshot to a configured VLM provider and return typed findings.
|
||
# @PRE Screenshot artifact is valid (not invalid); VlmAnalysis profile is registered and current.
|
||
# @POST Returns typed VlmFinding[] with model/prompt provenance; raw response stored under redaction.
|
||
# @SIDE_EFFECT External VLM API call; raw response persisted as separate DraftArtifact.
|
||
# @SIDE_EFFECT Logging (REASON before submission; REFLECT with finding count after).
|
||
# @DATA_CONTRACT DraftArtifactRef + VlmAnalysis -> VlmFinding[]
|
||
# @INVARIANT VLM findings are advisory observations, not deterministic assertions; they must not alter metric baseline truth.
|
||
# @INVARIANT Prompt template version and hash are recorded per analysis; stale prompts block analysis.
|
||
# @TEST_EDGE stale_prompt -> 422 with STALE_PROMPT code.
|
||
# @TEST_EDGE vlm_timeout -> step inconclusive; existing findings retained.
|
||
# @TEST_EDGE empty_response -> findings array empty; step status inconclusive with reason.
|
||
# @RATIONALE Typed findings make VLM output auditable and reviewable; raw text would require parsing and is not reproducible.
|
||
# @REJECTED Embedding VLM findings directly as assertion results — they are observations for human review, not deterministic pass/fail.
|
||
# #endregion ScenarioGraph.Vlm.Analyze
|
||
|
||
# #region ScenarioGraph.Human.Disposition [C:3] [TYPE Function] [SEMANTICS scenario,human,checkpoint,disposition]
|
||
# @ingroup ScenarioGraph
|
||
# @BRIEF Record human disposition on one or more VLM findings.
|
||
# @PRE Human checkpoint step is active; finding ids are unresolved.
|
||
# @POST Each finding disposition is set exactly once; step transitions according to policy.
|
||
# @SIDE_EFFECT Audit record of disposition decision.
|
||
# @SIDE_EFFECT Logging (REASON before disposition; REFLECT with outcome after).
|
||
# @RATIONALE Typed dispositions keep VLM review auditable while preserving graph immutability.
|
||
# @REJECTED Free-text disposition with no audit record — unreviewable and non-reproducible.
|
||
# @DATA_CONTRACT HumanDispositionRequest -> ScenarioStep (updated)
|
||
# @TEST_EDGE double_disposition -> 409.
|
||
# @TEST_EDGE disposition_blank -> accepted for dismiss/inconclusive; confirm requires non-blank comment.
|
||
# @INVARIANT Disposition never alters the scenario graph structure or step ordering.
|
||
# #endregion ScenarioGraph.Human.Disposition
|
||
|
||
# #region AgentChat.Tools.ScenarioGraph [C:4] [TYPE Module] [SEMANTICS scenario,agent,tools,compiler]
|
||
# @defgroup ScenarioGraph Thin agent tools that submit bounded intent and display compiler/validator results.
|
||
# @RELATION DEPENDS_ON -> [ScenarioGraph.Api]
|
||
# @RATIONALE Agent tools stay thin: the agent explains intent, the deterministic compiler owns the graph.
|
||
# @REJECTED Agent-side graph construction with free-form tool selection — bypasses validation and determinism.
|
||
# @INVARIANT Agent cannot submit executable code, custom tool categories, raw expected metrics, or artifact paths.
|
||
# #endregion AgentChat.Tools.ScenarioGraph
|
||
|
||
#endregion DashboardScenarioModel.Modules
|