ss-tools/specs/044-dashboard-scenario-execution/traceability.md

Traceability: Scenario Execution Engine (044)

Story	Requirement	Model	API operationId	Contract	Task	Test	Actual status / gap
US1 Start	SCEX-FR-001/008	ScenarioRun	scenarioRun.start	Execution.Start, Execution.Runner.QueuedDispatch	T006-T008, T024	test_runner, test_scenario_queued_dispatch, test_scenario_scheduler_callbacks, test_scenario_runs_api, test_scenario_automation_api, test_live_execution_binding	[~] HTTP/automation start and replay persist queued/pending rows without request-time dispatch; only the scheduler composition's durable queued->running CAS walks its winner. Fixed scheduler callback registration, database-edge containment, and repeat-tick terminal side-effect idempotency are unit-proven. Automated human plans are rejected before they reach CAS/walker; a manual human graph reaches HumanCheckpoint only after that dispatcher claim. Approval-to-real live dispatch remains unproven.
US2 Dispatch	SCEX-FR-002/006/009	ScenarioStepRun, LiveExecutionBinding	scenarioRun.step	Execution.Dispatch, Execution.LiveCompositionRoot	T009-T011, T024	test_dispatch, test_scenario_executors, test_live_execution_binding	[~] Browser/Superset/Screenshot use fail-safe typed adapter boundaries: no explicit adapter success means no PASS; invalid evidence digest/ref remains inconclusive. Lifespan bootstraps settings.scenario_live_execution_bindings through the existing SupersetClient, exact model and durable storage, so configured Superset dispatch invokes 037 and stores the exact raw-byte digest/ref; mismatched/unavailable providers make no I/O call. Browser safe-checkpoint and Screenshot durable-evidence registration are supported but no provider is deployed, so enabled bindings return stable configured-unavailable codes.
BrowserProvider	SCEX-FR-016..023	BrowserProviderActionContract, BrowserOperationReceipt, BrowserEvidenceReceipt	scenarioRun.step	ScenarioExecution.BrowserProvider	T028-T034, T040-T042, T042b	test_provider_browser_*, Browser PREPROD canaries	[~] Contract completeness is 90/100: all actions, risk classes, limits, lifecycle, checkpoint/recovery, ownership, cancellation/reconciliation, readiness and canary gates are specified. Runtime provider, shared capacity integration, deployment registration and real canary evidence remain open.
US3 Human	SCEX-FR-004/010	ScenarioRun(waiting_human)	scenarioRun.humanDecision	Execution.SuspendForHuman, Execution.Resume	T012-T014, T025	test_human_resume, test_scenario_runs_api	[~] persisted HumanCheckpoint and infrastructure-resume continuations advance only the missing DAG frontier; completed steps are not re-run. Full live-composition closure remains pending.
Manual-only boundary	SCEX-FR-004a	ScenarioRun, HumanCheckpoint	—	Execution.Runner.Start, RunnerPlan.Derive	T027	test_scenario_manual_run_only, test_scenario_automation_api	[x] Trusted scheduled/deploy/release/ETL/API origins reject persisted human revisions before idempotency or any run/gate/notification/queue side effect. Manual origin remains eligible; HumanCheckpoint is not an approval gate.
Automated human prohibition	SCEX-FR-004a/013, SCAUTO-FR-001/007	RunnerPlan.manual_run_only	—	Execution.Runner.TriggerSource, Automation.Trigger	T027, 046 T016/T019	test_scenario_manual_run_only, test_scenario_automation_api, test_scenario_automation_trigger	[x] Every trusted non-manual origin, including scheduled, deploy, release, ETL, API and background recovery, is rejected before idempotency/run/gate/notification/queue/dispatch side effects. Only authenticated manual origin may create a human-containing run.
US4 Lifecycle	SCEX-FR-005/006	ScenarioRun(status, cancel deadline), ScenarioArtifact(active projection)	scenarioRun.cancel, scenarioRun.retry	Execution.Lifecycle.Cancel, Execution.Lifecycle.Timeout, Execution.Lifecycle.Retry, Execution.Runner.ContinueAfterRetry	T014g, T015-T016, T025	test_scenario_lifecycle, test_scenario_cancel_timeout, test_scenario_retry_closure	[~] Strict eligible retry invalidates the persisted target/descendant closure, archives old step attempts, and retains artifact rows only as inactive historical provenance. Cancellation pins a durable bounded drain deadline; deadline finalization expires leases and retires active projections without deleting audit evidence. A timeout during adapter I/O wins over a late PASS and materializes every otherwise-lazy pinned-plan descendant as blocked, so continuation cannot dispatch it; it emits one idempotent inconclusive signal. Full live-I/O composition remains fail-closed/unproven.
US5 Snapshot/API	SCEX-FR-003/007	ScenarioExecutionResult	scenarioRun.detail, scenarioRun.events	Execution.RunnerPlan, Execution.Runner.CrashRecovery	T017-T020, T025	test_result, test_api, test_scenario_runner_walker, test_scenario_crash_recovery	[~] API/projection and real-digest artifact integrity checks exist. Server-driven crash recovery uses only the persisted run/RunnerPlan and expired lease: completed work is not rerun, safe claims get a new attempt with active evidence retired to history, unsafe claims require reconciliation, and browser recovery without a pinned safe checkpoint is non-pass without adapter I/O. Rejected terminal contexts reuse their idempotent signal. Production live-executor provenance remains unproven.
Terminal signals	SCEX-FR-011	InvestigationQueueItem	—	Execution.Runner.TerminalSignal	T026	test_scenario_terminal_signals	[x] Failed/blocked/inconclusive terminal runs emit one idempotent immutable 047 queue input with run/artifact provenance; passed runs emit none. Producer-only ingestion starts no case, AgentRun, chat, remediation action, or recurrence classification.
Gate/RBAC	SCEX-FR-008	ScenarioRun, ActionApprovalGate	scenarioRun.start	Execution.EnvironmentPolicy, Execution.Runner.Start	T019	test_scenario_runner, test_scenario_runs_api, test_scenario_automation_api, test_scenario_automation_trigger	[~] Server ConfigManager classifies every target before persistence: client flags cannot select PROD, unknown targets create no run-side effect, and every trusted source enters the same durable pending_approval gate boundary. HTTP/trigger paths remain persistence-only and dispatcher excludes pending gates. A dedicated real APScheduler scheduled-PROD integration test remains coverage debt; this is not a dispatch bypass.
Provider protocol	SCEX-FR-017..023	ProviderExecutionContext, ProviderOperationReceipt, ProviderEvidenceReceipt	—	ScenarioExecution.ProviderProtocol, ProviderOperations, ProviderOperations.Observability	T028-T031, T040-T042	test_provider_contract, test_provider_health, deployment health checks	[ ] New production gate: common context/result, operation receipts, ownership proof, cancellation/reconciliation, health/readiness and startup registration are specified but not implemented.
Capacity	SCEX-FR-024	CapacityLease, ExecutionCapacityManager	—	ScenarioExecution.CapacityManager	T032, T041-T042	test_provider_capacity, scheduler/worker integration	[ ] New production gate: shared environment-scoped capacity admission is specified but current 046 checks do not close it.
Agent evaluation	SCEX-FR-025	AgentEvaluation, DecisionPolicy	—	ScenarioExecution.ProviderCatalog, data-model AgentEvaluation and DecisionPolicy	T039, T041-T042	test_provider_agent_evaluation	[ ] New production gate: bounded provider, immutable evaluation evidence and deterministic policy mapping are specified but absent.
Success criteria	SC-001	RunnerPlan, ScenarioStepRun	—	ScenarioExecution.RunnerPlan.Derive, ScenarioExecution.Dispatch	T004-T011, T041	canonical fixture order and duplicate-dispatch assertions	[~] Existing fixture/lifecycle coverage passes; exact 100% canonical dispatch evidence remains a release gate.
Success criteria	SC-002	HumanCheckpoint, ScenarioRun	scenarioRun.humanDecision	ScenarioExecution.HumanCheckpoint, ScenarioExecution.Resume	T012-T014, T025	human CAS/frontier tests	[x] Human checkpoint CAS and missing-frontier resume are verified; live provider closure remains separate.
Success criteria	SC-003	ScenarioRun.cancel_drain_deadline_at	scenarioRun.cancel	ScenarioExecution.Cancel	T015-T016, T025, T041	100 cancellation trials plus scheduler finalizer	[~] Bounded drain is unit-proven; required 100-trial production evidence is open.
Success criteria	SC-004	RunnerPlan, worker lease, operation receipt	scenarioRun.detail	ScenarioExecution.Runner.CrashRecovery, ScenarioExecution.ProviderOperations	T014c, T025, T030, T041	safe/unsafe recovery and reconciliation tests	[~] Safe/unsafe persisted recovery is verified; provider operation reconciliation is not implemented.
Success criteria	SC-005	immutable run provenance	scenarioRun.result	Execution.RunnerPlan, Execution.Result	T017, T020, T041	revision-edit immutability and evidence receipt tests	[~] Snapshot fields exist; complete receipt-level byte identity evidence is open.
Success criteria	SC-006	ActionApprovalGate, ExecutorRegistry	scenarioRun.start	Execution.ActionApprovalGate, ScenarioExecution.ExecutorRegistry	T014d, T019, T041	PROD no-call and human-not-executor tests	[x] Current no-call and registry rejection tests pass.
Success criteria	SC-007..010	Provider protocol/health/capacity	—	ScenarioExecution.ProviderProtocol, ProviderOperations.Observability, CapacityManager	T028-T042	common/provider-specific contract and deployment profiles	[ ] New production gate not implemented.
Success criteria	SC-011	release evidence record	—	ScenarioExecution.ProviderProtocol	T022, T041-T042	full profiles, PostgreSQL and semantic audit outputs	[ ] Release evidence package is incomplete.

N/A: Registry (042), Editor (043), Monitor UX (045), Automation (046), Analytics (047).

Cross-Spec Pipeline Traceability

Stage / requirement	044 authority	Handoff / ownership	Evidence
persisted revision is the only launch program input	ScenarioExecution.RunPreflight	042 ScenarioRevision -> preflight handle	test_scenario_runner, test_scenario_automation_api
server-owned preflight digest and failure boundary	ScenarioExecution.RunPreflight	blocks before plan, lease, dispatch or provider I/O	test_scenario_runner, policy/binding tests
deterministic plan and plan digest	ScenarioExecution.RunnerPlan.Derive	preflight -> pinned RunnerPlan -> run	test_scenario_runner_plan, test_scenario_dispatch
queued/pending approval continuation	ScenarioExecution.Start, ScenarioExecution.ActionApprovalGate	approved pending run -> queued; denial/expiry -> blocked	test_scenario_queued_dispatch, test_scenario_automation_api
ScenarioRun snapshot and idempotency	ScenarioExecution.Start / ScenarioRun model	exact revision/content/target/principal/binding snapshot	test_scenario_runs_api, test_scenario_scheduler_callbacks
evidence ownership and authoritative StepOutcome	ScenarioExecution.ProviderProtocol, ScenarioStepRun	provider receipt -> Artifact(owner_type=scenario_run) -> StepOutcome	test_provider_contract, test_scenario_terminal_signals
terminal signal to 047	ScenarioExecution.TerminalSignal	failed/blocked/inconclusive -> idempotent 047 queue input; passed -> none	test_scenario_terminal_signals
no reference artifact authority	ScenarioExecution.RunnerPlan.Derive	runner.plan.json is diagnostic/reference only	test_scenario_runner_plan, crash-recovery tests
authoring promotion admission	ScenarioExecution.AuthoringAdmission	042 promoted revision -> 044 preflight	raw-artifact rejection, candidate-without-promotion, no-run sandbox tests

All rows are required together with the 038/042/050 matrices. Provider, capacity, PostgreSQL, or live-composition gaps keep the coordinated release gate NO-GO, even when lifecycle unit tests pass.

Authoring E2E is additionally NO-GO until persistent workspace exploration, typed proposal conversion, user diff review, 042 handle-based save, and 044 promoted-revision-only admission are evidenced. Sandbox security and code-backed provider readiness are separate gates and remain unimplemented unless explicitly proven.

Provider production boundary: Existing typed unavailable outcomes and exact Superset binding tests prove only the fail-closed boundary. Production readiness additionally requires T028-T042 and real startup/dependency health checks for every enabled live provider.

BrowserProvider boundary: The BrowserProvider is contract-ready at 90/100 but implementation-ready only after T034, T040-T042 and T042b. A typed unavailable result remains the correct behavior until the PREPROD read-only/timeout/cleanup/reconstruction canaries produce retained operation and evidence receipts.

Cross-spec production boundary: 044 readiness depends on 036 authority/evidence, 037 query and baseline provenance, 038 executable graph identity, 041 lineage target state, 042 registry revisions, 046 automation dispatch and 047 terminal-signal/case ingestion. 039/043/045 are operator continuity dependencies: they do not authorize execution, but incomplete typed API/SSE/revision flows prevent a complete production workflow. Current dependency-weighted aggregate for 036-047 is approximately 67/100 and remains NO-GO.

Full production tool boundary: There is no reduced preview target. Browser/Screenshot, controlled non-PROD mutation, AgentEvaluation/DecisionPolicy, automated schedules/triggers, case investigation, analytics and remediation are mandatory capability gates. Missing or unproven capability blocks GO; only human-containing automated revisions remain prohibited and PROD mutation remains policy-forbidden.

Verification boundary (2026-08-24): the available local profile is 246 backend tests, 59 provider/ lifecycle edge tests, scoped Ruff/compile and prototype validation. Real PostgreSQL migration checks, provider contract T028-T042, live Browser/Screenshot composition, scheduler deployment and Axiom index rebuild remain open.

Production acceptance traceability — 2026-09-08

Historical rows above identify prior tests/code only; removed agent UI paths are retired. The following audited gates are implemented=false / OPEN, independent of local suite totals.

Requirement	Domain contract / DTO	Task	Falsifiable acceptance	State
SCEX-FR-028	Production baseline-backed evaluation; data model	T043	Baseline set/version/catalog/release/commit/IDs/digests affect idempotency; moving catalog after admission cannot change plan/result; legacy unpinned result is ineligible.	OPEN
SCEX-FR-028	Production baseline-backed evaluation; data model	T044	GET/HEAD prove same ACL/status/headers; MIME/digest/length checked before bytes, cross-owner hidden, expired410, corrupt409, traversal/range/oversize rejected.	OPEN
SCEX-FR-028	Production baseline-backed evaluation; data model	T045	Startup/readiness/start-loop and shutdown/drain/cancel/reconcile survive fault injection; unknown effect quarantines capacity; late response cannot win.	OPEN
SCEX-FR-028	Production baseline-backed evaluation; data model	T046	End-to-end real browser→capture→durable artifact→deterministic comparison→optional immutable evaluation→policy→result; all required evidence present before PASS.	OPEN
SCEX-FR-028; external-MCP-only UI	manual editor/review; read-only evidence	production tasks	No frontend agent prompt/chat/assistant editing/proposal generation/workspace/start/handoff routes or requests; human approval remains usable.	OPEN

Sources: production gap, coverage gap, baseline gap. Spec schema/static checks prove contract structure only; live canary/runtime closure and optional approved performance baseline are not claimed.

Live-binding admin surface + catalog publisher — evidence 2026-09-11

Two deployment/operator seams added by commits 792bb125 + d04927bd; both are server-owned surfaces outside the 050 MCP catalog (no spec row existed before this note):

Seam	Contract	Surface	Evidence
Api.ScenarioLiveBindings	ScenarioExecution.LiveBinding + Core.ConfigManager	admin REST GET/PUT/PATCH /api/scenario-live-bindings (admin:settings); validates LiveExecutionBinding.from_snapshot + DashboardQueryModel (env/dashboard/fingerprint) and writes settings.scenario_live_execution_bindings atomically	backend/tests/api/test_scenario_live_bindings_api.py (7); first live exercise: re-registered ss-prod-d11-live-001 with the correct principal, then the T029m run adopted it server-side
Tooling.PublishCatalog	T045/D7 published-catalog source (PUBLISHED_CATALOG_*)	CLI backend/src/scripts/publish_catalog.py — pre-validates catalog bytes, then Gitea contents PUT (typed auth/conflict/unavailable)	backend/tests/scripts/test_publish_catalog.py (7, offline); publication itself blocked on a valid Gitea PAT

Cross-reference: the live external MCP replay that exercised the binding path end-to-end (and exposed the identity-less-compiled-step defect) is docs/2026-09-11-sales-prod-mcp-replay.md (050 T029m / E2E-EXT-002, CLOSED).