docs(scenarios): define production provider contracts
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# 044 Scenario Execution — Session State
|
||||
|
||||
> **Updated**: 2026-08-21
|
||||
> **Updated**: 2026-08-21 17:12 +03:00
|
||||
> **Purpose**: Durable handoff for the current implementation/review session. This is a
|
||||
> decision and verification ledger; `tasks.md` and `traceability.md` remain the canonical
|
||||
> feature backlog and requirement matrix.
|
||||
@@ -11,6 +11,27 @@ Bring the 044 Scenario Execution Engine into material conformance with its contr
|
||||
preserving fail-closed live I/O and GRACE-Poly invariants. Every code change is followed by an
|
||||
independent verifier pass and semantic curation.
|
||||
|
||||
## Readiness Assessment
|
||||
|
||||
Production readiness is weighted toward real external execution because local fail-closed tests
|
||||
cannot prove that a configured Browser/Superset/Screenshot/DB path performs authorized I/O and
|
||||
produces durable evidence.
|
||||
|
||||
| Category | Weight | Score | Weighted contribution | Evidence / limiting factor |
|
||||
|---|---:|---:|---:|---|
|
||||
| Real live composition and external evidence | 50% | 25/100 | 12.5 | Exact injected 037 Superset binding is proven; Browser/Screenshot providers remain unavailable; no real deployment run |
|
||||
| DAG, RunnerPlan and executor behavior | 15% | 87/100 | 13.05 | Pinned descriptors, DAG closure, typed executors, read-only `sql_evidence` and bounded transform pass; required browser/screenshot integrations remain partial |
|
||||
| Security and fail-closed policy | 15% | 86/100 | 12.9 | Server-owned PROD gate, CAS decisions, descriptor allowlist and no-I/O rejection paths are covered |
|
||||
| Retry, cancel, timeout and crash recovery | 10% | 84/100 | 8.4 | Persisted lifecycle and terminal cleanup profiles pass |
|
||||
| Deployment, PostgreSQL migrations and operations | 5% | 62/100 | 3.1 | One Alembic head; real PostgreSQL upgrade/check not run because environment has placeholder DATABASE_URL |
|
||||
| Tests and independent verification | 5% | 82/100 | 4.1 | 242 backend 044 tests, scoped Ruff/compile pass, prototype validation passes; semantic rebuild unavailable |
|
||||
| **Weighted total** | **100%** | | **54.05/100** | **NO-GO** |
|
||||
|
||||
**Hard production gate:** live composition must score at least 70/100. The current live score is
|
||||
25/100, so the feature remains **NO-GO for full production**, regardless of the 53/100 weighted
|
||||
total. It is suitable only for internal fail-safe preview/shadow mode with unavailable providers
|
||||
explicitly surfaced as non-pass.
|
||||
|
||||
## Architecture Decisions Already Implemented
|
||||
|
||||
- HTTP and 046 start paths are persistence-only. The scheduler-owned queued dispatcher is the
|
||||
@@ -58,8 +79,13 @@ The independent review found these material gaps. The first is fixed; the second
|
||||
4. **P1 — fixed (2026-08-21)**: Queued-dispatch exceptions now terminalize through a shared cleanup
|
||||
path that retires current evidence projections, marks active steps inconclusive, expires leases,
|
||||
and emits the existing idempotent terminal side effects.
|
||||
5. **P1 — pending**: `sql_evidence`, bounded transform, AgentEvaluation/DecisionPolicy and global
|
||||
capacity management remain absent despite the complete 044 target contract.
|
||||
5. **P1 — partial**: `sql_evidence` now has a version-pinned read-only action descriptor and
|
||||
exact 037 binding adapter with durable raw-response evidence. Bounded transform,
|
||||
AgentEvaluation/DecisionPolicy and global capacity management remain absent despite the
|
||||
complete 044 target contract. Bounded transform now has a version-pinned `transform_result`
|
||||
action with only `select_fields`, `sort_rows`, and `limit_rows`, hard row/operation limits and
|
||||
canonical normalization; arbitrary expressions/code remain rejected. Existing 046 candidate
|
||||
capacity/dedup checks do not constitute the 044 dispatcher capacity boundary.
|
||||
6. **P2 — pending**: Quickstart paths/counts and a stale automation migration-head test must be
|
||||
aligned with the current linear Alembic chain.
|
||||
|
||||
@@ -115,3 +141,310 @@ linear Alembic head, lint/compile/diff checks, independent verifier result, then
|
||||
- Screenshot evidence validation now accepts one or more provider-issued refs only when every ref
|
||||
has a valid non-zero SHA-256 and the first ref agrees with the declared `sha256`; focused live
|
||||
executor/composition profile: 36 passed.
|
||||
- Revalidated after commit `ffa4d6a8`: full available 044 backend profile is **242 passed**;
|
||||
scoped Ruff and targeted compile pass; `validate_static.py` passes; `alembic heads` reports the
|
||||
single head `e3f4a5b6c7d8`.
|
||||
- The subsequent auth/UI commit `1a5c1473` and current unrelated frontend working-tree changes
|
||||
are excluded from this 044 readiness score.
|
||||
- Added `sql_evidence` as a registered `capture_sql_evidence` action. It reuses only the exact
|
||||
server-owned 037 binding and persists raw response evidence; caller SQL, static result metadata,
|
||||
missing digest, or missing durable ref cannot pass. Full available 044 profile is now
|
||||
**244 passed** after updating the registry fingerprint fixture.
|
||||
- Added bounded `transform_result` action and executor. It applies only declarative field selection,
|
||||
deterministic string sorting and row limiting to completed table results, then canonicalizes via
|
||||
the existing 037 normalizer. No `eval`, `exec`, arbitrary imports, SQL or unbounded output path
|
||||
exists. Full available 044 profile is now **246 passed**; scoped Ruff and compile pass.
|
||||
|
||||
## Contract Hardening Update (2026-08-21)
|
||||
|
||||
The specification was expanded from a fail-closed adapter boundary into a production provider contract.
|
||||
The new normative requirements are SCEX-FR-017..025 and tasks T028..T042.
|
||||
|
||||
- Every provider now has a common execution context/result, immutable operation receipt, effect state,
|
||||
ownership/evidence receipt, cancellation and reconciliation protocol.
|
||||
- Provider-specific obligations are explicit for Browser, Superset API, SQL evidence, XLSX, Assertion,
|
||||
Transform, Screenshot, Report, Artifact and AgentEvaluation.
|
||||
- Provider lifecycle now includes resource ownership, server-owned limits, capacity leases, cleanup,
|
||||
liveness/readiness/dependency health, startup registration and redacted observability.
|
||||
- AgentEvaluation is bounded by immutable 038 spec and deterministic DecisionPolicy; it cannot directly
|
||||
set StepOutcome, mutate the graph, consume HumanCheckpoint or schedule work.
|
||||
- ExecutionCapacityManager is a shared environment-scoped admission boundary; no provider I/O is lawful
|
||||
without an atomic lease.
|
||||
|
||||
This hardening is specification-only. T028-T042 remain unimplemented, so production status remains
|
||||
NO-GO. Existing typed-unavailable and exact Superset tests prove only the fail-closed boundary, not the
|
||||
new provider production gate. The Axiom semantic index was not rebuilt in this session.
|
||||
|
||||
## Orthogonal Edge-Case Verification (2026-08-21 18:13 +03:00)
|
||||
|
||||
- Full available 044 registry/API profile: **246 passed**.
|
||||
- Provider and lifecycle edge profile (executors, live binding, cancel/timeout, crash recovery, worker,
|
||||
queued dispatch): **59 passed**.
|
||||
- Full registry scenario edge profile: **186 passed**.
|
||||
- Scoped Ruff and compile checks for execution/API modules: passed.
|
||||
- Prototype state validation: passed.
|
||||
- The first edge command referenced a nonexistent `test_scenario_live_execution_binding.py`; it was
|
||||
corrected to the existing `test_live_execution_binding.py` before the successful 59-test run.
|
||||
- No new provider protocol, capacity, AgentEvaluation, operation receipt, health or deployment tests
|
||||
exist yet; therefore these results do not close T028-T042.
|
||||
|
||||
## Cross-Spec Production Readiness: 036-047 (2026-08-21)
|
||||
|
||||
044 production readiness is not a single-feature property. The execution chain is:
|
||||
|
||||
`036 authority/evidence/gates -> 037 query/baseline -> 038 executable scenario model -> 042 registry ->
|
||||
043 editor -> 044 runner/providers -> 046 automation -> 047 triage/analytics -> 045 monitor`, with
|
||||
`039 scenario authoring UI` consuming 036/037/038 and feeding 042/043.
|
||||
|
||||
The following scores are categorical implementation-readiness estimates, not claims of completed
|
||||
verification. A score reflects contract completeness, runtime closure, external integration evidence and
|
||||
operator/deployment readiness. A neighboring spec may score well in isolation and still block 044 when
|
||||
its missing boundary is on the execution critical path.
|
||||
|
||||
| Spec | Domain | Contract | Runtime | External/deploy | Operational | Readiness | Effect on 044 |
|
||||
|---|---|---:|---:|---:|---:|---:|---|
|
||||
| 036 | Agent runs, evidence, gates | 90 | 82 | 70 | 80 | **81/100** | Strong substrate; 044 still needs provider-specific ownership receipts and live deployment proof |
|
||||
| 037 | Superset baseline/query engine | 88 | 82 | 68 | 78 | **79/100** | Exact query envelope is reusable; migration/API and real PostgreSQL/live checks remain limiting |
|
||||
| 038 | Scenario graph/compiler/model | 92 | 88 | 75 | 84 | **85/100** | Executable graph and pinned specs are a strong source of truth; AgentEvaluation runtime remains open in 044 |
|
||||
| 039 | Scenario authoring UI | 78 | 65 | 45 | 58 | **62/100** | Non-blocking for backend execution, but incomplete API binding weakens authoring-to-run continuity |
|
||||
| 040 | Load testing/capacity patterns | 88 | 78 | 65 | 76 | **77/100** | Useful capacity precedent, but not a substitute for shared 044 CapacityManager |
|
||||
| 041 | Dataset lineage/blast radius | 88 | 76 | 60 | 72 | **74/100** | Required for target/staleness/blast-radius provenance; fan-out and production sync evidence remain material |
|
||||
| 042 | Scenario registry/revisions | 86 | 72 | 55 | 68 | **70/100** | Direct blocker: runtime 037/041 staleness and 036 signal integration remain open |
|
||||
| 043 | Scenario editor | 82 | 68 | 50 | 62 | **66/100** | Not required for dispatcher execution, but immutable revision production flow depends on 042 and open E2E/policy checks |
|
||||
| 044 | Scenario execution/providers | 86 | 58 | 25 | 62 | **54/100** | Current hard NO-GO; provider contract T028-T042 and live composition are open |
|
||||
| 045 | Run monitor/results UX | 78 | 65 | 40 | 58 | **61/100** | Operator-facing production blocker: typed launch alignment and 047 handoff are incomplete |
|
||||
| 046 | Scenario automation/operations | 82 | 58 | 40 | 55 | **59/100** | Direct production blocker for scheduled/triggered runs; startup reload, event dispatch and persisted scheduler semantics are open |
|
||||
| 047 | Investigation queue/analytics | 82 | 60 | 35 | 58 | **59/100** | Direct terminal-flow blocker: canonical signal ingestion, immutable case evidence and conformant closure are open |
|
||||
|
||||
### Dependency-weighted aggregate
|
||||
|
||||
For a full 036-047 production surface, the practical aggregate is approximately **67/100**. The
|
||||
execution-critical subset `{036,037,038,041,042,044,046,047}` is approximately **68/100**. Both remain
|
||||
below a production GO because 044 live composition, 046 automation, and 047 terminal ingestion are hard
|
||||
gates rather than optional UX debt.
|
||||
|
||||
### Category impact on the 044 production score
|
||||
|
||||
| Cross-spec category | Current contribution to 044 | Required closure | Expected 044 effect |
|
||||
|---|---:|---|---:|
|
||||
| Authority, RBAC, approval and evidence substrate (036) | 80/100 | Preserve 036 gate/CAS semantics and bind provider receipts to 036 evidence | +2 to +4 weighted |
|
||||
| Superset query, normalization and baseline provenance (037) | 72/100 | Real PostgreSQL/live query verification; exact binding remains mandatory | +3 to +5 |
|
||||
| Executable graph and immutable program identity (038) | 82/100 | Close AgentEvaluationSpec/DecisionPolicy runtime boundary | +2 to +4 |
|
||||
| Target lineage and staleness (041) | 55/100 | Production sync/fan-out and pinned blast-radius snapshot | +3 to +6 |
|
||||
| Registry/revision source of truth (042) | 58/100 | Staleness subscription, health signal and final independent verification | +4 to +7 |
|
||||
| Operator authoring/editor continuity (039/043) | 45/100 | API binding, policy/E2E and revision save-to-run proof | +1 to +3; not a live-I/O gate |
|
||||
| Monitor and human operations (045) | 48/100 | Typed 044 launch/SSE contract and real evidence/047 handoff | +2 to +4 |
|
||||
| Automation and scheduling (046) | 42/100 | Event dispatch, startup schedule reload, persisted APScheduler behavior and shared capacity | +5 to +9 |
|
||||
| Triage, cases and analytics (047) | 35/100 | Canonical signal ingestion, immutable evidence, case CAS and closure | +3 to +6 |
|
||||
| Provider runtime contracts (new T028-T042) | 25/100 | Receipts, ownership, cancellation/reconciliation, health, deployment, capacity | +15 to +25 |
|
||||
|
||||
### Hard gates for full production GO
|
||||
|
||||
The following are conjunctive, not averaged away:
|
||||
|
||||
1. 044 Browser and Screenshot providers perform authorized live I/O and produce durable owned evidence.
|
||||
2. 044 shared CapacityManager admits every provider operation before I/O and survives retry/crash paths.
|
||||
3. 046 persisted schedules/triggers execute through 044 queued CAS with startup reload and real scheduler verification.
|
||||
4. 047 ingests one immutable terminal signal, preserves run/evidence provenance, and opens/closes cases through CAS without changing run truth.
|
||||
5. 042 registry revision/staleness state is production-connected to 037/041 and emits canonical 036 signals.
|
||||
6. 036/037/041 PostgreSQL migration and deployment checks pass against a real PostgreSQL environment.
|
||||
7. 045/039/043 operator paths use typed contracts end-to-end; no agent prose or compatibility payload substitutes for API truth.
|
||||
|
||||
### Interpretation
|
||||
|
||||
Implementing only the new 044 provider contracts would likely raise 044 from **54/100 to 80-88/100**,
|
||||
but the complete 036-047 product would remain around **67/100** until 042, 046 and 047 close their
|
||||
cross-spec runtime boundaries. Conversely, completing 045/039 UI polish without 046/047 and live 044
|
||||
composition would not materially change the production decision.
|
||||
|
||||
## Requirements Clarification Review (2026-08-24)
|
||||
|
||||
The documentation package was reviewed for ambiguous, non-measurable and cross-document requirements.
|
||||
The following consistency changes were applied:
|
||||
|
||||
- SC-001..011 now define exact thresholds for dispatch order, checkpoint uniqueness, 100 cancellation
|
||||
trials, recovery identity, provenance immutability, provider readiness and release evidence.
|
||||
- SCEX-FR-017..025 now have explicit MUST semantics, named evidence expectations and measurable ownership,
|
||||
capacity, cancellation, health and deployment rules.
|
||||
- OpenAPI now requires core run/step fields, strict step status enum, typed execution provenance and
|
||||
evidence receipts; cancel responses distinguish `202` request, `200` already-finalized and `409` invalid state.
|
||||
- Quickstart commands now match repository paths and current verified counts: 246 full 044 tests and
|
||||
59 provider/lifecycle edge tests. PostgreSQL and provider contract profiles are explicitly separate
|
||||
release gates.
|
||||
- Checklist expanded from FR-001..010 to FR-001..025 and SC-001..011, including per-provider and cross-
|
||||
spec production gates.
|
||||
- UX reference defines observable deadlines, event replay, capacity-unavailable, reconciliation-required,
|
||||
ownership failure and redaction behavior.
|
||||
- Plan, data model, tasks and traceability now use the same 5-second scheduler tolerance, 64-hex digest,
|
||||
one-receipt-per-operation and zero-open-P0/P1 release rules.
|
||||
|
||||
Verification after documentation update: OpenAPI YAML parsed, GRACE anchors balanced, `git diff --check`
|
||||
passed, and the available 044 regression profile passed **246 tests**. This remains documentation
|
||||
clarification, not implementation of T028-T042 or a production GO.
|
||||
|
||||
## Automated Human Checkpoint Clarification (2026-08-24)
|
||||
|
||||
Confirmed invariant: a HumanCheckpoint MUST NOT exist in an automated scheduled or triggered run.
|
||||
`manual_run_only=true` is rejected before idempotency lookup and before creation of any ScenarioRun,
|
||||
ActionApprovalGate, notification, queue item or dispatcher claim for scheduled, deploy, release, ETL, API
|
||||
and background-recovery origins. Only the authenticated manual route may create a human-containing run and
|
||||
later reach `waiting_human`.
|
||||
|
||||
This is covered by the existing manual-only tests and is now stated identically in 044 `spec.md`,
|
||||
`data-model.md`, 046 `spec.md`, 046 quickstart/checklist/traceability and the 044 cross-spec traceability.
|
||||
PROD ActionApprovalGate remains valid only for eligible automated revisions without human checkpoints; it
|
||||
never substitutes for or bypasses HumanCheckpoint.
|
||||
|
||||
## Production Scope Assessment (2026-08-24)
|
||||
|
||||
This assessment distinguishes the full feature roadmap from a production-usable scope. The full 036-047
|
||||
surface remains approximately **67/100** and NO-GO. A narrower production rollout is possible only as an
|
||||
explicitly constrained release profile; it must not be presented as the complete automated scenario product.
|
||||
|
||||
### Orthogonal scope categories
|
||||
|
||||
| Category | Included capability | Current readiness | Production scope decision |
|
||||
|---|---|---:|---|
|
||||
| Scenario authoring | Agent creates typed 038 graph/draft; validator rejects unsafe graph; analyst reviews | 78/100 | Include after revision-save E2E; agent cannot activate or execute directly |
|
||||
| Registry and revision identity | 042 immutable revision/current activation and run pinning | 70/100 | Include only with stale-state policy and activation CAS; no git-file-only truth |
|
||||
| Read-only manual execution | Manual analyst start, exact RunnerPlan, assertion/transform/XLSX/Superset read paths | 72/100 | Candidate for controlled production/shadow rollout after real PostgreSQL and Superset deployment proof |
|
||||
| Browser execution | Browser navigation/read/inspection with safe checkpoint and owned evidence | 90/100 contract, 28/100 runtime | Contract now includes per-action schemas/risk, limits, lifecycle, receipts, recovery and canaries; exclude until implementation and PREPROD proof |
|
||||
| Screenshot evidence | Durable principal/RLS-bound capture and receipt | 20/100 | Exclude; legacy ScreenshotService paths are not evidence |
|
||||
| Mutation execution | Browser/test-data mutation and cleanup/reconciliation | 15/100 | Exclude entirely from initial production scope; PROD mutation remains prohibited |
|
||||
| AgentEvaluation | Bounded model call plus immutable evaluation and DecisionPolicy | 15/100 | Exclude; no model-backed execution decisions in initial rollout |
|
||||
| Human checkpoints | Manual-only analyst disposition | 85/100 | Include only for manual runs; prohibited in scheduled/triggered/background runs |
|
||||
| Retry/recovery | Persisted lifecycle, safe retry, unsafe reconciliation block, timeout closure | 78/100 | Include for pure/read-only and explicitly retry-safe actions; unsafe external effects remain blocked |
|
||||
| Scheduled automation | 046 cron/deploy/release/ETL/API triggers | 42/100 | Exclude from initial production; 046 scheduler/subscriber lifecycle is not closed |
|
||||
| Notifications | Completion/failure/stale/triage domain events | 35/100 | Exclude as a production promise until lifecycle wiring is verified |
|
||||
| Run monitor | 045 typed SSE, provenance, human action and recovery UI | 58/100 | Include as operator surface only after typed 044 API alignment; not evidence of backend readiness |
|
||||
| Investigation queue | 047 queue ingestion and explicit analyst case opening | 45/100 | Include only read-only queue projection if signal ingestion is proven; agent case closure remains excluded |
|
||||
| Analytics/health | Flakiness, health, trends and recurring failures | 40/100 | Exclude from release acceptance; derived analytics cannot be operational truth yet |
|
||||
| Capacity/admission | Shared 044 CapacityManager across workloads | 25/100 | Hard blocker; no production provider I/O without this boundary |
|
||||
| Security/authority | Server-owned environment class, PROD gate, RBAC, exact bindings | 86/100 | Include; mandatory invariant for every rollout profile |
|
||||
| Deployment/operations | PostgreSQL migrations, readiness, health, observability, rollback | 45/100 | Hard blocker until real deployment evidence exists |
|
||||
|
||||
### Recommended production profile: `manual-readonly-preview`
|
||||
|
||||
The only defensible near-term production scope is:
|
||||
|
||||
1. Agent-authored, validator-approved, immutable scenario revisions.
|
||||
2. Authenticated manual starts only; no schedules, triggers or background execution.
|
||||
3. PREPROD or explicitly non-production environments only until real PROD evidence is complete.
|
||||
4. Read-only Superset/SQL evidence through the exact 037 binding, bounded transform, assertion and
|
||||
server-owned XLSX artifact intake.
|
||||
5. HumanCheckpoint allowed only for manual runs.
|
||||
6. Retry limited to pure/idempotent read-only actions; unknown external effects become non-pass and are
|
||||
never replayed automatically.
|
||||
7. 045 monitor for typed status/provenance and 047 queue projection only where the corresponding APIs
|
||||
are verified.
|
||||
8. Explicitly surface unavailable Browser/Screenshot/AgentEvaluation/automation capabilities as disabled
|
||||
or non-pass, never as successful execution.
|
||||
|
||||
Estimated readiness of this constrained profile: **72-78/100**, conditional on closing PostgreSQL,
|
||||
CapacityManager and deployment-readiness gates. It is a preview/shadow production profile, not full
|
||||
production automation.
|
||||
|
||||
### Explicitly excluded from production scope
|
||||
|
||||
- unattended scheduled, deploy, release, ETL or API-triggered execution;
|
||||
- any human-containing revision in an automated origin;
|
||||
- BrowserProvider and ScreenshotProvider until real owned evidence is proven;
|
||||
- browser or data mutation, including fixture mutation and cleanup side effects;
|
||||
- AgentEvaluation as a runtime decision source;
|
||||
- automatic retry after unknown external effects;
|
||||
- 047 agent-led remediation and unresolved case closure;
|
||||
- analytics-derived health as an authoritative deployment or release decision.
|
||||
|
||||
### Full production scope gate
|
||||
|
||||
The complete feature may be called production-ready only when the constrained profile additionally has:
|
||||
real Browser/Screenshot providers, common provider receipts, shared CapacityManager, AgentEvaluation and
|
||||
DecisionPolicy, closed 046 scheduler/event lifecycle, canonical 047 signal/case closure, 042 staleness
|
||||
integration, real PostgreSQL migration checks and end-to-end agent-authoring-to-execution-to-investigation
|
||||
evidence. Until then, the correct release label is `manual-readonly-preview`, not `production-ready`.
|
||||
|
||||
### BrowserProvider contract reassessment
|
||||
|
||||
BrowserProvider is now **90/100 contract-complete** and **28/100 runtime-complete**. The contract closes
|
||||
the prior ambiguity around every action, read-only versus mutation risk, limits, isolated context lifecycle,
|
||||
safe checkpoint/reconstruction, evidence ownership, cancellation, reconciliation, health and deployment.
|
||||
The score must not be interpreted as production enablement: T034, T040-T042, T042b and a real PREPROD
|
||||
canary remain release blockers. Until those pass, BrowserProvider stays typed unavailable and remains outside
|
||||
`manual-readonly-preview`.
|
||||
|
||||
## Contract Package Reassessment: 042-047 (2026-08-24)
|
||||
|
||||
This reassessment separates two questions:
|
||||
|
||||
- **Contract/system description score**: how completely the documents define a production system,
|
||||
including inputs, outputs, invariants, failure behavior, recovery, security, operations and measurable
|
||||
release evidence.
|
||||
- **Production proof score**: how much of that described system is implemented and externally verified.
|
||||
|
||||
| Spec | Contract/system description | Production proof | Key contract gap | Production blocker |
|
||||
|---|---:|---:|---|---|
|
||||
| 042 Registry | **88/100** | **70/100** | Staleness/health signal timing and cross-spec subscription semantics need stricter operational evidence | 037/041 runtime staleness integration and final independent verification |
|
||||
| 043 Editor | **84/100** | **66/100** | Save/activate/revision API and policy are clear, but end-to-end conflict and accessibility evidence are incomplete | Depends on 042 activation/staleness and 044 typed launch contract |
|
||||
| 044 Execution | **91/100** | **54/100** | Provider contract is now detailed; common runtime protocol/capacity/live providers remain unimplemented | Browser/Screenshot runtime, receipts, CapacityManager, PostgreSQL, deployment canaries |
|
||||
| 045 Monitor | **83/100** | **61/100** | Typed event/provenance UX is strong, but API schema alignment and event replay evidence remain incomplete | Depends on 044 runtime truth and 047 investigation ingestion |
|
||||
| 046 Automation | **84/100** | **59/100** | Trigger, schedule, dedup, retention and manual-human prohibition are clear; scheduler operational semantics need live proof | Persisted scheduler reload, event subscribers, notifications, capacity and real cron/trigger run |
|
||||
| 047 Analytics | **86/100** | **59/100** | Queue/case/health/flakiness contracts are clear; evidence snapshot and closure invariants are not runtime-closed | Canonical signal ingestion, immutable case evidence, verification/reconciliation closure |
|
||||
|
||||
### Functional contract categories
|
||||
|
||||
| Category | Contract completeness | Production proof | Assessment |
|
||||
|---|---:|---:|---|
|
||||
| Scenario identity, revision pinning and stale policy | 88/100 | 69/100 | 042/043 define the model; runtime staleness propagation remains incomplete |
|
||||
| Agent-authored graph/editor workflow | 84/100 | 65/100 | Draft/validation/save/activate are described; full agent-to-activation E2E is not proven |
|
||||
| Deterministic execution and provider boundary | 93/100 | 54/100 | 044 is contractually strongest; live provider protocol is the largest implementation gap |
|
||||
| Browser/evidence ownership | 90/100 | 28/100 | BrowserProvider contract is detailed, but no deployed provider/canary receipt exists |
|
||||
| Human checkpoint and manual-only policy | 94/100 | 85/100 | Strongly specified and tested; automated human revisions are rejected pre-create |
|
||||
| Retry, timeout, cancel and crash recovery | 88/100 | 78/100 | Runner lifecycle is strong; provider-level reconciliation is absent |
|
||||
| Scheduled/event automation | 84/100 | 42/100 | 046 requirements are clear; operational scheduler/subscriber lifecycle is not demonstrated |
|
||||
| Monitoring, SSE and operator recovery | 83/100 | 58/100 | 045 renders typed state, but depends on incomplete 044/047 boundaries |
|
||||
| Investigation/case/triage workflow | 86/100 | 45/100 | 047 producer signal exists; durable case evidence and closure are missing |
|
||||
| Analytics, health and recurrence | 82/100 | 40/100 | Deterministic formulas are described; end-to-end historical aggregation and 042 feedback are open |
|
||||
| Security, authority and RBAC | 89/100 | 80/100 | Server-owned gates and object-level access are well covered; deployment audit remains required |
|
||||
| Deployment, health, observability and rollback | 76/100 | 38/100 | Requirements exist but real PostgreSQL, provider readiness, scheduler and rollback evidence are open |
|
||||
|
||||
### Recalculated package scores
|
||||
|
||||
Using contract completeness as a descriptive index, the 042-047 package scores **86/100**. This means
|
||||
the documents describe most of the intended production system with clear boundaries and measurable
|
||||
requirements.
|
||||
|
||||
Using production proof and hard-gate weighting, the package scores **58/100**. The score is lower than the
|
||||
previous 67/100 estimate for 036-047 because this narrower 042-047 package gives greater weight to the
|
||||
unclosed execution, automation and case-closure boundaries rather than to the mature 036/037 substrate.
|
||||
|
||||
The complete package remains **NO-GO**. Hard gates are conjunctive:
|
||||
|
||||
1. 042 revision/staleness state is connected to 037/041 and emits canonical signal evidence.
|
||||
2. 043 save/activate flow is verified end to end against 042 and 044.
|
||||
3. 044 provider runtime passes common receipts, capacity, cancellation/reconciliation and BrowserProvider
|
||||
PREPROD canaries; BrowserProvider contract completeness is 90/100 but runtime remains 28/100.
|
||||
4. 045 consumes typed 044 events with gap-free replay and renders evidence/provenance without compatibility
|
||||
payloads.
|
||||
5. 046 persisted scheduler reload, event subscribers, dedup, notifications and trigger-to-044 execution
|
||||
are verified in a real deployment; human-containing revisions remain pre-create rejected for automation.
|
||||
6. 047 ingests canonical signals, persists immutable evidence snapshots, enforces verification/reconciliation
|
||||
before resolution and preserves immutable run truth.
|
||||
7. Real PostgreSQL migration/deployment checks, readiness, observability and rollback evidence pass.
|
||||
|
||||
### Production interpretation
|
||||
|
||||
The package defines a complete production tool, not a reduced profile. BrowserProvider, Screenshot,
|
||||
mutation, AgentEvaluation, unattended automation and agent-led remediation are mandatory capability gates;
|
||||
missing proof keeps the system NO-GO rather than creating a preview release.
|
||||
|
||||
## Superseding Product Scope Decision (2026-08-24)
|
||||
|
||||
The earlier `manual-readonly-preview` recommendation is superseded and must not be used as a release target.
|
||||
The product target is the complete production tool: agent authoring and activation, all trusted manual and
|
||||
automated trigger origins, Browser/Screenshot providers, controlled non-PROD fixture mutation,
|
||||
AgentEvaluation/DecisionPolicy, monitoring, automation, investigation cases, analytics and remediation.
|
||||
|
||||
These are mandatory capability gates, not permanent exclusions. A missing or unproven capability blocks
|
||||
production GO. The only policy exclusions are: HumanCheckpoint-containing revisions cannot be automated and
|
||||
PROD mutation is prohibited. Unknown external effects cannot retry before reconciliation. This decision
|
||||
overrides earlier preview/shadow wording in this historical session ledger.
|
||||
|
||||
@@ -1,39 +1,78 @@
|
||||
# Requirements Checklist: Scenario Execution Engine (044)
|
||||
|
||||
**Purpose**: Verify SCEX-FR-001..010 completeness. | **Created**: 2026-08-07
|
||||
**Purpose**: Verify SCEX-FR-001..025 and SC-001..011 with reproducible evidence.
|
||||
**Updated**: 2026-08-24
|
||||
|
||||
> **Factual audit 2026-08-20:** `[x]` requires current production evidence, `[~]` means partial
|
||||
> code exists, `[ ]` means missing integration or proof. Synthetic PASS executors are not evidence.
|
||||
> `[x]` means implementation plus named verification evidence is complete. `[~]` means code exists but
|
||||
> at least one required integration, deployment or production proof is missing. `[ ]` means absent.
|
||||
> A synthetic adapter result, caller-provided digest, path or fixture-only PASS is not production evidence.
|
||||
|
||||
## Run Model (FR-001/003/007)
|
||||
## Run Model and Determinism
|
||||
|
||||
- [~] CHK001 ScenarioRun first-class, pinned to scenario_id (UUID) + revision_id (UUID) + content_hash + env + param snapshot
|
||||
- [ ] CHK002 Deterministic; no LLM per step; agent not in hot path
|
||||
- [ ] CHK003 Recoverable by scenario_run_id; provenance for reproducibility
|
||||
- [~] CHK001 `ScenarioRun` stores UUID scenario/revision, 64-hex content/program hashes, environment,
|
||||
immutable parameter bindings, target snapshot and principal fingerprint.
|
||||
- [x] CHK002 RunnerPlan derives exact `{tool, action}` descriptors from the pinned 038 registry and
|
||||
rejects missing, altered, unknown or hash-mismatched descriptors before lease/provider I/O.
|
||||
- [~] CHK003 Runner walks the canonical graph in exact topological order and dispatches each completed
|
||||
logical step at most once per attempt.
|
||||
- [~] CHK004 Recovery by `scenario_run_id` preserves RunnerPlan hash and completed-step set; unknown
|
||||
effects never retry without reconciliation.
|
||||
|
||||
## Execution (FR-002/006/009)
|
||||
## Provider Contract
|
||||
|
||||
- [~] CHK004 Dispatch by step.tool to typed executor
|
||||
- [ ] CHK005 Executors reuse 037/038/036; no second Playwright/LLM/SQL stack
|
||||
- [ ] CHK006 Step attempt/retry/timeout/ref binding/error_code/artifact_refs
|
||||
- [ ] CHK007 Failure propagation blocks dependents
|
||||
- [ ] CHK005 Every provider receives context containing run, step, attempt, descriptor/binding/principal
|
||||
fingerprints, deadline, idempotency key, capacity lease and trace id.
|
||||
- [ ] CHK006 Every provider creates an append-only operation receipt before I/O and returns effect state,
|
||||
retry disposition, operation id and stable reason code.
|
||||
- [ ] CHK007 Every evidence ref has an immutable receipt binding owner tuple, provider/version, descriptor,
|
||||
content type, byte length and verified non-zero SHA-256.
|
||||
- [ ] CHK008 No provider performs I/O before a durable shared CapacityLease; lease expiry/release is
|
||||
idempotent and observable.
|
||||
- [ ] CHK009 Cancellation acknowledges `stopped|completed|unknown`; unknown effect blocks retry and PASS
|
||||
until reconciliation or terminal non-pass closure.
|
||||
- [ ] CHK010 Duplicate invocation, late response, malformed result, cleanup failure and reconciliation
|
||||
are covered for every enabled provider.
|
||||
- [ ] CHK011 Each enabled provider exposes separate liveness, readiness and dependency health with
|
||||
secret-free diagnostics and a registration/capability fingerprint.
|
||||
|
||||
## Human (FR-004/010)
|
||||
## Provider-Specific Obligations
|
||||
|
||||
- [ ] CHK008 Human step suspends run (waiting_human), persists, creates HumanCheckpoint (confirm/false_positive/inconclusive)
|
||||
- [ ] CHK009 Resume from token without rerunning completed steps
|
||||
- [ ] CHK010 human excluded from executor registry (lifecycle control)
|
||||
- [~] CHK012 Browser provider contract specifies isolated context, server auth binding, safe checkpoint,
|
||||
cleanup, mutation reconciliation, per-action risk classification, 120s/30s/3-page/25MiB/10MiB
|
||||
limits and PREPROD canaries; runtime provider and canary evidence remain open.
|
||||
- [ ] CHK013 Superset/SQL provider proves exact 037 model, database, principal/RLS and raw-response digest/ref;
|
||||
caller SQL and metadata cannot pass.
|
||||
- [ ] CHK014 XLSX provider accepts only server-owned verified artifacts and enforces archive/cell/formula limits.
|
||||
- [ ] CHK015 Assertion/Transform providers are deterministic, bounded and network/code-free.
|
||||
- [ ] CHK016 Screenshot provider atomically commits durable owned evidence and cleans temporary resources.
|
||||
- [ ] CHK017 Report/Artifact providers use immutable manifests, templates, producer receipts and digest verification.
|
||||
- [ ] CHK018 AgentEvaluation uses immutable 038 spec, pinned provider/model/prompt, bounded budget and
|
||||
evidence allowlist; DecisionPolicy alone produces StepOutcome.
|
||||
|
||||
## Lifecycle (FR-005)
|
||||
## Human and Lifecycle
|
||||
|
||||
- [ ] CHK011 Statuses queued/running/waiting_human/blocked/cancel_requested/cancelled/passed/failed/inconclusive
|
||||
- [ ] CHK012 Cancel drains in-flight within bounded window
|
||||
- [x] CHK019 Human step is lifecycle control only: `waiting_human`, exactly one pending checkpoint, CAS decision.
|
||||
- [x] CHK020 Resume continues only the missing DAG frontier and never reruns completed logical steps.
|
||||
- [x] CHK021 Lifecycle enum includes pending_approval, queued, running, waiting_human, blocked,
|
||||
cancel_requested, cancelled, passed, failed and inconclusive.
|
||||
- [x] CHK022 Cancellation has persisted drain deadline; terminal run has zero active lease, running step or
|
||||
active evidence projection.
|
||||
- [x] CHK023 Retry invalidates downstream closure, creates a new attempt and preserves historical evidence.
|
||||
|
||||
## Safety (FR-008)
|
||||
## Authority, API and Operations
|
||||
|
||||
- [ ] CHK013 PROD gated (ActionApprovalGate) before execution
|
||||
- [ ] CHK014 RBAC scenario:run vs scenario:run:prod
|
||||
- [x] CHK024 Server-owned environment policy creates PROD approval before dispatch and ignores client PROD flags.
|
||||
- [~] CHK025 API schemas require immutable run identity, status enums, step outcomes and execution provenance.
|
||||
- [ ] CHK026 Startup registers enabled providers only after dependency/readiness checks; unready providers
|
||||
admit zero new operations.
|
||||
- [ ] CHK027 Real PostgreSQL `alembic check` and `alembic upgrade head` pass on the deployment database.
|
||||
- [x] CHK028 Full available 044 backend profile passes: registry/API suite, provider/lifecycle profile,
|
||||
scoped Ruff/compile and prototype validation.
|
||||
|
||||
## Success Criteria
|
||||
## Release Gates
|
||||
|
||||
- [ ] CHK015 SC-001..006 verified (deterministic dispatch, human resume, cancel drain, recovery, immutable snapshot, human-not-executor)
|
||||
- [ ] CHK029 SC-001..011 each has reproducible command output or deployment evidence linked in traceability.
|
||||
- [ ] CHK030 No unresolved P0/P1 row remains in 044 or its execution-critical dependencies 036, 037, 038,
|
||||
041, 042, 046 and 047.
|
||||
- [ ] CHK031 Browser and Screenshot providers perform authorized live I/O and produce owned durable evidence
|
||||
in a real deployment run.
|
||||
|
||||
@@ -111,4 +111,258 @@ EXECUTOR_MAP = {browser: BrowserExecutor(ActionRegistry), superset_api, xlsx, as
|
||||
# @INVARIANT descriptor, not executor/tool default, declares {idempotent, retry_safe, side_effect_key, timeout, mutation risk}.
|
||||
# @INVARIANT mutation actions require immutable mutation_contract; PROD mutation is rejected; mutation retries default false.
|
||||
# #endregion ScenarioExecution.ExecutorRegistry
|
||||
|
||||
# #region ScenarioExecution.ProviderProtocol [C:5] [TYPE ADR] [SEMANTICS scenario,execution,provider,protocol,lifecycle,ownership,observability]
|
||||
# @ingroup ScenarioExecution
|
||||
# @BRIEF Common production contract for every external or resource-owning scenario provider.
|
||||
# @DATA_CONTRACT ProviderExecutionContext -> ProviderExecutionResult -> ScenarioStepRun/Artifact
|
||||
# @PRE Provider is registered by trusted startup composition, declares a stable provider_id,
|
||||
# provider_version, supported action descriptors, resource limits, health status, and
|
||||
# cancellation capabilities before accepting a run.
|
||||
# @POST Every invocation is bound to run_id, logical_step_id, attempt, descriptor fingerprint,
|
||||
# execution principal fingerprint, deadline, idempotency key, capacity lease, and trace_id;
|
||||
# result is persisted exactly once for that attempt or is reconciled before retry.
|
||||
# @SIDE_EFFECT External I/O, provider-owned resources, durable evidence, metrics, structured events.
|
||||
# @INVARIANT A provider cannot select authority, environment, principal, action policy, retry policy,
|
||||
# or capacity outside the pinned descriptor and server-owned composition.
|
||||
# @INVARIANT A provider result is accepted only when ownership proof binds every output/evidence ref
|
||||
# to run_id, step_id, attempt, provider_id/version, descriptor fingerprint and digest.
|
||||
# @INVARIANT Provider timeout/cancellation never implies rollback; unknown external effect state is
|
||||
# reconciled or terminalized non-pass before another attempt may start.
|
||||
# @INVARIANT Health failure prevents new claims but does not erase active leases or evidence history.
|
||||
# @REJECTED A generic callable-only provider contract was rejected — it hides resources, ownership,
|
||||
# cancellation, reconciliation and deployability obligations.
|
||||
|
||||
ProviderExecutionContext = {
|
||||
"run_id": "UUID",
|
||||
"logical_step_id": "UUID",
|
||||
"attempt": "positive integer",
|
||||
"descriptor_fingerprint": "sha256",
|
||||
"binding_ref": "opaque server-owned ref",
|
||||
"execution_principal_fingerprint": "sha256",
|
||||
"deadline_at": "RFC3339 timestamp",
|
||||
"idempotency_key": "opaque stable key",
|
||||
"capacity_lease_id": "opaque lease ref",
|
||||
"trace_id": "opaque trace ref",
|
||||
"cancellation": {"requested": "bool", "deadline_at": "RFC3339|null"},
|
||||
}
|
||||
|
||||
ProviderExecutionResult = {
|
||||
"status": "passed|failed|inconclusive|blocked",
|
||||
"reason_code": "stable taxonomy code",
|
||||
"provider_id": "stable id",
|
||||
"provider_version": "immutable version",
|
||||
"operation_id": "provider operation ref",
|
||||
"output_refs": "owned output refs",
|
||||
"evidence": "owned evidence refs + sha256 + media/type metadata",
|
||||
"effect_state": "none|completed|not_started|unknown|requires_reconciliation",
|
||||
"retry_disposition": "never|safe|after_reconciliation|manual_only",
|
||||
"observability": "trace/span/metric dimensions",
|
||||
}
|
||||
|
||||
# Provider lifecycle is: registered -> health_checked -> capacity_claimed -> admitted -> running ->
|
||||
# completed|failed|cancel_requested -> finalized|reconciliation_required. A provider MUST release
|
||||
# capacity and ephemeral resources in finalized/reconciliation_required, while durable evidence and
|
||||
# operation receipts remain immutable. A provider MUST expose readiness (can accept new work), liveness
|
||||
# (process responsive), and dependency health (external system reachable/authenticated) separately.
|
||||
#
|
||||
# Cancellation is cooperative first: runner marks cancellation_requested and invokes provider cancel
|
||||
# with operation_id. The provider acknowledges stopped|completed|unknown by the deadline. Unknown means
|
||||
# no retry and no PASS until reconcile(operation_id, idempotency_key) returns a terminal receipt or the
|
||||
# run is terminalized inconclusive/blocked. Retry is a new attempt with a new attempt id but the same
|
||||
# logical side-effect identity where the descriptor requires it.
|
||||
# #endregion ScenarioExecution.ProviderProtocol
|
||||
|
||||
# #region ScenarioExecution.ProviderCatalog [C:5] [TYPE ADR] [SEMANTICS scenario,execution,provider,catalog,browser,superset,xlsx,screenshot,agent]
|
||||
# @ingroup ScenarioExecution
|
||||
# @BRIEF Per-provider obligations and production acceptance criteria.
|
||||
# @INVARIANT Every catalog entry has input/output schemas, limits, ownership proof, failure taxonomy,
|
||||
# cancel/reconcile semantics, health checks, deployment registration and falsifiable tests.
|
||||
|
||||
Provider obligations:
|
||||
|
||||
- `BrowserProvider`: owns an isolated browser context for one run/lease; authenticates only through
|
||||
the server-owned auth binding; accepts only registry-pinned actions and typed inputs; emits an
|
||||
operation receipt, safe-checkpoint ref, page/dashboard provenance and durable evidence refs where
|
||||
applicable; closes context on finalization. It MUST reject unsafe recovery without a pinned safe
|
||||
checkpoint, report mutation effects as `completed|unknown`, and reconcile by operation id before retry.
|
||||
- `SupersetProvider`: invokes only the exact 037 query envelope/model/principal/RLS binding; enforces
|
||||
request, response, timeout and row/byte limits; records request/response fingerprints and raw-byte
|
||||
evidence ownership. 403/422 are non-retryable policy/input failures, 429/5xx are bounded retryable
|
||||
transport failures, and timeout/cancel with unknown server execution requires reconciliation by the
|
||||
external request id before retry.
|
||||
- `SqlEvidenceProvider`: is read-only and executes an immutable 038 SqlEvidenceSpec; runtime may bind
|
||||
typed parameters only. It MUST prove database identity, query-model fingerprint, principal/RLS
|
||||
fingerprint, raw response digest and durable evidence ref. Caller SQL or metadata never qualifies.
|
||||
- `XlsxProvider`: consumes a server-owned immutable artifact ref, not unbounded caller bytes; validates
|
||||
content digest, workbook type, archive/resource limits, allowed sheets/formulas/external links and
|
||||
canonical normalization. Parsing is local and idempotent; source artifact ownership is retained.
|
||||
- `AssertionProvider`: is pure and deterministic over canonical inputs; invokes 037 comparison with a
|
||||
pinned policy/version, has no external resources, and emits no PASS without complete inputs.
|
||||
- `TransformProvider`: executes only the bounded 038 TransformSpec DSL with hard operation, row,
|
||||
column and output limits; no code, SQL, imports or network. It is deterministic and retry-safe.
|
||||
- `ScreenshotProvider`: captures through a server-owned browser/capture binding and writes evidence
|
||||
atomically to durable storage. Each ref must contain an ownership receipt for run/step/attempt,
|
||||
binding, capture profile, media type, byte length and SHA-256. Temp files are cleaned after commit
|
||||
or failure; a path or raw bytes alone are never evidence.
|
||||
- `ReportProvider`: renders a version-pinned template from an immutable evidence manifest; output is
|
||||
deterministic for the same manifest/template, size-bounded, durably stored and digest-verified.
|
||||
Partial render/write is cleaned or marked reconciliation-required.
|
||||
- `ArtifactProvider`: registers only content refs created by an authorized producer or verified source
|
||||
artifact. It must not mint evidence from caller-supplied digest strings; ownership, digest, retention
|
||||
and active/inactive projection are immutable and audit-visible.
|
||||
- `AgentEvaluationProvider`: executes only a declared immutable 038 AgentEvaluationSpec, with explicit
|
||||
provider/model/prompt versions, input manifest hash, token/cost/time limits and evidence allowlist.
|
||||
It emits immutable AgentEvaluation; it cannot mutate the graph, invoke actions, consume checkpoints,
|
||||
schedule work or set ScenarioResult. DecisionPolicy alone maps evaluation plus deterministic evidence
|
||||
to StepOutcome.
|
||||
|
||||
# @REJECTED Treating provider availability as successful execution was rejected — unavailable,
|
||||
# unhealthy, missing-capacity and missing-evidence outcomes are explicit non-pass states.
|
||||
# #endregion ScenarioExecution.ProviderCatalog
|
||||
|
||||
# #region ScenarioExecution.BrowserProvider [C:5] [TYPE Module] [SEMANTICS scenario,execution,browser,playwright,actions,checkpoint,mutation]
|
||||
# @ingroup ScenarioExecution
|
||||
# @BRIEF Production contract for isolated BrowserProvider execution through the 038 ActionRegistry.
|
||||
# @RELATION IMPLEMENTS -> [ScenarioExecution.ProviderProtocol]
|
||||
# @RELATION DEPENDS_ON -> [ScenarioExecution.CapacityManager]
|
||||
# @RELATION DEPENDS_ON -> [ScenarioExecution.ProviderOperations]
|
||||
# @PRE Startup registration supplies provider/version, browser versions, catalog/binding fingerprints,
|
||||
# limits, health and cancellation capability. A valid CapacityLease and operation receipt exist
|
||||
# before browser/context creation.
|
||||
# @POST Every action returns typed output and an immutable operation receipt; declared evidence has an
|
||||
# EvidenceReceipt; context/pages/downloads/temp files are closed or accounted before finalization.
|
||||
# @SIDE_EFFECT Browser process/context, authenticated session, navigation, downloads, evidence and only
|
||||
# descriptor-authorized mutation effects.
|
||||
# @INVARIANT Authority comes only from the exact persisted binding and 038 descriptor; URLs, IDs, sessions,
|
||||
# cookies and request metadata cannot select credentials or principal.
|
||||
# @INVARIANT Mutation requires mutation_contract, fixture lease, target keys, precondition hash, cleanup
|
||||
# policy and retry_safe=false; PROD mutation is rejected before browser invocation.
|
||||
# @INVARIANT Crash recovery never revives a dead context; it reconstructs from a safe checkpoint. Unknown
|
||||
# mutation effects require reconciliation before retry.
|
||||
# @INVARIANT Browser PASS requires action receipt, output-shape validation and binding/principal/evidence
|
||||
# provenance; page/session metadata alone cannot produce PASS.
|
||||
# @REJECTED Process-global Playwright pages were rejected because they permit cross-run cookies and target
|
||||
# leakage. Dead-context replay was rejected because external effect state is unknowable.
|
||||
|
||||
BrowserProviderActionContract = {
|
||||
"action": "exact registered 038 action",
|
||||
"input_schema": "versioned typed JSON schema",
|
||||
"output_schema": "versioned typed JSON schema",
|
||||
"risk": "read_only|evidence|mutation",
|
||||
"context_timeout_ms": 120000,
|
||||
"action_timeout_ms": 30000,
|
||||
"max_pages": 3,
|
||||
"max_download_bytes": 26214400,
|
||||
"max_screenshot_bytes": 10485760,
|
||||
"checkpoint_policy": "none|safe_reconstruction_point|required",
|
||||
"idempotency": "idempotent|keyed|non_idempotent",
|
||||
"retry_safe": "bool",
|
||||
"mutation_contract": "required for risk=mutation",
|
||||
}
|
||||
|
||||
Per-action policy:
|
||||
|
||||
- `open_dashboard`: typed dashboard/release/binding input; output canonical URL, dashboard fingerprint and
|
||||
`dashboard_open` checkpoint; read-only.
|
||||
- `navigate_tab`: allowlisted tab/route token only; output selected tab and `tab_navigated` checkpoint;
|
||||
external origins are rejected.
|
||||
- `apply_native_filter`: typed query-model filter input; output normalized state and `filters_applied`
|
||||
checkpoint; saving a filter is a separate mutation action.
|
||||
- `inspect_filter_state`, `inspect_columns`, `scroll_to`, `wait_for_state`: typed bounded read-only
|
||||
actions; timeout is inconclusive and cannot manufacture a checkpoint.
|
||||
- `apply_table_filter`: typed normalized table filter; it cannot persist dashboard state.
|
||||
- `extract_table`: bounded selector/projection; max 10,000 rows, 100 columns and 10 MiB output; declared
|
||||
evidence requires an EvidenceReceipt.
|
||||
- `click`: locator plus explicit effect class; submit/save/delete/edit behavior is mutation, and ambiguous
|
||||
clicks are rejected.
|
||||
- `select_rows`: exact row keys/locator; selection-only is read-only and never proves mutation completion.
|
||||
- `edit_row`: exact fixture/record keys, field allowlist and precondition hash; fixture lease, cleanup and
|
||||
reconciliation are mandatory; prohibited in PROD.
|
||||
- `bulk_edit`: same mutation requirements, max 100 record keys and explicit rollback/cleanup plan;
|
||||
prohibited in PROD.
|
||||
- `download`: allowlisted artifact type and max bytes; output is a server-owned artifact ref/digest, never
|
||||
a local path.
|
||||
- `refresh`: bounded read-only render operation; it is not evidence of data correctness without a separate
|
||||
evidence action.
|
||||
|
||||
Resource lifecycle is `lease_claimed -> context_created -> authenticated -> action_running ->
|
||||
checkpointed|effect_recorded -> cleanup_started -> receipt_finalized -> context_closed`. At most one
|
||||
active context exists per `(run_id, capacity_lease_id)`, with max three pages, 25 MiB downloads and 10 MiB
|
||||
screenshots. Context/authentication has a 120-second limit; each action has a 30-second limit unless the
|
||||
descriptor pins a lower value. Cleanup runs on success, failure, timeout and cancellation; cleanup failure
|
||||
prevents PASS and marks mutation reconciliation required.
|
||||
|
||||
`cancel(operation_id)` stops new actions, closes the page/context by the persisted drain deadline and
|
||||
returns `stopped|completed|unknown` with checkpoint/effect state. Read-only work may retry after clean close;
|
||||
unknown mutation work cannot retry until `reconcile(operation_id, idempotency_key)` verifies target state.
|
||||
Recovery never attaches to the old process: it reconstructs from `dashboard_open`, `filters_applied`,
|
||||
`tab_navigated` or another descriptor-declared safe checkpoint and records `reconstruction_replay=true`.
|
||||
|
||||
Readiness is `ready` only when browser executable/version, auth binding, allowed origins, evidence storage,
|
||||
cleanup worker and cancellation capability checks pass. Health exposes provider/version, catalog/binding
|
||||
fingerprints, origin hash, limits and dependency status, excluding cookies, credentials, secret URLs and
|
||||
page content. Deployment requires one read-only PREPROD canary and one forced timeout/cleanup canary before
|
||||
enabling the provider; mutation requires a separate fixture-lease canary.
|
||||
|
||||
Acceptance profile: 100% pass for schema rejection, cross-run isolation, unauthorized-origin rejection,
|
||||
binding/lease failures, timeout, cancellation, duplicate/late response, safe recovery, unsafe mutation
|
||||
recovery, cleanup failure, ownership mismatch, size limits, health redaction and startup registration.
|
||||
A real PREPROD canary must produce one owned evidence receipt and one reconstruction trace.
|
||||
# #endregion ScenarioExecution.BrowserProvider
|
||||
|
||||
# #region ScenarioExecution.CapacityManager [C:5] [TYPE Module] [SEMANTICS scenario,execution,capacity,quota,lease,concurrency]
|
||||
# @ingroup ScenarioExecution
|
||||
# @BRIEF Environment-scoped allocator shared by ScenarioRun and neighboring workloads.
|
||||
# @PRE Claim contains environment, workload_class, priority, requested units, provider_id, run/step,
|
||||
# deadline and idempotency context; capacity policy is server-owned.
|
||||
# @POST Atomic claim returns a lease or `CAPACITY_UNAVAILABLE`; lease heartbeat, expiry, release and
|
||||
# forced reconciliation are durable and idempotent.
|
||||
# @INVARIANT No provider starts external I/O before a valid capacity lease; retries claim separately.
|
||||
# @INVARIANT Capacity exhaustion creates queued/blocked state according to policy, never an unbounded
|
||||
# provider call or a client-controlled quota increase.
|
||||
# @SIDE_EFFECT Durable lease/quota records, admission metrics, provider concurrency accounting.
|
||||
# @REJECTED Independent per-feature concurrency limits were rejected — they permit cross-workload
|
||||
# starvation and bypass the environment-scoped allocator.
|
||||
def claim_capacity(context: ProviderExecutionContext, request): ...
|
||||
def heartbeat_capacity(lease_id: str): ...
|
||||
def release_capacity(lease_id: str): ...
|
||||
# #endregion ScenarioExecution.CapacityManager
|
||||
|
||||
# #region ScenarioExecution.ProviderOperations [C:5] [TYPE Module] [SEMANTICS scenario,execution,provider,operation,cancel,retry,reconcile]
|
||||
# @ingroup ScenarioExecution
|
||||
# @BRIEF Durable operation receipts and recovery protocol for provider side effects.
|
||||
# @PRE Operation receipt is created before external I/O with operation_id, idempotency_key,
|
||||
# descriptor/provider fingerprints and capacity lease.
|
||||
# @POST Every operation reaches completed, failed, cancelled, or reconciliation_required; no receipt
|
||||
# is overwritten, and a late provider response is stored as historical evidence only.
|
||||
# @INVARIANT Retry cannot begin while effect_state is unknown or reconciliation_required.
|
||||
# @SIDE_EFFECT Operation receipt, cancellation request, reconciliation record and audit events.
|
||||
def request_provider_cancel(operation_id: str, deadline_at: str): ...
|
||||
def reconcile_provider_operation(operation_id: str, idempotency_key: str): ...
|
||||
# #endregion ScenarioExecution.ProviderOperations
|
||||
|
||||
# #region ScenarioExecution.ProviderOperations.Observability [C:4] [TYPE Module] [SEMANTICS scenario,execution,provider,health,metrics,tracing]
|
||||
# @ingroup ScenarioExecution
|
||||
# @BRIEF Required provider health, readiness, trace and metric surface.
|
||||
# @POST Provider exposes liveness, readiness, dependency health, registration fingerprint and metrics
|
||||
# for admission, latency, timeout, cancellation, retry, reconciliation, capacity and evidence.
|
||||
# @INVARIANT Health output never includes credentials, cookies, raw SQL, capture bytes or secret payloads.
|
||||
# @SIDE_EFFECT Structured events and metrics correlated by trace_id/run_id/step_id/operation_id.
|
||||
# @TEST_EDGE dependency_down->not_ready; auth_failure->unavailable; stale_registration->not_ready;
|
||||
# health_payload_redaction->no_secret_leak.
|
||||
# #endregion ScenarioExecution.ProviderOperations.Observability
|
||||
|
||||
# #region ScenarioExecution.ReleaseEvidence [C:5] [TYPE ADR] [SEMANTICS scenario,execution,release,verification,evidence,thresholds]
|
||||
# @ingroup ScenarioExecution
|
||||
# @BRIEF Define the measurable evidence required to declare 044 production-ready.
|
||||
# @POST Release evidence contains command output or deployment records for SC-001..SC-011, the full
|
||||
# available backend profile, provider contract profile, PostgreSQL migration verification and
|
||||
# prototype validation.
|
||||
# @INVARIANT A local unit profile cannot close a live-provider, PostgreSQL, scheduler or cross-spec gate.
|
||||
# @INVARIANT Production GO requires 100% pass for all mandatory profiles, zero unresolved P0/P1 rows in
|
||||
# 044 and execution-critical dependencies, and no enabled provider without readiness evidence.
|
||||
# @TEST_EDGE missing_profile->NO_GO; stale_profile->NO_GO; provider_not_ready->NO_GO;
|
||||
# unresolved_dependency_P1->NO_GO.
|
||||
# #endregion ScenarioExecution.ReleaseEvidence
|
||||
#endregion ScenarioExecution.Modules
|
||||
|
||||
@@ -93,7 +93,10 @@ paths:
|
||||
summary: Cancel a run (bounded drain)
|
||||
security: [{ bearerAuth: [] }]
|
||||
parameters: [{ name: run_id, in: path, required: true, schema: { type: string } }]
|
||||
responses: { "200": { description: Cancelled } }
|
||||
responses:
|
||||
"202": { description: Cancellation requested; persisted drain deadline returned, content: { application/json: { schema: { $ref: "#/components/schemas/ScenarioRun" } } } }
|
||||
"200": { description: Cancellation already finalized, content: { application/json: { schema: { $ref: "#/components/schemas/ScenarioRun" } } } }
|
||||
"409": { description: Run cannot be cancelled from its current terminal state }
|
||||
/api/scenario-runs/{run_id}/resume:
|
||||
post:
|
||||
operationId: scenarioRun.resume
|
||||
@@ -136,6 +139,7 @@ components:
|
||||
schemas:
|
||||
ScenarioRun:
|
||||
type: object
|
||||
required: [id, scenario_id, scenario_revision_id, environment_id, status, parameter_bindings, target_snapshot, execution_principal_fingerprint, verification_program_hash, action_registry_version, step_runs]
|
||||
properties:
|
||||
id: { type: string }
|
||||
scenario_id: { type: string }
|
||||
@@ -151,12 +155,13 @@ components:
|
||||
step_runs: { type: array, items: { $ref: "#/components/schemas/ScenarioStepRun" } }
|
||||
ScenarioStepRun:
|
||||
type: object
|
||||
required: [id, logical_step_id, step_position, attempt, status, outputs, artifact_refs, error_code, step_outcome]
|
||||
properties:
|
||||
id: { type: string }
|
||||
logical_step_id: { type: string }
|
||||
step_position: { type: integer }
|
||||
attempt: { type: integer }
|
||||
status: { type: string }
|
||||
status: { type: string, enum: [queued, running, waiting_human, passed, failed, inconclusive, blocked, skipped, cancelled] }
|
||||
outputs: { type: object }
|
||||
artifact_refs: { type: array, items: { type: string } }
|
||||
error_code: { type: string, nullable: true }
|
||||
@@ -187,7 +192,7 @@ components:
|
||||
status: { type: string }
|
||||
step_counts: { type: object, additionalProperties: { type: integer } }
|
||||
failures: { type: array, items: { type: object } }
|
||||
provenance: { type: object }
|
||||
provenance: { $ref: "#/components/schemas/ExecutionProvenance" }
|
||||
analytics_context_key: { type: string }
|
||||
step_outcomes: { type: array, items: { $ref: "#/components/schemas/StepOutcome" } }
|
||||
agent_evaluation_summary: { type: object }
|
||||
@@ -201,6 +206,36 @@ components:
|
||||
agent_evaluation_ids: { type: array, items: { type: string } }
|
||||
decision_policy_id: { type: [string, "null"] }
|
||||
decision_policy_version: { type: [string, "null"] }
|
||||
ExecutionProvenance:
|
||||
type: object
|
||||
required: [scenario_revision_id, scenario_content_hash, verification_program_hash, runner_version, target_snapshot, parameter_bindings, execution_principal_fingerprint, analytics_context_key]
|
||||
properties:
|
||||
scenario_revision_id: { type: string }
|
||||
scenario_content_hash: { type: string, pattern: '^[0-9a-fA-F]{64}$' }
|
||||
verification_program_hash: { type: string, pattern: '^[0-9a-fA-F]{64}$' }
|
||||
runner_version: { type: string, minLength: 1 }
|
||||
target_snapshot: { $ref: "#/components/schemas/TargetSnapshot" }
|
||||
parameter_bindings: { type: array, items: { $ref: "#/components/schemas/ParameterBinding" } }
|
||||
execution_principal_fingerprint: { type: string, minLength: 1 }
|
||||
analytics_context_key: { type: string, pattern: '^[0-9a-fA-F]{64}$' }
|
||||
evidence_receipts: { type: array, items: { $ref: "#/components/schemas/EvidenceReceipt" } }
|
||||
EvidenceReceipt:
|
||||
type: object
|
||||
required: [evidence_ref, owner_type, owner_id, run_id, logical_step_id, attempt, operation_id, provider_id, provider_version, descriptor_fingerprint, content_type, byte_length, sha256]
|
||||
properties:
|
||||
evidence_ref: { type: string, minLength: 1 }
|
||||
owner_type: { type: string, const: scenario_run }
|
||||
owner_id: { type: string }
|
||||
run_id: { type: string }
|
||||
logical_step_id: { type: string }
|
||||
attempt: { type: integer, minimum: 1 }
|
||||
operation_id: { type: string, minLength: 1 }
|
||||
provider_id: { type: string, minLength: 1 }
|
||||
provider_version: { type: string, minLength: 1 }
|
||||
descriptor_fingerprint: { type: string, pattern: '^[0-9a-fA-F]{64}$' }
|
||||
content_type: { type: string, minLength: 1 }
|
||||
byte_length: { type: integer, minimum: 1 }
|
||||
sha256: { type: string, pattern: '^[0-9a-fA-F]{64}$' }
|
||||
AgentEvaluationSummary:
|
||||
type: object
|
||||
required: [evaluation_id, logical_step_id, verdict, confidence, model_id, prompt_template_version]
|
||||
|
||||
@@ -22,7 +22,11 @@ server-side to build the existing `SupersetClient`, then registers the exact mod
|
||||
Browser and screenshot providers are process-local registrations and are never serialized in this
|
||||
record; absent registrations yield stable configured-unavailable outcomes.
|
||||
|
||||
If the selected revision contains a `human` step, it is derived as `manual_run_only=true`: it may start only from the authenticated analyst manual-run route. Scheduler, deploy/ETL/API trigger and any background runner are ineligible; no HumanCheckpoint may be skipped or defaulted.
|
||||
If the selected revision contains a `human` step, it is derived as `manual_run_only=true`: it may start
|
||||
only from the authenticated analyst manual-run route. Scheduler, deploy/release/ETL/API trigger and any
|
||||
background runner or recovery worker are ineligible. The eligibility check occurs before idempotency,
|
||||
ScenarioRun, ActionApprovalGate, notification, queue insertion and dispatcher CAS; no HumanCheckpoint may
|
||||
be created, skipped, defaulted or converted into an automated approval path.
|
||||
|
||||
For a PROD request, the service atomically creates `ScenarioRun(status=pending_approval)` and `ActionApprovalGate(owner_type=scenario_run, owner_id=run_id, operation=scenario_execution)`. Approval transitions only `pending_approval → queued`; denial/expiry transitions to `blocked`. Schedulers and external triggers therefore receive a durable run/intent, never an unusable 403. `trigger_source` is set only by the trusted entry point (manual route, scheduler, deploy connector, or API key), never by a bearer client field. Idempotency uses canonical execution-request hash: same key + same hash returns the existing run; same key + different hash returns `409 IDEMPOTENCY_KEY_REUSED`.
|
||||
|
||||
@@ -118,6 +122,110 @@ The registry resolves `ActionExecutionDescriptor -> executor`; the following lis
|
||||
|
||||
`BrowserExecutor` implements only registered actions: `open_dashboard`, `navigate_tab`, `apply_native_filter`, `inspect_filter_state`, `apply_table_filter`, `extract_table`, `scroll_to`, `inspect_columns`, `click`, `select_rows`, `edit_row`, `bulk_edit`, `download`, `refresh`, `wait_for_state`. ScreenshotService is evidence infrastructure only, not the browser action executor. It validates the registry-declared inputs/outputs/risk/timeout/idempotency before dispatch.
|
||||
|
||||
## Provider runtime protocol
|
||||
|
||||
Every provider implements the following logical records. They are persisted or emitted as immutable
|
||||
operation/evidence receipts; a Python callback alone is not a sufficient provider implementation.
|
||||
|
||||
`ProviderExecutionContext`: `run_id`, `logical_step_id`, `attempt`, `descriptor_fingerprint`,
|
||||
`live_binding_ref`, `execution_principal_fingerprint`, `deadline_at`, `idempotency_key`,
|
||||
`capacity_lease_id`, `trace_id`, `cancellation_requested`, `cancellation_deadline_at`.
|
||||
|
||||
`ProviderOperationReceipt`: `operation_id`, context identity, `provider_id`, `provider_version`,
|
||||
`started_at`, `finished_at`, `effect_state`, `external_request_id?`, `resource_refs`,
|
||||
`cleanup_state`, `reconciliation_state`, `result_digest?`. Receipts are append-only; late responses are
|
||||
historical records and cannot mutate the winning attempt.
|
||||
|
||||
`ProviderEvidenceReceipt`: `evidence_ref`, `owner_type=scenario_run`, `owner_id`, `run_id`,
|
||||
`logical_step_id`, `attempt`, `operation_id`, `provider_id/version`, `descriptor_fingerprint`,
|
||||
`binding_ref`, `execution_principal_fingerprint`, `content_type`, `byte_length`, `sha256`,
|
||||
`retention_class`, `created_at`. The durable store must atomically commit the content and receipt or
|
||||
return no usable ref.
|
||||
|
||||
Provider state is `registered|health_checked|capacity_claimed|admitted|running|completed|failed|
|
||||
cancel_requested|finalized|reconciliation_required`. A provider exposes separate liveness, readiness and
|
||||
dependency-health checks and a redacted capability/limit document. Readiness gates new admissions only;
|
||||
it does not remove active leases or evidence.
|
||||
|
||||
Cancellation calls `cancel(operation_id)` and records `stopped|completed|unknown`. `unknown` means the
|
||||
provider cannot prove whether an external effect happened: retry is forbidden until
|
||||
`reconcile(operation_id, idempotency_key)` returns a terminal receipt. If reconciliation is unavailable
|
||||
before the bounded recovery deadline, the runner terminalizes `inconclusive` or `blocked` according to
|
||||
effect risk. Capacity is released only after receipt and cleanup state are durable.
|
||||
|
||||
### Provider-specific schemas and obligations
|
||||
|
||||
| Provider | Input boundary | Required result/evidence | Resource and recovery rules |
|
||||
|---|---|---|---|
|
||||
| Browser | 038 action descriptor, server-owned auth/session binding, typed action input | operation receipt, safe checkpoint, page/action provenance and durable evidence where declared | isolated context, bounded pages/downloads, cleanup on finalization, safe-checkpoint replay only |
|
||||
| Superset API | exact 037 query envelope and pinned model/principal/RLS | request/response fingerprints, raw response SHA-256 and durable ref | request/response limits; 429/5xx bounded retry; unknown timeout reconciles external request |
|
||||
| SQL evidence | immutable 038 SqlEvidenceSpec plus typed ParameterBindings | DB identity, model/principal/RLS fingerprints, raw response digest/ref | read-only; caller SQL rejected; reconcile before retry |
|
||||
| XLSX | verified server-owned artifact ref | source ownership/digest and canonical normalized output | archive/cell/formula limits; local parse is idempotent |
|
||||
| Assertion | canonical values plus pinned 037 ComparisonPolicy | deterministic input manifest, policy/version and StepOutcome | pure, no external resources, retry-safe |
|
||||
| Transform | bounded 038 TransformSpec DSL | source manifest, transform fingerprint and canonical output digest | hard operation/row/column/output limits; no code/SQL/network |
|
||||
| Screenshot | bound capture session/profile and durable evidence policy | atomic evidence receipt with media type, length and SHA-256 | temp cleanup; unknown capture reconciles before retry |
|
||||
| Report | pinned template and immutable evidence manifest | template/manifest/output digest and durable ref | bounded deterministic renderer; partial writes reconcile or discard |
|
||||
| Artifact | authorized producer ref or verified source artifact | producer receipt, owner tuple, digest and retention | never accept caller digest alone; registration idempotent |
|
||||
| AgentEvaluation | immutable 038 spec, evidence allowlist, provider/model/prompt versions | immutable AgentEvaluation and raw response artifact receipt | token/cost/time limits; verdict cannot directly set StepOutcome |
|
||||
|
||||
### BrowserProvider action contract
|
||||
|
||||
Each browser step persists the exact 038 action descriptor plus `input_schema_version`,
|
||||
`output_schema_version`, `risk`, `checkpoint_policy`, `idempotency`, `retry_safe`, limits and mutation
|
||||
contract fingerprint. Defaults are context/authentication timeout `120000ms`, action timeout `30000ms`,
|
||||
maximum `3` pages, maximum download `26214400` bytes and maximum screenshot `10485760` bytes. A descriptor
|
||||
may lower a limit but cannot raise it from scenario payload.
|
||||
|
||||
`open_dashboard`, `navigate_tab`, filter application, inspection, scroll, extraction, refresh and wait are
|
||||
read-only/evidence actions; `click` is rejected unless its effect class is explicit; `edit_row` and
|
||||
`bulk_edit` are mutation actions; `select_rows` is selection-only; `download` produces a server-owned
|
||||
artifact ref, never a local path. `extract_table` is bounded to 10,000 rows, 100 columns and 10 MiB output.
|
||||
|
||||
Browser resource state is `lease_claimed|context_created|authenticated|action_running|checkpointed|
|
||||
effect_recorded|cleanup_started|receipt_finalized|context_closed`. One active context is allowed per
|
||||
`(run_id, capacity_lease_id)` and cleanup failure prevents PASS. Recovery never attaches to a dead context;
|
||||
it reconstructs from a safe checkpoint and records `reconstruction_replay=true`. Unknown mutation effects
|
||||
are held for reconciliation and cannot retry.
|
||||
|
||||
Readiness requires browser executable/version, auth binding, allowed-origin, evidence-storage, cleanup-worker
|
||||
and cancellation-capability checks. Deployment requires a read-only PREPROD canary and forced timeout/
|
||||
cleanup canary before enabling; mutation requires a separate fixture-lease canary.
|
||||
|
||||
## AgentEvaluation and DecisionPolicy integration
|
||||
|
||||
The AgentEvaluation provider claims capacity as `workload_class=agent_evaluation`, creates an immutable
|
||||
input manifest from deterministic evidence refs, executes within the declared budget, and stores the raw
|
||||
response before publishing a verdict. Provider/model/prompt versions and manifest hash are mandatory.
|
||||
`DecisionPolicy(policy_id, version)` consumes only the immutable evaluation record plus deterministic
|
||||
evidence and emits StepOutcome. Low confidence, missing evidence or disagreement follow the pinned
|
||||
policy table; the model cannot select a policy, mutate the graph, invoke a provider action, consume a
|
||||
HumanCheckpoint or schedule downstream work.
|
||||
|
||||
## ExecutionCapacityManager integration
|
||||
|
||||
Before any provider I/O, the dispatcher calls the shared allocator with `environment`, `workload_class`,
|
||||
`provider_id`, `priority`, `requested_units`, `quota`, `reserved_capacity`, run/step and deadline. The
|
||||
atomic result is a durable `CapacityLease`; no lease means queued or blocked according to policy and no
|
||||
provider invocation. Heartbeat, expiry, release and forced reconciliation are idempotent. Retries claim
|
||||
new leases, while active unknown operations retain their lease until reconciliation or terminal closure.
|
||||
|
||||
## Verification thresholds
|
||||
|
||||
- `scenario_content_hash`, `verification_program_hash`, `descriptor_fingerprint`, `query_model_fingerprint`,
|
||||
`execution_principal_fingerprint`, `rls_security_fingerprint` and evidence `sha256` are lowercase or
|
||||
uppercase hexadecimal SHA-256 values with exactly 64 characters.
|
||||
- `attempt` starts at 1 and increases by exactly 1 for each new logical-step attempt. Historical attempts
|
||||
are immutable; only one attempt may be the active projection for a logical step.
|
||||
- `byte_length` is a positive integer equal to the durable content length. A zero-byte evidence object is
|
||||
never eligible for PASS.
|
||||
- `deadline_at` is later than `started_at`; a result received after the deadline cannot become the winning
|
||||
outcome. The runner records the late result as historical operation data.
|
||||
- CapacityLease validity is `[claimed_at, expires_at]`; provider I/O requires `now < expires_at` and a
|
||||
matching run/step/attempt/provider context. Lease release is idempotent: repeated release changes no
|
||||
terminal state or accounting total.
|
||||
- Cancellation finalization must occur no later than `cancel_drain_deadline_at + 5 seconds` under the
|
||||
default scheduler interval. Any exception is a release failure and keeps the run non-GO.
|
||||
|
||||
## Mutation authorization and execution policy
|
||||
|
||||
Authorization is evaluated per run and step as `environment_class + scenario risk profile + action mutation profile`. Read-only PROD steps require the ScenarioExecution approval policy. Every mutation requires the immutable 038 mutation contract, scoped target keys, precondition evidence, side-effect identity, cleanup/reconciliation outcome and `retry_safe=false` unless the registry proves an idempotent compensating action. Mutating browser steps in PROD are prohibited. Non-PROD test-data mutation may be delegated only inside an authorized fixture lease; this mutation policy is separate from PROD execution approval.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Implementation Plan: Scenario Execution Engine
|
||||
|
||||
**Branch**: `044-dashboard-scenario-execution` | **Date**: 2026-08-07 | **Spec**: spec.md | **Status**: Not production-complete — factual audit pending remediation
|
||||
**Branch**: `044-dashboard-scenario-execution` | **Date**: 2026-08-07 | **Spec**: spec.md | **Status**: Not production-complete — provider contract and runtime integration gate pending
|
||||
|
||||
> **Implementation audit, 2026-08-20:** run models/API/primitives exist; synthetic executors, missing
|
||||
> executor imports and insufficient walker tests prevent the plan from being considered delivered.
|
||||
@@ -14,12 +14,21 @@ Implement deterministic ScenarioRun/ScenarioStepRun orchestration for the valida
|
||||
**Language/Version**: Python 3.13+ (backend), TypeScript DTOs (frontend consumed by 045)
|
||||
**Primary Dependencies**: FastAPI, SQLAlchemy, Pydantic 2; reuse 037 metric_executor/comparison, 038 capture, 036 evidence/HITL, 040 RunnerPool pattern, browser/xlsx infra
|
||||
**Storage**: new tables `scenario_runs`, `scenario_step_runs`
|
||||
**Testing**: pytest (unit/contract/integration for executor dispatch, resume, cancel, timeout)
|
||||
**Testing**: pytest (unit/contract/integration for executor dispatch, provider lifecycle, ownership,
|
||||
capacity, health, deployment, resume, cancel, timeout and reconciliation)
|
||||
**Frontend Architecture**: DTOs only (run monitor is 045)
|
||||
**Performance Goals**: step dispatch < 100ms overhead; deterministic topological order; bounded drain on cancel
|
||||
**Constraints**: deterministic (no LLM per step); human not an executor; PROD gated; immutable revision snapshot
|
||||
**Performance Goals**: step dispatch overhead <100ms p95 on the canonical 100-step fixture excluding
|
||||
provider latency; SSE event replay has zero gaps/duplicates by sequence; cancellation finalizes within
|
||||
the persisted drain deadline plus one 5-second scheduler interval
|
||||
**Constraints**: deterministic (no LLM per step); human not an executor; PROD gated; immutable revision
|
||||
snapshot; no provider I/O without shared capacity lease; unknown external effects require reconciliation
|
||||
**Scale**: up to 100-step graphs, concurrent scenario runs per environment bounded
|
||||
|
||||
**Release thresholds**: 100% canonical fixture dispatch order; 100/100 cancellation trials within the
|
||||
bounded drain tolerance; 100% provider I/O preceded by CapacityLease and operation receipt; 100% accepted
|
||||
evidence refs with ownership receipt and valid SHA-256; all enabled providers ready; PostgreSQL migration
|
||||
check/upgrade pass; no unresolved P0/P1 execution-critical dependency row.
|
||||
|
||||
## Constitution Check
|
||||
|
||||
| Principle | Result |
|
||||
@@ -44,6 +53,9 @@ specs/044-dashboard-scenario-execution/
|
||||
backend/src/models/scenario_run.py # ScenarioRun, ScenarioStepRun
|
||||
backend/src/services/dashboard_testing/execution/
|
||||
│ ├── runner.py runner_plan.py dispatch.py executor_registry.py
|
||||
│ ├── provider_protocol.py provider_evidence.py provider_operations.py provider_health.py capacity.py
|
||||
│ ├── provider_contracts.py provider_bootstrap.py
|
||||
│ ├── providers/ (browser.py superset.py xlsx.py assertion.py transform.py screenshot.py report.py artifact.py agent_evaluation.py)
|
||||
│ ├── executors/ (browser.py superset.py xlsx.py assertion.py screenshot.py report.py artifact.py)
|
||||
│ ├── human.py resume.py cancel.py retry.py
|
||||
└── api/routes/dashboard_testing/scenario_runs.py # POST/start/cancel/resume/human/GET/events
|
||||
@@ -59,10 +71,14 @@ backend/src/services/dashboard_testing/execution/
|
||||
6. Cancel/retry/timeout + lifecycle.
|
||||
7. Immutable snapshot + provenance; REST routes; frontend DTOs.
|
||||
8. PROD gate + RBAC + regression gates.
|
||||
9. Provider protocol, ownership receipts, lifecycle/cancellation/reconciliation, health/deployment
|
||||
readiness, shared capacity and bounded AgentEvaluation/DecisionPolicy integration.
|
||||
|
||||
## API and Schema
|
||||
|
||||
- `contracts/openapi.yaml` — POST /scenario-runs, cancel, resume, human/decision, GET detail, GET events (SSE).
|
||||
- `contracts/modules.md` — ProviderProtocol, ProviderCatalog, ProviderOperations, ProviderHealth and
|
||||
CapacityManager contracts.
|
||||
|
||||
## Traceability
|
||||
|
||||
@@ -77,4 +93,6 @@ traceability.md maps Story → model → operationId → contract → task → t
|
||||
|
||||
## Complexity Tracking
|
||||
|
||||
No exception planned. Runner/dispatch are C5 but decomposed (runner, dispatch, executors, human, cancel). Do not collapse into one oversized orchestrator.
|
||||
No exception planned. Runner/dispatch/provider lifecycle are C5 but decomposed (runner, dispatch,
|
||||
provider protocol, operations, health, capacity, per-provider adapters). Do not collapse into one
|
||||
oversized orchestrator.
|
||||
|
||||
@@ -1,50 +1,79 @@
|
||||
# Quickstart: Scenario Execution Engine (044)
|
||||
|
||||
> **Factual audit 2026-08-20:** the targeted service profile independently reverified 35 passes.
|
||||
> Browser/Superset/Screenshot are fail-safe: they cannot synthesize PASS without explicit typed
|
||||
> adapter success, and evidence requires a real valid digest/ref. This is not proof of real live
|
||||
> Playwright/Superset/Screenshot composition.
|
||||
> The Superset binding slice persists an immutable identity snapshot and, under an injected exact
|
||||
> resolver, invokes the existing 037 query envelope and stores its exact raw-byte digest/ref.
|
||||
> **Verification status (2026-08-24):** the available SQLite/unit profile passes, but this is not
|
||||
> production evidence. Browser and Screenshot providers remain unavailable by default. Production GO
|
||||
> additionally requires provider contract tests, real PostgreSQL migration checks and live deployment proof.
|
||||
|
||||
## Prereqs
|
||||
- 042 registry + DB migrated (scenario_runs, scenario_step_runs)
|
||||
- Backend venv
|
||||
## Prerequisites
|
||||
|
||||
## Commands
|
||||
- PostgreSQL reachable through a real `DATABASE_URL`.
|
||||
- `backend/.venv` activated.
|
||||
- `SERVICE_JWT` set when using Docker Compose.
|
||||
- 042 registry and 044 migrations applied.
|
||||
- Enabled live bindings configured only through server-owned startup settings.
|
||||
|
||||
## Available local verification
|
||||
|
||||
```bash
|
||||
cd backend && source .venv/bin/activate
|
||||
alembic upgrade head
|
||||
python -m pytest -v tests/services/dashboard_testing/execution/
|
||||
python -m pytest -v tests/api/test_scenario_runs.py
|
||||
python -m ruff check src/services/dashboard_testing/execution/
|
||||
cd backend
|
||||
source .venv/bin/activate
|
||||
python -m pytest -q \
|
||||
tests/services/dashboard_testing/registry/test_scenario_*.py \
|
||||
tests/api/test_scenario_runs_api.py \
|
||||
tests/api/test_scenario_automation_api.py \
|
||||
tests/api/test_scenario_analytics_api.py
|
||||
python -m pytest -q \
|
||||
tests/services/dashboard_testing/registry/test_scenario_executors.py \
|
||||
tests/services/dashboard_testing/registry/test_live_execution_binding.py \
|
||||
tests/services/dashboard_testing/registry/test_scenario_cancel_timeout.py \
|
||||
tests/services/dashboard_testing/registry/test_scenario_crash_recovery.py \
|
||||
tests/services/dashboard_testing/registry/test_scenario_worker.py \
|
||||
tests/services/dashboard_testing/registry/test_scenario_queued_dispatch.py
|
||||
python -m ruff check src/services/dashboard_testing/execution src/api/routes/dashboard_testing/scenario_runs.py
|
||||
python -m compileall -q src/services/dashboard_testing/execution src/api/routes/dashboard_testing/scenario_runs.py
|
||||
cd ..
|
||||
python3 specs/044-dashboard-scenario-execution/prototype/validate_static.py
|
||||
```
|
||||
|
||||
## Exit Gates
|
||||
- [ ] Fixture scenario runs end-to-end, dispatching every step to correct executor
|
||||
- [x] Persisted worker lease/idempotency checks reject unsafe reclaim and live-lease conflicts
|
||||
- [x] Human and infrastructure resume advance only the missing DAG frontier; completed steps are not rerun
|
||||
- [x] Browser/Superset/Screenshot do not manufacture PASS; invalid/missing evidence digest/ref is inconclusive
|
||||
- [x] Superset live binding persists ref/snapshot; exact resolver/model match invokes 037 and stores exact digest/ref
|
||||
- [ ] Cancel terminates `cancelled` with bounded drain
|
||||
- [x] Server-driven crash recovery by `scenario_run_id` uses only the persisted RunnerPlan and
|
||||
expired worker claims: only idempotent/retry-safe frontier work gets a new attempt; unsafe
|
||||
effects require reconciliation, and browser work without a pinned browser-safe checkpoint is
|
||||
typed non-pass without replay. Retired evidence remains historical, and repeated rejected
|
||||
terminal contexts reuse the idempotent queue signal.
|
||||
- [ ] Results reference immutable revision + provenance
|
||||
- [ ] human not invoked as executor; PROD gated
|
||||
- [ ] T022 full scope, scoped ruff, semantic rebuild, and prototype states covered
|
||||
Expected current local evidence:
|
||||
|
||||
## Verification Boundary
|
||||
- full available 044 profile: `246 passed`;
|
||||
- provider/lifecycle edge profile: `59 passed`;
|
||||
- prototype static validation: `passed`;
|
||||
- scoped Ruff and compile: `passed`.
|
||||
|
||||
- The exact API test file has a 29-pass result outside this sandbox.
|
||||
- In this sandbox FastAPI TestClient is blocked by AnyIO self-pipe `EPERM`; this is an execution
|
||||
restriction, not an application failure.
|
||||
- Startup composition supplies config-owned 037 resolver/client/model/storage for enabled exact
|
||||
bindings. Browser and Screenshot providers remain absent and fail closed; prototype validation
|
||||
remains unresolved.
|
||||
- Before deployment, PostgreSQL must run and verify `alembic upgrade head` for the additive
|
||||
`c1d2e3f4a5b6` live-binding columns/index; the isolated migration test does not replace that
|
||||
deployment prerequisite.
|
||||
## Production verification
|
||||
|
||||
```bash
|
||||
cd backend
|
||||
source .venv/bin/activate
|
||||
alembic check
|
||||
alembic upgrade head
|
||||
python -m pytest -q --run-integration tests/integration/
|
||||
```
|
||||
|
||||
Run the provider contract profile only after T028-T042 exists:
|
||||
|
||||
```bash
|
||||
python -m pytest -q tests/services/dashboard_testing/registry/test_provider_contract.py
|
||||
python -m pytest -q tests/services/dashboard_testing/registry/test_provider_*.py
|
||||
```
|
||||
|
||||
## Measurable exit gates
|
||||
|
||||
1. SC-001..011 each has a passing named test or deployment evidence record.
|
||||
2. 100/100 cancellation trials terminate by `cancel_drain_deadline_at + 5 seconds`.
|
||||
3. 100% of provider I/O attempts have a valid CapacityLease and operation receipt.
|
||||
4. 100% of accepted evidence refs have an ownership receipt and verified SHA-256.
|
||||
5. 100% of unknown external effects are reconciled or terminalized non-pass before retry.
|
||||
6. Every enabled provider has passing liveness, readiness and dependency-health checks.
|
||||
7. Browser and Screenshot perform one real authorized deployment run with durable evidence.
|
||||
8. No unresolved P0/P1 traceability row remains in 044 or execution-critical dependencies 036, 037, 038,
|
||||
041, 042, 046 and 047.
|
||||
|
||||
## Current boundary
|
||||
|
||||
The local profile proves fail-closed adapters, exact Superset binding behavior, lifecycle closure and
|
||||
prototype state coverage. It does not prove Browser/Screenshot live composition, shared provider capacity,
|
||||
AgentEvaluation runtime, real PostgreSQL migration validity, scheduler deployment behavior or 047 case
|
||||
ingestion.
|
||||
|
||||
@@ -68,3 +68,31 @@ See data-model.md: `ScenarioRun`, `ScenarioStepRun`, `RunnerPlan`, `ScenarioExec
|
||||
| V. RBAC Enforcement | PASS — scenario:run, scenario:run:prod |
|
||||
| VII. Test-Driven C3+ | PASS — executor dispatch/resume/cancel tests first |
|
||||
| VIII. Attention-Optimized | PASS — `[SEMANTICS scenario,execution,...]` |
|
||||
|
||||
## R6. Provider protocol
|
||||
|
||||
**Decision**: Treat every provider as a resource-owning operation boundary with a common execution
|
||||
context, immutable operation/evidence receipts, explicit effect state, cancellation, reconciliation,
|
||||
health and deployment capabilities.
|
||||
|
||||
**Rationale**: A typed callback validates the result envelope but cannot prove ownership, cleanup,
|
||||
unknown external effects, capacity admission or deployability.
|
||||
|
||||
**Alternatives**: provider-specific ad hoc callbacks (rejected: inconsistent recovery and evidence);
|
||||
runner-only lifecycle (rejected: provider resources and external effects remain invisible).
|
||||
|
||||
**Impact**: T028-T042; no live provider is production-ready until the common contract suite passes.
|
||||
|
||||
## R7. Capacity and AgentEvaluation
|
||||
|
||||
**Decision**: All providers claim the shared environment-scoped `ExecutionCapacityManager` before I/O.
|
||||
AgentEvaluation is a bounded provider producing immutable evidence; deterministic `DecisionPolicy` is
|
||||
the only mapper to `StepOutcome`.
|
||||
|
||||
**Rationale**: Independent feature quotas permit starvation; direct model verdicts violate deterministic
|
||||
aggregation and human checkpoint authority.
|
||||
|
||||
**Alternatives**: per-provider semaphore (rejected: no global fairness or durable lease); direct verdict
|
||||
aggregation (rejected: nondeterministic and bypasses policy).
|
||||
|
||||
**Impact**: CapacityLease, provider operation receipts, T032 and T039 are blocking production work.
|
||||
|
||||
@@ -121,6 +121,34 @@
|
||||
- **SCEX-FR-015**: `AgentEvaluation` MUST be a separate immutable runtime record and DecisionPolicy MUST deterministically map it plus deterministic evidence to StepOutcome. A bare model verdict never directly sets ScenarioResult.
|
||||
- **SCEX-FR-016**: Browser actions and mutation safety MUST use the same versioned 038 ActionRegistry/mutation contract. Mutating browser steps in PROD are prohibited; test-data mutation needs fixture scope, record keys, side-effect/retry and cleanup policy independent of PROD approval.
|
||||
|
||||
- **SCEX-FR-017**: Every provider MUST implement common `ProviderExecutionContext` and
|
||||
`ProviderExecutionResult` contracts including run/step/attempt, descriptor and binding fingerprints,
|
||||
principal fingerprint, deadline, idempotency key, capacity lease, trace identity, provider/version,
|
||||
operation receipt, effect state and retry disposition.
|
||||
- **SCEX-FR-018**: Provider output ownership MUST be proven by an immutable receipt binding every output
|
||||
or evidence ref to run, step, attempt, provider/version, descriptor, binding and principal fingerprints,
|
||||
content type, byte length and verified SHA-256. A path, caller digest, session id, raw bytes or model
|
||||
metadata alone is never evidence.
|
||||
- **SCEX-FR-019**: Providers MUST own isolated resources and enforce server-owned concurrency, size,
|
||||
timeout/deadline, cleanup and durable-write atomicity limits.
|
||||
- **SCEX-FR-020**: Cancellation MUST be operation-aware: durable request plus provider cancel by
|
||||
`operation_id`, acknowledged as `stopped|completed|unknown`. `unknown` forbids retry and PASS until
|
||||
reconciliation or terminal non-pass closure.
|
||||
- **SCEX-FR-021**: Retry MUST create a new attempt and capacity lease while preserving descriptor side-
|
||||
effect identity. Retry is allowed only when declared safe/idempotent or after reconciliation; late
|
||||
responses remain historical.
|
||||
- **SCEX-FR-022**: Providers MUST expose separate liveness, readiness and dependency-health checks,
|
||||
registration fingerprint, version, capabilities and redacted reason codes. Telemetry contains no
|
||||
credentials, cookies, raw SQL or capture bytes.
|
||||
- **SCEX-FR-023**: Deployment MUST register providers only from trusted startup configuration and verify
|
||||
versions, descriptors, bindings, storage, dependencies, limits and capability health before admission.
|
||||
- **SCEX-FR-024**: `ExecutionCapacityManager` MUST be the shared environment-scoped admission boundary
|
||||
for ScenarioRun, AgentRun, VerificationRun and LoadRun. No provider performs I/O without an atomic
|
||||
lease; exhaustion queues or blocks, and lease lifecycle is durable and idempotent.
|
||||
- **SCEX-FR-025**: `AgentEvaluationProvider` MUST execute only immutable 038 AgentEvaluationSpec with
|
||||
pinned provider/model/prompt versions, bounded budget and evidence allowlist. It emits immutable
|
||||
AgentEvaluation; DecisionPolicy alone maps it plus deterministic evidence to StepOutcome.
|
||||
|
||||
### Live Execution Composition Root (SCEX-LIVE-COMPOSITION)
|
||||
|
||||
`LiveExecutionBinding` persisted on `ScenarioRun` is identity-only: a binding reference plus immutable
|
||||
@@ -166,12 +194,34 @@ startup registers such a provider; no path or raw capture metadata is treated as
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- **SC-001**: A fixture scenario runs end-to-end, dispatching every step to the correct executor, in deterministic order.
|
||||
- **SC-002**: Human checkpoint suspends the run and resumes from the exact next step without rerunning completed steps.
|
||||
- **SC-003**: 100% of cancel requests terminate with `cancelled` and bounded drain.
|
||||
- **SC-004**: Runs are recoverable by `scenario_run_id` after disconnect.
|
||||
- **SC-005**: Results reference an immutable revision snapshot + full provenance; editing the scenario after a run never changes that run's results.
|
||||
- **SC-006**: `human` is not invoked as an executor; PROD runs are gated.
|
||||
- **SC-001**: The canonical fixture graph (`fixtures/graph.json`) executes with 100% of declared
|
||||
executable steps dispatched in pinned topological order; the test MUST assert the exact dispatch
|
||||
sequence and zero duplicate dispatches for completed logical steps.
|
||||
- **SC-002**: For every human-checkpoint fixture, the run reaches `waiting_human` within one dispatcher
|
||||
cycle, persists exactly one pending checkpoint, and after a valid CAS decision resumes the next frontier
|
||||
without dispatching any previously passed logical step.
|
||||
- **SC-003**: For 100/100 repeated cancellation attempts across queued, running and waiting states, the
|
||||
run reaches `cancelled` no later than `cancel_drain_deadline_at + 1 scheduler interval`; terminal rows
|
||||
contain zero running steps, active leases or active evidence projections.
|
||||
- **SC-004**: A run recovered by `scenario_run_id` after worker/process interruption produces the same
|
||||
completed-step set and pinned RunnerPlan hash as before interruption; unsafe unknown effects produce no
|
||||
retry and a terminal `blocked` or `inconclusive` result.
|
||||
- **SC-005**: After a revision edit, 100% of fields in the prior run's revision/content/program,
|
||||
parameter, target, principal and evidence provenance remain byte-identical to their launch snapshot.
|
||||
- **SC-006**: Across all registry actions, `human` has zero executor resolution/invocation records and
|
||||
every configured PROD run has a pending approval gate before the first provider I/O event.
|
||||
- **SC-007**: Every provider passes a common contract suite for ownership, limits, lifecycle,
|
||||
cancellation, retry, reconciliation, health and deployment registration.
|
||||
- **SC-008**: No external provider performs I/O without a shared capacity lease; leases are released or
|
||||
reconciled after success, failure, cancellation, timeout and crash recovery.
|
||||
- **SC-009**: AgentEvaluation produces immutable evidence and DecisionPolicy produces the only accepted
|
||||
StepOutcome mapping for evaluation-backed steps.
|
||||
- **SC-010**: Every enabled live provider has a startup readiness record with provider/version,
|
||||
capability fingerprint, dependency checks and redacted diagnostics; an unready provider admits zero
|
||||
new operations until readiness is restored.
|
||||
- **SC-011**: The 044 backend profile, provider contract profile, PostgreSQL migration check and prototype
|
||||
validation each publish reproducible command output; a production GO requires 100% pass in all four
|
||||
profiles and no unresolved P0/P1 traceability row.
|
||||
|
||||
## Clarifications
|
||||
|
||||
@@ -181,6 +231,39 @@ startup registers such a provider; no path or raw capture metadata is treated as
|
||||
- Q: Is human an executor? → A: No. It is a runner-lifecycle suspend/resume control; the executor registry covers browser/superset_api/xlsx/assertion/screenshot/report/artifact.
|
||||
- Q: How does this differ from VerificationRun/AgentRun? → A: ScenarioRun executes the user-created DashboardTestScenario DAG; AgentRun is the creation run; VerificationRun is release-pipeline category verification. Three distinct run concepts.
|
||||
|
||||
### Session 2026-08-24 — measurable requirement clarifications
|
||||
|
||||
- **Requirement language**: `MUST` is a release-blocking requirement; `SHOULD` is an explicitly tracked
|
||||
non-blocking recommendation; `MAY` is optional. Every MUST is verified by a named test, migration
|
||||
check, health check or deployment evidence record in `traceability.md`.
|
||||
- **Bounded drain**: the provider cancellation drain is the persisted interval
|
||||
`cancel_drain_deadline_at - cancel_requested_at`; finalization tolerance is one scheduler interval.
|
||||
The default scheduler interval is 5 seconds, so the default acceptance tolerance is 5 seconds.
|
||||
- **Timeout**: provider timeout is measured from `started_at` to the pinned descriptor deadline. A late
|
||||
response received after the deadline is historical only and cannot change the winning StepOutcome.
|
||||
- **Ownership**: an evidence ref is usable only after an immutable receipt exists with the complete
|
||||
owner tuple (`run_id`, logical step, attempt, operation, provider/version, descriptor, binding and
|
||||
principal fingerprints), content metadata and verified SHA-256.
|
||||
- **Capacity**: no adapter callback, network request, browser action, capture, model call or artifact
|
||||
write may begin before a durable CapacityLease is acquired. Lease expiry is not permission to retry an
|
||||
unknown external effect.
|
||||
- **Production readiness**: `NO-GO` remains the default. `GO` requires all SC-001..011 evidence,
|
||||
enabled-provider readiness, real PostgreSQL migration verification and zero open P0/P1 rows in the
|
||||
044 and cross-spec traceability matrices.
|
||||
- **Automated human prohibition**: `manual_run_only=true` is a pre-create eligibility failure for every
|
||||
trusted non-manual source. Scheduled, deploy, release, ETL, API and background-recovery paths MUST NOT
|
||||
create a ScenarioRun, ActionApprovalGate, notification, queue item or dispatcher claim. Only the
|
||||
authenticated manual route may create the run and later reach `waiting_human`.
|
||||
|
||||
### Full production tool boundary
|
||||
|
||||
There is no reduced preview release target. The package describes one complete production tool. Agent-authored
|
||||
typed scenarios, immutable revision activation, manual and automated triggers, Browser/Screenshot providers,
|
||||
controlled non-PROD fixture mutation, AgentEvaluation/DecisionPolicy, monitoring, investigation cases,
|
||||
analytics and remediation are mandatory capabilities with independent release gates. A missing or unproven
|
||||
capability blocks production GO; it is not silently downgraded to preview. Human-containing revisions remain
|
||||
manual-run-only and PROD mutation remains prohibited by policy.
|
||||
|
||||
## Implementation Status & MVP Debt (factual audit 2026-08-20)
|
||||
|
||||
ScenarioRun/ScenarioStepRun models, lifecycle/API primitives, a runner plan, and a fail-safe executor
|
||||
@@ -235,8 +318,17 @@ boundary now exist. They do not yet constitute the specified production executio
|
||||
- `[~]` The targeted service profile independently reverified 35 passes. The exact API file has a
|
||||
29-pass result outside this sandbox; inside it FastAPI TestClient is blocked by AnyIO self-pipe
|
||||
`EPERM`, a sandbox restriction rather than an application failure.
|
||||
- `[ ]` Live production composition adapters, full T022 scope, recurring-episode classification from
|
||||
terminal signals, and an Axiom
|
||||
semantic-index rebuild remain unresolved; SCEX-FR-001..016 are not globally closed.
|
||||
- `[ ]` Live production composition adapters, provider contract suite, AgentEvaluation/DecisionPolicy,
|
||||
shared ExecutionCapacityManager, full T022 scope, recurring-episode classification from
|
||||
terminal signals, and an Axiom
|
||||
semantic-index rebuild remain unresolved; SCEX-FR-001..025 are not globally closed.
|
||||
|
||||
### BrowserProvider readiness reassessment
|
||||
|
||||
The BrowserProvider contract is now specified at **90/100 contract completeness**. This score covers the
|
||||
per-action schema/risk catalog, resource limits, isolated context lifecycle, safe checkpoints, ownership
|
||||
receipts, cancellation/reconciliation, mutation policy, readiness checks and mandatory canaries. It does
|
||||
not claim runtime implementation: T028-T034, T040-T042, T042b and a real PREPROD canary remain required before
|
||||
the BrowserProvider can be called production-ready.
|
||||
|
||||
#endregion ScenarioExecution.Spec
|
||||
|
||||
@@ -87,10 +87,72 @@
|
||||
## Phase 8 — Polish
|
||||
|
||||
- [x] T021 [P] Belief-runtime instrumentation for C5 runner/dispatch
|
||||
- [ ] T022 Run quickstart-equivalent full scenario backend scope, scoped ruff, ATTN/orphan static audit and semantic rebuild.
|
||||
- [ ] T022 Run quickstart-equivalent full scenario backend scope, scoped Ruff, ATTN/orphan static audit,
|
||||
real PostgreSQL `alembic check`/`upgrade head`, and semantic rebuild. Completion requires command
|
||||
output attached to the release record and zero unresolved P0/P1 rows.
|
||||
- [x] T023 **Prototype validation**: every declared @UX_STATE is reachable via `prototype/index.html`.
|
||||
Proof: `python specs/044-dashboard-scenario-execution/prototype/validate_static.py`.
|
||||
|
||||
## Phase 9 — Provider Production Contract
|
||||
|
||||
- [ ] T028 [P] [US2] Implement common `ProviderExecutionContext`, operation receipts and typed
|
||||
`ProviderExecutionResult` schemas in `backend/src/services/dashboard_testing/execution/provider_protocol.py`.
|
||||
- [ ] T029 [P] [US2] Add provider ownership receipts and atomic evidence commit contract in
|
||||
`backend/src/services/dashboard_testing/execution/provider_evidence.py`.
|
||||
- [ ] T030 [P] [US2] Add provider operation lifecycle, cancellation and reconciliation contracts in
|
||||
`backend/src/services/dashboard_testing/execution/provider_operations.py`.
|
||||
- [ ] T031 [P] [US2] Add provider liveness/readiness/dependency health and redacted telemetry contract in
|
||||
`backend/src/services/dashboard_testing/execution/provider_health.py`.
|
||||
- [ ] T032 [US2] Integrate atomic shared `ExecutionCapacityManager` admission, heartbeat, expiry, release
|
||||
and reconciliation with dispatcher claims in `backend/src/services/dashboard_testing/execution/capacity.py`.
|
||||
@INVARIANT: no provider I/O without a capacity lease; retries claim a new lease.
|
||||
- [ ] T033 [P] [US2] Add per-provider input/output schemas, limits and error taxonomy in
|
||||
`backend/src/services/dashboard_testing/execution/provider_contracts.py` for browser, Superset API,
|
||||
SQL evidence, XLSX, assertion, transform, screenshot, report and artifact.
|
||||
- [ ] T034 [US2] Implement BrowserProvider resource ownership, safe-checkpoint replay, cancel and
|
||||
reconciliation adapter in `backend/src/services/dashboard_testing/execution/providers/browser.py`.
|
||||
Required action catalog: open_dashboard, navigate_tab, apply_native_filter, inspect_filter_state,
|
||||
apply_table_filter, extract_table, scroll_to, inspect_columns, click, select_rows, edit_row,
|
||||
bulk_edit, download, refresh, wait_for_state. Required limits: 120s context/auth, 30s action, 3
|
||||
pages, 25 MiB downloads, 10 MiB screenshots; mutation actions require fixture lease and cleanup.
|
||||
- [ ] T035 [US2] Implement ScreenshotProvider atomic durable evidence adapter with cleanup and ownership
|
||||
receipts in `backend/src/services/dashboard_testing/execution/providers/screenshot.py`.
|
||||
- [ ] T036 [US2] Harden SupersetProvider and SqlEvidenceProvider request limits, error taxonomy,
|
||||
external request reconciliation and principal/RLS evidence in
|
||||
`backend/src/services/dashboard_testing/execution/providers/superset.py`.
|
||||
- [ ] T037 [P] [US2] Implement server-owned XLSX artifact intake and resource limits in
|
||||
`backend/src/services/dashboard_testing/execution/providers/xlsx.py`.
|
||||
- [ ] T038 [P] [US2] Implement deterministic ReportProvider and ArtifactProvider durable registration
|
||||
with manifest/digest ownership in `backend/src/services/dashboard_testing/execution/providers/artifacts.py`.
|
||||
- [ ] T039 [US2] Implement AgentEvaluationProvider and deterministic DecisionPolicy integration in
|
||||
`backend/src/services/dashboard_testing/execution/providers/agent_evaluation.py`.
|
||||
@INVARIANT: model verdict never directly sets ScenarioResult or consumes HumanCheckpoint.
|
||||
- [ ] T040 [US2] Add startup deployment registration and readiness preflight for all provider capabilities
|
||||
in `backend/src/services/dashboard_testing/execution/provider_bootstrap.py`.
|
||||
- [ ] T041 [US2] Add common provider contract tests for unavailable/dependency failure/capacity
|
||||
exhaustion/timeout/cancel/duplicate/late response/ownership mismatch/malformed result/cleanup and
|
||||
reconciliation in `backend/tests/services/dashboard_testing/registry/test_provider_contract.py`.
|
||||
- [ ] T042 [US2] Add provider-specific contract tests in
|
||||
`backend/tests/services/dashboard_testing/registry/test_provider_*.py` and real deployment health
|
||||
checks for Superset, Browser and Screenshot bindings.
|
||||
- [ ] T042b [US2] Run BrowserProvider PREPROD canaries and retain evidence in
|
||||
`specs/044-dashboard-scenario-execution/evidence/browser-provider/`: read-only action canary,
|
||||
forced timeout/cleanup canary, safe-checkpoint reconstruction trace and readiness/health payload.
|
||||
`GO` requires all BrowserProvider acceptance vectors to pass and one owned evidence receipt.
|
||||
Contract target before runtime enablement: **90/100**; runtime score remains below target until
|
||||
T034, T040-T042 and this canary task are complete.
|
||||
|
||||
## Requirement evidence rules
|
||||
|
||||
- A task claiming provider PASS MUST assert a valid CapacityLease, operation receipt and ownership receipt;
|
||||
mocked callback success alone is insufficient.
|
||||
- A cancellation/retry task MUST include `unknown` external effect, late response and duplicate request
|
||||
cases, with the expected durable state asserted after the scheduler interval.
|
||||
- A deployment task MUST record provider/version, capability fingerprint, readiness result and redacted
|
||||
dependency diagnostics for every enabled binding.
|
||||
- A cross-spec task is complete only when its named dependency row in `traceability.md` is `[x]`; local
|
||||
unit tests cannot close an unresolved 036/037/041/042/046/047 runtime boundary.
|
||||
|
||||
## Audit Follow-ups (2026-08-20)
|
||||
|
||||
- [~] T024 Replace synthetic default executor outcomes with typed adapter boundaries for required
|
||||
|
||||
@@ -4,16 +4,49 @@
|
||||
|-------|-------------|-------|------------------|----------|------|------|---------------------|
|
||||
| US1 Start | SCEX-FR-001/008 | ScenarioRun | scenarioRun.start | Execution.Start, Execution.Runner.QueuedDispatch | T006-T008, T024 | test_runner, test_scenario_queued_dispatch, test_scenario_scheduler_callbacks, test_scenario_runs_api, test_scenario_automation_api, test_live_execution_binding | `[~]` HTTP/automation start and replay persist queued/pending rows without request-time dispatch; only the scheduler composition's durable queued->running CAS walks its winner. Fixed scheduler callback registration, database-edge containment, and repeat-tick terminal side-effect idempotency are unit-proven. Automated human plans are rejected before they reach CAS/walker; a manual human graph reaches HumanCheckpoint only after that dispatcher claim. Approval-to-real live dispatch remains unproven. |
|
||||
| US2 Dispatch | SCEX-FR-002/006/009 | ScenarioStepRun, LiveExecutionBinding | scenarioRun.step | Execution.Dispatch, Execution.LiveCompositionRoot | T009-T011, T024 | test_dispatch, test_scenario_executors, test_live_execution_binding | `[~]` Browser/Superset/Screenshot use fail-safe typed adapter boundaries: no explicit adapter success means no PASS; invalid evidence digest/ref remains inconclusive. Lifespan bootstraps `settings.scenario_live_execution_bindings` through the existing `SupersetClient`, exact model and durable storage, so configured Superset dispatch invokes 037 and stores the exact raw-byte digest/ref; mismatched/unavailable providers make no I/O call. Browser safe-checkpoint and Screenshot durable-evidence registration are supported but no provider is deployed, so enabled bindings return stable configured-unavailable codes. |
|
||||
| BrowserProvider | SCEX-FR-016..023 | BrowserProviderActionContract, BrowserOperationReceipt, BrowserEvidenceReceipt | scenarioRun.step | ScenarioExecution.BrowserProvider | T028-T034, T040-T042, T042b | test_provider_browser_*, Browser PREPROD canaries | `[~]` Contract completeness is **90/100**: all actions, risk classes, limits, lifecycle, checkpoint/recovery, ownership, cancellation/reconciliation, readiness and canary gates are specified. Runtime provider, shared capacity integration, deployment registration and real canary evidence remain open. |
|
||||
| US3 Human | SCEX-FR-004/010 | ScenarioRun(waiting_human) | scenarioRun.humanDecision | Execution.SuspendForHuman, Execution.Resume | T012-T014, T025 | test_human_resume, test_scenario_runs_api | `[~]` persisted HumanCheckpoint and infrastructure-resume continuations advance only the missing DAG frontier; completed steps are not re-run. Full live-composition closure remains pending. |
|
||||
| Manual-only boundary | SCEX-FR-004a | ScenarioRun, HumanCheckpoint | — | Execution.Runner.Start, RunnerPlan.Derive | T027 | test_scenario_manual_run_only, test_scenario_automation_api | `[x]` Trusted scheduled/deploy/release/ETL/API origins reject persisted human revisions before idempotency or any run/gate/notification/queue side effect. Manual origin remains eligible; HumanCheckpoint is not an approval gate. |
|
||||
| Automated human prohibition | SCEX-FR-004a/013, SCAUTO-FR-001/007 | RunnerPlan.manual_run_only | — | Execution.Runner.TriggerSource, Automation.Trigger | T027, 046 T016/T019 | test_scenario_manual_run_only, test_scenario_automation_api, test_scenario_automation_trigger | `[x]` Every trusted non-manual origin, including scheduled, deploy, release, ETL, API and background recovery, is rejected before idempotency/run/gate/notification/queue/dispatch side effects. Only authenticated manual origin may create a human-containing run. |
|
||||
| US4 Lifecycle | SCEX-FR-005/006 | ScenarioRun(status, cancel deadline), ScenarioArtifact(active projection) | scenarioRun.cancel, scenarioRun.retry | Execution.Lifecycle.Cancel, Execution.Lifecycle.Timeout, Execution.Lifecycle.Retry, Execution.Runner.ContinueAfterRetry | T014g, T015-T016, T025 | test_scenario_lifecycle, test_scenario_cancel_timeout, test_scenario_retry_closure | `[~]` Strict eligible retry invalidates the persisted target/descendant closure, archives old step attempts, and retains artifact rows only as inactive historical provenance. Cancellation pins a durable bounded drain deadline; deadline finalization expires leases and retires active projections without deleting audit evidence. A timeout during adapter I/O wins over a late PASS and materializes every otherwise-lazy pinned-plan descendant as blocked, so continuation cannot dispatch it; it emits one idempotent inconclusive signal. Full live-I/O composition remains fail-closed/unproven. |
|
||||
| US5 Snapshot/API | SCEX-FR-003/007 | ScenarioExecutionResult | scenarioRun.detail, scenarioRun.events | Execution.RunnerPlan, Execution.Runner.CrashRecovery | T017-T020, T025 | test_result, test_api, test_scenario_runner_walker, test_scenario_crash_recovery | `[~]` API/projection and real-digest artifact integrity checks exist. Server-driven crash recovery uses only the persisted run/RunnerPlan and expired lease: completed work is not rerun, safe claims get a new attempt with active evidence retired to history, unsafe claims require reconciliation, and browser recovery without a pinned safe checkpoint is non-pass without adapter I/O. Rejected terminal contexts reuse their idempotent signal. Production live-executor provenance remains unproven. |
|
||||
| Terminal signals | SCEX-FR-011 | InvestigationQueueItem | — | Execution.Runner.TerminalSignal | T026 | test_scenario_terminal_signals | `[x]` Failed/blocked/inconclusive terminal runs emit one idempotent immutable 047 queue input with run/artifact provenance; passed runs emit none. Producer-only ingestion starts no case, AgentRun, chat, remediation action, or recurrence classification. |
|
||||
| Gate/RBAC | SCEX-FR-008 | ScenarioRun, ActionApprovalGate | scenarioRun.start | Execution.EnvironmentPolicy, Execution.Runner.Start | T019 | test_scenario_runner, test_scenario_runs_api, test_scenario_automation_api, test_scenario_automation_trigger | `[~]` Server ConfigManager classifies every target before persistence: client flags cannot select PROD, unknown targets create no run-side effect, and every trusted source enters the same durable pending_approval gate boundary. HTTP/trigger paths remain persistence-only and dispatcher excludes pending gates. A dedicated real APScheduler scheduled-PROD integration test remains coverage debt; this is not a dispatch bypass. |
|
||||
| Provider protocol | SCEX-FR-017..023 | ProviderExecutionContext, ProviderOperationReceipt, ProviderEvidenceReceipt | — | ScenarioExecution.ProviderProtocol, ProviderOperations, ProviderOperations.Observability | T028-T031, T040-T042 | test_provider_contract, test_provider_health, deployment health checks | `[ ]` New production gate: common context/result, operation receipts, ownership proof, cancellation/reconciliation, health/readiness and startup registration are specified but not implemented. |
|
||||
| Capacity | SCEX-FR-024 | CapacityLease, ExecutionCapacityManager | — | ScenarioExecution.CapacityManager | T032, T041-T042 | test_provider_capacity, scheduler/worker integration | `[ ]` New production gate: shared environment-scoped capacity admission is specified but current 046 checks do not close it. |
|
||||
| Agent evaluation | SCEX-FR-025 | AgentEvaluation, DecisionPolicy | — | ScenarioExecution.ProviderCatalog, data-model AgentEvaluation and DecisionPolicy | T039, T041-T042 | test_provider_agent_evaluation | `[ ]` New production gate: bounded provider, immutable evaluation evidence and deterministic policy mapping are specified but absent. |
|
||||
| Success criteria | SC-001 | RunnerPlan, ScenarioStepRun | — | ScenarioExecution.RunnerPlan.Derive, ScenarioExecution.Dispatch | T004-T011, T041 | canonical fixture order and duplicate-dispatch assertions | `[~]` Existing fixture/lifecycle coverage passes; exact 100% canonical dispatch evidence remains a release gate. |
|
||||
| Success criteria | SC-002 | HumanCheckpoint, ScenarioRun | scenarioRun.humanDecision | ScenarioExecution.HumanCheckpoint, ScenarioExecution.Resume | T012-T014, T025 | human CAS/frontier tests | `[x]` Human checkpoint CAS and missing-frontier resume are verified; live provider closure remains separate. |
|
||||
| Success criteria | SC-003 | ScenarioRun.cancel_drain_deadline_at | scenarioRun.cancel | ScenarioExecution.Cancel | T015-T016, T025, T041 | 100 cancellation trials plus scheduler finalizer | `[~]` Bounded drain is unit-proven; required 100-trial production evidence is open. |
|
||||
| Success criteria | SC-004 | RunnerPlan, worker lease, operation receipt | scenarioRun.detail | ScenarioExecution.Runner.CrashRecovery, ScenarioExecution.ProviderOperations | T014c, T025, T030, T041 | safe/unsafe recovery and reconciliation tests | `[~]` Safe/unsafe persisted recovery is verified; provider operation reconciliation is not implemented. |
|
||||
| Success criteria | SC-005 | immutable run provenance | scenarioRun.result | Execution.RunnerPlan, Execution.Result | T017, T020, T041 | revision-edit immutability and evidence receipt tests | `[~]` Snapshot fields exist; complete receipt-level byte identity evidence is open. |
|
||||
| Success criteria | SC-006 | ActionApprovalGate, ExecutorRegistry | scenarioRun.start | Execution.ActionApprovalGate, ScenarioExecution.ExecutorRegistry | T014d, T019, T041 | PROD no-call and human-not-executor tests | `[x]` Current no-call and registry rejection tests pass. |
|
||||
| Success criteria | SC-007..010 | Provider protocol/health/capacity | — | ScenarioExecution.ProviderProtocol, ProviderOperations.Observability, CapacityManager | T028-T042 | common/provider-specific contract and deployment profiles | `[ ]` New production gate not implemented. |
|
||||
| Success criteria | SC-011 | release evidence record | — | ScenarioExecution.ProviderProtocol | T022, T041-T042 | full profiles, PostgreSQL and semantic audit outputs | `[ ]` Release evidence package is incomplete. |
|
||||
|
||||
N/A: Registry (042), Editor (043), Monitor UX (045), Automation (046), Analytics (047).
|
||||
|
||||
**Verification boundary (2026-08-20):** the targeted service profile independently reverified 35
|
||||
passes. The exact API file has a 29-pass result outside this sandbox; inside it, FastAPI TestClient
|
||||
is blocked by AnyIO self-pipe `EPERM`, a sandbox restriction rather than an application failure.
|
||||
T022, prototype validation, live production composition adapters, and the Axiom index remain open.
|
||||
**Provider production boundary:** Existing typed unavailable outcomes and exact Superset binding tests
|
||||
prove only the fail-closed boundary. Production readiness additionally requires T028-T042 and real
|
||||
startup/dependency health checks for every enabled live provider.
|
||||
|
||||
**BrowserProvider boundary:** The BrowserProvider is contract-ready at 90/100 but implementation-ready
|
||||
only after T034, T040-T042 and T042b. A typed unavailable result remains the correct behavior until the
|
||||
PREPROD read-only/timeout/cleanup/reconstruction canaries produce retained operation and evidence receipts.
|
||||
|
||||
**Cross-spec production boundary:** 044 readiness depends on 036 authority/evidence, 037 query and
|
||||
baseline provenance, 038 executable graph identity, 041 lineage target state, 042 registry revisions,
|
||||
046 automation dispatch and 047 terminal-signal/case ingestion. 039/043/045 are operator continuity
|
||||
dependencies: they do not authorize execution, but incomplete typed API/SSE/revision flows prevent a
|
||||
complete production workflow. Current dependency-weighted aggregate for 036-047 is approximately
|
||||
**67/100** and remains NO-GO.
|
||||
|
||||
**Full production tool boundary:** There is no reduced preview target. Browser/Screenshot, controlled
|
||||
non-PROD mutation, AgentEvaluation/DecisionPolicy, automated schedules/triggers, case investigation,
|
||||
analytics and remediation are mandatory capability gates. Missing or unproven capability blocks GO; only
|
||||
human-containing automated revisions remain prohibited and PROD mutation remains policy-forbidden.
|
||||
|
||||
**Verification boundary (2026-08-24):** the available local profile is 246 backend tests, 59 provider/
|
||||
lifecycle edge tests, scoped Ruff/compile and prototype validation. Real PostgreSQL migration checks,
|
||||
provider contract T028-T042, live Browser/Screenshot composition, scheduler deployment and Axiom index
|
||||
rebuild remain open.
|
||||
|
||||
@@ -9,7 +9,11 @@
|
||||
- **Context**: Browser; from scenario detail "Run" or from 046 automation.
|
||||
|
||||
## 2. Happy Path
|
||||
Analyst clicks Run, picks environment/params/revision (Run Configuration), confirms (PROD gate if applicable), and the run starts. The live monitor (045) shows progress. A human checkpoint pauses the run for a decision; the analyst confirms and the run resumes.
|
||||
Analyst selects environment, revision and parameters, then submits one idempotent start request. The
|
||||
server returns `201` with `queued` or `pending_approval` within 2 seconds at p95 under the fixture load.
|
||||
For PROD, the first provider I/O event is impossible before approval. The monitor (045) renders every
|
||||
typed event by sequence; reconnecting with `Last-Event-ID` replays each missing event exactly once.
|
||||
A human checkpoint creates one pending CAS checkpoint; a valid decision resumes only the missing frontier.
|
||||
|
||||
## 3. Interaction Surface (044 backend-driven)
|
||||
- **POST /scenario-runs** — start with run configuration.
|
||||
@@ -21,11 +25,23 @@ Analyst clicks Run, picks environment/params/revision (Run Configuration), confi
|
||||
Run configuration UI and live monitor UI are spec 045. 044 supplies the run/step state and endpoints.
|
||||
|
||||
## 4. Error Experience (044 invariants)
|
||||
- PROD without approval → blocked (gate), no dispatch.
|
||||
- Timeout/retry-exhausted → step failed/inconclusive.
|
||||
- Runner crash → recoverable by run id.
|
||||
- PROD without approval -> `pending_approval`; zero provider I/O events and denial/expiry -> `blocked`.
|
||||
- Capacity unavailable -> `queued` or `blocked` with `CAPACITY_UNAVAILABLE`; zero provider I/O.
|
||||
- Timeout -> typed non-pass outcome; late response cannot alter the displayed winning attempt.
|
||||
- Cancellation -> `cancel_requested` with visible deadline, then `cancelled` within the configured drain
|
||||
deadline plus one 5-second scheduler interval.
|
||||
- Unknown external effect -> `reconciliation_required`; retry control is disabled until reconciliation
|
||||
or terminal non-pass closure.
|
||||
- Runner crash -> recoverable by run id; completed steps remain completed and unsafe effects are not replayed.
|
||||
- Missing/mismatched evidence ownership -> non-pass with stable reason code and an auditable receipt gap.
|
||||
|
||||
## 5. Tone & Voice
|
||||
## 5. Verification States
|
||||
- Each provider operation exposes provider/version, operation id, attempt, status, reason code and evidence
|
||||
receipt state. Secret values, cookies, raw SQL and capture bytes are never rendered.
|
||||
- The UI distinguishes `passed`, `failed`, `blocked`, `inconclusive`, `waiting_human`, `cancel_requested`
|
||||
and `cancelled`; it never maps unavailable or pending states to PASS.
|
||||
|
||||
## 6. Tone & Voice
|
||||
- **Style**: Technical. **Terminology**: "Scenario run" distinct from "Release verification" and "Load tests".
|
||||
|
||||
#endregion ScenarioExecution.UxReference
|
||||
|
||||
@@ -7,7 +7,8 @@
|
||||
|
||||
## Schedule & Trigger (FR-001/002)
|
||||
|
||||
- [ ] CHK001 Runs triggerable by manual, PREPROD deploy, release-created, ETL, scheduled, API
|
||||
- [ ] CHK001 Runs triggerable by manual, PREPROD deploy, release-created, ETL, scheduled, API only when
|
||||
persisted `manual_run_only=false`
|
||||
- [ ] CHK002 Reuses 037 trigger framework + APScheduler (no parallel scheduler)
|
||||
- [ ] CHK003 Scheduled/triggered runs pin revision + env at trigger time
|
||||
|
||||
@@ -20,7 +21,8 @@
|
||||
- [ ] CHK005 Concurrency/dedup prevent redundant parallel runs (same env+revision)
|
||||
- [ ] CHK006 Schedule/deployment-window overlap blocked or warning-gated
|
||||
- [ ] CHK007 Retention prunes while preserving provenance/artifacts
|
||||
- [ ] CHK008 PROD scheduled runs gated (036)
|
||||
- [ ] CHK008 Eligible PROD scheduled runs gated (036); human-containing revisions rejected before any
|
||||
run/gate/queue/notification/dispatch side effect
|
||||
|
||||
## Metrics & Boundary (FR-008/009)
|
||||
|
||||
|
||||
@@ -18,10 +18,13 @@ python -m ruff check src/services/dashboard_testing/automation/
|
||||
```
|
||||
|
||||
## Exit Gates
|
||||
- [ ] Each trigger type (deploy/release/ETL/schedule/api) starts a run with pinned revision+env
|
||||
- [ ] Notification events emitted for completed/failed/blocked/human/stale/flaky
|
||||
- [ ] Each trigger type (deploy/release/ETL/schedule/api) starts an eligible run with pinned revision+env;
|
||||
a `manual_run_only=true` revision is rejected before any durable side effect
|
||||
- [ ] Notification events emitted for completed/failed/blocked/stale/flaky; no `human-action-required`
|
||||
event is emitted by automation
|
||||
- [ ] Concurrency/dedup prevents redundant parallel runs; overlap blocked/warned
|
||||
- [ ] Retention prunes preserving provenance/artifacts
|
||||
- [ ] PROD scheduled runs gated
|
||||
- [ ] Eligible PROD automated runs create ActionApprovalGate before dispatch; human-containing revisions
|
||||
are rejected before gate/run/queue/notification creation
|
||||
- [ ] Zero writes to 037 baseline catalog
|
||||
- [ ] ruff clean; prototype states covered
|
||||
|
||||
@@ -91,13 +91,19 @@
|
||||
|
||||
### Functional
|
||||
|
||||
- **SCAUTO-FR-001**: Scenario runs MUST be triggerable by manual, PREPROD deploy, release-created, ETL-completed, scheduled, and API triggers, reusing the 037 trigger framework.
|
||||
- **SCAUTO-FR-001**: Scenario runs MUST be triggerable by manual, PREPROD deploy, release-created,
|
||||
ETL-completed, scheduled, and API triggers, reusing the 037 trigger framework. This applies only to
|
||||
revisions whose persisted RunnerPlan has `manual_run_only=false`.
|
||||
- **SCAUTO-FR-002**: Scheduled runs MUST pin the scenario revision and environment at trigger time. A newly saved candidate MUST NOT be adopted until 042 activation; a pinned rule MUST reject an ineligible revision.
|
||||
- **SCAUTO-FR-003**: The system MUST emit domain notification events: completed, failed, blocked, scenario-stale, repeated-flaky-failure. `human-action-required` is excluded because HumanCheckpoint scenarios are manual-run-only.
|
||||
- **SCAUTO-FR-004**: Concurrency and deduplication policies MUST prevent redundant parallel runs on the same environment+revision.
|
||||
- **SCAUTO-FR-005**: Schedule/deployment-window overlap MUST be blocked or warning-gated per policy.
|
||||
- **SCAUTO-FR-006**: Retention MUST prune old runs per policy while preserving provenance and referenced artifacts.
|
||||
- **SCAUTO-FR-007**: PROD-classified scheduled runs MUST require an approval gate (036).
|
||||
- **SCAUTO-FR-007**: A PROD-classified automated run for an eligible revision (`manual_run_only=false`)
|
||||
MUST create an ActionApprovalGate before dispatch. A revision with `manual_run_only=true` MUST be
|
||||
rejected before idempotency lookup, ScenarioRun creation, ActionApprovalGate creation, notification,
|
||||
queue insertion or dispatcher CAS for every non-manual origin, including scheduled, deploy, release,
|
||||
ETL, API and background recovery. Approval MUST NOT convert a human checkpoint into an automated run.
|
||||
- **SCAUTO-FR-008**: Operational metrics MUST include scheduled run counts, success rate, trigger distribution, and repeated-failure alerts.
|
||||
- **SCAUTO-FR-009**: Scenario automation MUST NOT write into the 037 baseline catalog or release verification pipeline; scenario runs remain distinct.
|
||||
- **SCAUTO-FR-010**: Schedules, trigger rules, and policies MUST be fully manageable (CRUD + enable/disable) via an Automation Management UI.
|
||||
@@ -112,7 +118,9 @@
|
||||
|
||||
- **ScenarioSchedule**: Cron/daily schedule binding a scenario, revision, environment, and policy.
|
||||
- **ScenarioTriggerRule**: Event-based trigger (deploy/release/ETL) mapping to a scenario run.
|
||||
- **NotificationEvent**: Domain event for automation outcomes (completed/failed/blocked/human/stale/flaky).
|
||||
- **NotificationEvent**: Domain event for automation outcomes (completed/failed/blocked/stale/flaky).
|
||||
`human-action-required` MUST NOT be emitted by automation because human-containing revisions are
|
||||
ineligible before run creation.
|
||||
- **AutomationPolicy**: Concurrency cap, deduplication window, overlap rule, retention, PROD gating.
|
||||
|
||||
## Success Criteria
|
||||
@@ -122,13 +130,18 @@
|
||||
- **SC-003**: No redundant parallel runs on the same environment+revision under concurrency/dedup policy.
|
||||
- **SC-004**: Overlap with deployment windows is blocked or warning-gated.
|
||||
- **SC-005**: Zero writes to the 037 baseline catalog by automation.
|
||||
- **SC-006**: PROD scheduled runs are gated.
|
||||
- **SC-006**: 100/100 automated start attempts for a `manual_run_only=true` revision across scheduled,
|
||||
deploy, release, ETL, API and background origins produce no ScenarioRun, gate, notification, queue or
|
||||
dispatcher side effect; 100/100 eligible PROD automated attempts create a gate before dispatch.
|
||||
|
||||
## Clarifications
|
||||
|
||||
### Session 2026-08-07
|
||||
|
||||
- Q: New scheduler? → A: No. Reuse the 037 trigger framework + existing APScheduler.
|
||||
- Q: Can a scheduled run pause at HumanCheckpoint? → A: No. Human-containing revisions are manual-run-only
|
||||
and are rejected before any durable automation side effect. The PROD approval gate is an authorization
|
||||
gate for eligible automated revisions, never a substitute for HumanCheckpoint.
|
||||
- Q: Where do results go? → A: Scenario runs (044), never into 037 baseline/verification.
|
||||
|
||||
## Implementation Status & MVP Debt (factual audit 2026-08-20)
|
||||
|
||||
@@ -2,7 +2,7 @@
|
||||
|
||||
| Story | Requirement | Model | API operationId | Contract | Task | Test | Actual status / gap |
|
||||
|-------|-------------|-------|------------------|----------|------|------|---------------------|
|
||||
| US1 Schedule | SCAUTO-FR-001/002/007 | ScenarioSchedule, ScenarioTriggerRule, ActionApprovalGate | automation.schedule, automation.trigger | Automation.Schedule, Automation.Trigger, Execution.EnvironmentPolicy, Execution.Runner.QueuedDispatch | T003-T005, T016, T019 | test_trigger, test_scenario_runner, test_scenario_runs_api, test_scenario_automation_api, test_scenario_manual_run_only, test_scenario_queued_dispatch, test_scenario_scheduler_callbacks | `[~]` Event dispatcher passes its server-owned deploy/release/ETL origin into 044 before creation; persisted human revisions reject manual-run-only before a row reaches the dispatch CAS. ConfigManager alone classifies PROD for every source: client flags cannot downgrade it, unknown targets have no side effect, and a PROD intent persists pending_approval plus its durable gate before dispatcher CAS. Fixed 044 queue/cancel callbacks are unit-proven to register exact IDs/interval/singleton-coalescing options, contain database-edge errors, and preserve exact terminal side effects on repeat ticks. Dedicated real scheduled-PROD callback integration, cron firing, a live scheduler process, and long-running subscribers remain open. |
|
||||
| US1 Schedule | SCAUTO-FR-001/002/007 | ScenarioSchedule, ScenarioTriggerRule, ActionApprovalGate | automation.schedule, automation.trigger | Automation.Schedule, Automation.Trigger, Execution.EnvironmentPolicy, Execution.Runner.QueuedDispatch | T003-T005, T016, T019 | test_trigger, test_scenario_runner, test_scenario_runs_api, test_scenario_automation_api, test_scenario_manual_run_only, test_scenario_queued_dispatch, test_scenario_scheduler_callbacks | `[~]` Any persisted human-containing revision is rejected before idempotency, ScenarioRun, gate, notification, queue or dispatcher CAS for scheduled/deploy/release/ETL/API/background origins; manual origin remains eligible. Eligible PROD automation creates ActionApprovalGate before dispatch. Dedicated real scheduled-PROD callback integration, cron firing, a live scheduler process and long-running subscribers remain open. |
|
||||
| US2 Notify | SCAUTO-FR-003 | NotificationEvent | automation.notify | Automation.Notify | T006-T007 | test_notify | `[ ]` helper has no run/staleness lifecycle caller. |
|
||||
| US3 Policy | SCAUTO-FR-004/005/006/007 | AutomationPolicy | automation.policy | Automation.ApplyPolicy, Automation.Retention | T008-T010 | test_policy | `[~]` pure helpers exist; persisted runtime enforcement is unproven. |
|
||||
| US4 Metrics | SCAUTO-FR-008 | — | automation.metrics | Automation.Metrics | T011 | test_metrics | `[~]` aggregate helper exists; source events/runs are not fully wired. |
|
||||
|
||||
Reference in New Issue
Block a user