docs(specs): unify production acceptance matrix and live residual gates
This commit is contained in:
@@ -143,6 +143,15 @@ Contract: [CatalogRevision and baseline lifecycle](contracts/catalog-lifecycle.m
|
||||
- [ ] T083 [P0/P1/P2] Capture bytes/profile/review are server-owned; forged content refs/hashes/approved booleans cannot approve; required raw bytes remain held. Implement at the existing 037 domain boundary; verify with independent hardcoded fixtures and retain command/evidence references in traceability.md.
|
||||
- [ ] T084 [P0/P1/P2] Rebaseline supersedes via CAS; retire/invalidate preserve historical pins; explicit Git commit/push failure returns publish_failed and retry reconciles without duplicate revision. Implement at the existing 037 domain boundary; verify with independent hardcoded fixtures and retain command/evidence references in traceability.md.
|
||||
|
||||
## Design Amendment implementation tasks — curated/scenario-derived baselines (2026-09-17/18)
|
||||
|
||||
- [x] T085 [P0] Extend candidate schema with `kind=scenario_transform`, mandatory `scenario_run_id` + bounded `transform_expression`, metric policy/capture anchoring and visual-field rejection. **CLOSED:** `67c7b1d8`; 18 focused + 1549 domain tests green.
|
||||
- [ ] T086 [P0] Service wiring: create a scenario_transform candidate only from a server-owned 044 transform artifact/ref; preserve run/expression/input manifests in approval and catalog records; forged caller value/ref rejected.
|
||||
- [ ] T087 [P0] BaselineSelectionProposal service/MCP DTO: facts-only ranked coordinates, operator curation, max 50 gating entries, default+business filter contexts, tolerance rationale and CAS.
|
||||
- [ ] T088 [P0] Observatory catalog tier: default-filter broad metric snapshot; non-gating by schema/validator/RunnerPlan; negative observatory→PASS test and drift-only 047 signal.
|
||||
- [ ] T089 [P1] Period rollover/staleness: stale baseline blocks run, agent proposes candidate, operator approves; no automatic rebaseline.
|
||||
- [ ] T090 [P1] Batch capture/review/approval UI/API for curated scope; inheritance diff between release revisions.
|
||||
|
||||
Frontend boundary for this package: manual CRUD/editor, human review/approval, monitoring and read-only evidence/evaluation only; all agent interaction is external MCP. No agent chat/prompt/assistant editing/proposal generation/workspace/start/handoff controls. Runtime removal is OPEN, not performed by this spec refresh. Optional approved performance baseline is outside scope.
|
||||
|
||||
#endregion SupersetBaselineEngine.Tasks
|
||||
|
||||
@@ -43,6 +43,10 @@ Historical rows above identify prior tests/code only; removed agent UI paths are
|
||||
| AGBASE-FR-014 | [CatalogRevision and baseline lifecycle](contracts/catalog-lifecycle.md); [data model](data-model.md) | [T083](tasks.md) | Capture bytes/profile/review are server-owned; forged content refs/hashes/approved booleans cannot approve; required raw bytes remain held. | OPEN |
|
||||
| AGBASE-FR-014 | [CatalogRevision and baseline lifecycle](contracts/catalog-lifecycle.md); [data model](data-model.md) | [T084](tasks.md) | Rebaseline supersedes via CAS; retire/invalidate preserve historical pins; explicit Git commit/push failure returns publish_failed and retry reconciles without duplicate revision. | OPEN |
|
||||
| AGBASE-FR-014; external-MCP-only UI | manual editor/review; read-only evidence | [production tasks](tasks.md) | No frontend agent prompt/chat/assistant editing/proposal generation/workspace/start/handoff routes or requests; human approval remains usable. | OPEN |
|
||||
| AGBASE-FR-017 | Scenario-derived CandidateRequest | [T085](tasks.md) | kind=scenario_transform requires run/expression/capture provenance and rejects visual fields. | **CLOSED schema 2026-09-18** — `67c7b1d8`; 18 focused + 1549 domain tests. Service wiring is T086 OPEN. |
|
||||
| AGBASE-FR-016 | BaselineSelectionProposal | [T087](tasks.md) | Facts-only ≤50 curated gating coordinates, operator review/CAS/filter contexts/tolerance. | OPEN. |
|
||||
| AGBASE-FR-015 | Observatory tier | [T088](tasks.md) | Observatory metrics cannot enter BaselineSelectionPin or PASS; drift signal only. | OPEN. |
|
||||
| AGBASE-FR-018 | Staleness/rollover | [T089](tasks.md) | Period rollover stales and blocks; candidate+operator approval; no auto-rebaseline. | OPEN. |
|
||||
|
||||
Sources: [production gap](../../docs/reports/ss-prod-agentic-e2e-production-gap-2026-09-08.md), [coverage gap](../../docs/reports/ss-prod-agentic-e2e-spec-coverage-2026-09-08.md), [baseline gap](../../docs/reports/ss-prod-agentic-e2e-baseline-gap-2026-09-08.md). Spec schema/static checks prove contract structure only; live canary/runtime closure and optional approved performance baseline are not claimed.
|
||||
|
||||
|
||||
@@ -172,6 +172,17 @@ Contract: [Canonical executable chain](contracts/decision-policy.md).
|
||||
- [ ] T061 [P0/P1/P2] Every browser action/alias maps to one descriptor; sql_evidence/transform enum parity and disabled capability fail before I/O. Implement at the existing 038 domain boundary; verify with independent hardcoded fixtures and retain command/evidence references in traceability.md.
|
||||
- [ ] T062 [P0/P1/P2] All 15 policy precedence rows, threshold equality, malformed output and criterion disagreement use hardcoded outcomes; no dynamic HumanCheckpoint. Implement at the existing 038 domain boundary; verify with independent hardcoded fixtures and retain command/evidence references in traceability.md.
|
||||
|
||||
## Design Amendment implementation tasks — 2026-09-17/18
|
||||
|
||||
- [x] T063 [P0] Distinct `action_inputs` + typed `set_step_inputs`; canonical `inputs:list[Ref]` and old bytes preserved. **CLOSED:** `534d488f` + `f123210b`; 602+12 green.
|
||||
- [x] T064 [P0] No embedded numeric business truth outside baseline refs; bounded per-action validator. **CLOSED:** `534d488f`/`f123210b`.
|
||||
- [x] T065 [P0] Registry 038.5.0 disabled-action discipline; ACTION_DISABLED before I/O. **CLOSED:** `d29959f6`.
|
||||
- [x] T066 [P0] Stop new human_checkpoint classification; unsupported carve-out. **CLOSED:** `a59f5e3c`, 1577 green.
|
||||
- [x] T067 [P0] Typed `set_step_evaluation` + AgentEvaluationSpec/code-token guard. **CLOSED:** `87a90f6f`, 815 green.
|
||||
- [ ] T068 [P0] Bounded TransformSpec executor over declared refs; byte-stable output/resource bounds; no SQL/Python/shell/network.
|
||||
- [ ] T069 [P0] DecisionPolicy single-shot/best-of-N schema and aggregation; quorum/tie/error/confidence; no evidence-free PASS; 044 binding.
|
||||
- [ ] T070 [P1] BaselineSelectionProposal DTO: ≤50 gating coordinates, filter contexts, rationale/class/tolerance; observatory non-gating.
|
||||
|
||||
Frontend boundary for this package: manual CRUD/editor, human review/approval, monitoring and read-only evidence/evaluation only; all agent interaction is external MCP. No agent chat/prompt/assistant editing/proposal generation/workspace/start/handoff controls. Runtime removal is OPEN, not performed by this spec refresh. Optional approved performance baseline is outside scope.
|
||||
|
||||
#endregion DashboardScenarioModel.Tasks
|
||||
|
||||
@@ -100,6 +100,12 @@ Historical rows above identify prior tests/code only; removed agent UI paths are
|
||||
| AGSCN-FR-015 | [Canonical executable chain](contracts/decision-policy.md); [data model](data-model.md) | [T061](tasks.md) | Every browser action/alias maps to one descriptor; sql_evidence/transform enum parity and disabled capability fail before I/O. | OPEN |
|
||||
| AGSCN-FR-015 | [Canonical executable chain](contracts/decision-policy.md); [data model](data-model.md) | [T062](tasks.md) | All 15 policy precedence rows, threshold equality, malformed output and criterion disagreement use hardcoded outcomes; no dynamic HumanCheckpoint. | OPEN |
|
||||
| AGSCN-FR-015; external-MCP-only UI | manual editor/review; read-only evidence | [production tasks](tasks.md) | No frontend agent prompt/chat/assistant editing/proposal generation/workspace/start/handoff routes or requests; human approval remains usable. | OPEN |
|
||||
| AGSCN-FR-016/023 | action_inputs/editor/validator | [T063-T064](tasks.md) | Canonical inputs refs preserved; bounded action_inputs; embedded metric literal rejected; MCP parity. | **CLOSED 2026-09-18** — `534d488f` + `f123210b`; 602+12 green. |
|
||||
| AGSCN-FR-019/020 | capability mapper/compile | [T066](tasks.md) | New graph emits no human_checkpoint; unsafe/missing contexts unsupported; legacy runtime retained. | **CLOSED** — `a59f5e3c`, 1577 green. |
|
||||
| AGSCN-FR-021/034 | AgentEvaluationSpec/editor op | [T067](tasks.md) | set_step_evaluation pins strict spec and executable-code tokens reject. | **CLOSED** — `87a90f6f`, 815 green. |
|
||||
| AGSCN-FR-018 | TransformSpec | [T068](tasks.md) | Bounded deterministic DSL runtime over declared refs; stable digest/resource bounds; no executable code. | OPEN. |
|
||||
| AGSCN-FR-022 | DecisionPolicy | [T069](tasks.md) | single-shot/best-of-N/quorum/tie/provider-error/confidence deterministic outcomes; no evidence-free PASS. | OPEN. |
|
||||
| AGBASE-FR-016 | Baseline selection schema | [T070](tasks.md) | ≤50 curated gating coordinates; filter contexts/rationale/tolerance; observatory non-gating. | OPEN. |
|
||||
|
||||
Sources: [production gap](../../docs/reports/ss-prod-agentic-e2e-production-gap-2026-09-08.md), [coverage gap](../../docs/reports/ss-prod-agentic-e2e-spec-coverage-2026-09-08.md), [baseline gap](../../docs/reports/ss-prod-agentic-e2e-baseline-gap-2026-09-08.md). Spec schema/static checks prove contract structure only; live canary/runtime closure and optional approved performance baseline are not claimed.
|
||||
|
||||
|
||||
@@ -118,5 +118,11 @@ SCEX-FR-028: [Production baseline-backed evaluation](../contracts/production-cha
|
||||
- [x] CHK034 Startup/readiness/start-loop and shutdown/drain/cancel/reconcile survive fault injection; unknown effect quarantines capacity; late response cannot win. Evidence: [T045](../tasks.md), [traceability](../traceability.md). **CLOSED 2026-09-17**: fault-injection canary `prototype/fault_injection_canary.py` (real provider loop/capacity/receipt CAS/cancel lifecycle on real PostgreSQL) **16/16 GREEN** — `evidence/fault-injection/fault-injection-canary-20260917T105541Z.json`.
|
||||
- [~] CHK035 End-to-end real browser→capture→durable artifact→deterministic comparison→optional immutable evaluation→policy→result; all required evidence present before PASS. Evidence: [T046](../tasks.md), [traceability](../traceability.md). Partial 2026-09-17: the chain up to policy is PROVEN live (canary v2/v4; browser canaries 6/6); the graph-level terminal PASS is blocked by the walker-level evaluation→comparison binding (`policy_inputs_from_outcome` reads `evaluation_input` only from the `agent_evaluation` step), diagnosed with executable proof; closure = binding + live v4 rerun.
|
||||
- [ ] CHK036 Negative product UI test: no agent chat/prompt/assistant editing/proposal-generation/typical-operation-to-agent/workspace/start/handoff controls or agent invocation routes/requests; manual CRUD/editor/human review/read-only results remain usable.
|
||||
- [x] CHK037 Wave-2 structural browser actions have unit failure matrices and retained live PREPROD evidence: `assert_dom`, `inspect_filter_options`, `navigate_tabs`, `wait_for_selector`. **CLOSED 2026-09-18:** `4274e403`, unit 811; `evidence/browser-provider/readonly-canary-20260918T125451Z.json` **4/4 PASS** + PNG.
|
||||
- [ ] CHK038 Native filter search/clear/date have retained live evidence showing selected/filter-cleared/date state and chart-data response; offline implementation `683347be`/791 green only.
|
||||
- [ ] CHK039 Provider shutdown leaves zero pending Playwright/asyncio tasks after live canary and soak (`PROVIDER-SHUTDOWN-001`, T052).
|
||||
- [ ] CHK040 Mutation PASS requires pinned independent SQL readback equal to provider post_rows; mismatch is reconciliation_required/blocked; live mutate+restore proof closes `MUT-RECON-001` (T053).
|
||||
- [ ] CHK041 Hot/cold tiering proves verify-before-delete, holds, WebP markers, lossy pixel-compare refusal and idempotent sweep (T054).
|
||||
- [ ] CHK042 Scheduled zero-human E-class run reaches terminal PASS/FAIL/inconclusive from pinned AgentEvaluation+DecisionPolicy; no runtime human step and no evidence-free PASS (T051).
|
||||
|
||||
Schema/static success alone is not runtime completion. Optional approved performance baseline is outside scope.
|
||||
|
||||
@@ -120,7 +120,7 @@ The registry resolves `ActionExecutionDescriptor -> executor`; the following lis
|
||||
- artifact -> generic artifact register (owner_type=scenario_run)
|
||||
- human -> EXCLUDED (runner-lifecycle HumanCheckpoint control, not an executor)
|
||||
|
||||
`BrowserExecutor` implements only registered actions: `open_dashboard`, `navigate_tab`, `apply_native_filter`, `inspect_filter_state`, `apply_table_filter`, `pagination`, `navigate_dashboard`, `extract_table`, `scroll_to`, `inspect_columns`, `click`, `select_rows`, `edit_row`, `bulk_edit`, `download`, `refresh`, `wait_for_state`. ScreenshotService is evidence infrastructure only, not the browser action executor. It validates the registry-declared inputs/outputs/risk/timeout/idempotency before dispatch.
|
||||
`BrowserExecutor` implements registered actions: `open_dashboard`, `navigate_tab`, `navigate_tabs`, `apply_native_filter`, `inspect_filter_state`, `inspect_filter_options`, `apply_table_filter`, `extract_table`, `scroll_to`, `inspect_columns`, `click`, `select_rows`, `assert_dom`, `wait_for_selector`, `edit_row`, `bulk_edit`, `download`, `refresh`, `wait_for_state`. `pagination` and `navigate_dashboard` remain registry-declared but disabled until their drivers and live canary gates close. ScreenshotService is evidence infrastructure only, not the browser action executor. It validates the registry-declared inputs/outputs/risk/timeout/idempotency before dispatch. Composite `navigate_tabs` is bounded to the visible tab nav and returns the visited labels; `assert_dom` evaluates only declared structural DOM predicates and cannot embed business metric truth; `inspect_filter_options` is observe-only; `wait_for_selector` is bounded and fail-closed.
|
||||
|
||||
## Provider runtime protocol
|
||||
|
||||
|
||||
@@ -551,4 +551,36 @@ Normative contract: [Production baseline-backed evaluation](contracts/production
|
||||
- `EVID-TIER-001`: verify-before-delete, hold-правила, lossy-pixel-compare отказ, round-trip digest.
|
||||
- `LLM-INJ-001`: инъекция в evidence не меняет вердикт.
|
||||
|
||||
## Live-Canary Amendment — Provider shutdown and mutation reconciliation (2026-09-18)
|
||||
|
||||
> Источник: owner-authorized PREPROD canaries против `https://ss-prod.bebesh.ru`, retained evidence
|
||||
> `evidence/browser-provider/readonly-canary-20260918T125144Z*.{json,png}`,
|
||||
> `readonly-canary-20260918T125451Z*.{json,png}`, `readonly-canary-20260918T125542Z*.{json,png}`.
|
||||
> Статус: wave-2 observe **4/4 PASS**; mutation mutate+restore provider path **2/2 PASS**, но
|
||||
> два эксплуатационных residual держат SOAK/mutation reconciliation OPEN.
|
||||
|
||||
8. **SCEX-FR-037 (Playwright shutdown completion).** ProviderEventLoop shutdown MUST close every
|
||||
run-scoped BrowserSession/page/context/browser/Playwright connection and await their tasks before
|
||||
the asyncio loop stops. После shutdown запрещены `Task was destroyed but it is pending!` и
|
||||
незавершённые `Connection.run()` tasks. Приёмка: leak-negative test + повтор live canary/soak с
|
||||
нулём pending asyncio tasks и нулём незакрытых browser contexts. Незавершённый teardown —
|
||||
typed non-pass readiness/soak failure, даже если action evidence уже сохранено.
|
||||
9. **SCEX-FR-038 (Independent mutation reconciliation).** Мутационный StepOutcome НЕ МОЖЕТ стать
|
||||
`passed` только по provider `post_rows`/optimistic readback. Нужна независимая server-owned
|
||||
verification read по тому же pinned environment/database/schema/table/key/filter/principal
|
||||
identity и тому же transaction/cache profile, записанная в receipt. Provider/SQL mismatch
|
||||
(например post_rows=82.75, independent SQL=82.74) → `reconciliation_required` + blocked/non-pass;
|
||||
retry/следующая мутация запрещены до reconcile. Restore должен доказать исходное значение тем же
|
||||
независимым readback и закрыть cleanup checkpoint. Cache/view/transaction identities входят в
|
||||
receipt provenance; сравнение разных физических слоёв запрещено.
|
||||
|
||||
### Дополнительные гейты (2026-09-18)
|
||||
- `BSC-ASSERT-001` live-part **CLOSED 2026-09-18**: `wait_for_selector`, `navigate_tabs`,
|
||||
`assert_dom`, `inspect_filter_options` → **4/4 PASS** на dashboard 11, retained JSON+PNG
|
||||
`readonly-canary-20260918T125451Z*`.
|
||||
- `PROVIDER-SHUTDOWN-001`: SCEX-FR-037 — unit leak-negative + live/soak no pending tasks.
|
||||
- `MUT-RECON-001`: SCEX-FR-038 — provider readback == independent SQL; mismatch fail-closed;
|
||||
mutate+restore live evidence. Canary `readonly-canary-20260918T125542Z.json` фиксирует первый
|
||||
mismatch и потому НЕ закрывает этот гейт.
|
||||
|
||||
#endregion ScenarioExecution.Spec
|
||||
|
||||
@@ -343,6 +343,18 @@ Contract: [Production baseline-backed evaluation](contracts/production-chain.md)
|
||||
- [x] T045 [P0/P1/P2] Startup/readiness/start-loop and shutdown/drain/cancel/reconcile survive fault injection; unknown effect quarantines capacity; late response cannot win. Implement at the existing 044 domain boundary; verify with independent hardcoded fixtures and retain command/evidence references in traceability.md. **Status (2026-09-10):** startup/readiness live-verified (provider loop ready, readiness preflight; browser-probe deadlock fixed `ba2f1f45`); capacity leases claimed/released across the live canaries. Fault-injection drain/cancel/reconcile canary not yet exercised. **Status (2026-09-17): CLOSED — fault-injection canary GREEN (16/16).** New harness `specs/044-dashboard-scenario-execution/prototype/fault_injection_canary.py` runs the REAL provider runtime + capacity manager + receipt CAS + cancel lifecycle against real PostgreSQL (`canary_044`), injecting faults at each boundary; evidence `evidence/fault-injection/fault-injection-canary-20260917T105541Z.json`. Vectors: submit before start / expired deadline → typed `PROVIDER_LOOP_NOT_RUNNING` / `PROVIDER_SUBMIT_DEADLINE` in bounded time; slow coroutine cancelled at the caller deadline (`PROVIDER_SUBMIT_DEADLINE`, `cancelled` flag observed); shutdown during in-flight work unwinds the caller on its own deadline and the loop reports not-running (no hang); a crashed (never-released) lease **quarantines the slot** (second claim → `CAPACITY_UNAVAILABLE`) until `reconcile_expired_leases` frees exactly those units and capacity is restored (SC-008 crash recovery); unknown effect → receipt `reconciliation_required` + `effect_state=unknown`, then `reconcile_provider_operation` resolves it; a late response appends history **without** changing the terminal status and a reconcile on a terminal receipt is refused (`PROVIDER_OPERATION_TERMINAL`); cancel opens the drain window (`draining`), the deadline finalizer terminalizes with 0 running steps and 0 unexpired worker leases, and the capacity lease of a cancelled run is **reconciled after TTL, never silently dropped**; immediate cancel terminalizes in one pass (0 running steps, 0 unexpired worker leases). Offline references: `tests/services/dashboard_testing/registry/test_provider_operations.py` (receipt CAS, late response, reconcile), `test_scenario_cancel_timeout.py` (6 cancel/timeout), `test_dispatch_capacity_lifecycle.py`, `test_provider_capacity.py` — all green (41 + 47 passed on 2026-09-17).
|
||||
- [~] T046 [P0/P1/P2] End-to-end real browser→capture→durable artifact→deterministic comparison→optional immutable evaluation→policy→result; all required evidence present before PASS. Implement at the existing 044 domain boundary; verify with independent hardcoded fixtures and retain command/evidence references in traceability.md. **Status (2026-09-11):** the chain real browser→capture→durable artifact→immutable evaluation→policy→result is PROVEN live; a deterministic comparison step and a live baseline pin are now both in live graphs (canary v4: `compare_to_baseline` deterministically passes; pin resolved from the published envelope and stamped into `AgentEvaluation.baseline_pin`, `docs/reports/agentic-runtime-live-canary-v4-baseline-pin-2026-09-11.md`). Residual: a graph-level terminal PASS (the baseline-semantic policy marks comparisons without a bound evaluation as typed `EVALUATION_UNAVAILABLE`; `evaluate-visual` itself passes). **Diagnosis (2026-09-17, executable proof):** the gap is a walker-level binding, not the policy truth table. `walker.py` computes `decide_step_outcome(policy_inputs_from_outcome(outcome, step_meta, integrity), pinned_policy)` PER STEP using only that step's own outcome; `policy_inputs_from_outcome` binds the evaluation ONLY from `step_outcome["evaluation_input"]`, which the evaluation adapter emits solely on the `agent_evaluation` step (`evaluation_adapter.py:379`). The `assertion compare_to_baseline` step therefore always sees `evaluation=None`; `derive_runner_plan` enables mandatory mode as soon as the graph contains an `agent_evaluation` step (per the invariant "evaluation_mode stays disabled while no registry tool can emit an AgentEvaluation"), so every normative step — including the compare step — resolves to `EVALUATION_UNAVAILABLE` even though `evaluate-visual` passes. Verified on HEAD: (A) compare-only outcome + mandatory policy → `inconclusive ["EVALUATION_UNAVAILABLE"]`; (B) same + disabled → `passed ["BASELINE_PASS"]`; (C) same + bound `evaluation_input` (succeeded/pass/0.9) + mandatory → `passed ["BASELINE_AND_SEMANTIC_PASS"]`. The canonical chain (`contracts/production-chain.md` §8) puts the optional declared evaluation BETWEEN the comparison and the pinned DecisionPolicy, so the decision must observe both; today it is computed per-step in isolation and never sees the sibling evaluation. Design options for the closure (next packet; both keep the pinned truth table unchanged): (1) dependency-ordered binding — the compare step depends on the covering `evaluate-visual` step (v4's graph has them as siblings) and the walker injects the persisted `AgentEvaluation` (via that step's `agent_evaluation_ids`) as `evaluation_input` for the dependent comparison decision; (2) deferred decision — the walker defers the policy decision of comparison steps until their covering evaluation step is persisted, then computes one decision on the aggregated inputs. Either path then requires a live rerun of the v4-style graph ending in terminal `passed` as the closing evidence.
|
||||
|
||||
## Design Amendment implementation tasks — 2026-09-17/18
|
||||
|
||||
- [x] T047 [P0] `assert_dom`, `inspect_filter_options`, `navigate_tabs`, `wait_for_selector`: **CLOSED** `4274e403`; unit `811 passed`; live dashboard 11 **4/4 PASS**, `evidence/browser-provider/readonly-canary-20260918T125451Z.json` + PNG.
|
||||
- [x] T048 [P0] Native filter `search_text`, `mode=set|clear`, date-picker, checkpoint/replay and click URL/popup: **CLOSED offline** `683347be`, `791 passed`; live filter canary stays `BSC-FILT-001`.
|
||||
- [x] T049 [P0] Stop new HumanStep emission: **CLOSED** `a59f5e3c`, `1577 passed`.
|
||||
- [x] T050 [P0] Typed `set_step_evaluation`: **CLOSED** `87a90f6f`, `815 passed`.
|
||||
- [ ] T051 [P0] Bind AgentEvaluation output/ref to comparison; scheduled real-LLM terminal PASS; single-shot/best-of-N and confidence/no-evidence policy.
|
||||
- [ ] T052 [P0] SCEX-FR-037 Playwright shutdown; leak-negative + live/soak `PROVIDER-SHUTDOWN-001`.
|
||||
- [ ] T053 [P0] SCEX-FR-038 independent mutation readback; mismatch→reconciliation_required/blocked; `MUT-RECON-001`.
|
||||
- [ ] T054 [P1] Evidence tiering: TranscodeReceipt, hot_runs=2, holds, verify-before-delete, WebP marker, sweep.
|
||||
- [ ] T055 [P1] Implement/canary `navigate_dashboard`; checkpoint segment/filter reset/old checkpoint rejection.
|
||||
|
||||
Frontend boundary for this package: manual CRUD/editor, human review/approval, monitoring and read-only evidence/evaluation only; all agent interaction is external MCP. No agent chat/prompt/assistant editing/proposal generation/workspace/start/handoff controls. Runtime removal is OPEN, not performed by this spec refresh. Optional approved performance baseline is outside scope.
|
||||
|
||||
#endregion ScenarioExecution.Tasks
|
||||
|
||||
@@ -105,6 +105,11 @@ Historical rows above identify prior tests/code only; removed agent UI paths are
|
||||
| SCEX-FR-028 | [Production baseline-backed evaluation](contracts/production-chain.md); [data model](data-model.md) | [T046](tasks.md) | End-to-end real browser→capture→durable artifact→deterministic comparison→optional immutable evaluation→policy→result; all required evidence present before PASS. | **PARTIAL 2026-09-17 — root cause diagnosed with executable proof.** Chain real browser→capture→durable artifact→immutable evaluation→policy→result is PROVEN live; live baseline pin PROVEN (canary v4). Gap: the walker binds the evaluation per STEP (`policy_inputs_from_outcome` reads only `step_outcome["evaluation_input"]`, emitted solely by the `agent_evaluation` step), so an `assertion compare_to_baseline` step resolves to `EVALUATION_UNAVAILABLE` while mandatory mode is on; proven: compare-only+mandatory → `inconclusive [EVALUATION_UNAVAILABLE]`, +bound evaluation → `passed [BASELINE_AND_SEMANTIC_PASS]`. Closure = evaluation→comparison binding (two options in [tasks.md](tasks.md) T046) + live rerun of the v4 graph to terminal `passed`. |
|
||||
| SCEX-FR-028; external-MCP-only UI | manual editor/review; read-only evidence | [production tasks](tasks.md) | No frontend agent prompt/chat/assistant editing/proposal generation/workspace/start/handoff routes or requests; human approval remains usable. | OPEN (050 CHK003/CHK004 row; tracked with 043 T015/T023). |
|
||||
|
||||
| SCEX-FR-037 | ProviderEventLoop/BrowserSession shutdown | [T052](tasks.md) | Live shutdown/soak leaves zero pending Playwright Connection.run tasks and zero open contexts. | OPEN — 2026-09-18 wave-2 canary retained `Task was destroyed but it is pending!`. |
|
||||
| SCEX-FR-038 | Mutation receipt/reconciliation | [T053](tasks.md) | Provider post_rows equals independent pinned SQL readback; mismatch blocks; restore proves original. | OPEN — 2026-09-18 mutation canary recorded 82.75 provider vs 82.74 independent SQL. |
|
||||
| SCEX-FR-031..034 | EvidenceTier/TranscodeReceipt/Retention | [T054](tasks.md) | Hot/cold archive, holds, digest chain, lossy refusal and idempotent sweep. | OPEN — contract exists, implementation/evidence absent. |
|
||||
| SCEX-FR-028 / live observe | BrowserProvider wave-2 | [T047](tasks.md) | Four observe actions pass on real Superset with durable evidence. | CLOSED 2026-09-18 — 4/4 live, JSON+PNG `readonly-canary-20260918T125451Z`. |
|
||||
|
||||
Sources: [production gap](../../docs/reports/ss-prod-agentic-e2e-production-gap-2026-09-08.md), [coverage gap](../../docs/reports/ss-prod-agentic-e2e-spec-coverage-2026-09-08.md), [baseline gap](../../docs/reports/ss-prod-agentic-e2e-baseline-gap-2026-09-08.md). Spec schema/static checks prove contract structure only; live canary/runtime closure and optional approved performance baseline are not claimed.
|
||||
|
||||
## Live-binding admin surface + catalog publisher — evidence 2026-09-11
|
||||
@@ -125,6 +130,9 @@ the identity-less-compiled-step defect) is `docs/2026-09-11-sales-prod-mcp-repla
|
||||
|
||||
| Canary / evidence file | Vectors | Task |
|
||||
|---|---|---|
|
||||
| `evidence/browser-provider/readonly-canary-20260918T125144Z.json` (+PNG) | Live Superset dashboard 11: open_dashboard / wait_for_state / refresh → 3/3 PASS | T047 prerequisite |
|
||||
| `evidence/browser-provider/readonly-canary-20260918T125451Z.json` (+PNG) | Wave-2 live observe: wait_for_selector / navigate_tabs / assert_dom / inspect_filter_options → 4/4 PASS; filter Region 19 options | T047 `[x]`, BSC-ASSERT-001 live CLOSED |
|
||||
| `evidence/browser-provider/readonly-canary-20260918T125542Z.json` (+PNG) | Mutation mutate+restore provider path 2/2; independent SQL mismatch retained (82.75 vs 82.74) → MUT-RECON-001 OPEN | T053 `[ ]` |
|
||||
| `evidence/browser-provider/readonly-canary-20260916T*.json` (+PNG) | 6/6 GREEN under Wave-C architecture: read-only actions, forced timeout/cleanup, mutation row_edit+restore, reconciliation sweep, safe-checkpoint reconstruction, scheduler soak | T042b `[x]` |
|
||||
| `evidence/artifact-content/artifact-content-canary-20260917T085438Z.json` | 11/11 GREEN: GET/HEAD ACL+header parity, byte-exact body, anon 401, no-VIEW 403, unknown/foreign 404, expired 410, corrupt 409, bad-MIME 409, oversized 413, missing 409, Range 416 | T044 `[x]` |
|
||||
| `evidence/fault-injection/fault-injection-canary-20260917T105541Z.json` | 16/16 GREEN: loop refusals, deadline cancellation, shutdown drain, capacity quarantine + TTL reconcile, receipt reconciliation_required, late response history-only, cancel drain/terminalize | T045 `[x]` |
|
||||
|
||||
@@ -157,6 +157,14 @@ Contract: [Parity, retention and rollout](contracts/production-operations.md).
|
||||
RetentionReceipts ACL (16 passed in file). Remaining: 5/15/50-tab canaries vs versioned ops-v1
|
||||
SLO thresholds (D-wave, after T034 catalog).
|
||||
|
||||
## Design Amendment implementation tasks — operations (2026-09-17/18)
|
||||
|
||||
- [ ] T022 [P0] Wire terminal non-pass, queue/case lifecycle and baseline-stale events to the existing notification contract; define channel/SLA N and retain delivery receipts (`NOTIFY-001`, `INV-NOTIFY-001`).
|
||||
- [ ] T023 [P0] Poisoned-run policy: N identical infra failures → durable quarantine/DLQ; no infinite retry; operator recovery and notification.
|
||||
- [ ] T024 [P0] 72-hour multi-process soak with three restart storms; zero duplicate runs/gates/outbox; queue-depth/capacity SLO retained (`SCHED-SOAK-001`).
|
||||
- [ ] T025 [P1] `hot_runs=2` retention policy and tiering scheduler integration; holds for active run/open case/pending baseline/manual hold; deletion only after verified archive.
|
||||
- [ ] T026 [P1] 5/15/50-tab cost/load/timeout/cancel canaries against versioned capacity thresholds; close prior T021 residual.
|
||||
|
||||
Frontend boundary for this package: manual CRUD/editor, human review/approval, monitoring and read-only evidence/evaluation only; all agent interaction is external MCP. No agent chat/prompt/assistant editing/proposal generation/workspace/start/handoff controls. Runtime removal is OPEN, not performed by this spec refresh. Optional approved performance baseline is outside scope.
|
||||
|
||||
#endregion ScenarioAutomation.Tasks
|
||||
|
||||
@@ -20,5 +20,8 @@ Historical rows above identify prior tests/code only; removed agent UI paths are
|
||||
| SCAUTO-FR-019 | [Parity, retention and rollout](contracts/production-operations.md); [data model](data-model.md) | [T020](tasks.md) | Crash/replay/concurrent due events create one pinned run/gate/outbox and preserve approval before dispatcher CAS. | DONE (2026-09-12): baseline pin in due identity + typed conflict on changed canonical request — `test_scheduled_scenario_start.py` (BaselineStamp/BaselineConflict/BaselineNextDue); concurrent double-callback convergence, crash-between-run-and-gate rollback+retry, terminal transition+queue+outbox single transaction, approval-before-dispatcher-CAS — `test_due_admission_atomicity.py`; evidence: `python -m pytest -q tests/services/dashboard_testing/registry/` (466 passed). |
|
||||
| SCAUTO-FR-019 | [Parity, retention and rollout](contracts/production-operations.md); [data model](data-model.md) | [T021](tasks.md) | Retention honors baseline/case holds; deletion receipt proves bytes removed; 5/15/50-tab cost/load/timeout/cancel canaries meet versioned SLO thresholds. | PARTIAL (2026-09-12): holds (`apply_holds` — approved-baseline fail-closed, active ops, analytics window на run_metadata) + durable deletion pipeline (`automation/deletions.py`, модель + миграция `0024`, mark→eligible→delete-bytes→verify→tombstone, идемпотентный retry, never-deleted-while-bytes-survive) + sweep job + ACL-проекция в `GET /retention` — `test_scenario_retention_holds.py` (11), `test_scenario_automation_api.py` (16). OPEN: canaries vs ops-v1 SLO (нужен T034 каталог + live стенд). |
|
||||
| SCAUTO-FR-019; external-MCP-only UI | manual editor/review; read-only evidence | [production tasks](tasks.md) | No frontend agent prompt/chat/assistant editing/proposal generation/workspace/start/handoff routes or requests; human approval remains usable. | OPEN |
|
||||
| SCAUTO-FR-022 | NotificationEvent callers | [T022](tasks.md) | terminal/queue/case/stale events produce bounded notification receipts within declared SLA. | OPEN — helper exists; callers/channel contract not wired. |
|
||||
| SCAUTO-FR-023/024 | Quarantine/DLQ/soak/capacity SLO | [T023/T024/T026](tasks.md) | poisoned-run quarantine, recovery, 72h restart soak, zero duplicates, 5/15/50-tab thresholds. | OPEN — T021 canary residual. |
|
||||
| SCAUTO-FR-020 | Evidence tier retention | [T025](tasks.md) | hot_runs=2/holds/tiering scheduler integrated with durable deletion. | OPEN — retention deletion exists; WebP tier not wired. |
|
||||
|
||||
Sources: [production gap](../../docs/reports/ss-prod-agentic-e2e-production-gap-2026-09-08.md), [coverage gap](../../docs/reports/ss-prod-agentic-e2e-spec-coverage-2026-09-08.md), [baseline gap](../../docs/reports/ss-prod-agentic-e2e-baseline-gap-2026-09-08.md). Spec schema/static checks prove contract structure only; live canary/runtime closure and optional approved performance baseline are not claimed.
|
||||
|
||||
@@ -82,18 +82,22 @@ Contract: [Atomic triage and exact context](contracts/atomic-triage.md).
|
||||
> Правило то же: `[ ]` только с retained evidence; гейты `INV-MCP-001`, `INV-FIND-001`, `INV-ATOMIC-001`,
|
||||
> `INV-NOTIFY-001` держат NO-GO до закрытия. Спека: Design Amendment 2026-09-17 (SCAN-FR-016..022).
|
||||
|
||||
- [ ] T022 MCP read tools `list_investigation_queue` / `get_investigation_case`: bounded проекции,
|
||||
- [x] T022 MCP read tools `list_investigation_queue` / `get_investigation_case`: bounded проекции,
|
||||
human-readable заголовок, evidence refs+digests без байтов; RBAC view; zero side effects
|
||||
(SCAN-FR-016/019). Гейт `INV-MCP-001` (read-часть).
|
||||
- [ ] T023 `record_case_note` + linked-AgentRun provenance (human-only write, triage) и
|
||||
`propose_case_disposition` (pull-only; решение — human CAS) (SCAN-FR-017/018). Гейт
|
||||
`INV-MCP-001` (полный: queue→read→note→propose; service-principal decide denied).
|
||||
- [ ] T024 Case index UI + queue human-readable проекция + in_case-видимость + фильтры/пагинация
|
||||
(SCAN-FR-020). Гейт `INV-FIND-001`.
|
||||
- [ ] T025 CaseWorkspace: рендер linked_run_ids/evidence/tier-маркеры/scenario-ссылки, i18n
|
||||
timeline (SCAN-FR-021).
|
||||
(SCAN-FR-016/019). **CLOSED 2026-09-18:** `23414f9a` + catalog-fix `5eea72bc` (2.4.0),
|
||||
focused 73 green; master post-merge 869 green.
|
||||
- [x] T023 `record_case_note` + linked-AgentRun provenance (human-only write, triage) и
|
||||
`propose_case_disposition` (pull-only; решение — human CAS) (SCAN-FR-017/018). **CLOSED
|
||||
2026-09-18:** `23414f9a`/`5eea72bc`; service principal denied; disposition enum resolved|accepted.
|
||||
- [x] T024 Case index UI + queue human-readable проекция + in_case-видимость + фильтры/пагинация
|
||||
(SCAN-FR-020). **CLOSED 2026-09-18:** frontend `e7925d0e`, backend live-gap fix `c9e6e642`;
|
||||
3360 vitest/lint/build; live browser `/dashboard-testing/analytics/cases` verified.
|
||||
- [x] T025 CaseWorkspace: рендер linked_run_ids/evidence/tier-маркеры/scenario-ссылки, i18n
|
||||
timeline (SCAN-FR-021). **CLOSED:** `e7925d0e`, frontend 3360 green; live case workspace verified.
|
||||
- [ ] T026 Queue/case notifications wiring через 046 (SCAN-FR-022 = SCAUTO-FR-022). Гейт `INV-NOTIFY-001`.
|
||||
- [ ] T027 Browser E2E: queue→case→disposition живым прогоном (изолированный e2e-стенд).
|
||||
- [ ] T028 PostgreSQL concurrency acceptance for T019/T020/T021: signal/disposition/open races,
|
||||
one active episode/case, rollback on stale CAS, exact-context comparability and immutable evidence.
|
||||
|
||||
Frontend boundary for this package: manual CRUD/editor, human review/approval, monitoring and read-only evidence/evaluation only; all agent interaction is external MCP. No agent chat/prompt/assistant editing/proposal generation/workspace/start/handoff controls. Runtime removal is OPEN, not performed by this spec refresh. Optional approved performance baseline is outside scope.
|
||||
|
||||
|
||||
@@ -19,5 +19,11 @@ Historical rows above identify prior tests/code only; removed agent UI paths are
|
||||
| SCAN-FR-015 | [Atomic triage and exact context](contracts/atomic-triage.md); [data model](data-model.md) | [T020](tasks.md) | Concurrent signal/disposition/open replay yields one active episode/case; retained run/evaluation evidence never mutates. | OPEN |
|
||||
| SCAN-FR-015 | [Atomic triage and exact context](contracts/atomic-triage.md); [data model](data-model.md) | [T021](tasks.md) | Same environment but different baseline family/release/RLS/principal cannot compare; hardcoded eligible sequence separates regression/flaky/model variance. | OPEN |
|
||||
| SCAN-FR-015; external-MCP-only UI | manual editor/review; read-only evidence | [production tasks](tasks.md) | No frontend agent prompt/chat/assistant editing/proposal generation/workspace/start/handoff routes or requests; human approval remains usable. | OPEN |
|
||||
| SCAN-FR-016/019 | MCP investigation read projections | [T022](tasks.md) | tools/call list/get are bounded, owner/RBAC scoped, zero side effects; evidence refs/digests only. | **CLOSED 2026-09-18** — `23414f9a` + catalog `5eea72bc`; focused 73 green, master 869 green. |
|
||||
| SCAN-FR-017/018 | Append-only note/proposal | [T023](tasks.md) | User-principal note/proposal appends AgentAction; service principal denied; proposal never closes case. | **CLOSED 2026-09-18** — same commits/tests; enum resolved|accepted pinned. |
|
||||
| SCAN-FR-020 | Case index/queue findability | [T024](tasks.md) | queued+in_case visible; human-readable title; case index recovers open case; filters/pagination. | **CLOSED 2026-09-18** — `e7925d0e` + backend `c9e6e642`; 3360 frontend green + live browser. |
|
||||
| SCAN-FR-021 | Case workspace evidence/links | [T025](tasks.md) | linked runs/scenario/evidence refs and localized timeline render without agent controls. | **CLOSED 2026-09-18** — `e7925d0e`; live case workspace verified. |
|
||||
| SCAN-FR-022 | Notifications | [T026](tasks.md) | queue created / case opened / disposition → durable notification receipt within SLA. | OPEN (`INV-NOTIFY-001`). |
|
||||
| SCAN-FR-015 | Atomic concurrency/exact context | [T019-T021, T028](tasks.md) | PostgreSQL race fixtures and rollback/exact-context evidence. | OPEN (`INV-ATOMIC-001`). |
|
||||
|
||||
Sources: [production gap](../../docs/reports/ss-prod-agentic-e2e-production-gap-2026-09-08.md), [coverage gap](../../docs/reports/ss-prod-agentic-e2e-spec-coverage-2026-09-08.md), [baseline gap](../../docs/reports/ss-prod-agentic-e2e-baseline-gap-2026-09-08.md). Spec schema/static checks prove contract structure only; live canary/runtime closure and optional approved performance baseline are not claimed.
|
||||
|
||||
@@ -96,4 +96,12 @@ Contract: [Contract-complete public parity](contracts/modules.md).
|
||||
- [x] T045 [P0/P1/P2] consume/publish failures return typed errors/pending state with no legacy fallback; every prerequisite is externally MCP-reachable. Implement at the existing 050 domain boundary; verify with independent hardcoded fixtures and retain command/evidence references in traceability.md. **Status (2026-09-11): CLOSED.** Consume path typed fail-closed (`bbbd4ccf`, live-proven); publication is now externally MCP-reachable through the curated gated tool `publish_baseline_catalog` (human-only, `scenario` RUN_PROD, `requires_approval`) executing the 037 publication-worker contract via `ScenarioExecution.PublicationWorker` + durable `publication_operations` (migration `0023`): idempotency by key, envelope validation (resolver-canonical), branch-head CAS, commit/published receipts, failure retention + reconcile of the same commit; REST parity `POST/GET /api/catalog-publications`. LIVE evidence (`docs/reports/agentic-runtime-live-mcp-publish-t045-2026-09-11.md`): MCP gate `approval_required` → `decide_approval` → dispatch → `published` (commit `abce63db…`, receipt head 3f782574→abce63db) + moved-HEAD canary → typed `PUBLISH_HEAD_MOVED` with a durable `publish_failed` row. Offline: `tests/services/dashboard_testing/execution/test_publication_worker.py`, `tests/api/test_catalog_publications_api.py`, `tests/test_mcp_ops_parity.py`.
|
||||
- [x] T046 [P0/P1/P2] Fresh external-client chain preserves authoritative context and complete baseline pin without raw ORM/REST repair; no frontend agent controls/routes/requests. Implement at the existing 050 domain boundary; verify with independent hardcoded fixtures and retain command/evidence references in traceability.md. **Status (2026-09-11): CLOSED.** Context authority: fresh REST chain (canary v2 `4eebfab3`) and fresh MCP chain (T029m) preserved the server-resolved binding and verified context (`context_authority=verified`). Complete baseline pin: (a) the MCP-external chain with `--baseline` (`live_mcp_replay.py --baseline`, no raw ORM seeding) compiled the baseline capability, started with the published-catalog selector, and carried a full `runner_plan.baseline_pin` (set/version/digest from the published envelope); (b) the REST canary v4 (`ace916a0…`, then re-run `adeabe63…`) proved the walker stamps that same pin into the persisted `AgentEvaluation.baseline_pin` (`plan pin == record pin`, strict equality asserted) — `docs/reports/agentic-runtime-live-canary-v4-baseline-pin-2026-09-11.md`. No raw ORM/REST pin repair; frontend agent controls remain absent (2026-09-10 evidence).
|
||||
|
||||
## Design Amendment implementation tasks — 2026-09-17/18
|
||||
|
||||
- [x] T047 [P0] Investigation MCP tools + RBAC/human-only writes + catalog 2.4.0. **CLOSED:** `23414f9a` + `5eea72bc`, 73 focused green.
|
||||
- [x] T048 [P0] `set_step_inputs` and `set_step_evaluation` through propose_graph_revision/EditOperation parity. **CLOSED:** `534d488f`/`f123210b` + `87a90f6f`.
|
||||
- [ ] T049 [P1] `propose_baseline_selection` MCP surface for curated coordinates/review handle; observatory non-gating.
|
||||
- [ ] T050 [P1] `exploration_step` read-only sandbox action with CAS/receipts; reuse 044 browser session; no second Playwright stack.
|
||||
- [ ] T051 [P0] Complete REST/MCP lifecycle/error parity (prior T044 residual), including disabled-action/investigation tools.
|
||||
|
||||
Frontend boundary for this package: manual CRUD/editor, human review/approval, monitoring and read-only evidence/evaluation only; all agent interaction is external MCP. No agent chat/prompt/assistant editing/proposal generation/workspace/start/handoff controls. Runtime removal is OPEN, not performed by this spec refresh. Optional approved performance baseline is outside scope.
|
||||
|
||||
@@ -10,6 +10,10 @@ Historical rows above identify prior tests/code only; removed agent UI paths are
|
||||
| MCPX-FR-030 | [Contract-complete public parity](contracts/modules.md); [data model](data-model.md) | [T045](tasks.md) | consume/publish failures return typed errors/pending state with no legacy fallback; every prerequisite is externally MCP-reachable. | CLOSED (gated MCP `publish_baseline_catalog` + 037 publication worker, REST parity; live gate→CAS→commit→receipt chain 2026-09-11) |
|
||||
| MCPX-FR-030 | [Contract-complete public parity](contracts/modules.md); [data model](data-model.md) | [T046](tasks.md) | Fresh external-client chain preserves authoritative context and complete baseline pin without raw ORM/REST repair; no frontend agent controls/routes/requests. | CLOSED (MCP `--baseline` chain carries the plan pin without raw ORM; REST canary proves the record stamping) |
|
||||
| MCPX-FR-030; external-MCP-only UI | manual editor/review; read-only evidence | [production tasks](tasks.md) | No frontend agent prompt/chat/assistant editing/proposal generation/workspace/start/handoff routes or requests; human approval remains usable. | OPEN |
|
||||
| MCPX-FR-031 | Investigation tools | [T047](tasks.md) | queue/case read, human-only note/proposal, bounded refs, catalog/RBAC parity. | **CLOSED 2026-09-18** — `23414f9a`/`5eea72bc`, 73 focused green. |
|
||||
| MCPX-FR-032 | Authoring input/evaluation ops | [T048](tasks.md) | set_step_inputs/set_step_evaluation pass external proposal/diff validation. | **CLOSED** — `534d488f`/`f123210b`/`87a90f6f`. |
|
||||
| MCPX-FR-033 | Exploration step | [T050](tasks.md) | sandbox read-only interactive step with CAS/receipts and no second Playwright stack. | OPEN. |
|
||||
| MCPX-FR-034 | Prompt dangerous-content profile | [T051](tasks.md) | REST/MCP validation/error parity plus injection negative tests. | PARTIAL — code-token guard exists; dedicated parity/LLM injection gate OPEN. |
|
||||
|
||||
Sources: [production gap](../../docs/reports/ss-prod-agentic-e2e-production-gap-2026-09-08.md), [coverage gap](../../docs/reports/ss-prod-agentic-e2e-spec-coverage-2026-09-08.md), [baseline gap](../../docs/reports/ss-prod-agentic-e2e-baseline-gap-2026-09-08.md). Spec schema/static checks prove contract structure only; live canary/runtime closure and optional approved performance baseline are not claimed.
|
||||
|
||||
|
||||
72
specs/PRODUCTION-ACCEPTANCE-MATRIX-043-050.md
Normal file
72
specs/PRODUCTION-ACCEPTANCE-MATRIX-043-050.md
Normal file
@@ -0,0 +1,72 @@
|
||||
#region ProductionAcceptance.Matrix043050 [C:5] [TYPE ADR] [SEMANTICS production,acceptance,gates,scenario,baseline,execution,monitor,automation,analytics,mcp]
|
||||
@BRIEF Unified conjunctive production acceptance matrix for the current 037/038/044/045/046/047/050 scenario-testing chain after the 2026-09-17/18 amendments and live canaries.
|
||||
@RELATION DEPENDS_ON -> [SupersetBaselineEngine.Spec]
|
||||
@RELATION DEPENDS_ON -> [DashboardScenarioModel.Spec]
|
||||
@RELATION DEPENDS_ON -> [ScenarioExecution.Spec]
|
||||
@RELATION DEPENDS_ON -> [ScenarioRunMonitor.Spec]
|
||||
@RELATION DEPENDS_ON -> [ScenarioAutomation.Spec]
|
||||
@RELATION DEPENDS_ON -> [ScenarioAnalytics.Spec]
|
||||
@RELATION DEPENDS_ON -> [McpInterface.Spec]
|
||||
@RATIONALE Prior acceptance rows were fragmented across seven feature packages and WORKSTATE; a single conjunctive table is required so a local unit PASS cannot be mistaken for production GO while a live, safety or operations gate remains OPEN.
|
||||
@REJECTED Taking the newest historical checklist status per package was rejected — many older `[x]` rows predate drift amendments, zero-human policy, evidence tiering and current live findings.
|
||||
@INVARIANT Production GO requires every P0 row CLOSED and every P1 row either CLOSED or explicitly accepted by a signed release decision; CLOSED requires retained executable evidence, never prose or schema shape alone.
|
||||
|
||||
## 1. Status vocabulary
|
||||
|
||||
| State | Meaning |
|
||||
|---|---|
|
||||
| `CLOSED` | Falsifiable verifier executed against the current contract; retained command/output or live evidence is linked. |
|
||||
| `PARTIAL` | Implementation/evidence exists but at least one named acceptance vector remains open. |
|
||||
| `OPEN` | Missing implementation or missing executable proof. |
|
||||
| `BLOCKED` | Cannot execute until an external prerequisite/decision exists. |
|
||||
| `DEFERRED` | Explicit post-GA item; it cannot hide a P0 requirement. |
|
||||
|
||||
**GO rule:** `GO ⇔ all P0=CLOSED ∧ all accepted P1 have signed disposition ∧ full suites/migrations/semantic audit green on the release commit ∧ no unresolved critical security or reconciliation finding`.
|
||||
|
||||
## 2. Unified matrix
|
||||
|
||||
| Gate | Priority | Owner spec/task | Falsifiable acceptance | Current evidence | State |
|
||||
|---|---:|---|---|---|---|
|
||||
| `G-BSL-CATALOG` | P0 | 037 T082–T084 | Catalog CAS, one head under concurrent approval, server-owned capture bytes, rebaseline/supersede/retire/publish failure reconcile without duplicate revision. | Existing catalog lifecycle + publication canaries; new curated amendments not complete. | OPEN |
|
||||
| `G-BSL-DERIVED` | P0 | 037 T085–T090; 038 T068/T070 | Bounded transform output creates `scenario_transform` candidate from server artifact; curated ≤50 gating coordinates; operator approval; observatory entry cannot reach PASS; period stale→blocked→candidate. | Candidate schema/provenance `67c7b1d8`, 1549 green. Service/selection/observatory not implemented. | PARTIAL |
|
||||
| `G-SCENARIO-AUTHOR` | P0 | 038 T063–T070 | Typed action/evaluation ops; no embedded metric truth; no new HumanStep; disabled actions reject before I/O; TransformSpec/DecisionPolicy schemas deterministic. | `534d488f`,`f123210b`,`d29959f6`,`a59f5e3c`,`87a90f6f`; unit suites green. Transform/best-of-N open. | PARTIAL |
|
||||
| `G-BROWSER-CORE` | P0 | 044 T047/T048 | Live Superset browser actions: open/wait/refresh + assert_dom/inspect_filter_options/navigate_tabs/wait_for_selector; owned durable evidence; no fabricated PASS. | `readonly-canary-20260918T125144Z*` 3/3; `125451Z*` 4/4; unit 811. | CLOSED |
|
||||
| `G-BROWSER-FILTERS` | P0 | 044 T048 / CHK038 | Live `search_text`, set, clear, date-picker; inspect state before/after; chart-data response; clear checkpoint replay. | Code `683347be`, 791 unit. No retained live search/clear/date evidence. | OPEN |
|
||||
| `G-BROWSER-CROSS-DASH` | P1 | 044 T055 | navigate_dashboard driver, filter reset, new checkpoint segment, old checkpoint rejected, cross-dashboard fixture. | Registry disabled: pending driver. | OPEN |
|
||||
| `G-E2E-EVALUATION` | P0 | 038 T069; 044 T043/T046/T051 | Browser→artifact→baseline comparison→bound immutable AgentEvaluation→pinned DecisionPolicy→terminal PASS/FAIL/inconclusive; scheduled zero-human real-LLM run; no evidence-free PASS. | Browser/evaluation/pin proven separately; `set_step_evaluation` `87a90f6f`; walker binding and best-of-N open. | PARTIAL |
|
||||
| `G-LLM-SECURITY` | P0 | 038 AGSCN-FR-022; 044 SCEX-FR-035 | Evidence-as-data; zero tools; strict output; prompt/evidence injection cannot force PASS; malformed/low-confidence/disagreement→non-pass. | Code-token op guard `87a90f6f`; `LLM-INJ-001` live/runtime proof open. | PARTIAL |
|
||||
| `G-PROVIDER-SHUTDOWN` | P0 | 044 T052 / SCEX-FR-037 | After canary and soak: zero pending Playwright `Connection.run`, page/context/browser tasks; shutdown bounded. | Live canary emitted pending-task warnings. | OPEN |
|
||||
| `G-MUTATION-RECON` | P0 | 044 T053 / SCEX-FR-038 | Provider post_rows equals independent pinned SQL readback; mismatch→reconciliation_required/blocked; restore independently verified. | `125542Z` retained provider mutate/restore but SQL mismatch 82.75/82.74. | OPEN |
|
||||
| `G-EVIDENCE-TIER` | P1 | 044 T054; 046 T025 | `hot_runs=2`, holds, TranscodeReceipt source→target digest, verify-before-delete, lossless/lossy marker, lossy pixel refusal, idempotent sweep. | Contract only. | OPEN |
|
||||
| `G-ARTIFACT-CONTENT` | P0 | 044 T044 | Auth/ACL/status/headers/digest/MIME/length/expired/corrupt/traversal/range. | 2026-09-17 live artifact canary 11/11. | CLOSED |
|
||||
| `G-CAPACITY-FAULTS` | P0 | 044 T045 | Startup/readiness/deadline/cancel/drain/quarantine/reconcile/late-response fault matrix. | 2026-09-17 live fault canary 16/16. | CLOSED |
|
||||
| `G-MONITOR` | P1 | 045 RUNMON-FR-015..017 | Launch gates usable; legacy checkpoint read; new runs no waiting-human; tiered evidence markers; reload/reconnect. | UI unit/build partially proven; tiering absent. | PARTIAL |
|
||||
| `G-SCHEDULER-CORE` | P0 | 046 T019/T020 | REST/MCP automation parity, ownership, disabled/legacy HumanStep rejection, idempotent crash/replay/concurrent due events. | T019/T020 closed with existing tests. | CLOSED |
|
||||
| `G-NOTIFICATIONS` | P0 | 046 T022; 047 T026 | Failed/inconclusive run, queue created, case opened/disposition, stale baseline each produce durable notification receipt within declared SLA/channel. | Contract/events exist; caller/channel wiring absent. | OPEN |
|
||||
| `G-DLQ-SOAK` | P0 | 046 T023/T024/T026 | N identical infra failures→quarantine; recovery; queue-depth/capacity SLO; 72h soak + 3 restarts + 5/15/50-tab canaries, zero duplicates. | Short/fault canaries only; 72h and size canaries open. | OPEN |
|
||||
| `G-INVESTIGATION-MCP` | P0 | 047 T022/T023 | MCP queue/get/note/proposal bounded, RBAC/owner, service writes denied, proposal never closes case. | `23414f9a`,`5eea72bc`; 73 green. | CLOSED |
|
||||
| `G-INVESTIGATION-UI` | P1 | 047 T024/T025/T027 | queued+in_case findability, case index, linked runs/evidence/i18n, browser queue→case→disposition. | `e7925d0e`,`c9e6e642`; 3360/build + live queue/case navigation. Disposition browser E2E open. | PARTIAL |
|
||||
| `G-INVESTIGATION-ATOMIC` | P0 | 047 T019–T021/T028 | PostgreSQL concurrency: one active episode/case; stale CAS rollback; exact-context comparability; immutable evidence. | Local primitives; retained concurrency evidence open. | OPEN |
|
||||
| `G-MCP-PARITY` | P0 | 050 T044/T047/T048/T051 | REST/MCP error/auth/lifecycle parity, catalog/visibility mirrors, service cannot decide human gates; all prerequisites externally reachable. | AgentRun/authoring/investigation tools and live external chain proven; dedicated error-shape matrix open. | PARTIAL |
|
||||
| `G-MCP-BASELINE-SELECTION` | P1 | 050 T049 | Agent proposes bounded curated baseline scope; operator review handle; server capture; observatory non-gating. | Contract only. | OPEN |
|
||||
| `G-MCP-EXPLORATION` | P1 | 050 T050 | Read-only exploration_step in isolated workspace with CAS/receipts; reuses 044 session; no raw code or second Playwright stack. | Fixed one-shot exploration exists; interactive operation absent. | OPEN |
|
||||
| `G-STORAGE-BUDGET` | P1 | 044/046 tiering | Measured hot/cold artifact volume and quota alarms meet the approved retention budget for 20-table dashboards. | Estimates only; no telemetry-based acceptance. | OPEN |
|
||||
| `G-SECURITY-OPS` | P0 | 044/046/050 | Local provider perimeter, artifact ACL, secret-redaction, key expiry/rotation, PII retention policy, no credentials in evidence. | Artifact ACL/provenance proven; key expiry/PII/archive policy open. | PARTIAL |
|
||||
| `G-DR-ROLLBACK` | P1 | cross-cutting | Restore DB + baseline Git + evidence store; migration rollback/roll-forward runbook; recovery RTO/RPO exercise. | No unified exercise. | OPEN |
|
||||
| `G-FULL-REGRESSION` | P0 | release | Full backend pytest, frontend vitest/lint/build, ruff, compileall, PostgreSQL Alembic heads/upgrade/check, isolated browser E2E on release commit. | Scoped runs green; no current full-suite release gate after 2026-09-18 changes. | OPEN |
|
||||
| `G-SEMANTIC-HEALTH` | P0 | release | Axiom verify/rebuild: 0 parse warnings; no new naked critical blocks/unresolved critical relations; docs/nav rebuilt. | Pending after current spec/code amendments. | OPEN |
|
||||
| `G-RELEASE-SIGNOFF` | P0 | release owner | All P0 CLOSED, P1 dispositions signed, unified matrix frozen with evidence refs and commit; NO-GO→GO recorded. | Product remains NO-GO. | OPEN |
|
||||
|
||||
## 3. Release sequence
|
||||
|
||||
1. Close P0 runtime safety first: `G-PROVIDER-SHUTDOWN`, `G-MUTATION-RECON`, `G-BROWSER-FILTERS`.
|
||||
2. Close correctness truth chain: TransformSpec + curated/observatory baseline + E2E evaluation (`G-BSL-DERIVED`, `G-E2E-EVALUATION`, `G-LLM-SECURITY`).
|
||||
3. Close operations: notifications, DLQ/soak, atomic investigation, MCP parity.
|
||||
4. Run full regression, migration and semantic health on one release commit.
|
||||
5. Freeze this matrix; obtain release-owner signoff.
|
||||
|
||||
## 4. Matrix maintenance rule
|
||||
|
||||
Any future spec amendment that adds a production invariant MUST add/update one row here in the same commit. A row state may move to `CLOSED` only when its owner package traceability links retained executable evidence; WORKSTATE narrative alone is insufficient.
|
||||
|
||||
#endregion ProductionAcceptance.Matrix043050
|
||||
Reference in New Issue
Block a user