ss-tools/specs/036-agent-test-stabilization/spec.md

#region AgentTestStabilization.Spec [C:3] [TYPE ADR] [SEMANTICS spec,requirements,agent,test-scenarios,stabilization] @BRIEF Stabilize the existing Gradio/LangGraph agent runtime so it can safely support long-running dashboard test scenario generation. @RELATION DEPENDS_ON -> [ADR-0001] @RELATION DEPENDS_ON -> [ADR-0003] @RELATION DEPENDS_ON -> [ADR-0005] @RELATION DEPENDS_ON -> [ADR-0006] @RELATION DEPENDS_ON -> [ADR-0008] @RATIONALE Dashboard test scenario generation produces durable artifacts and baseline approval flows, so the agent must be a recoverable execution surface rather than a best-effort chat stream. @REJECTED Starting dashboard scenario generation directly on top of unstabilized chat behavior — rejected because failures would be indistinguishable across Gradio transport, context passing, artifact preview, and HITL gates.

Navigation (DSA Indexer keywords)

@SEMANTICS: spec, requirements, feature, agent, gradio, langgraph, scenario, artifact, hitl, progress, dashboard-testing

Feature Branch: 036-agent-test-stabilization
Created: 2026-07-07 | Status: Draft
Input: "Stabilize the existing Gradio LangGraph agent runtime so it can support long running dashboard test scenario generation with stable dashboard UIContext, explicit scenario intent, structured progress events, draft artifact preview, recoverable agent run identifiers, and HITL confirmation for repository writes or baseline approvals."

User Scenarios

Story 1 — Dashboard Context Run Start (P1)

Why P1: Dashboard scenario generation must start from a precise dashboard context, environment, and intent without relying on user prose.

Independent Test: Open /agent from a dashboard route with query context and verify the agent run exposes the exact dashboard object, env, route, and scenario intent in debug/status metadata.

Acceptance:

1. Given a user clicks "Создать сценарий тестирования" on a dashboard When /agent opens Then UIContext contains objectType=dashboard, objectId, objectName, envId, route, contextVersion, and intent=build_dashboard_test_scenario.
2. Given the user opens /agent from two different dashboards in sequence When each run starts Then the second run uses the new dashboard context and never reuses stale objectId or envId.
3. Given malformed or oversized context is supplied When the agent run starts Then invalid context is rejected or discarded with visible recovery and audit metadata, not injected into the agent prompt.

---

Story 2 — Recoverable Long Agent Runs (P1)

Why P1: Scenario generation may inspect Superset, produce draft artifacts, and wait for user approval; it cannot be a fragile one-shot response.

Independent Test: Start a simulated long scenario run, disconnect/reload /agent, and verify the user can see run status and draft artifact references by run_id.

Acceptance:

1. Given a scenario run starts When the backend accepts the request Then the UI receives a stable agent_run_id correlated with conversation id and dashboard context.
2. Given the Gradio stream drops during a run When the user reconnects Then the latest run status, progress stage, and recoverable draft references are visible.
3. Given no first observable event arrives within the configured timeout When timeout elapses Then the UI shows actionable recovery instead of endless thinking.

---

Story 3 — Structured Progress and Draft Artifacts (P2)

Why P2: The UI must render scenario progress and artifact preview from structured events, not parse natural-language chat text.

Independent Test: Run a fixture-based scenario generation stub and verify progress stages and draft artifacts render from metadata events only.

Acceptance:

1. Given a run progresses through context, inspect, scenario, parameters, generate, validate, and save stages When events stream Then the UI updates each stage from structured metadata.
2. Given draft artifacts are produced When the agent reports them Then the user can preview artifact names, types, warnings, and validation status before any repository write.
3. Given an artifact contains warnings When preview renders Then warnings are visible and block silent save.

---

Story 4 — HITL Write and Approval Gates (P2)

Why P2: Generated files and approved baselines affect future regression truth and must never be committed by the agent without human approval.

Independent Test: Trigger a repository-write or baseline-approval action and verify confirm_required appears with target path, operation type, risks, and denial path.

Acceptance:

1. Given the agent proposes saving generated artifacts When save is requested Then a confirmation card lists target paths, file count, risk level, and warnings before the write.
2. Given the agent proposes approving a baseline candidate When approval is requested Then confirmation requires a user-visible reason and shows provenance.
3. Given the user denies a write or approval When denial is submitted Then no artifact is written or approved, and the chat records an explicit cancellation outcome.

---

Edge Cases

• JWT expires mid-run → next protected operation stops with session-expired recovery and does not retry privileged actions.
• User lacks permission to save artifacts or approve baselines → permission_denied appears instead of confirm_required.
• Multiple tabs start runs for the same dashboard → each run has an independent agent_run_id; artifact preview does not cross-contaminate.
• Existing 033/035 agent chat functions remain available → dashboard test intent does not break ordinary chat, tool cards, or confirmation flows.

Requirements

Functional

• AGSTAB-FR-001: /agent MUST accept and validate dashboard UIContext with explicit intent=build_dashboard_test_scenario while preserving backward compatibility with existing 035 uicontext payload position.
• AGSTAB-FR-002: Every long-running agent operation for dashboard testing MUST expose a stable agent_run_id correlated with conversation id, user id, dashboard context, and current progress stage.
• AGSTAB-FR-003: The agent runtime MUST emit structured progress metadata for scenario-oriented stages; UI MUST NOT infer these stages by parsing prose.
• AGSTAB-FR-004: Draft artifacts MUST be previewable before repository writes and must include type, name, intended path, validation status, warnings, and producing run id.
• AGSTAB-FR-005: Repository writes and baseline approvals MUST require HITL confirmation; approval must capture a user-supplied reason.
• AGSTAB-FR-006: Permission-denied operations MUST emit permission_denied metadata and never present a confirm button for unauthorized actions.
• AGSTAB-FR-007: The existing Gradio/LangGraph chat, streaming, confirmation, context, and guardrail flows from 033/035 MUST remain compatible.
• AGSTAB-FR-008: A live smoke test MUST verify context → stream → tool event → draft artifact preview → confirmation/denial recovery.

Key Entities

• DashboardScenarioUIContext: Versioned context payload identifying the dashboard, environment, source route, and scenario-generation intent.
• AgentRun: Recoverable execution unit for long-running agent work; owns progress stage, events, draft artifact references, and terminal status.
• ScenarioProgressEvent: Structured metadata event representing a stage or step in scenario generation.
• DraftArtifactRef: Previewable reference to generated content that is not yet persisted as an approved repository artifact.
• ApprovalGate: HITL confirmation envelope for repository writes and baseline approvals.

Success Criteria

• SC-001: 100% of dashboard scenario runs in test fixtures expose a valid agent_run_id before the first tool action.
• SC-002: Context handoff from dashboard page to /agent is verified for at least three dashboard ids with no stale-object reuse.
• SC-003: Structured progress events render all required stages without text parsing in frontend tests.
• SC-004: Repository write and baseline approval attempts are blocked until confirmation in 100% of permissioned test cases.
• SC-005: Existing 035 agent context/guardrails tests continue to pass unchanged or with documented fixture updates only.

#endregion AgentTestStabilization.Spec