busya
b95df37cd5
- Move agent code from backend/src/agent/ to agent/src/ss_tools/agent/ - Extract shared stdlib-only utilities to shared/src/ss_tools/shared/ - Add #region/#endregion contracts to all ~140 functions (INV_1 compliance) - Update docker files, entrypoint, build scripts for new package layout - Backend now imports ss_tools.shared._llm_health (no gradio/langchain deps) - Add specs for 036-039 feature plans
108 lines
8.4 KiB
Markdown
108 lines
8.4 KiB
Markdown
#region AgentTestStabilization.Spec [C:3] [TYPE ADR] [SEMANTICS spec,requirements,agent,test-scenarios,stabilization]
|
|
@BRIEF Stabilize the existing Gradio/LangGraph agent runtime so it can safely support long-running dashboard test scenario generation.
|
|
@RELATION DEPENDS_ON -> [ADR-0001]
|
|
@RELATION DEPENDS_ON -> [ADR-0003]
|
|
@RELATION DEPENDS_ON -> [ADR-0005]
|
|
@RELATION DEPENDS_ON -> [ADR-0006]
|
|
@RELATION DEPENDS_ON -> [ADR-0008]
|
|
@RATIONALE Dashboard test scenario generation produces durable artifacts and baseline approval flows, so the agent must be a recoverable execution surface rather than a best-effort chat stream.
|
|
@REJECTED Starting dashboard scenario generation directly on top of unstabilized chat behavior — rejected because failures would be indistinguishable across Gradio transport, context passing, artifact preview, and HITL gates.
|
|
|
|
## Navigation (DSA Indexer keywords)
|
|
@SEMANTICS: spec, requirements, feature, agent, gradio, langgraph, scenario, artifact, hitl, progress, dashboard-testing
|
|
|
|
**Feature Branch**: `036-agent-test-stabilization`
|
|
**Created**: 2026-07-07 | **Status**: Draft
|
|
**Input**: "Stabilize the existing Gradio LangGraph agent runtime so it can support long running dashboard test scenario generation with stable dashboard UIContext, explicit scenario intent, structured progress events, draft artifact preview, recoverable agent run identifiers, and HITL confirmation for repository writes or baseline approvals."
|
|
|
|
## User Scenarios
|
|
|
|
### Story 1 — Dashboard Context Run Start (P1)
|
|
|
|
**Why P1**: Dashboard scenario generation must start from a precise dashboard context, environment, and intent without relying on user prose.
|
|
|
|
**Independent Test**: Open `/agent` from a dashboard route with query context and verify the agent run exposes the exact dashboard object, env, route, and scenario intent in debug/status metadata.
|
|
|
|
**Acceptance**:
|
|
1. **Given** a user clicks "Создать сценарий тестирования" on a dashboard **When** `/agent` opens **Then** `UIContext` contains `objectType=dashboard`, `objectId`, `objectName`, `envId`, `route`, `contextVersion`, and `intent=build_dashboard_test_scenario`.
|
|
2. **Given** the user opens `/agent` from two different dashboards in sequence **When** each run starts **Then** the second run uses the new dashboard context and never reuses stale `objectId` or `envId`.
|
|
3. **Given** malformed or oversized context is supplied **When** the agent run starts **Then** invalid context is rejected or discarded with visible recovery and audit metadata, not injected into the agent prompt.
|
|
|
|
---
|
|
|
|
### Story 2 — Recoverable Long Agent Runs (P1)
|
|
|
|
**Why P1**: Scenario generation may inspect Superset, produce draft artifacts, and wait for user approval; it cannot be a fragile one-shot response.
|
|
|
|
**Independent Test**: Start a simulated long scenario run, disconnect/reload `/agent`, and verify the user can see run status and draft artifact references by `run_id`.
|
|
|
|
**Acceptance**:
|
|
1. **Given** a scenario run starts **When** the backend accepts the request **Then** the UI receives a stable `agent_run_id` correlated with conversation id and dashboard context.
|
|
2. **Given** the Gradio stream drops during a run **When** the user reconnects **Then** the latest run status, progress stage, and recoverable draft references are visible.
|
|
3. **Given** no first observable event arrives within the configured timeout **When** timeout elapses **Then** the UI shows actionable recovery instead of endless thinking.
|
|
|
|
---
|
|
|
|
### Story 3 — Structured Progress and Draft Artifacts (P2)
|
|
|
|
**Why P2**: The UI must render scenario progress and artifact preview from structured events, not parse natural-language chat text.
|
|
|
|
**Independent Test**: Run a fixture-based scenario generation stub and verify progress stages and draft artifacts render from metadata events only.
|
|
|
|
**Acceptance**:
|
|
1. **Given** a run progresses through context, inspect, scenario, parameters, generate, validate, and save stages **When** events stream **Then** the UI updates each stage from structured metadata.
|
|
2. **Given** draft artifacts are produced **When** the agent reports them **Then** the user can preview artifact names, types, warnings, and validation status before any repository write.
|
|
3. **Given** an artifact contains warnings **When** preview renders **Then** warnings are visible and block silent save.
|
|
|
|
---
|
|
|
|
### Story 4 — HITL Write and Approval Gates (P2)
|
|
|
|
**Why P2**: Generated files and approved baselines affect future regression truth and must never be committed by the agent without human approval.
|
|
|
|
**Independent Test**: Trigger a repository-write or baseline-approval action and verify `confirm_required` appears with target path, operation type, risks, and denial path.
|
|
|
|
**Acceptance**:
|
|
1. **Given** the agent proposes saving generated artifacts **When** save is requested **Then** a confirmation card lists target paths, file count, risk level, and warnings before the write.
|
|
2. **Given** the agent proposes approving a baseline candidate **When** approval is requested **Then** confirmation requires a user-visible reason and shows provenance.
|
|
3. **Given** the user denies a write or approval **When** denial is submitted **Then** no artifact is written or approved, and the chat records an explicit cancellation outcome.
|
|
|
|
---
|
|
|
|
### Edge Cases
|
|
- JWT expires mid-run → next protected operation stops with session-expired recovery and does not retry privileged actions.
|
|
- User lacks permission to save artifacts or approve baselines → `permission_denied` appears instead of `confirm_required`.
|
|
- Multiple tabs start runs for the same dashboard → each run has an independent `agent_run_id`; artifact preview does not cross-contaminate.
|
|
- Existing 033/035 agent chat functions remain available → dashboard test intent does not break ordinary chat, tool cards, or confirmation flows.
|
|
|
|
## Requirements
|
|
|
|
### Functional
|
|
|
|
- **AGSTAB-FR-001**: `/agent` MUST accept and validate dashboard `UIContext` with explicit `intent=build_dashboard_test_scenario` while preserving backward compatibility with existing 035 `uicontext` payload position.
|
|
- **AGSTAB-FR-002**: Every long-running agent operation for dashboard testing MUST expose a stable `agent_run_id` correlated with conversation id, user id, dashboard context, and current progress stage.
|
|
- **AGSTAB-FR-003**: The agent runtime MUST emit structured progress metadata for scenario-oriented stages; UI MUST NOT infer these stages by parsing prose.
|
|
- **AGSTAB-FR-004**: Draft artifacts MUST be previewable before repository writes and must include type, name, intended path, validation status, warnings, and producing run id.
|
|
- **AGSTAB-FR-005**: Repository writes and baseline approvals MUST require HITL confirmation; approval must capture a user-supplied reason.
|
|
- **AGSTAB-FR-006**: Permission-denied operations MUST emit `permission_denied` metadata and never present a confirm button for unauthorized actions.
|
|
- **AGSTAB-FR-007**: The existing Gradio/LangGraph chat, streaming, confirmation, context, and guardrail flows from 033/035 MUST remain compatible.
|
|
- **AGSTAB-FR-008**: A live smoke test MUST verify context → stream → tool event → draft artifact preview → confirmation/denial recovery.
|
|
|
|
### Key Entities
|
|
|
|
- **DashboardScenarioUIContext**: Versioned context payload identifying the dashboard, environment, source route, and scenario-generation intent.
|
|
- **AgentRun**: Recoverable execution unit for long-running agent work; owns progress stage, events, draft artifact references, and terminal status.
|
|
- **ScenarioProgressEvent**: Structured metadata event representing a stage or step in scenario generation.
|
|
- **DraftArtifactRef**: Previewable reference to generated content that is not yet persisted as an approved repository artifact.
|
|
- **ApprovalGate**: HITL confirmation envelope for repository writes and baseline approvals.
|
|
|
|
## Success Criteria
|
|
|
|
- **SC-001**: 100% of dashboard scenario runs in test fixtures expose a valid `agent_run_id` before the first tool action.
|
|
- **SC-002**: Context handoff from dashboard page to `/agent` is verified for at least three dashboard ids with no stale-object reuse.
|
|
- **SC-003**: Structured progress events render all required stages without text parsing in frontend tests.
|
|
- **SC-004**: Repository write and baseline approval attempts are blocked until confirmation in 100% of permissioned test cases.
|
|
- **SC-005**: Existing 035 agent context/guardrails tests continue to pass unchanged or with documented fixture updates only.
|
|
|
|
#endregion AgentTestStabilization.Spec
|