busya
ac95beb1a0
Fact-check the dashboard-testing spec packages against actual code and amend the full speckit document set (spec, plan, research, tasks, traceability, quickstart) so documented status matches reality: - 037: discrete metric tools work, but deploy hooks do not create VerificationRun and GET read-API endpoints are missing (T080-T081) - 038: compiler/validator work; VLM _default_submit and capture dispatch remain runtime stubs that never call LLMClient/ScreenshotService (T057-T059) - 039: UI components exist, but api/dashboard-testing.ts is unbound and pipeline views are not wired to pages; depends on 037 read-API (T054-T058) - 040: run_load_run never invokes RunnerPool, so load runs execute zero Superset requests; must wire 037 executor (T075-T079) - 041: backend index works, but no /lineage frontend and lineage_index stays opt-in (T045-T048) - 036: confirmed operational, relations to reused plugin modules fixed 19 open closure tasks total; region pairs balanced.
150 lines
9.6 KiB
Markdown
150 lines
9.6 KiB
Markdown
# Implementation Plan: Dashboard Load Testing
|
||
|
||
**Branch**: `040-dashboard-load-testing` | **Date**: 2026-07-22 | **Spec**: [spec.md](spec.md)
|
||
**Input**: Clarified feature specification from `/specs/040-dashboard-load-testing/spec.md`
|
||
|
||
## Summary
|
||
|
||
Build a dashboard load-testing surface around bounded async workers, not raw HTTP connection count. A LoadRun is one TaskManager task with a per-run execution queue; an environment capacity registry and semaphore coordinate all active runs. The runner uses the 037 Superset-native chart-data path, preserves Superset cache metadata from the JSON response, computes queue/resource/upstream timings separately, uses bounded result processing to avoid GIL distortion, and stores a pinned 041 blast-radius snapshot for configuration and PROD approval. Load runs are strictly isolated from baseline mutation.
|
||
|
||
## Technical Context
|
||
|
||
**Language/Version**: Python 3.13+ backend; TypeScript frontend with Svelte 5 runes-only
|
||
**Primary Dependencies**: FastAPI 0.126, SQLAlchemy 2.0.45, APScheduler 3.11.2 for scheduled entry boundaries, httpx via existing AsyncAPIClient, Pydantic 2.x; SvelteKit 2.49, Svelte 5.56, Vite, Tailwind; no new runtime dependency
|
||
**Storage**: PostgreSQL 16; new load profile/run/execution/aggregate/finding tables plus existing TaskManager persistence; no baseline catalog writes
|
||
**Testing**: pytest unit/contract/integration tests; Vitest L1 Screen Model tests and L2 UX tests; Playwright for worker monitor/profile flow; ruff and frontend lint
|
||
**Target Platform**: Linux Docker deployment, modern desktop browsers
|
||
**Project Type**: FastAPI REST/WebSocket backend + SvelteKit SPA frontend
|
||
**Frontend Architecture**: `.svelte.ts` Screen Models, `$state/$derived/$effect`, typed DTOs, components bind to models; ordinary AgentChat and 036 AgentRun remain separate
|
||
**Performance Goals**: upstream p95/p99 measured independently from queue/resource wait; progress events ≤4 Hz; result flush ≤50 rows or 1s; no more than effective worker cap in flight; profile/matrix preview <200ms after metadata available
|
||
**Constraints**: PROD default cap=5, DEV/PREPROD=10, absolute ceiling=25, reserve 5 shared pool slots; direct SQL/raw query context forbidden; bounded sample 100 rows/256 KiB; 041 fingerprint required for blast-radius binding; RBAC default-deny; no implementation phase in this planning request
|
||
**Scale/Scope**: 100 dashboards/environment, 300 charts, 25 maximum load workers per environment, 500 selected variation combinations per profile by default
|
||
|
||
## Constitution Check
|
||
|
||
*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.* — **PASS with one explicit implementation prerequisite**
|
||
|
||
| Principle | Status | Evidence / Guardrail |
|
||
|---|---|---|
|
||
| I. Semantic Contract First | ✅ | C3–C5 contracts in `contracts/modules.md`, hierarchical `LoadTesting.*` IDs and cross-stack edges |
|
||
| II. Decision Memory | ✅ | Spec `@RATIONALE/@REJECTED`; research R1–R10; HTTP pool, per-execution tasks, latency inference, full normalization rejected |
|
||
| III. External Orchestrator | ✅ | 037 Superset-native adapter remains sole execution boundary; no Superset plugin or SQL path |
|
||
| IV. Module Discipline | ✅ | Runner, capacity, cache, bounded processing, breaker separated; target each module <400 LOC and CC≤10 |
|
||
| V. RBAC Enforcement | ✅ | `dashboard:loadtest:execute` and `dashboard:loadtest:prod`; gate hash includes profile/cap/fingerprint |
|
||
| VI. Svelte 5 Runes Only | ✅ | LoadProfileModel/LoadRunModel `.svelte.ts`, no legacy stores in new surface |
|
||
| VII. Test-Driven C3+ | ✅ | Quickstart is falsifiable; C4/C5 contracts require rejection-path tests and GIL/cache/cap edge tests |
|
||
| VIII. Attention Optimization | ✅ | Anchor density, dotted IDs, shared `load-testing` semantic tags, bounded contract blocks |
|
||
| Repository prerequisite | ⚠️ MUST FIX DURING IMPLEMENTATION | `client_registry.get_client()` creates per-env semaphore but does not pass it to `AsyncAPIClient`; without this, `connection_pool_size` is declarative only. This is a foundational task, not a silent assumption. |
|
||
|
||
**Post-Phase-1 re-check**: PASS — the prerequisite is explicitly represented in traceability and must be verified before load execution.
|
||
|
||
## Project Structure
|
||
|
||
### Documentation
|
||
|
||
```text
|
||
specs/040-dashboard-load-testing/
|
||
├── spec.md
|
||
├── ux_reference.md
|
||
├── checklists/requirements.md
|
||
├── plan.md
|
||
├── research.md
|
||
├── data-model.md
|
||
├── contracts/modules.md
|
||
├── quickstart.md
|
||
└── traceability.md
|
||
```
|
||
|
||
### Source Code (implementation target; not changed in this planning phase)
|
||
|
||
```text
|
||
backend/src/
|
||
├── models/load_testing.py
|
||
├── schemas/load_testing.py
|
||
├── services/load_testing/
|
||
│ ├── capacity.py
|
||
│ ├── profile.py
|
||
│ ├── matrix.py
|
||
│ ├── runner_pool.py
|
||
│ ├── timing.py
|
||
│ ├── cache_metadata.py
|
||
│ ├── bounded_result.py
|
||
│ ├── breaker.py
|
||
│ ├── aggregates.py
|
||
│ └── comparison.py
|
||
├── api/routes/load_testing.py
|
||
├── core/utils/client_registry.py # pass existing semaphore into AsyncAPIClient
|
||
└── plugins/load_testing.py # one TaskManager task per LoadRun
|
||
|
||
frontend/src/
|
||
├── lib/types/load-testing.ts
|
||
├── lib/api/load-testing.ts
|
||
├── lib/models/Dashboards.LoadProfileModel.svelte.ts
|
||
├── lib/models/Dashboards.LoadRunModel.svelte.ts
|
||
├── lib/components/dashboards/load-testing/
|
||
└── routes/load-runs/[id]/
|
||
```
|
||
|
||
**Structure Decision**: Backend domain under `services/load_testing/`, separate from 037 correctness engine. `core/utils/client_registry.py` gets only the semaphore wiring prerequisite; no shared-client replacement because auth/CSRF session continuity is an existing invariant. Frontend is a dedicated workspace/monitor, not Agent Workspace.
|
||
|
||
## Phase 0 Outputs
|
||
|
||
| Artifact | Decision |
|
||
|---|---|
|
||
| `research.md` R1 | Async worker queue + env capacity registry; one TaskManager task per run |
|
||
| R2 | Effective cap derives from stage, override, pool size minus 5 reserved slots, ceiling 25; existing semaphore wiring prerequisite |
|
||
| R3 | Separate queue/resource/upstream timing; upstream percentiles only |
|
||
| R4 | Superset response-body cache metadata authoritative |
|
||
| R5 | Raw digest + bounded sample/count; no full 037 normalization in load path |
|
||
| R6 | Batched persistence and ≤4 Hz progress |
|
||
| R7 | Minimum-20 rolling breaker window; error/p99 thresholds |
|
||
| R8 | Deterministic matrix and seeded sampling |
|
||
| R9 | Pinned 041 blast-radius snapshot |
|
||
| R10 | Stage/RBAC/approval gate hash semantics |
|
||
|
||
## Phase 1 Outputs
|
||
|
||
| Artifact | Coverage |
|
||
|---|---|
|
||
| `data-model.md` | Profile, variation, run, execution, aggregates, findings, breaker, blast radius, invariants |
|
||
| `contracts/modules.md` | Backend C3–C5 contracts and frontend Screen Models with UX states |
|
||
| `quickstart.md` | 10-step falsifiable verification sequence |
|
||
| `traceability.md` | LOAD-FR-001..019 to contracts and tests; cross-spec dependencies |
|
||
|
||
## Complexity Tracking
|
||
|
||
No constitution violations. The repository semaphore wiring is an explicit prerequisite, not a justification for bypassing the constitution.
|
||
|
||
## MVP Runtime Closure (audit 2026-08-07)
|
||
|
||
**Status correction**: Факт-чекинг кода показал, что `run_load_run` в `backend/src/plugins/load_testing.py` **не вызывает `RunnerPool`** — фазы RAMP→STEADY→DRAIN→COMPLETED проходят без единого Superset-запроса; `execute_chart_data` из 037 в load-пути не используется (нарушение LOAD-FR-001). Задачи T001–T074 покрывают модули и API, но runtime-исполнение не замкнуто.
|
||
|
||
**Closure tasks (tasks.md Phase 9, T075–T079):**
|
||
- T075 — Wire `RunnerPool` into `run_load_run` (executor + env semaphore + breaker + on_result persist).
|
||
- T076 — 037-native executor adapter `services/load_testing/executor.py` (delegates to `SupersetClient.ChartData.Execute`, bounded normalization).
|
||
- T077 — Share per-env capacity semaphore with `client_registry` admission.
|
||
- T078 — Circuit breaker evaluation between executions → `circuit_breaker_abort` with preserved partials.
|
||
- T079 — Persist per-execution LoadExecution so `/status` and `/compare` return real data.
|
||
|
||
**Exit rule**: 040 не объявляется operational, пока quickstart шаги 3–7 не выполняются против реального (или фикстурного, но через executor) Superset-потока, а не hold/sleep-перехода.
|
||
|
||
## ADR Continuity
|
||
|
||
- ADR-0001: `services/load_testing/`, `models/load_testing.py`, `api/routes/load_testing.py` follow module layout.
|
||
- ADR-0003: all Superset interaction remains in existing client boundary.
|
||
- ADR-0005: new permissions are explicit and default-deny; PROD uses 036 gate.
|
||
- ADR-0006: new UI uses Svelte 5 Runes and model-first components.
|
||
- ADR-0011: async worker execution stays on FastAPI event loop; no `asyncio.run()` and no blocking `AsyncJobRunner.run()` from the event-loop thread.
|
||
- 036/037/041 continuity: one TaskManager run, 037 chart-data semantics, 041 fingerprint pinning; no baseline mutation.
|
||
|
||
## Verification Gates (implementation phase only)
|
||
|
||
1. Backend load-testing unit/contract tests and existing client-registry regressions.
|
||
2. `python -m ruff check .`.
|
||
3. Frontend Vitest/L2 tests, lint, build, and Playwright profile/run monitor flow.
|
||
4. Quickstart steps 1–10.
|
||
5. Semantic contract audit and runtime instrumentation audit for C4/C5 runner/breaker flows.
|
||
6. **MVP gate (new)**: run lifecycle test asserts executor invoked per queued item against real chart-data context; breaker abort preserves partials.
|
||
|
||
#endregion DashboardLoadTesting.Plan
|