ss-tools/specs/040-dashboard-load-testing/tasks.md

#region DashboardLoadTesting.Tasks [C:3] [TYPE ADR] [SEMANTICS tasks,load-testing,implementation] @BRIEF Ordered TDD backlog for dashboard load testing with bounded async workers, deterministic variations, Superset cache metadata, and 041 blast-radius snapshots.

Input: all documents in specs/040-dashboard-load-testing/ Prerequisites: clarified spec, plan, research R1–R10, data-model, contracts/modules.md, quickstart, 041 LIN-FR-013 read contract

Phase 1 — Setup: Fixtures, DTOs, Permissions

• T001 Create canonical dashboard/chart/query response fixtures under specs/040-dashboard-load-testing/fixtures/: scalar chart, bounded table chart, two filters, two viewports, repeated identical coordinates, Superset cache hit/miss/bypassed/disabled/unknown responses, and 041 shared-dataset blast-radius snapshot.
• T002 [P] Create canonical environment policy fixtures under specs/040-dashboard-load-testing/fixtures/environments/: DEV/PREPROD pool=20, PROD pool=10, pool=5 edge, stage/is_production combinations, per-env override, and reserve slots.
• T003 Materialize canonical fixtures into backend/tests/fixtures/load_testing/ and frontend/src/lib/models/fixtures/load-testing/.
• T004 [P] Define extra-forbid Pydantic DTOs in backend/src/schemas/load_testing.py for profiles, matrix preview, runs, executions, aggregates, findings, cache metadata, and typed errors.
• T005 [P] Define matching TypeScript DTOs in frontend/src/lib/types/load-testing.ts with cross-stack field parity and no SQL/raw endpoint/query_context fields.
• T006 [P] Register dashboard:loadtest:execute and dashboard:loadtest:prod in backend/src/services/rbac_permission_catalog.py; add default-deny role mappings and permission fixtures.
• T007 Add load testing navigation/action labels and state copy to frontend/src/lib/i18n/locales/ru/load-testing.json and frontend/src/lib/i18n/locales/en/load-testing.json; register locale resources without deriving state from localized strings.

Phase 2 — Foundational: Persistence, Client Capacity, Task Boundary

• T008 Write failing migration/model tests in backend/tests/models/test_load_testing.py for LoadProfile revisioning, immutable LoadVariation coordinates, terminal LoadRun status, LoadExecution provenance, aggregate uniqueness, and ConsistencyFinding linkage.
• T009 Add ORM models in backend/src/models/load_testing.py and Alembic migration under backend/alembic/versions/ for profile/run/variation/execution/aggregate/finding records and indexes.
• T010 Write failing client-registry tests in backend/tests/core/test_client_registry_load_capacity.py proving Environment.connection_pool_size semaphore is passed into AsyncAPIClient, shared ordinary/load calls respect the same semaphore, and shutdown closes the shared client.
• T011 Fix backend/src/core/utils/client_registry.py to pass the existing per-environment semaphore into AsyncAPIClient; preserve shared auth/CSRF client behavior and do not create a second client/pool. @PRE Registry client is initialized for the environment. @POST AsyncAPIClient acquires/releases the shared semaphore around every request. @INVARIANT No double-acquire path is introduced in the load runner. @TEST_EDGE pool slot exhaustion waits then releases; shutdown leaves no live client.
• T012 Write failing TaskManager boundary tests in backend/tests/services/load_testing/test_task_boundary.py proving one LoadRun creates exactly one TaskManager task, executions are not tasks, cancellation reaches the run, and task events remain aggregate-level.
• T013 Add load testing plugin/task entrypoint in backend/src/plugins/load_testing.py; create one observable TaskManager task per LoadRun and connect lifecycle cancellation/status persistence.
• T014 [P] Add belief-runtime test helpers and instrumentation fixtures under backend/tests/services/load_testing/conftest.py for belief_scope, logger.reason, logger.reflect, and terminal event coverage.

Checkpoint: Models migrate; shared client semaphore is active; one run maps to one TaskManager task; no user story execution yet.

Phase 3 — User Story 1: Configure Load Profile and Variations (P1)

Goal: Validate profile inputs, resolve effective capacity, expand deterministic variations, and preview 041 blast radius without dispatching requests. Independent Test: Fixture profile returns exact matrix/request estimate, cap notice, and dependent-dashboard snapshot; invalid axes/values are rejected with recovery markers.

Tests First

• T015 [P] [US1] Write failing profile validation tests in backend/tests/services/load_testing/test_profile.py for closed axes, typed filters, required iterations/duration, circuit-breaker bounds, max variation cap, and no raw SQL/query context.
• T016 [P] [US1] Write failing capacity tests in backend/tests/services/load_testing/test_capacity.py for DEV/PREPROD=10, PROD=5, ceiling=25, reserve=5, per-env override clamping, pool=5 rejection, and multi-run environment sharing.
• T017 [P] [US1] Write failing matrix tests in backend/tests/services/load_testing/test_matrix.py for canonical axis ordering, stable variation ids, cartesian expansion, seeded reservoir sampling, theoretical vs selected counts, and unknown-axis rejection.
• T018 [P] [US1] Write failing 041 integration contract tests in backend/tests/services/load_testing/test_blast_radius_contract.py for fingerprint pinning, dependent-dashboard projection, stale snapshot warning, and changed fingerprint requiring refresh.
• T019 [P] [US1] Write failing L1 model tests in frontend/src/lib/models/tests/Dashboards.LoadProfileModel.test.ts for idle/validating/matrix_ready/gate_required/start_error states, cap notices, NEEDS_CONTEXT, and pinned blast-radius snapshot.
• T020 [P] [US1] Write failing profile editor L2 tests in frontend/src/lib/components/dashboards/load-testing/tests/LoadProfileEditor.ux.test.ts for accessible controls, variation errors, matrix preview, cap clamp, and dependent-dashboard warning.

Implementation

• T021 [US1] Implement backend/src/services/load_testing/capacity.py. @PRE Environment stage and pool configuration are valid. @POST Returns effective cap and clamp/rejection reasons; never exceeds profile, environment, reserve-adjusted pool, or ceiling. @INVARIANT PROD=5, DEV/PREPROD=10, ceiling=25, reserve=5.
• T022 [US1] Implement backend/src/services/load_testing/matrix.py. @PRE Axes are closed and normalized. @POST Returns byte-deterministic variation matrix and request estimate; over-cap expansion is seeded and bounded. @TEST_EDGE shuffled input ordering produces identical bytes.
• T023 [US1] Implement backend/src/services/load_testing/profile.py to validate profile against 037 DashboardQueryModel and 041 pinned dependents response; no request dispatch.
• T024 [US1] Implement backend/src/api/routes/load_testing.py profile validation and matrix preview endpoints with RBAC and typed errors.
• T025 [US1] Implement frontend/src/lib/models/Dashboards.LoadProfileModel.svelte.ts and frontend/src/lib/components/dashboards/load-testing/LoadProfileEditor.svelte bound to typed API DTOs.
• T026 [US1] Add dashboard entry action and load-testing route integration in frontend/src/routes/dashboards/[id]/components/DashboardHeader.svelte and frontend/src/routes/load-testing/.

Checkpoint: Profile preview works independently; zero Superset execution requests are dispatched during configuration.

Phase 4 — User Story 2: Controlled Parallel Execution (P1)

Goal: Execute one validated matrix through bounded workers with clean ramp, steady, stop, drain, recovery, and aggregate progress. Independent Test: Concurrency=5 fixture never exceeds cap; queue/resource/upstream timings are disjoint; stop drains and preserves partial results.

Tests First

• T027 [P] [US2] Write failing worker-pool tests in backend/tests/services/load_testing/test_runner_pool.py for worker cap, per-run FIFO, shared env capacity across runs, one worker registry, and no execution TaskManager tasks.
• T028 [P] [US2] Write failing timing tests in backend/tests/services/load_testing/test_timing.py injecting queue/resource/upstream delays; assert separate timing fields and upstream-only percentile inputs.
• T029 [P] [US2] Write failing lifecycle tests in backend/tests/services/load_testing/test_run_lifecycle.py for ramp steps, steady state, user stop, bounded drain, browser-independent continuation, terminal immutability, and partial result durability.
• T030 [P] [US2] Write failing persistence batch tests in backend/tests/services/load_testing/test_persistence.py for ≤50/1s flush thresholds, aggregate updates, reconnect snapshot, and no full response blob storage.
• T031 [P] [US2] Write failing L1 model tests in frontend/src/lib/models/tests/Dashboards.LoadRunModel.test.ts for queued/ramping/steady/draining/terminal states, worker registry, queue depth, reconnect, and partial results.
• T032 [P] [US2] Write failing L2 monitor tests in frontend/src/lib/components/dashboards/load-testing/tests/LoadRunMonitor.ux.test.ts for worker status, phase badges, stop action, drain feedback, error counts, and keyboard accessibility.

Implementation

• T033 [US2] Implement backend/src/services/load_testing/timing.py with monotonic queue/resource/upstream clocks and nearest-rank percentile aggregation.
• T034 [US2] Implement backend/src/services/load_testing/runner_pool.py. @PRE Validated profile, gate if required, pinned 041 fingerprint, effective capacity. @POST In-flight workers never exceed cap; one LoadRun terminal state and durable partial results. @SIDE_EFFECT Superset chart-data calls, batched DB writes, aggregate progress events. @INVARIANT Acquire load capacity before entering shared client request; breaker/stop prevents new queue intake. @TEST_EDGE cancellation before take leaves item unexecuted; in-flight timeout drains without new dispatch.
• T035 [US2] Implement backend/src/services/load_testing/persistence.py with bounded execution batches and aggregate snapshots.
• T036 [US2] Implement backend/src/api/routes/load_testing.py start/status/stop/results endpoints and reconnect by load_run_id.
• T037 [US2] Implement frontend/src/lib/models/Dashboards.LoadRunModel.svelte.ts and frontend/src/lib/components/dashboards/load-testing/LoadRunMonitor.svelte.
• T038 [US2] Add WebSocket or existing task-event subscription binding for aggregate progress at ≤4 Hz; never emit one UI event per execution.

Checkpoint: Worker run survives browser disconnect, stops cleanly, and reports partial results without exceeding cap.

Phase 5 — User Story 3: Circuit Breaker and PROD Safety Gate (P1)

Goal: Abort degraded runs safely and prevent unapproved PROD dispatch. Independent Test: Fault injection trips breaker after minimum window; denied or stale PROD gate dispatches zero requests.

Tests First

• T039 [P] [US3] Write failing breaker tests in backend/tests/services/load_testing/test_breaker.py for minimum 20 samples, 25% error threshold, p99 multiplier, absolute p99 fallback, trigger metadata, drain, and preserved partial results.
• T040 [P] [US3] Write failing gate/RBAC tests in backend/tests/services/load_testing/test_gate.py for DEV/PREPROD permission, PROD permission, reason requirement, profile/cap/fingerprint hash binding, denial, expiry, replay, and payload mutation.
• T041 [P] [US3] Write failing frontend gate tests in frontend/src/lib/components/dashboards/load-testing/tests/LoadApprovalGate.ux.test.ts for impact summary, reason, deny, stale fingerprint, and no confirmation control on permission denial.

Implementation

• T042 [US3] Implement backend/src/services/load_testing/breaker.py. @PRE Completed execution window has policy and required sample count. @POST Tripped breaker closes intake, starts drain, records metric/threshold, and preserves partial results. @TEST_EDGE fewer than 20 samples never triggers p99 abort.
• T043 [US3] Implement backend/src/services/load_testing/gates.py using 036 ApprovalGate semantics; bind profile revision, effective cap, request estimate, environment, and 041 fingerprint.
• T044 [US3] Add PROD gate and permission guards to backend/src/api/routes/load_testing.py; revalidate hash immediately before dispatch.
• T045 [US3] Add breaker badge, approval card, and recovery states to frontend/src/lib/components/dashboards/load-testing/.
• T046 [US3] Add rejected-path regression tests proving unapproved PROD, stale gate, and arbitrary-SQL payloads dispatch zero requests.

Checkpoint: Circuit breaker and PROD safety are independently verifiable; no unauthorized request reaches Superset.

Phase 6 — User Story 4: Results, Cache, and Consistency (P2)

Goal: Report upstream latency, typed errors, authoritative cache state, bounded result metadata, and consistency findings without baseline mutation. Independent Test: Fixture run returns percentiles, cache states, divergent digest finding, and unchanged baseline catalog hash.

Tests First

• T047 [P] [US4] Write failing cache mapping tests in backend/tests/services/load_testing/test_cache_metadata.py for Superset body hit/miss/bypassed/disabled/unknown and inconsistent metadata; latency changes must not affect mapping.
• T048 [P] [US4] Write failing bounded-result tests in backend/tests/services/load_testing/test_bounded_result.py for raw SHA-256, 10k-row response, 100-row/256KiB cap, row count, and no call to full 037 normalization.
• T049 [P] [US4] Write failing consistency tests in backend/tests/services/load_testing/test_consistency.py for identical coordinate digest divergence, row-order-only allowance, and finding classification=flakiness.
• T050 [P] [US4] Write failing baseline-isolation tests in backend/tests/services/load_testing/test_baseline_isolation.py asserting catalog hash, source_response_hash, candidates, and immutability status are unchanged.
• T051 [P] [US4] Write failing results UI tests in frontend/src/lib/components/dashboards/load-testing/tests/LoadResults.ux.test.ts for p50/p90/p95/p99, cache counts, errors, consistency finding detail, and bounded samples.

Implementation

• T052 [US4] Implement backend/src/services/load_testing/cache_metadata.py using Superset response-body fields: is_cached, cache_key, cached_dttm, queried_dttm, cache_timeout. @POST Maps to hit/miss/bypassed/disabled/unknown with cache_state_source="superset_response". @INVARIANT Latency inference forbidden.
• T053 [US4] Implement backend/src/services/load_testing/bounded_result.py. @POST Raw digest, row count, sample ≤100 rows/256KiB; full 037 normalization not invoked. @TEST_EDGE GIL/event-loop distortion >10% requires escalation, not silent ProcessPool introduction.
• T054 [US4] Implement backend/src/services/load_testing/aggregates.py and consistency finding persistence; use digest-based comparison and separate error rates.
• T055 [US4] Implement frontend/src/lib/components/dashboards/load-testing/LoadResults.svelte and LoadComparison.svelte.
• T056 [US4] Add baseline read-only boundary assertions to backend/src/services/load_testing/ and expose explicit “baselines not modified” result state.

Checkpoint: Results and cache provenance are queryable; baseline catalog remains byte-identical.

Phase 7 — User Story 5: Blast-Radius Comparison and Scheduling (P2)

Goal: Compare runs and expose shared-dataset impact with the 041 pinned read model; warn about overlap windows. Independent Test: Two same-profile runs plus dependent-dashboard probe show latency deltas, cache metadata, and probe coverage bound to fingerprints.

Tests First

• T057 [P] [US5] Write failing comparison tests in backend/tests/services/load_testing/test_comparison.py for complete/partial runs, per-chart deltas, cache-state attribution, and consistency-finding deltas.
• T058 [P] [US5] Write failing blast-radius scheduling tests in backend/tests/services/load_testing/test_schedule_policy.py for dependent deployment/maintenance overlap, fingerprint pinning, and warning/block policy.
• T059 [P] [US5] Write failing comparison UI tests in frontend/src/lib/components/dashboards/load-testing/tests/LoadComparison.ux.test.ts for run selector, latency deltas, cache annotations, dependent dashboards, and probe coverage.

Implementation

• T060 [US5] Implement backend/src/services/load_testing/comparison.py with partial-run-compatible aggregates and cache-state provenance.
• T061 [US5] Implement backend/src/services/load_testing/schedule_policy.py; query 041 dependent dashboard windows and block/warning conflicts per policy.
• T062 [US5] Add comparison and blast-radius endpoints to backend/src/api/routes/load_testing.py.
• T063 [US5] Complete frontend/src/lib/components/dashboards/load-testing/LoadComparison.svelte and blast-radius result panels.

Checkpoint: Same-profile comparison works with partial current runs; blast-radius scope remains pinned and visible.

Phase 8 — Integration, Accessibility, and Quality Gates

• T064 Add frontend/src/lib/models/tests/Dashboards.LoadModels.integration.test.ts covering profile → gate → run → reconnect → results.
• T065 Add frontend/e2e/tests/dashboard-load-testing.e2e.js covering dashboard entry → matrix preview → approval/deny → worker monitor → stop/reconnect → results.
• T066 Add backend/tests/integration/test_dashboard_load_testing_superset.py using Superset chart-data fixtures or Testcontainers; verify response cache metadata is preserved.
• T067 Add backend/tests/integration/test_load_testing_client_capacity.py proving shared semaphore wiring, reserve slots, multiple runs, and ordinary request fairness.
• T068 Add frontend responsive and accessibility tests for 1366×768 and narrow viewport profile/monitor/results screens.
• T069 Run quickstart.md steps 1–10 and record results.
• T070 Run backend load-testing tests, existing client-registry/migration/task regressions, and python -m ruff check ..
• T071 Run frontend tests, lint, build, and Playwright.
• T072 [P] Audit rejected paths: direct SQL, HTTP-only cap, per-execution TaskManager tasks, latency cache inference, full normalization in load path, baseline mutation, unapproved PROD dispatch.
• T073 [P] Audit ATTN_1–4, exact anchor pairs, C4/C5 @RATIONALE/@REJECTED, belief-runtime markers, unresolved relations, and 041 fingerprint integration.
• T074 [P] Update 041/039 traceability notes with LoadRun/FleetReport consumer links after contracts are implemented.

Phase 9 — MVP Gap Closure (audit 2026-08-07)

Context: Задачи выше закрыты, но факт-чекинг кода выявил, что run_load_run в backend/src/plugins/load_testing.py не вызывает RunnerPool — он переводит фазы RAMP→STEADY→DRAIN→COMPLETED без единого реального Superset-запроса. Это нарушает LOAD-FR-001 (обязанность идти через 037 executor) и делает фичу «нагрузка без нагрузки». Ниже — обязательные задачи закрытия.

• T075 [P] [US2] Wire RunnerPool into run_load_run: in backend/src/plugins/load_testing.py, replace the hold/sleep body with RunnerPool(capacity=run.effective_concurrency, executor=<037 query executor>, env_capacity_semaphore=<shared per-env semaphore>, breaker=<run breaker>, on_result=<persist LoadExecution>) and drain execution_specs through it. @POST: each queued execution produces a LoadExecution row via the 037 executor; in-flight never exceeds cap; phase transitions preserved. @TEST: backend/tests/services/load_testing/test_run_lifecycle.py — assert executor called once per item with real chart-data context. DONE: run_load_run now calls _resolve_execution_scope → _run_bounded_pool (RunnerPool wired). Verified by test_executor_runtime.py::TestRunLoadRunRealExecution::test_run_real_execution_persists_load_executions (2 items → 2 LoadExecution rows, run → COMPLETED).
• T076 [P] [US2] Add the 037-native executor adapter backend/src/services/load_testing/executor.py: async def execute_superset_chart(env, chart_id, filters_context) -> LoadExecution delegating to src.core.superset_client._chart_data.execute_chart_data + bounded normalization (LOAD-FR-018). @REJECTED: direct HTTP/SQL in the load path — must reuse 037 SupersetClient.ChartData.Execute. @TEST: fixture chart-data response → bounded digest + cache metadata preserved (reuses 037 fixtures). DONE: executor.py created (execute_superset_chart via execute_dashboard_query_envelope, build_execution_items). Tested in test_executor_runtime.py (delegation + missing chart/dataset raise + deterministic items).
• T077 [P] [US2] Share the per-environment capacity semaphore with the existing client_registry admission so load runs respect the same pool as ordinary requests (LOAD-FR-002 note in 040 traceability). @TEST: concurrent ordinary request + load run never exceed env cap (extend test_load_testing_client_capacity.py). DONE: _run_bounded_pool acquires client_registry.get_semaphore(env) and passes it as env_capacity_semaphore to RunnerPool — same shared per-env limit as ordinary calls.
• T078 [P] [US3] Wire the run-level circuit breaker into run_load_run evaluation between executions: abort to circuit_breaker_abort with partial results preserved when thresholds breach (LOAD-FR-004). @TEST: fault-injection run trips breaker mid-drain; partial LoadExecutions remain queryable. DONE: _build_breaker returns a CircuitBreaker; _run_bounded_pool records each outcome/latency; run_load_run transitions to CIRCUIT_BREAKER_ABORT when breaker.tripped. Partial executions persist before the terminal transition.
• T079 [P] [US4] Persist per-execution results from on_result into the LoadExecution store so /status and /compare return real latency/cache/consistency data instead of seeded aggregates. @TEST: completed run exposes per-chart p50/p90 + cache-state; consistency violation detected on divergent hashes. DONE: write_load_executions (batched LoadExecution persist) added to persistence.py; _run_bounded_pool drains results through it. Verified by test_run_real_execution_persists_load_executions.

Dependencies

T001–T007 → T008–T014 foundational → US1 → US2 → US3/US4; US5 follows US4 and 041 read-model availability. Phase 8 follows all stories. Phase 9 (T075–T079) is the MVP runtime closure: it must be merged before 040 can be claimed operational. Tests precede implementation within each story. T011 must pass before claiming any environment capacity result. T047/T052 require the 037 chart-data adapter to preserve cache metadata.

Parallel Opportunities

• T002 ∥ T004 ∥ T005 ∥ T006 ∥ T007
• T008 ∥ T010 ∥ T012 ∥ T014
• T015 ∥ T016 ∥ T017 ∥ T018 ∥ T019 ∥ T020
• T027 ∥ T028 ∥ T029 ∥ T030 ∥ T031 ∥ T032
• T039 ∥ T040 ∥ T041
• T047 ∥ T048 ∥ T049 ∥ T050 ∥ T051
• T057 ∥ T058 ∥ T059
• T064 ∥ T066 ∥ T067 ∥ T068

Story Verification Criteria

Story	Verification
US1	Matrix/cap/041 preview; no dispatch; deterministic bytes; quickstart 1–2
US2	Worker cap, disjoint timings, stop/drain/reconnect; quickstart 3, 6, 10
US3	Breaker and PROD gate; zero dispatch on deny/stale; quickstart 7–8
US4	Cache body mapping, bounded digest/sample, consistency, baseline hash; quickstart 4–5, 9
US5	Partial comparison, cache attribution, dependent-dashboard probe coverage, schedule overlap

Rejected-Path Coverage

• Direct SQL/raw query context: T046, T072.
• HTTP pool as sole concurrency control: T016, T027, T067, T072.
• One TaskManager task per execution: T012, T027, T072.
• Latency-based cache inference: T047, T072.
• Full 037 normalization in load path: T048, T053, T072.
• Baseline/catalog mutation: T050, T056, T072.
• Unapproved PROD dispatch: T040, T046, T072.

#endregion DashboardLoadTesting.Tasks