busya/ss-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
455a56856feea2b6cc1e9dc0e7caf0f922d0e94f
ss-tools/specs/040-dashboard-load-testing/spec.md

21 KiB
Raw Blame History

#region DashboardLoadTesting.Spec [C:3] [TYPE ADR] [SEMANTICS spec,requirements,load-testing,dashboard-testing,variations] @BRIEF Load testing interface for dashboards: bounded parallel execution of Superset-native checks across declarative variation axes, with circuit breaker, latency metrics, consistency detection, and dataset blast-radius awareness. @RELATION DEPENDS_ON -> [Doc.Adr.ADR0001] @RELATION DEPENDS_ON -> [Doc.Adr.ADR0003] @RELATION DEPENDS_ON -> [Doc.Adr.ADR0005] @RELATION DEPENDS_ON -> [Doc.Adr.ADR0006] @RELATION DEPENDS_ON -> [AgentTestStabilization.Spec] @RELATION DEPENDS_ON -> [SupersetBaselineEngine.Spec] @RATIONALE Dashboard correctness (037) proves a query returns the right value once; it does not prove the dashboard survives concurrent load, nor that repeated identical executions return identical results. A dedicated load surface reuses the 037 Superset-native executor so load fidelity matches production execution semantics — a separate load stack would diverge exactly like the rejected direct-SQL path. @REJECTED Treating load runs as 036 AgentRun instances — rejected because load execution is non-conversational, fan-out by design, and would pollute run/recovery semantics with thousands of pseudo-conversations. @REJECTED Writing load results into the 037 baseline catalog — rejected because load executions measure latency and consistency, not truth; polluting baselines with load samples would corrupt immutability detection. @REJECTED Unbounded client-declared concurrency — rejected because a single misconfigured run could saturate the Superset/KXD connection pool and degrade production for all users.

Navigation (DSA Indexer keywords)

@SEMANTICS: spec, requirements, feature, load-testing, concurrency, variations, circuit-breaker, latency, blast-radius, dashboard-testing

Feature Branch: 040-dashboard-load-testing Created: 2026-07-22 | Status: Draft Input: "Интерфейс для нагрузочного тестирования дашбордов — параллельный запуск нескольких проверок на одном дашборде + вариации. Учитывает blast-radius: общий dataset, cache warming одного дашборда влияет на latency зависимых дашбордов."

Clarifications

Session 2026-07-22

  • Q1 (execution model): Worker pool, not HTTP-connection cap. Per-run async worker queue + env-level semaphore (sum of workers across active runs per env ≤ max_concurrent_per_env). Workers check circuit breaker between executions → clean drain. Run registered as ONE TaskManager task (observable in Task Center); executions are not tasks. Pool-wait = explicit queue-wait metric. → LOAD-FR-002/016/017 added.
  • Q2 (concurrency caps): prod=5, default (dev/preprod)=10, absolute ceiling=25 (not client-overridable); per-env override load_test_max_concurrent in environment config, clamped to ceiling. Ramp defaults: prod 1→2→3→5; others 1→3→5→10. → LOAD-FR-002 amended.
  • Q3 (GIL analysis): Async single-loop model kept — at cap 25 and Superset latency 0.5–3s, workers are ~95–99% I/O-bound (~40% of one core CPU). GIL contention risk exists only for large table charts (10k+ rows, ~200–500ms continuous parse+normalize). Mitigation B1+B2: load path uses bounded normalization (response sha256 hash + row count + first-N-row sample for display; hash suffices for consistency checks); full 037 normalization stays in the verification path. ProcessPool offload (B3) deferred unless fixtures show >10% GIL distortion. → LOAD-FR-018 added.
  • Q4 (cache state, source-audited): Authoritative source is Superset /api/v1/chart/data JSON body (result[].is_cached, cache_key, cached_dttm, queried_dttm, cache_timeout) — verified against /home/busya/dev/superset/superset/common/query_context_processor.py, charts/data/api.py, and charts/schemas.py. State mapping: hit if is_cached=true; bypassed if request.force=true; disabled if cache_timeout=-1; miss if is_cached=null + cached_dttm=null + force=false; unknown if fields absent/inconsistent. Latency-based probe inference rejected (DB buffer cache/network jitter are confounders). → LOAD-FR-012 amended, LOAD-FR-019 added.

User Scenarios

Story 1 — Configure Load Profile With Variations (P1)

Why P1: Analysts must declare what to load (dashboard, charts), how hard (concurrency, ramp), and across which axes (filters, viewport, role) before any execution.

Independent Test: Build a profile for a fixture dashboard with 2 variation axes and verify the preview shows exact request count, chart coverage, and blast-radius warning listing dependent dashboards sharing datasets.

Acceptance:

  1. Given a dashboard and environment When a load profile is configured Then the user selects concurrency target (bounded by server cap), execution mode (iterations or duration), ramp-up strategy, and variation axes (filters, viewport, role, time range).
  2. Given filter variations are declared When the profile is validated Then every variation value is checked against authoritative Superset filter metadata; unknown values produce NEEDS_CONTEXT markers, never invented values.
  3. Given the dashboard's charts share datasets with other dashboards When the profile preview renders Then a blast-radius panel lists dependent dashboard count and warns that cache warming/clearing affects them.

Story 2 — Controlled Parallel Execution (P1)

Why P1: Load execution must be bounded, observable, and stoppable — never a fire-and-forget request flood.

Independent Test: Run a fixture load profile with concurrency=5 and verify in-flight count never exceeds the cap, ramp-up is staged, and live progress shows running/completed/failed per variation.

Acceptance:

  1. Given a started load run When execution proceeds Then in-flight requests never exceed the server-enforced concurrency cap for that environment, and ramp-up increases load in declared steps.
  2. Given a running load run When the user requests stop Then in-flight requests complete or time out within a bounded drain window, no new requests are dispatched, and the run terminates with a stopped_by_user status.
  3. Given executions hit Superset When responses return Then every result is recorded with chart id, variation coordinates, latency, and typed error taxonomy (403/404/422/5xx/timeout) inherited from 037.

Story 3 — Circuit Breaker and PROD Safety Gate (P1)

Why P1: A load run against degraded infrastructure must self-terminate before it amplifies an incident; PROD runs require explicit human approval.

Independent Test: Simulate error-rate exceeding threshold mid-run and verify automatic abort with typed terminal reason; verify PROD runs cannot start without a 036-style approval gate.

Acceptance:

  1. Given error rate exceeds the declared threshold OR p99 latency exceeds N× the profile baseline When the circuit breaker evaluates Then the run aborts with terminal status circuit_breaker_abort including the triggering metric and threshold.
  2. Given a load profile targets a PROD-classified environment When start is requested Then an approval gate shows concurrency ceiling, estimated request volume, blast-radius dependents, and requires a user-supplied reason.
  3. Given the user denies the PROD gate When denial is submitted Then no request is dispatched and the denial is recorded in the run audit trail.

Story 4 — Results: Latency, Errors, Consistency (P2)

Why P2: Load value is in the analysis — percentiles per chart, error breakdown, and detection of non-deterministic results under parallelism.

Independent Test: Complete a fixture run with injected latency variance and one consistency violation; verify percentiles, error taxonomy counts, and the violation are reported without touching baseline state.

Acceptance:

  1. Given a completed run When results render Then per-chart p50/p90/p95/p99 latency, throughput, and error counts by taxonomy category are shown.
  2. Given two executions in one run share identical (chart, normalized filters) coordinates When their normalized results diverge Then a consistency_violation finding is recorded with both result hashes — flagged as flakiness/race, never as a baseline event.
  3. Given a load run completes When the 037 baseline catalog is inspected Then no baseline entry, candidate, source_response_hash, or immutability status has changed.

Story 5 — Blast-Radius and Cross-Run Comparison (P2)

Why P2: Cache effects cross dashboard boundaries; analysts need to compare runs and see shared-dataset impact explicitly.

Independent Test: Run the same profile twice (cold vs warm cache) and a dependent-dashboard probe; verify the comparison view shows latency delta and attributes the shift to shared-dataset cache state.

Acceptance:

  1. Given two runs of the same profile When comparison opens Then per-chart latency deltas are shown with cache-state annotation (cold/warm/unknown) derived from response metadata.
  2. Given dashboards share a dataset with the tested dashboard When results render Then the blast-radius panel links each dependent dashboard and marks whether it was probed during the run window.
  3. Given a run targets a dashboard whose datasets serve PROD dashboards When scheduling is configured Then the schedule policy warns about cache-interference windows, not just absolute concurrency.

Edge Cases

  • Superset connection pool exhaustion in the environment → executor queues within cap, records pool_wait latency separately, never bypasses the cap.
  • Variation references a filter value that exists in PREPROD but not PROD → validation marks the variation NEEDS_CONTEXT per environment; the run may proceed with the valid subset only.
  • Circuit breaker trips during ramp-up → partial results are preserved and the run is analyzable; abort is not data loss.
  • Identical chart+filters executed concurrently return different row ordering → normalization (037) canonicalizes before consistency comparison; ordering alone is not a violation.
  • User closes the browser mid-run → the run continues server-side; reconnecting shows live status by run id (recovery semantics from 036 US2).
  • Scheduled load run overlaps a deployment window for a blast-radius-dependent dashboard → schedule policy blocks or warning-gates the overlap.

Requirements

Functional

  • LOAD-FR-001: All load executions MUST go through the 037 Superset-native chart/dataset execution path; direct SQL, generated SQL, and raw query_context injection are forbidden (inherits 037 @REJECTED).
  • LOAD-FR-002: Concurrency MUST be enforced via a worker pool model: per-run async worker queue plus an env-level semaphore guaranteeing the sum of workers across all active runs on one environment never exceeds max_concurrent_per_env. Defaults: PROD-classified=5, dev/preprod=10, absolute ceiling=25 (not client-overridable); per-env override load_test_max_concurrent clamped to ceiling. Client values above the cap are clamped with a visible notice. Ramp defaults: PROD 1→2→3→5; others 1→3→5→10.
  • LOAD-FR-016: Each load run MUST be registered as exactly ONE TaskManager task (Task Center observability); individual executions are queue items, never tasks (no Reports flooding). Workers MUST check the circuit breaker between executions — in-flight completes, new executions are not taken (clean drain semantics).
  • LOAD-FR-017: Worker state MUST be observable per run: workers busy/idle, current execution per worker, queue depth — powering the live progress of LOAD-FR-008.
  • LOAD-FR-018: Load-path result processing MUST use bounded normalization: full-response sha256 hash + row count + first-N-row sample for display. Consistency checks (LOAD-FR-006) operate on hashes; full 037 normalization remains in the verification path only. This bounds continuous GIL hold per execution (large table charts) and keeps measured latency attributable to Superset, not to the load pipeline.
  • LOAD-FR-003: Load executions MUST NOT create, update, or invalidate 037 baseline entries, candidates, source_response_hash, immutability status, or visual baselines. Load artifacts live in a separate result store.
  • LOAD-FR-004: Every load run MUST have a circuit breaker with declared thresholds (error-rate %, p99 multiplier); breach aborts the run with typed terminal status and preserved partial results.
  • LOAD-FR-005: Starting a run against a PROD-classified environment MUST require an approval gate (036 semantics) showing concurrency ceiling, estimated request volume, blast-radius dependents, and a mandatory reason. Denial dispatches nothing.
  • LOAD-FR-006: Executions sharing identical (chart id, normalized filter context) within one run MUST be compared after 037 normalization; divergence MUST produce a consistency_violation finding, classified as flakiness — never as baseline drift.
  • LOAD-FR-007: Variation axes MUST be declarative and closed: filters, viewport, role, time_range. Unknown axes or values MUST fail validation; missing filter values MUST surface as NEEDS_CONTEXT, never invented.
  • LOAD-FR-008: Every load run MUST expose a stable load_run_id with live progress (per-variation running/completed/failed), recoverable after disconnect, with typed terminal statuses: completed, stopped_by_user, circuit_breaker_abort, failed.
  • LOAD-FR-009: The system MUST resolve the reverse index dataset → dependent dashboards for the tested dashboard and display it as a blast-radius panel at configure time, at the PROD gate, and in results.
  • LOAD-FR-010: Variation expansion MUST be deterministic for the same (dashboard query model, axes, seed): cartesian for ≤ declared cap, seeded sampling above it; the expanded matrix MUST be previewable before start.
  • LOAD-FR-011: Ramp-up MUST be staged (declared steps to target concurrency); steady state and drain phases MUST be distinguishable in progress and metrics.
  • LOAD-FR-012: Result records MUST include per-execution provenance: environment, dashboard, chart, variation coordinates, normalized filters hash, HTTP latency, queue-wait time, cache-state metadata, and Superset error taxonomy. Cache-state metadata comes authoritatively from /api/v1/chart/data response body: is_cached, cache_key, cached_dttm, queried_dttm, cache_timeout; mapped to hit|miss|bypassed|disabled|unknown with cache_state_source="superset_response".
  • LOAD-FR-019: Cache-state mapping MUST be: hit iff is_cached=true; bypassed iff request force=true; disabled iff cache_timeout=-1; miss iff is_cached=null AND cached_dttm=null AND force=false; unknown when fields are absent/inconsistent. Latency-based cache inference is forbidden because DB buffer cache, connection reuse, and network jitter are confounders.
  • LOAD-FR-013: Scheduled load runs MUST evaluate overlap against deployment/maintenance windows of blast-radius-dependent dashboards and block or warning-gate conflicts per policy.
  • LOAD-FR-014: RBAC MUST distinguish dashboard:loadtest:execute (PREPROD/staging) from dashboard:loadtest:prod (PROD-classified); unauthorized actors see permission_denied, never a confirm control (036 gate semantics).
  • LOAD-FR-015: Comparison of two runs of the same profile MUST show per-chart latency deltas and consistency-finding deltas; comparison MUST NOT require both runs to be complete (partial-current vs baseline-run allowed).
  • LOAD-FR-020: An opened InvestigationCase MAY ask the agent to construct/profile/compare a load experiment and autonomously run policy-permitted diagnostics. The agent MUST NOT bypass caps, mutate an active ramp, suppress a circuit breaker, or replace a required ActionApprovalGate.
  • LOAD-FR-021: Circuit-breaker aborts and consistency findings MUST emit idempotent 036 InvestigationSignals with blast-radius and run provenance; 047 creates/updates Queue items and MUST NOT automatically start agent work.

Key Entities

  • LoadProfile: Declarative configuration — dashboard, environment, concurrency target, ramp steps, execution mode (iterations|duration), variation axes with values, circuit-breaker thresholds, schedule policy. Versioned and reusable.
  • LoadVariation: One expanded coordinate set from the variation matrix — concrete filter values, viewport, role, time range. Immutable once the run starts.
  • LoadRun: Recoverable execution instance of a profile: id, status, phase (ramp|steady|drain|terminal), per-variation progress, circuit-breaker state, approval linkage for PROD.
  • LoadExecution: Single Superset-native request record: variation coordinates, chart, HTTP latency, queue-wait, bounded result hash/count/sample, typed outcome, and authoritative cache metadata (is_cached, cache_key, cached_dttm, queried_dttm, cache_timeout, mapped cache_state). Never writes to baseline state.
  • ConsistencyViolation: Finding that identical (chart, filters) executions diverged post-normalization; carries both result hashes; classified flakiness, not baseline drift.
  • BlastRadiusReport: Reverse-index projection — dataset ids used by the tested dashboard, dependent dashboards per dataset, cache-interference warning level, probe coverage.
  • CircuitBreakerPolicy: Thresholds (error-rate %, p99 multiplier), evaluation window, abort semantics, preserved-partial guarantee.
  • LoadRunComparison: Delta view of two runs — per-chart latency shift, cache-state attribution, consistency-finding delta.
  • LoadRunnerPool: Per-environment worker pool — async workers executing queue items, env-level semaphore enforcing max_concurrent_per_env across all active runs, worker registry (busy/idle, current execution, queue depth), breaker-checked between executions.
  • ExecutionQueue: Per-run FIFO of LoadExecution items; pool-wait measured as queue-residence time; drain = stop enqueue + bounded wait for in-flight.

Success Criteria

  • SC-001: In 100% of fixture runs, in-flight requests never exceed the configured cap, including during ramp-up and drain.
  • SC-002: Circuit breaker aborts within one evaluation window of threshold breach in 100% of fault-injection tests; partial results remain fully queryable.
  • SC-003: Zero mutations to the 037 baseline catalog across the entire load-test fixture suite (verified by catalog hash before/after).
  • SC-004: Variation matrix expansion is byte-deterministic across repeated expansions and shuffled axis ordering.
  • SC-005: Blast-radius panel lists 100% of dependent dashboards for fixture shared datasets at configure time and in results.
  • SC-006: PROD gate blocks 100% of unapproved PROD starts; denial produces zero dispatched requests.
  • SC-007: Consistency violation detection catches 100% of injected divergent-result fixtures and produces zero false positives on ordering-only differences.

Implementation Status & MVP Debt (audit 2026-08-07)

Facts (code check, not tasks.md):

  • ✅ Задачи T001–T074 закрыты: API routes, runner pool module, breaker, matrix, capacity, PROD gate, сравнение, фронт-модели.
  • 🔴 Главный MVP-долг: backend/src/plugins/load_testing.py::run_load_run не вызывает RunnerPool — он лишь переводит фазы RAMP→STEADY→DRAIN→COMPLETED и спит hold_seconds, не отправляя ни одного запроса к Superset. execute_chart_data из 037 в load-пути не используется (нарушение LOAD-FR-001). RunnerPool.run() нигде не инстанцируется в production-пути.
  • 🔴 Следствие: /status и /compare отдают seeded/пустые данные, circuit breaker не оценивается на реальном потоке, consistency-детектор не имеет входных результатов.

Закрытие: обязательные задачи T075–T079 в tasks.md Phase 9 (executor через 037, shared semaphore с client_registry, breaker, персистенция LoadExecution). Фича не может быть объявлена operational до их мерджа.

Runtime Closure Status (2026-08-07, resolved)

  • ✅ T075–T079 закрыты: run_load_run теперь вызывает RunnerPool через _resolve_execution_scope + _run_bounded_pool; 037-native executor (services/load_testing/executor.py) исполняет chart-data; shared per-env semaphore (client_registry.get_semaphore); LoadExecution персистятся (write_load_executions); breaker → circuit_breaker_abort.
  • ✅ Verified: tests/services/load_testing/test_executor_runtime.py (5 тестов, включая real-execution→LoadExecution), полный tests/services/load_testing/ = 76 passed, ruff clean.
  • 🟡 Осталось для operational: живые/фикстурные Superset-прогоны (exit gate 6), интеграционный тест capacity с ordinary-запросами (T077 extension).

#endregion DashboardLoadTesting.Spec