Address external review findings on the 036-041 package:
- 041 LIN-FR-016/Q2: sync with research R2 — SQL-expression parsing uses
the in-repo sqlparse-based extractor (sqlglot rejected as new dep);
exact-confidence bounded to authoritative column/metric refs; note that
sqlparse is intentionally non-validating (no SQL AST guarantees)
- 041 LIN-FR-002/data-model: snapshot pinning documented as an optimistic
consistency token, not a historical edge-set store (mismatch -> stale
notice, no edge-set restore)
- 041 research: second R9 renamed R10 (collision with R9 labels/metrics)
- 041 tasks T017: '11 matrix rows' -> '12 data-rows' (actual matrix count)
- 041 plan: storage counts 5 -> 6 new tables + 1 additive FK (matches
data-model); decision-memory R1-R8 -> R1-R10
- 041 spec Status: Draft -> Ready for Implementation; CHK021 reworded
(cycles impossible by construction per LIN-FR-015)
- 038/039/040/041 validation.md: regenerate digest tables; withdraw
038 'IMPLEMENTATION COMPLETE' claim and clarify each PASS certifies
spec/contract validity only while runtime closure tasks stay open
Fact-check the dashboard-testing spec packages against actual code and
amend the full speckit document set (spec, plan, research, tasks,
traceability, quickstart) so documented status matches reality:
- 037: discrete metric tools work, but deploy hooks do not create
VerificationRun and GET read-API endpoints are missing (T080-T081)
- 038: compiler/validator work; VLM _default_submit and capture dispatch
remain runtime stubs that never call LLMClient/ScreenshotService
(T057-T059)
- 039: UI components exist, but api/dashboard-testing.ts is unbound and
pipeline views are not wired to pages; depends on 037 read-API
(T054-T058)
- 040: run_load_run never invokes RunnerPool, so load runs execute zero
Superset requests; must wire 037 executor (T075-T079)
- 041: backend index works, but no /lineage frontend and lineage_index
stays opt-in (T045-T048)
- 036: confirmed operational, relations to reused plugin modules fixed
19 open closure tasks total; region pairs balanced.
start_maintenance accepted any environment_id, creating a stuck PENDING
event that never transitioned for unknown environments. Add synchronous
404 guard (mirrors preview_dashboards), inject config_manager via Depends,
and cover with a regression test proving no event row is created.
Also fix mock_task_manager to await broadcast_maintenance_event (AsyncMock),
aligning the fixture with the production route's awaited call.
Add auto_end flag to maintenance_events: when set with an end_time, a
60s APScheduler scan dispatches the end task automatically. The scan
survives restarts and is de-duped by task_id; end_time alone stays
informational. Includes alembic migration, route/schema wiring, Svelte
checkbox with validation, examples, and backend + frontend tests.
The start endpoint declared 409 in OpenAPI responses but returned the
already_active idempotency hit as a plain 200. Now returns HTTP 409
Conflict with the declared MaintenanceAlreadyActiveResponse body
{maintenance_id, status: 'already_active'}.
Consumers updated to treat 409 already_active as idempotent success:
- bash example: 409 case in api_call
- python example: 409 branch in start_maintenance
- frontend form: info toast instead of error
New test: TestStartIdempotency verifies 409 + body + no new task
dispatched (naive datetimes to match SQLite tz-stripping).
Two defects fixed:
- ensure_banner_chart created an orphan 'Maintenance Banner' markdown chart
(polluted Charts menu and dashboard exports). The banner is now a native
MARKDOWN element in position_json; chart_id is a synthetic layout key.
- insert_banner_markdown_at_top blindly targeted GRID_ID; on ROOT->TABS
dashboards (FI-0085) GRID_ID is orphaned and the banner never rendered.
The layout is now normalized to ROOT->GRID_ID->[ROW-banner, ...] with
recursive parents update, matching the proven-working manual example.
Review-driven hardening:
- liveness check verifies reachability from ROOT (children graph), so
dashboards corrupted by the old bug self-heal on the next start.
- insert removes stale ROW-banner-*/MARKDOWN-banner-* keys (single banner).
- decision memory (@RATIONALE/@REJECTED) added to both modules.
- new ops script scripts/cleanup_maintenance_banner_charts.py (dry-run by
default) deletes already-created bogus banner charts in prod.
sqlparse raises SQLParseError above MAX_GROUPING_TOKENS=10000 tokens
(~25KB of typical SQL). The try/except fallback already handled it, but paid
~1s per oversized SQL for a parse doomed to fail. Add _SQLPARSE_SKIP_THRESHOLD
(30k chars) to bypass sqlparse for oversized text (~15x faster, 1.2s->0.08s for
a 212KB SQL) while keeping literal filtering for SQL under the threshold.
Tests: oversized-SQL skip-threshold behavior.
Discovery of virtual datasets now works, but a runtime blocker remained: any
virtual dataset whose SQL exceeds sqlparse's MAX_GROUPING_TOKENS (10000 tokens)
raised SQLParseError 'Maximum number of tokens exceeded (10000)' from
extract_tables_from_sql_span, which is called unguarded in the scan loop — one
oversized virtual dataset aborted the whole maintenance preview/start.
- extract_tables_from_sql_span now wraps sqlparse.parse + token walk in
try/except and falls back to regex-only extraction (keeping all schema.table
matches) instead of raising, so huge SQL no longer fails the scan.
- Tier-1 virtual filter uses value "" (not None) so the sql is_not_null filter
passes Superset's rison schema instead of always falling back to a full scan.
Tests: huge-SQL fallback (extractor) and huge-virtual-dataset scan resilience
(scanner). ADR-0020 updated with Decision 3.
Virtual (SQL) datasets were never matched, so maintenance discovery returned
0 affected dashboards. Two defects fixed:
- find_affected_dashboards filtered by is_sqllab_view, which is NOT a
filterable column in Superset's dataset list API (absent from search_columns),
so the query was rejected. Now discover virtual datasets via the filterable
sql column: primary server-side 'sql is_not_null' filter with a client-side
non-empty-sql scan as fallback (best-effort vs pagination cap), dedupe by id.
- AsyncAPIClient.request never called raise_for_status(), so rejected filters
(HTTP 400) were returned as bodies without a 'result' key and surfaced as
'Found 0 datasets', dead-coding the filtered->full-scan fallback. request()
now raises on non-2xx via the existing error mapper.
Tests cover both virtual-scan tiers, all fallback paths, the raise behavior,
and an end-to-end match with the real sql_table_extractor on production SQL.
Documented in ADR-0020.
Cache pip/npm dependency downloads via BuildKit cache mounts and skip
the postgres pull when the image is already present locally, so a
repeat ./build.sh bundle run does not download dependencies twice.
- Add Tabs atom (underline/segmented/card/pills + per-tab badge, parent-controlled value+onchange) and Switch atom
- Add success/warning/info/link variants to Button (link skips size classes to avoid cn() class conflicts)
- Add onPageChange to Pagination for 1-based currentPage pages
- Convert ~86 raw <button> across 17 routes/*/+page.svelte to atoms; keep 14 base-conflicting controls (backdrop, chips, accordion rows, destructive-colored links/actions) raw with @REJECTED docs
- Refine audit manual-button rule: flag pages with raw buttons only when no $lib/ui import and no '@REJECTED Raw <button>' exception
- Refine hasDocumentedException regex to require the comment itself to document the raw control
Show HelpTooltip (ⓘ) next to every variable configuration field in the
Settings UI explaining what the variable is and how it affects behavior:
- System tab: session timeouts, task retention, auth rate limit,
assistant history retention, translation baseline (replaces broken
hint prop on session inputs)
- Logging tab: agent log level, max file size, backup count, agent
view, hide routine infra
- Environments tab: default environment selector
- Input atom gains optional helpText prop rendering the tooltip next
to the label (reusable by other forms)
- 17 new i18n keys in en/ru with inline fallbacks
Move hardcoded constants into GlobalSettings and surface them in the
Settings UI: task retention, auth rate limit, assistant history retention,
translate baseline expiry, default environment, and extended logging fields.
- consolidated settings API: new fields in GET/PATCH with re-validation
through GlobalSettings (422 on out-of-range instead of silent persist)
- rate limiter policy read live from settings with 60s cache + lock-free
fast path; cache invalidated centrally in ConfigManager on auth policy
change (covers PATCH /settings/global and /consolidated)
- shared settings_provider.get_global_settings() replaces three copies of
the fallback pattern; scheduler baseline fallback derives from model
default
- remove dead GlobalSettings fields (pagination_limit, ff_dataset_*,
LLM_*_RETENTION_DAYS, GLOBAL_VALIDATION_WORKER_LIMIT, AppAsyncRuntimeConfig)
- SystemSettings blocks save on out-of-range values; LoggingSettings gains
max_bytes/backup_count/agent_view/hide_routine_infra/log_level_for_agents;
EnvironmentsTab gains default environment selector
- tests: rate limiter settings-driven policy, consolidated PATCH 422 paths,
System tab save-blocking UX test
- RBAC: ensure_admin_role() guarantees the Admin role carries is_admin=True
(startup self-heal + create_admin promotion + role-is_admin UI checkbox in
admin/roles); update_role refuses to strip is_admin from the last admin role.
- WS: broadcast_maintenance_event is now awaited (3 sites) so maintenance
events actually reach clients (was an un-awaited coroutine RuntimeWarning).
- Pagination: MAX_PAGINATION_PAGES cap + clear error in fetch_paginated_data
to stop runaway loops on huge environments.
- Discovery: find_affected_dashboards and translate datasource picker filter
datasets/dashboards server-side (table_name/id filters, opr operator per
Superset OpenAPI) instead of full scans that hit the pagination token cap;
fallback to full scan when filters are rejected; virtual-dataset dedupe.
WebSocket endpoints now accept then close with real codes (4001 auth, 4003
permission) so clients detect auth failure via event.code instead of an opaque
403 handshake, ending the infinite reconnect storm. _authenticate_websocket
logs the actual JWT/API-key failure reason. Frontend WS consumers stop on
auth rejection and use capped exponential backoff for transient failures.
async_network.request() routes proxy 502/503/504 (HTML) responses to
NetworkError so migration/maintenance surface a clean 503 instead of a
500 JSON-parse traceback.
Orthogonal evaluation of semantics-core/contracts/testing/python/svelte
and molecular-cot-logging invariants for LLM handoff. Scores each rule
across correctness, Doxygen/retrieval value, runtime observability,
agent utility, and compliance cost; separates strict gates from
ritual-prone practices.
- INV_3: capability_mapper.py — MapCaseImpl region closed with wrong ID
(MapCase); duplicate MapCase endregion removed; all region pairs now
match by EXACT ID (stack-verified)
- ATTN_3: tools_038 helpers (DualAuthHeaders/Post/GuardPermission) now
carry 'scenario' primary keyword in [SEMANTICS] for DSA grouping
- Full invariant audit: INV_1-8 + ATTN_1-4 verified (module <400,
CC<=10 via ruff C901=0, all non-root contracts <=150 lines)
- 108 tests green; index rebuilt
