busya/ss-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
Files
2136082d6dfac250c10c9cda97210595ed80b900
ss-tools/specs/037-superset-baseline-engine/tasks.md
busya 2136082d6d feat(037): Phase 7 — Visual Baseline Support (T039-T047) 
- T039-T040: Schema ready (visualEntry + visualPolicy already in JSON schema)
- T041: visual_baseline.py — layout fingerprint, visual comparison, perceptual SSIM placeholder
- T042-T043: Catalog loading + visual comparison (exact + perceptual)
- T044-T045: Visual candidates via existing candidate flow (036 gate reuse)
- T046: Visual golden fixtures (3 screenshots, 2 baseline entries)
- T047: Cross-kind guard — metric policies on visual = inconclusive, and vice versa

65/65 tests pass. SPEC 037 COMPLETE: 47/47 tasks.
2026-07-28 19:41:05 +03:00

7.0 KiB
Raw Blame History

#region SupersetBaselineEngine.Tasks [C:3] [TYPE ADR] [SEMANTICS tasks,baseline,implementation] @BRIEF Ordered TDD backlog for Superset-native baseline engine.

Phase 1 — Fixtures and DTO Foundation

  • T001 Create canonical dashboard/chart/dataset/native-filter Superset fixtures under specs/037-superset-baseline-engine/fixtures/superset/.
  • T002 [P] Create scalar, percent, date, table, empty, malformed, and locale result fixtures under specs/037-superset-baseline-engine/fixtures/results/.
  • T003 [P] Create valid/invalid/stale baseline catalog fixtures under specs/037-superset-baseline-engine/fixtures/baselines/.
  • T004 Materialize fixtures into backend/tests/fixtures/dashboard_testing/.
  • T005 Implement extra-forbid DTOs from contracts/dashboard-testing.openapi.yaml in backend/src/schemas/dashboard_testing.py.

Phase 2 — US1 Inspect Dashboard Query Model

  • T006 [US1] Write failing deterministic inspection tests in backend/tests/services/dashboard_testing/test_query_model.py.
  • T007 [US1] Write failing filter scope/type/hash tests in backend/tests/services/dashboard_testing/test_filters.py.
  • T008 [US1] Implement backend/src/services/dashboard_testing/query_model.py using authoritative SupersetClient metadata.
  • T009 [US1] Implement backend/src/services/dashboard_testing/filters.py with canonical typed values and deterministic hashes.
  • T010 [US1] Add fingerprint helpers in backend/src/services/dashboard_testing/fingerprints.py and cover metadata order invariance.

Checkpoint: Fixture dashboards yield byte-stable models and correct chart/filter scopes.

Phase 3 — US2 Superset-Native Execution

  • T01- [x] T011 [US2] Write failing no-SQL schema and payload tests in backend/tests/services/dashboard_testing/test_query_executor.py.
  • T01- [x] T012 [US2] Add backend/src/core/superset_client/_chart_data.py adapter for saved-chart POST /api/v1/chart/data.
  • T01- [x] T013 [US2] Implement backend/src/services/dashboard_testing/query_executor.py: reload authoritative metadata, scope filters, bound limits, typed errors.
  • T01- [x] T014 [US2] Add agent tools inspect_dashboard_query_model and execute_dashboard_result in agent/src/ss_tools/agent/tools.py as thin backend clients.
  • T01- [x] T015 [US2] Verify scenario intent tool pipeline includes these tools and excludes superset_execute_sql.

Checkpoint: Scalar and table fixtures execute through chart-data only; injected SQL/raw context cannot reach Superset.

Phase 4 — US3 Normalize and Compare

  • T016 [US3] Write failing normalization tests in backend/tests/services/dashboard_testing/test_normalization.py.
  • T017 [US3] Implement normalization.py with Decimal strings, ISO temporal values, percent metadata, and bounded tables.
  • T018 [US3] Write failing comparison policy tests in backend/tests/services/dashboard_testing/test_comparison.py.
  • T019 [US3] Implement exact/absolute/relative/range/row-set comparison in backend/src/services/dashboard_testing/comparison.py.
  • T020 [US3] Add empty/unsupported/duplicate-row-key cases that must return inconclusive to backend/tests/services/dashboard_testing/test_comparison.py.
  • T021 [US3] Add independent query/dataset/filter staleness tests in backend/tests/services/dashboard_testing/test_staleness.py.
  • T022 [US3] Add deterministic diff/evidence reference output tests in backend/tests/services/dashboard_testing/test_comparison.py.

Phase 5 — US4 Baseline Candidate Lifecycle

  • T023 [US4] Write failing JSON-schema/YAML/path tests in backend/tests/services/dashboard_testing/test_baseline_catalog.py.
  • T024 [US4] Implement safe catalog loader and deterministic writer in backend/src/services/dashboard_testing/baseline_catalog.py.
  • T025 [US4] Resolve repository only through GitService and reject traversal/symlink escape.
  • T026 [US4] Validate catalogs against contracts/baseline-catalog.schema.json before use/write.
  • T027 [US4] Write failing candidate provenance/discrepancy tests in backend/tests/services/dashboard_testing/test_candidates.py.
  • T028 [US4] Implement candidate creation in backend/src/services/dashboard_testing/candidates.py as 036 DraftArtifact; never update approved YAML.
  • T029 [US4] Implement request-baseline-approval endpoint in backend/src/api/routes/dashboard_testing.py using 036 request hash and reason_required.
  • T030 [US4] Implement approval consume callback in backend/src/services/dashboard_testing/candidates.py that atomically writes YAML and records gate id/reason.
  • T031 [US4] Cover stale-after-confirm, RBAC revoked, payload mutation, and replay in backend/tests/services/dashboard_testing/test_candidates.py.
  • T032 [US4] Add agent tools discover_baseline_candidate and request_baseline_approval without direct approve capability.

Phase 6 — API and Integration

  • T033 Add backend/src/api/routes/dashboard_testing.py matching OpenAPI and register router.
  • T034 Write API contract/RBAC tests in backend/tests/api/test_dashboard_testing.py.
  • T035 Add Superset 4.1.2 Testcontainers test in backend/tests/integration/test_dashboard_testing_superset.py.
  • T036 Add compatibility tests for existing translation preview chart-data usage in backend/tests/core/superset_client/test_chart_data.py.
  • T037 Run quickstart, backend full relevant tests, ruff, and OpenAPI/schema validation.
  • T038 Audit C3+ contracts, direct-SQL ban, async boundaries, ATTN_1–4, and unresolved relations.

Phase 7 — Visual Baseline Support (AGBASE-FR-010)

  • T039 [P] Write failing visual baseline schema validation tests in backend/tests/services/dashboard_testing/test_visual_baseline.py.
  • T040 Extend baseline-catalog.schema.json validation to accept visualEntry alongside metric entries; reject cross-kind policy usage.
  • T041 [P] Implement backend/src/services/dashboard_testing/visual_baseline.py: VisualComparisonPolicy (exact + perceptual), layout fingerprint computation, stale_visual_baseline detection.
  • T042 [P] Wire visual baseline loading into BaselineEngine.Catalog.Load; extend catalog YAML to support visual entries.
  • T043 Implement BaselineEngine.Visual.Compare: digest comparison + perceptual SSIM path; return stale_visual_baseline when layout fingerprint mismatches.
  • T044 [P] Implement BaselineEngine.Visual.Candidate: create draft visual candidate from reviewed screenshot artifact with mandatory human disposition.
  • T045 Add visual baseline approval flow reusing 036 gate; verify approval writes visual entry atomically alongside metric entries.
  • T046 Write visual baseline golden fixtures under specs/037-superset-baseline-engine/fixtures/visual/.
  • T047 Audit: visual baselines never use metric policies; metric baselines never use visual policies; cross-kind comparison returns inconclusive.

Dependencies

T001–T005 → US1 → US2 → US3; US4 depends on US3 and completed 036. API integration follows all domain contracts. Phase 7 depends on completed 036 Phase 8 (screenshot evidence artifacts).

#endregion SupersetBaselineEngine.Tasks