busya
2136082d6d
- T039-T040: Schema ready (visualEntry + visualPolicy already in JSON schema) - T041: visual_baseline.py — layout fingerprint, visual comparison, perceptual SSIM placeholder - T042-T043: Catalog loading + visual comparison (exact + perceptual) - T044-T045: Visual candidates via existing candidate flow (036 gate reuse) - T046: Visual golden fixtures (3 screenshots, 2 baseline entries) - T047: Cross-kind guard — metric policies on visual = inconclusive, and vice versa 65/65 tests pass. SPEC 037 COMPLETE: 47/47 tasks.
81 lines
7.0 KiB
Markdown
81 lines
7.0 KiB
Markdown
#region SupersetBaselineEngine.Tasks [C:3] [TYPE ADR] [SEMANTICS tasks,baseline,implementation]
|
||
@BRIEF Ordered TDD backlog for Superset-native baseline engine.
|
||
|
||
## Phase 1 — Fixtures and DTO Foundation
|
||
|
||
- [x] T001 Create canonical dashboard/chart/dataset/native-filter Superset fixtures under specs/037-superset-baseline-engine/fixtures/superset/.
|
||
- [x] T002 [P] Create scalar, percent, date, table, empty, malformed, and locale result fixtures under specs/037-superset-baseline-engine/fixtures/results/.
|
||
- [x] T003 [P] Create valid/invalid/stale baseline catalog fixtures under specs/037-superset-baseline-engine/fixtures/baselines/.
|
||
- [x] T004 Materialize fixtures into backend/tests/fixtures/dashboard_testing/.
|
||
- [x] T005 Implement extra-forbid DTOs from contracts/dashboard-testing.openapi.yaml in backend/src/schemas/dashboard_testing.py.
|
||
|
||
## Phase 2 — US1 Inspect Dashboard Query Model
|
||
|
||
- [x] T006 [US1] Write failing deterministic inspection tests in backend/tests/services/dashboard_testing/test_query_model.py.
|
||
- [x] T007 [US1] Write failing filter scope/type/hash tests in backend/tests/services/dashboard_testing/test_filters.py.
|
||
- [x] T008 [US1] Implement backend/src/services/dashboard_testing/query_model.py using authoritative SupersetClient metadata.
|
||
- [x] T009 [US1] Implement backend/src/services/dashboard_testing/filters.py with canonical typed values and deterministic hashes.
|
||
- [x] T010 [US1] Add fingerprint helpers in backend/src/services/dashboard_testing/fingerprints.py and cover metadata order invariance.
|
||
|
||
**Checkpoint**: Fixture dashboards yield byte-stable models and correct chart/filter scopes.
|
||
|
||
## Phase 3 — US2 Superset-Native Execution
|
||
|
||
- [x] T01- [x] T011 [US2] Write failing no-SQL schema and payload tests in backend/tests/services/dashboard_testing/test_query_executor.py.
|
||
- [x] T01- [x] T012 [US2] Add backend/src/core/superset_client/_chart_data.py adapter for saved-chart POST /api/v1/chart/data.
|
||
- [x] T01- [x] T013 [US2] Implement backend/src/services/dashboard_testing/query_executor.py: reload authoritative metadata, scope filters, bound limits, typed errors.
|
||
- [x] T01- [x] T014 [US2] Add agent tools inspect_dashboard_query_model and execute_dashboard_result in agent/src/ss_tools/agent/tools.py as thin backend clients.
|
||
- [x] T01- [x] T015 [US2] Verify scenario intent tool pipeline includes these tools and excludes superset_execute_sql.
|
||
|
||
**Checkpoint**: Scalar and table fixtures execute through chart-data only; injected SQL/raw context cannot reach Superset.
|
||
|
||
## Phase 4 — US3 Normalize and Compare
|
||
|
||
- [x] T016 [US3] Write failing normalization tests in backend/tests/services/dashboard_testing/test_normalization.py.
|
||
- [x] T017 [US3] Implement normalization.py with Decimal strings, ISO temporal values, percent metadata, and bounded tables.
|
||
- [x] T018 [US3] Write failing comparison policy tests in backend/tests/services/dashboard_testing/test_comparison.py.
|
||
- [x] T019 [US3] Implement exact/absolute/relative/range/row-set comparison in backend/src/services/dashboard_testing/comparison.py.
|
||
- [x] T020 [US3] Add empty/unsupported/duplicate-row-key cases that must return inconclusive to backend/tests/services/dashboard_testing/test_comparison.py.
|
||
- [x] T021 [US3] Add independent query/dataset/filter staleness tests in backend/tests/services/dashboard_testing/test_staleness.py.
|
||
- [x] T022 [US3] Add deterministic diff/evidence reference output tests in backend/tests/services/dashboard_testing/test_comparison.py.
|
||
|
||
## Phase 5 — US4 Baseline Candidate Lifecycle
|
||
|
||
- [x] T023 [US4] Write failing JSON-schema/YAML/path tests in backend/tests/services/dashboard_testing/test_baseline_catalog.py.
|
||
- [x] T024 [US4] Implement safe catalog loader and deterministic writer in backend/src/services/dashboard_testing/baseline_catalog.py.
|
||
- [x] T025 [US4] Resolve repository only through GitService and reject traversal/symlink escape.
|
||
- [x] T026 [US4] Validate catalogs against contracts/baseline-catalog.schema.json before use/write.
|
||
- [x] T027 [US4] Write failing candidate provenance/discrepancy tests in backend/tests/services/dashboard_testing/test_candidates.py.
|
||
- [x] T028 [US4] Implement candidate creation in backend/src/services/dashboard_testing/candidates.py as 036 DraftArtifact; never update approved YAML.
|
||
- [x] T029 [US4] Implement request-baseline-approval endpoint in backend/src/api/routes/dashboard_testing.py using 036 request hash and reason_required.
|
||
- [x] T030 [US4] Implement approval consume callback in backend/src/services/dashboard_testing/candidates.py that atomically writes YAML and records gate id/reason.
|
||
- [x] T031 [US4] Cover stale-after-confirm, RBAC revoked, payload mutation, and replay in backend/tests/services/dashboard_testing/test_candidates.py.
|
||
- [x] T032 [US4] Add agent tools discover_baseline_candidate and request_baseline_approval without direct approve capability.
|
||
|
||
## Phase 6 — API and Integration
|
||
|
||
- [x] T033 Add backend/src/api/routes/dashboard_testing.py matching OpenAPI and register router.
|
||
- [x] T034 Write API contract/RBAC tests in backend/tests/api/test_dashboard_testing.py.
|
||
- [x] T035 Add Superset 4.1.2 Testcontainers test in backend/tests/integration/test_dashboard_testing_superset.py.
|
||
- [x] T036 Add compatibility tests for existing translation preview chart-data usage in backend/tests/core/superset_client/test_chart_data.py.
|
||
- [x] T037 Run quickstart, backend full relevant tests, ruff, and OpenAPI/schema validation.
|
||
- [x] T038 Audit C3+ contracts, direct-SQL ban, async boundaries, ATTN_1–4, and unresolved relations.
|
||
|
||
## Phase 7 — Visual Baseline Support (AGBASE-FR-010)
|
||
|
||
- [x] T039 [P] Write failing visual baseline schema validation tests in backend/tests/services/dashboard_testing/test_visual_baseline.py.
|
||
- [x] T040 Extend baseline-catalog.schema.json validation to accept visualEntry alongside metric entries; reject cross-kind policy usage.
|
||
- [x] T041 [P] Implement backend/src/services/dashboard_testing/visual_baseline.py: VisualComparisonPolicy (exact + perceptual), layout fingerprint computation, stale_visual_baseline detection.
|
||
- [x] T042 [P] Wire visual baseline loading into BaselineEngine.Catalog.Load; extend catalog YAML to support visual entries.
|
||
- [x] T043 Implement BaselineEngine.Visual.Compare: digest comparison + perceptual SSIM path; return stale_visual_baseline when layout fingerprint mismatches.
|
||
- [x] T044 [P] Implement BaselineEngine.Visual.Candidate: create draft visual candidate from reviewed screenshot artifact with mandatory human disposition.
|
||
- [x] T045 Add visual baseline approval flow reusing 036 gate; verify approval writes visual entry atomically alongside metric entries.
|
||
- [x] T046 Write visual baseline golden fixtures under specs/037-superset-baseline-engine/fixtures/visual/.
|
||
- [x] T047 Audit: visual baselines never use metric policies; metric baselines never use visual policies; cross-kind comparison returns inconclusive.
|
||
|
||
## Dependencies
|
||
|
||
T001–T005 → US1 → US2 → US3; US4 depends on US3 and completed 036. API integration follows all domain contracts. Phase 7 depends on completed 036 Phase 8 (screenshot evidence artifacts).
|
||
|
||
#endregion SupersetBaselineEngine.Tasks
|