diff --git a/specs/030-dataset-lifecycle-workspace/contracts/modules.md b/specs/030-dataset-lifecycle-workspace/contracts/modules.md new file mode 100644 index 00000000..af45b1ae --- /dev/null +++ b/specs/030-dataset-lifecycle-workspace/contracts/modules.md @@ -0,0 +1,256 @@ +# Contracts: Dataset Lifecycle Workspace + +> Generated for speckit feature 030. All IDs scoped to this feature. Complexity per GRACE-Poly v2.6. Allowed predicates: DEPENDS_ON, CALLS, INHERITS, IMPLEMENTS, DISPATCHES, BINDS_TO. + +--- + +## Backend Contracts + +### #region MetricItem [C:1] [TYPE DataClass] +@BRIEF Pydantic DTO for a dataset metric — carries Superset metric metadata. +@RELATION DEPENDS_ON -> [DatasetDetailResponse] +**File:** `backend/src/api/routes/datasets.py` + +```python +class MetricItem(BaseModel): + id: int + metric_name: str + expression: str | None = None + verbose_name: str | None = None + description: str | None = None + metric_type: str | None = None +``` + +### #region StatsCounts [C:1] [TYPE DataClass] +@BRIEF Aggregate statistics for the Stats Bar — computed from full dataset list. +**File:** `backend/src/api/routes/datasets.py` + +```python +class StatsCounts(BaseModel): + total: int + unmapped_count: int + mapped_count: int + linked_count: int +``` + +### #region ColumnDescriptionUpdate [C:1] [TYPE DataClass] +@BRIEF Request DTO for inline-edit of a column description. +**File:** `backend/src/api/routes/datasets.py` + +```python +class ColumnDescriptionUpdate(BaseModel): + description: str +``` + +### #region MetricDescriptionUpdate [C:1] [TYPE DataClass] +@BRIEF Request DTO for inline-edit of a metric description. +**File:** `backend/src/api/routes/datasets.py` + +```python +class MetricDescriptionUpdate(BaseModel): + description: str +``` + +### #region DatasetDetailResponse [C:2] [TYPE DataClass] — MODIFIED +@BRIEF Detailed DTO for a dataset including columns, linked dashboards, and metrics. +@RELATION DEPENDS_ON -> [MetricItem] +**File:** `backend/src/api/routes/datasets.py` +**Modification:** Add `metrics: list[MetricItem]` and `metric_count: int`. + +### #region DatasetsResponse [C:2] [TYPE DataClass] — MODIFIED +@BRIEF Paginated response DTO for dataset listings with stats. +@RELATION DEPENDS_ON -> [StatsCounts] +**File:** `backend/src/api/routes/datasets.py` +**Modification:** Add `stats: StatsCounts`. + +### #region update_column_description [C:4] [TYPE Endpoint] +@BRIEF Save description for a single dataset column. Internal: GET full dataset from Superset, modify one column's description, PUT back. +@PRE: dataset_id and column_id must exist in the target environment. +@POST: Column description in Superset is updated. Response confirms success. +@SIDE_EFFECT: Mutates dataset metadata in upstream Superset instance via PUT. +@RELATION CALLS -> [SupersetClient.get_dataset] +@RELATION CALLS -> [SupersetClient.update_dataset] +@RATIONALE Must perform GET→modify→PUT because Superset has no PATCH for individual columns — only full object PUT with `override_columns=false`. +@REJECTED Direct PUT from frontend — rejected because frontend would need to handle full Superset payload structure. +**File:** `backend/src/api/routes/datasets.py` +**Route:** `PUT /api/datasets/{dataset_id}/columns/{column_id}/description` +**Method:** `PUT` +**Request:** `ColumnDescriptionUpdate` +**Response:** `{success: true, description: "new text"}` +**RBAC:** `has_permission("plugin:migration", "READ")` + +### #region update_metric_description [C:4] [TYPE Endpoint] +@BRIEF Save description for a single dataset metric. Mirror of update_column_description for metrics. +@PRE: dataset_id and metric_id must exist in the target environment. +@POST: Metric description in Superset is updated. +@SIDE_EFFECT: Mutates dataset metadata in upstream Superset instance via PUT. +@RELATION CALLS -> [SupersetClient.get_dataset] +@RELATION CALLS -> [SupersetClient.update_dataset] +**File:** `backend/src/api/routes/datasets.py` +**Route:** `PUT /api/datasets/{dataset_id}/metrics/{metric_id}/description` +**Request:** `MetricDescriptionUpdate` +**Response:** `{success: true, description: "new text"}` +**RBAC:** `has_permission("plugin:migration", "READ")` + +### #region get_datasets [C:4] [TYPE Endpoint] — MODIFIED +@BRIEF Fetch paginated dataset list with enhanced metadata, mapped fields, and stats counts. +@PRE: env_id must be valid. page >= 1, page_size in [1,100]. +@POST: Returns paginated datasets plus StatsCounts for the Stats Bar. +@SIDE_EFFECT: Computes aggregate counts from full dataset list (in-memory, no DB write). +@RELATION CALLS -> [resource_service.get_datasets_with_status] +@RATIONALE Stats counts returned in the same response as datasets to avoid an extra API call for the Stats Bar (FR-022). +**File:** `backend/src/api/routes/datasets.py` +**Modification:** Compute `stats` dict from full dataset list before pagination slicing, add to DatasetsResponse. + +### #region get_dataset_detail [C:3] [TYPE Endpoint] — MODIFIED +@BRIEF Get detailed dataset information including columns, linked dashboards, and metrics. +@RELATION CALLS -> [SupersetClient.get_dataset_detail] +**File:** `backend/src/api/routes/datasets.py` +**Modification:** Extract `dataset.get("metrics", [])` from Superset response, map to `MetricItem` list. + +### #region SupersetClient.get_dataset_detail [C:3] [TYPE Function] — MODIFIED +@BRIEF Extract dataset detail from Superset API including metrics. +@RELATION CALLS -> [SupersetClient.get_dataset] +**File:** `backend/src/core/superset_client/_datasets.py` +**Modification:** Add metrics extraction loop after columns loop. Add `metrics` and `metric_count` to result dict. + +--- + +## Frontend Contracts + +### #region StatsBar [C:3] [TYPE Component] [SEMANTICS ui,dataset,stats,filter] +@BRIEF Displays 4 aggregate metric tiles (All, Without mapping, With description, Linked) and emits filter selection. +@LAYER UI +@RELATION BINDS_TO -> [environmentContextStore] +@UX_STATE Loading -> 4 skeleton tiles pulsing. +@UX_STATE Loaded -> 4 tiles with numbers, active filter highlighted. +@UX_STATE Filtered -> Selected tile has accent border; list reflects filter. +@UX_STATE Empty -> All tiles show 0. +@UX_REACTIVITY Props -> `stats: StatsCounts`, `activeFilter: string|null`. +@UX_REACTIVITY Events -> `onfilter(filterKey)` dispatched on tile click. +**File:** `frontend/src/routes/datasets/StatsBar.svelte` + +### #region DatasetList [C:4] [TYPE Component] [SEMANTICS ui,dataset,card,list,pagination] +@BRIEF Renders paginated dataset cards with mapping progress bars, checkboxes, and quick actions. +@LAYER UI +@RELATION BINDS_TO -> [environmentContextStore] +@RELATION BINDS_TO -> [taskDrawerStore] +@UX_STATE Loading -> Skeleton cards. +@UX_STATE Loaded -> Cards with progress bars. +@UX_STATE Selecting -> Checkboxes visible, checked cards highlighted. +@UX_STATE Empty -> "No datasets found" after search/filter. +@UX_FEEDBACK Click card → dispatches dataset selection for detail panel. +@UX_FEEDBACK Click checkbox → updates selection set. +@UX_FEEDBACK Click quick action → opens Map Columns or Generate Docs modal. +@UX_REACTIVITY Props -> `datasets: DatasetCard[]`, `selectedIds: Set`, `activeFilter: string|null`. +@UX_REACTIVITY Events -> `onselect(datasetId)`, `onaction(dataset, action)`. +**File:** `frontend/src/routes/datasets/DatasetList.svelte` + +### #region DatasetPreview [C:3] [TYPE Component] [SEMANTICS ui,dataset,detail,columns,metrics] +@BRIEF Right panel showing dataset detail: header info, columns table, metrics table, linked dashboards. +@LAYER UI +@RELATION DEPENDS_ON -> [ColumnsTable] +@RELATION DEPENDS_ON -> [MetricsTable] +@UX_STATE NoSelection -> "Select a dataset from the list" placeholder. +@UX_STATE Loading -> Skeleton for header + 2 tables. +@UX_STATE Loaded -> Full detail with columns and metrics rendered. +@UX_STATE Error -> Error banner with retry. +@UX_REACTIVITY Props -> `dataset: DatasetDetailResponse | null`. +@UX_TEST: NoSelection -> {click: card, expected: Loading -> Loaded}. +**File:** `frontend/src/routes/datasets/DatasetPreview.svelte` + +### #region ColumnsTable [C:4] [TYPE Component] [SEMANTICS ui,dataset,columns,inline-edit] +@BRIEF Table of dataset columns with type chips, description, and inline-edit capability. +@LAYER UI +@RELATION CALLS -> [api_module] +@UX_STATE Display -> All rows in read mode. +@UX_STATE Editing -> One row in edit mode (textarea), others dimmed. +@UX_STATE Saving -> Spinner on save button. +@UX_STATE Error -> Red border on textarea + error message. +@UX_FEEDBACK verbose_name → shown bold; missing → column_name in italic as fallback. +@UX_FEEDBACK No description → "✏️ Add description" action prompt. +@UX_REACTIVITY Props -> `columns: ColumnRow[]`, `datasetId: number`. +@UX_REACTIVITY LocalState -> `isEditingRowId: number|null`. +@RATIONALE Inline-edit uses optimised state per row to minimise re-renders across the table. +@REJECTED Modal-per-column editing — rejected as too many clicks for repetitive workflow. +**File:** `frontend/src/routes/datasets/ColumnsTable.svelte` + +### #region MetricsTable [C:4] [TYPE Component] [SEMANTICS ui,dataset,metrics,inline-edit] +@BRIEF Table of dataset metrics with expression, description, and inline-edit capability. +@LAYER UI +@RELATION CALLS -> [api_module] +@UX_STATE Display -> All rows in read mode with expression shown under name. +@UX_STATE Editing -> One row in edit mode. +@UX_STATE Saving -> Spinner on save button. +@UX_FEEDBACK verbose_name → shown bold; missing → metric_name in italic as fallback. +@UX_FEEDBACK No description → shows expression as hint + "✏️ Add description" prompt. +@UX_REACTIVITY Props -> `metrics: MetricRow[]`, `datasetId: number`. +**File:** `frontend/src/routes/datasets/MetricsTable.svelte` + +### #region DatasetHub [C:5] [TYPE Page] — REWRITTEN +@BRIEF Dataset lifecycle workspace page — split-view with Stats Bar, card list, detail panel, and bulk actions. +@LAYER Page +@RELATION DEPENDS_ON -> [StatsBar] +@RELATION DEPENDS_ON -> [DatasetList] +@RELATION DEPENDS_ON -> [DatasetPreview] +@RELATION BINDS_TO -> [environmentContextStore] +@RELATION BINDS_TO -> [taskDrawerStore] +@PRE: Environment context is initialized before dataset loading. +@POST: Page renders split-view layout; dataset detail updates without full page reload. +@SIDE_EFFECT: Fetches datasets from API; subscribes to WebSocket `dataset.updated` events; opens modals for bulk actions. +@INVARIANT: Always shows datasets for the active environment from context store. +@DATA_CONTRACT: Receives DatasetsResponse (list + stats) and DatasetDetailResponse (detail panel). Sends ColumnDescriptionUpdate/MetricDescriptionUpdate on save. +@RATIONALE Split-view replaces separate list + detail pages to reduce navigation clicks and make the workspace feel integrated. +@REJECTED Multi-page navigation (list → detail separate routes) — rejected because it forces full page reload between selecting datasets, breaking flow. +**File:** `frontend/src/routes/datasets/+page.svelte` + +--- + +## Complexity Summary + +| Contract | Type | C | File | +|----------|------|---|------| +| MetricItem | DataClass | 1 | datasets.py | +| StatsCounts | DataClass | 1 | datasets.py | +| ColumnDescriptionUpdate | DataClass | 1 | datasets.py | +| MetricDescriptionUpdate | DataClass | 1 | datasets.py | +| DatasetDetailResponse (mod) | DataClass | 2 | datasets.py | +| DatasetsResponse (mod) | DataClass | 2 | datasets.py | +| get_dataset_detail (mod) | Endpoint | 3 | datasets.py | +| SupersetClient.get_dataset_detail (mod) | Function | 3 | _datasets.py | +| StatsBar | Component | 3 | StatsBar.svelte | +| DatasetPreview | Component | 3 | DatasetPreview.svelte | +| update_column_description | Endpoint | 4 | datasets.py | +| update_metric_description | Endpoint | 4 | datasets.py | +| get_datasets (mod) | Endpoint | 4 | datasets.py | +| DatasetList | Component | 4 | DatasetList.svelte | +| ColumnsTable | Component | 4 | ColumnsTable.svelte | +| MetricsTable | Component | 4 | MetricsTable.svelte | +| DatasetHub (rewrite) | Page | 5 | +page.svelte | + +--- + +## ADR Continuity + +| ADR | Status | This Feature | +|-----|--------|-------------| +| ADR-0001 (module layout) | ACTIVE | ✅ Follows `backend/src/api/routes/`, `frontend/src/routes/` placement | +| ADR-0002 (semantic protocol) | ACTIVE | ✅ All contracts use GRACE anchors with C1-C5 complexity | +| ADR-0003 (orchestrator pattern) | ACTIVE | ✅ Backend proxies Superset API, no direct frontend-to-Superset calls | +| ADR-0004 (plugin architecture) | ACTIVE | ✅ Mapping/docs tasks use existing plugin system (no new plugin) | +| ADR-0005 (auth RBAC) | ACTIVE | ✅ New endpoints gated by existing permission checks | +| ADR-0006 (frontend architecture) | ACTIVE | ✅ Svelte 5 runes, Tailwind, static SPA, vitest | +| ADR-0007 (rejected fromStore) | ACTIVE | ✅ Uses `$state`, avoids deprecated patterns | + +--- + +## Rejected Paths + +| Path | Reason | Guardrail | +|------|--------|-----------| +| PATCH endpoint on Superset directly | Superset v1 API has no PATCH for columns/metrics | `override_columns=false` with full-object PUT only | +| Direct frontend-to-Superset PUT | Violates orchestrator pattern (ADR-0003), exposes auth | All mutations go through ss-tools backend | +| Modal-per-row editing | Too many clicks for repetitive column description workflow | Inline edit via textarea toggle | +| Multi-page list→detail navigation | Full page reload between dataset selections breaks flow | Split-view within single SvelteKit route | +| Polling for Stats Bar refresh | Unnecessary network overhead, user chose WebSocket | WebSocket `dataset.updated` event | +| `override_columns=true` | Risk of deleting unmapped columns on PUT | Always `override_columns=false` (FR-023) | diff --git a/specs/030-dataset-lifecycle-workspace/data-model.md b/specs/030-dataset-lifecycle-workspace/data-model.md new file mode 100644 index 00000000..e5e81cc7 --- /dev/null +++ b/specs/030-dataset-lifecycle-workspace/data-model.md @@ -0,0 +1,182 @@ +# Data Model: Dataset Lifecycle Workspace + +## Entity Overview + +``` +GET /api/datasets GET /api/datasets/{id} +───────────────────────────────────────────────────────────── +DatasetsResponse DatasetDetailResponse +├── datasets: DatasetItem[] ├── ... (existing fields) +│ ├── id ├── metrics: MetricItem[] +│ ├── table_name │ ├── id +│ ├── schema │ ├── metric_name +│ ├── database │ ├── expression +│ ├── mapped_fields │ ├── verbose_name +│ └── last_task │ ├── description +├── stats: StatsCounts │ └── metric_type +│ ├── total └── metric_count +│ ├── unmapped_count +│ ├── mapped_count +│ └── linked_count +├── total, page, page_size, +│ total_pages (existing) +└── [unchanged pagination] + +PUT /api/datasets/{id}/columns/{col_id}/description +PUT /api/datasets/{id}/metrics/{metric_id}/description +───────────────────────────────────────────────────────────── +Request: ColumnDescriptionUpdate | MetricDescriptionUpdate + { description: string } + +Response: { success: true, description: string } +``` + +--- + +## Backend Pydantic Schemas (in `backend/src/api/routes/datasets.py`) + +### MetricItem (new) +```python +class MetricItem(BaseModel): + id: int + metric_name: str + expression: str | None = None + verbose_name: str | None = None + description: str | None = None + metric_type: str | None = None +``` + +### StatsCounts (new) +```python +class StatsCounts(BaseModel): + total: int # all datasets + unmapped_count: int # mapped_fields.mapped == 0 + mapped_count: int # mapped_fields.mapped > 0 + linked_count: int # linked_dashboard_count > 0 +``` + +### DatasetsResponse (modified) +```python +class DatasetsResponse(BaseModel): + datasets: list[DatasetItem] + stats: StatsCounts # NEW + total: int + page: int + page_size: int + total_pages: int +``` + +### DatasetDetailResponse (modified) +```python +class DatasetDetailResponse(BaseModel): + # ... existing fields unchanged ... + metrics: list[MetricItem] # NEW + metric_count: int # NEW +``` + +### ColumnDescriptionUpdate (new) +```python +class ColumnDescriptionUpdate(BaseModel): + description: str +``` + +### MetricDescriptionUpdate (new) +```python +class MetricDescriptionUpdate(BaseModel): + description: str +``` + +--- + +## Frontend Types (TypeScript / Svelte inferred) + +```typescript +interface StatsCounts { + total: number; + unmapped_count: number; + mapped_count: number; + linked_count: number; +} + +interface MetricItem { + id: number; + metric_name: string; + expression?: string; + verbose_name?: string; + description?: string; + metric_type?: string; +} + +// UI-only state types (not in API) +type ColumnRow = { + column_name: string; + verbose_name?: string; + type?: string; + description?: string; + is_dttm: boolean; + is_active: boolean; + id: number; + isEditing?: boolean; // UI state for inline edit +}; + +type MetricRow = { + metric_name: string; + expression?: string; + verbose_name?: string; + description?: string; + metric_type?: string; + id: number; + isEditing?: boolean; // UI state for inline edit +}; + +type StatsBarFilter = 'all' | 'unmapped' | 'mapped' | 'linked' | null; +``` + +--- + +## WebSocket Message + +```json +{ + "type": "dataset.updated", + "payload": { + "env_id": "ss-dev", + "dataset_ids": ["242377ce-bdbd-42df-85da-ca1340a72fbb"] + } +} +``` + +--- + +## State Transitions + +### Stats Bar Tile States +``` +Idle → (click tile) → Active (highlighted, list filtered) +Active → (click same tile) → Idle (filter cleared) +Active → (click different tile) → Active (new tile, list re-filtered) +``` + +### Inline Edit States +``` +Display → (click ✏️) → Editing (textarea visible) +Editing → (Save + API success) → Display (new description shown) +Editing → (Save + API error) → Editing (error message, textarea stays) +Editing → (Cancel) → Display (original description restored) +``` + +### Detail Panel States +``` +NoSelection → (click card) → Loading → Loaded +Loaded → (click different card) → Loading → Loaded +Loaded → (deselect / bulk mode) → NoSelection +Loading → (API error) → Error → (Retry) → Loading +``` + +### Bulk Selection States +``` +None → (check 1+ cards) → Selecting (bottom panel hidden for 1) +Selecting → (check 2+ cards) → BulkActive (bottom panel visible) +BulkActive → (uncheck all) → None +BulkActive → (click action) → Modal → (complete) → None +``` diff --git a/specs/030-dataset-lifecycle-workspace/plan.md b/specs/030-dataset-lifecycle-workspace/plan.md new file mode 100644 index 00000000..158b227b --- /dev/null +++ b/specs/030-dataset-lifecycle-workspace/plan.md @@ -0,0 +1,97 @@ +# Implementation Plan: Dataset Lifecycle Workspace + +**Branch**: `030-dataset-lifecycle-workspace` | **Date**: 2026-05-19 | **Spec**: [spec.md](./spec.md) +**Input**: Feature specification from `specs/030-dataset-lifecycle-workspace/spec.md` + +## Summary + +Переработка страницы датасетов (`/datasets`) из табличного вида в полноценное split-view рабочее пространство с: (1) Stats Bar с 4 агрегирующими метриками и клиентской фильтрацией, (2) split-view с карточками датасетов слева и детальной панелью справа, (3) inline-редактированием описаний колонок и метрик через новые ss-tools API эндпоинты, (4) WebSocket-автообновлением при завершении фоновых задач, (5) отображением метрик из Superset API. Изменения затрагивают бэкенд (3 новых DTO, 2 новых эндпоинта, модификация 2 существующих) и фронтенд (5 новых компонентов, переписанный +page.svelte). + +## Technical Context + +**Language/Version**: Python 3.9+ (backend), JavaScript/TypeScript — Svelte 5 runes (frontend) +**Primary Dependencies**: FastAPI 0.104+, Pydantic v2, SQLAlchemy (backend); SvelteKit 2.x, Svelte 5.x, Vite 7.x, Tailwind CSS 3.x (frontend) +**Storage**: PostgreSQL 16 (ss-tools own DB); no schema changes in this feature +**Testing**: pytest (backend), vitest + @testing-library/svelte 5.x (frontend) +**Target Platform**: Linux server (Docker), modern browsers +**Project Type**: web application (FastAPI REST + WebSocket backend, SvelteKit SPA frontend) +**Performance Goals**: Split-view dataset switch <500ms, inline-edit save <1s, initial Stats Bar load <200ms +**Constraints**: <400 lines per module (INV_7), RBAC on all new endpoints, Tailwind-only styling, Svelte 5 runes only +**Scale/Scope**: 33 datasets (current), single environment (dev), 5 new frontend components, 2 new API endpoints + +## Constitution Check + +*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.* + +| Principle | Status | Notes | +|-----------|--------|-------| +| Library-First | ✅ N/A | No new library extracted | +| Semantic Protocol (ADR-0002) | ✅ PASS | All C4/C5 contracts use belief_scope/reason/reflect markers. All components use @UX_STATE. @RATIONALE/@REJECTED on key decisions. | +| Orchestrator Pattern (ADR-0003) | ✅ PASS | Backend proxies Superset API; no direct frontend-to-Superset calls | +| Plugin Architecture (ADR-0004) | ✅ PASS | No new plugin; mapping/docs reuse existing MapperPlugin/DocumentationPlugin | +| Auth RBAC (ADR-0005) | ✅ PASS | New endpoints gated: `has_permission("plugin:migration","READ")` | +| Frontend Svelte 5 Runes (ADR-0006) | ✅ PASS | All components use $state/$derived/$props/$effect | +| Anti-Erosion (INV_7) | ✅ PASS | New components <400 lines each; no monolithic page | + +**No blocking conflicts detected.** + +## Project Structure + +### Documentation (this feature) +``` +specs/030-dataset-lifecycle-workspace/ +├── plan.md # This file +├── research.md # 10 research decisions +├── data-model.md # Pydantic schemas, frontend types, state machines +├── quickstart.md # Verification commands +├── contracts/ +│ └── modules.md # 16 contracts with complexity classes +├── spec.md # Feature specification (7 user stories, 24 FR) +├── ux_reference.md # UX interaction reference +└── checklists/ + └── requirements.md # 44 checklist items +``` + +### Source Code Changes + +```text +backend/ +├── src/ +│ ├── api/routes/ +│ │ └── datasets.py # +MetricItem, +StatsCounts, +2 endpoints, MOD DatasetsResponse +│ └── core/ +│ ├── superset_client/ +│ │ └── _datasets.py # MOD get_dataset_detail (metrics extraction) +│ └── task_manager.py # MOD on_task_complete (WebSocket event) +└── tests/ + └── __tests__/ + └── test_datasets.py # +tests for new endpoints and metrics + +frontend/ +├── src/ +│ ├── routes/datasets/ +│ │ ├── +page.svelte # REWRITE (split-view orchestrator, ~200 lines) +│ │ ├── StatsBar.svelte # NEW (4 metric tiles, ~80 lines) +│ │ ├── DatasetList.svelte # NEW (card grid, pagination, ~250 lines) +│ │ ├── DatasetPreview.svelte # NEW (detail panel, ~150 lines) +│ │ ├── ColumnsTable.svelte # NEW (inline-edit columns, ~180 lines) +│ │ └── MetricsTable.svelte # NEW (inline-edit metrics, ~150 lines) +│ └── lib/i18n/locales/ +│ ├── en/datasets.json # +11 keys +│ └── ru/datasets.json # +11 keys +└── tests/ + └── routes/datasets/ + └── dataset_workspace.test.js # +component tests for new components +``` + +## Complexity Tracking + +No violations. All components adhere to C1-C5 tier rules per ADR-0002. + +| Contract | C | Justification | +|----------|---|--------------| +| MetricItem, StatsCounts, *Update DTOs | 1 | Pure data classes, anchors only | +| DatasetDetailResponse, DatasetsResponse (mod) | 2 | DTOs with @PURPOSE | +| get_dataset_detail (mod), StatsBar, DatasetPreview | 3 | Multi-step logic with @RELATION | +| update_*_description endpoints, get_datasets (mod), DatasetList, ColumnsTable, MetricsTable | 4 | Stateful operations with @PRE/@POST/@SIDE_EFFECT | +| DatasetHub (+page.svelte rewrite) | 5 | Orchestrator with @DATA_CONTRACT, @INVARIANT, decision memory | diff --git a/specs/030-dataset-lifecycle-workspace/quickstart.md b/specs/030-dataset-lifecycle-workspace/quickstart.md new file mode 100644 index 00000000..c68fc362 --- /dev/null +++ b/specs/030-dataset-lifecycle-workspace/quickstart.md @@ -0,0 +1,100 @@ +# Quickstart: Dataset Lifecycle Workspace + +## Prerequisites + +- Backend virtualenv: `cd backend && source .venv/bin/activate` +- Frontend deps: `cd frontend && npm install` +- Superset environment configured (dev via http://localhost:8100) +- RBAC: user must have `plugin:migration:READ` permission + +## Backend Verification + +```bash +# Run backend tests +cd backend && source .venv/bin/activate && python -m pytest -v + +# Lint +python -m ruff check . + +# Run dev server +uvicorn src.main:app --reload --port 8000 +``` + +## Frontend Verification + +```bash +# Run frontend tests +cd frontend && npm run test + +# Lint +cd frontend && npm run lint + +# Build check +cd frontend && npm run build + +# Dev server (requires backend on :8000) +cd frontend && npm run dev +``` + +## Feature-Specific Verification + +### Backend API Tests + +```bash +# Test metrics in dataset detail response +curl -s http://localhost:8100/api/datasets/32?env_id=dev | jq '.metrics, .metric_count' + +# Test stats counts in dataset list +curl -s 'http://localhost:8100/api/datasets?env_id=dev&page=1&page_size=5' | jq '.stats' + +# Test column description inline-edit +curl -s -X PUT http://localhost:8100/api/datasets/32/columns/851/description \ + -H 'Content-Type: application/json' \ + -d '{"description":"Updated via API"}' + +# Test metric description inline-edit +curl -s -X PUT http://localhost:8100/api/datasets/32/metrics/70/description \ + -H 'Content-Type: application/json' \ + -d '{"description":"Count of all records"}' +``` + +### Frontend Component Tests + +```bash +# Run dataset-specific component tests +cd frontend && npx vitest run src/routes/datasets/ --reporter=verbose +``` + +### Browser Validation + +1. Open `http://localhost:8100/datasets` +2. Verify Stats Bar shows 4 tiles with counts +3. Click each tile — list filters correctly +4. Click a dataset card — detail panel loads with columns + metrics +5. Click ✏️ on a column — textarea appears, type description, save +6. Verify description persists after save +7. Check 2+ checkboxes — bottom panel appears +8. Click "Map Columns" — modal opens + +### Docker + +```bash +docker compose up --build +# Visit http://localhost:8100/datasets +``` + +## Key Files Changed/Created + +| File | Change | +|------|--------| +| `backend/src/api/routes/datasets.py` | +MetricItem, +StatsCounts, +2 endpoints, MOD DatasetsResponse/DatasetDetailResponse | +| `backend/src/core/superset_client/_datasets.py` | +metrics extraction in get_dataset_detail | +| `backend/src/core/task_manager.py` | +dataset.updated WebSocket event on task complete | +| `frontend/src/routes/datasets/+page.svelte` | REWRITE (split-view orchestrator) | +| `frontend/src/routes/datasets/StatsBar.svelte` | NEW | +| `frontend/src/routes/datasets/DatasetList.svelte` | NEW | +| `frontend/src/routes/datasets/DatasetPreview.svelte` | NEW | +| `frontend/src/routes/datasets/ColumnsTable.svelte` | NEW | +| `frontend/src/routes/datasets/MetricsTable.svelte` | NEW | +| `frontend/src/lib/i18n/locales/en/datasets.json` | +11 keys | +| `frontend/src/lib/i18n/locales/ru/datasets.json` | +11 keys | diff --git a/specs/030-dataset-lifecycle-workspace/research.md b/specs/030-dataset-lifecycle-workspace/research.md new file mode 100644 index 00000000..741c7947 --- /dev/null +++ b/specs/030-dataset-lifecycle-workspace/research.md @@ -0,0 +1,168 @@ +# Research: Dataset Lifecycle Workspace (030) + +## R1 — Backend: Metrics in DatasetDetailResponse + +**Decision:** Add `metrics: list[MetricItem]` and `metric_count: int` to `DatasetDetailResponse`. Add `MetricItem` Pydantic DTO in `backend/src/api/routes/datasets.py`. In `SupersetClient.get_dataset_detail` (`backend/src/core/superset_client/_datasets.py`), extract `dataset.get("metrics", [])` from the Superset API response. + +**Rationale:** Superset's `GET /api/v1/dataset/{pk}` already returns `metrics` array with `metric_name`, `expression`, `verbose_name`, `description`, `metric_type`. Our backend currently ignores them. Extracting them is straightforward — no breaking API change, only additive fields. + +**Alternatives Considered:** +- A separate `/datasets/{id}/metrics` endpoint → rejected, adds unnecessary API surface for data already present +- Client-side fetch from Superset directly → rejected (bypasses auth, breaks SSR parity) + +**Impact:** `DatasetDetailResponse` gains 2 new fields. `MetricItem` DTO ~8 lines. `get_dataset_detail` gains ~15 lines. Zero DB changes. No migration. + +--- + +## R2 — Backend: Inline-edit Endpoints + +**Decision:** Two new endpoints on ss-tools backend under existing `APIRouter(prefix="/api/datasets")`: +- `PUT /api/datasets/{id}/columns/{col_id}/description` (body: `{"description": "text"}`) +- `PUT /api/datasets/{id}/metrics/{metric_id}/description` (body: `{"description": "text"}`) + +Each performs: GET full dataset from Superset via `SupersetClient.get_dataset()` → modify target column/metric description → PUT back via `SupersetClient.update_dataset()` with `?override_columns=false`. + +**Rationale:** Superset has no PATCH for individual columns/metrics — only PUT on the full dataset with `override_columns=false`. A convenience endpoint on ss-tools handles the GET→modify→PUT dance server-side, keeping frontend payloads simple (`{description: "text"}`). + +**Alternatives Considered:** +- Frontend does GET→modify→PUT directly → rejected: requires frontend to handle full Superset payload, increases attack surface +- Single `PUT /api/datasets/{id}` with `{"columns":[{"id":...,"description":...}]}` → valid but more complex for frontend, reserved for future batch updates +- Optimistic save with rollback → rejected: adds complexity without significant UX benefit for description editing + +**Impact:** +2 route handlers in `datasets.py` (~50 lines each). Reuses existing `SupersetClient.get_dataset()` and `SupersetClient.update_dataset()`. RBAC: checked via existing `has_permission("plugin:migration","READ")` or new permission. + +--- + +## R3 — Backend: Stats Counts in GET /api/datasets + +**Decision:** `GET /api/datasets` response adds `stats` object with `total`, `unmapped_count`, `mapped_count`, `linked_count`. Computed by the existing `resource_service.get_datasets_with_status()` which already fetches ALL datasets before pagination. Stats are computed from the full (non-paginated) dataset list. + +**Rationale:** The Statistics Bar needs aggregate counts. Since the backend already fetches all datasets to compute `mapped_fields` enrichment, computing stats is O(n) over the already-loaded data — near-zero cost. + +**Alternatives Considered:** +- Separate `/api/datasets/stats` endpoint → rejected, adds unnecessary request +- Client-side counts (fetch all, count locally) → rejected, pagination would break counts + +**Impact:** `DatasetsResponse` DTO gains `stats: dict`. One additional aggregation loop in the route handler (~5 lines). No new DB queries. + +--- + +## R4 — Backend: WebSocket Event `dataset.updated` + +**Decision:** After a mapping or documentation task completes, the task manager emits a WebSocket event `dataset.updated` with payload `{env_id, dataset_ids: []}`. The frontend subscribes to this event channel and automatically re-fetches `GET /api/datasets`. + +**Rationale:** Existing WebSocket infrastructure already streams task logs to the task drawer. Extending with a completion event avoids polling overhead. The `task_manager` already has an `on_task_complete` hook. + +**Alternatives Considered:** +- Polling every N seconds → rejected (unnecessary network traffic, 33 datasets trivial but doesn't scale) +- Manual refresh only → rejected by user in clarification Q5 + +**Impact:** One additional WebSocket message type in task completion handler (~5 lines). Frontend subscription in `+page.svelte` (~10 lines). No new infrastructure. + +--- + +## R5 — Frontend: Component Architecture + +**Decision:** Replace current monolithic 974-line `+page.svelte` with composition of 5 leaf components: + +``` ++page.svelte (layout orchestrator, ~200 lines) +├── StatsBar.svelte (4 metric tiles, client-side filtering) +├── DatasetList.svelte (card list with pagination, checkbox selection) +├── DatasetPreview.svelte (right panel: header + columns + metrics) +│ ├── ColumnsTable.svelte (column rows with inline-edit ✏️) +│ └── MetricsTable.svelte (metric rows with inline-edit ✏️) +└── BulkActionPanel.svelte (reused from current implementation) +``` + +Each component follows ss-tools Svelte 5 conventions (runes `$state`, `$derived`, `$props`, Tailwind classes). + +**Rationale:** SS-tools fractal limit (INV_7): modules <400 lines, contract node cyclomatic complexity <10. The current 974-line page violates both. Decomposition into <400-line components is required by ADR-0002. + +**Alternatives Considered:** +- Keep monolithic + extract inline-edit as helper → rejected (still violates INV_7) +- Use SvelteKit layout groups for sub-panels → rejected (over-engineering; this is one page, not multi-route) + +**Impact:** ~5 new `.svelte` files in `frontend/src/routes/datasets/`. Existing `[id]/+page.svelte` remains as fallback but is largely superseded by `DatasetPreview.svelte`. + +--- + +## R6 — Frontend: Inline Edit Interaction + +**Decision:** On ✏️ click, the column/metric row enters `isEditing` state. A `