ss-tools/specs/030-dataset-lifecycle-workspace/research.md

Research: Dataset Lifecycle Workspace (030)

R1 — Backend: Metrics in DatasetDetailResponse

Decision: Add metrics: list[MetricItem] and metric_count: int to DatasetDetailResponse. Add MetricItem Pydantic DTO in backend/src/api/routes/datasets.py. In SupersetClient.get_dataset_detail (backend/src/core/superset_client/_datasets.py), extract dataset.get("metrics", []) from the Superset API response.

Rationale: Superset's GET /api/v1/dataset/{pk} already returns metrics array with metric_name, expression, verbose_name, description, metric_type. Our backend currently ignores them. Extracting them is straightforward — no breaking API change, only additive fields.

Alternatives Considered:

• A separate /datasets/{id}/metrics endpoint → rejected, adds unnecessary API surface for data already present
• Client-side fetch from Superset directly → rejected (bypasses auth, breaks SSR parity)

Impact: DatasetDetailResponse gains 2 new fields. MetricItem DTO ~8 lines. get_dataset_detail gains ~15 lines. Zero DB changes. No migration.

---

R2 — Backend: Inline-edit Endpoints

Decision: Two new endpoints on superset-tools backend under existing APIRouter(prefix="/api/datasets"):

• PUT /api/datasets/{id}/columns/{col_id}/description (body: {"description": "text"})
• PUT /api/datasets/{id}/metrics/{metric_id}/description (body: {"description": "text"})

Each performs: GET full dataset from Superset via SupersetClient.get_dataset() → modify target column/metric description → PUT back via SupersetClient.update_dataset() with ?override_columns=false.

Rationale: Superset has no PATCH for individual columns/metrics — only PUT on the full dataset with override_columns=false. A convenience endpoint on superset-tools handles the GET→modify→PUT dance server-side, keeping frontend payloads simple ({description: "text"}).

Alternatives Considered:

• Frontend does GET→modify→PUT directly → rejected: requires frontend to handle full Superset payload, increases attack surface
• Single PUT /api/datasets/{id} with {"columns":[{"id":...,"description":...}]} → valid but more complex for frontend, reserved for future batch updates
• Optimistic save with rollback → rejected: adds complexity without significant UX benefit for description editing

Impact: +2 route handlers in datasets.py (~50 lines each). Reuses existing SupersetClient.get_dataset() and SupersetClient.update_dataset(). RBAC: has_permission("plugin:migration","WRITE") for mutations.

---

R3 — Backend: Stats Counts + Server-Side Filtering in GET /api/datasets

Decision: GET /api/datasets accepts optional query parameter ?filter=unmapped|mapped|linked|all. When filter is present, the backend filters the full dataset list before pagination. Response always includes stats object with total, unmapped_count, mapped_count, linked_count — computed from the full (non-paginated, non-filtered) dataset list.

Rationale: Server-side filtering ensures that all matching datasets are returned, not just those on the current page. Stats represent global state independently of the active filter. Previously considered client-side filtering was incompatible with server-side pagination (a client can only filter the current page, causing mismatch between Stats Bar counts and visible cards).

Alternatives Considered:

• Separate /api/datasets/stats endpoint → rejected, adds unnecessary request
• Client-side counts (fetch all, count locally) → rejected, pagination would break counts

Impact: DatasetsResponse DTO gains stats: dict. One additional aggregation loop in the route handler (~5 lines). No new DB queries.

---

R4 — Backend: WebSocket Event dataset.updated

Decision: After a mapping or documentation task completes, the task manager emits a WebSocket event dataset.updated with payload {env_id, dataset_ids: []}. The frontend subscribes to this event channel and automatically re-fetches GET /api/datasets.

Rationale: Existing WebSocket infrastructure already streams task logs to the task drawer. Extending with a completion event avoids polling overhead. The task_manager already has an on_task_complete hook.

Alternatives Considered:

• Polling every N seconds → rejected (unnecessary network traffic, 33 datasets trivial but doesn't scale)
• Manual refresh only → rejected by user in clarification Q5

Impact: One additional WebSocket message type in task completion handler (~5 lines). Frontend subscription in +page.svelte (~10 lines). No new infrastructure.

---

R5 — Frontend: Component Architecture

Decision: Replace current monolithic 974-line +page.svelte with composition of 5 leaf components:

+page.svelte (layout orchestrator, ~200 lines)
├── StatsBar.svelte        (4 metric tiles, emits filter → parent calls API)
├── DatasetList.svelte     (card list with pagination, checkbox selection)
├── DatasetPreview.svelte  (presentational right panel: receives dataset/loading/error props; parent fetches detail API)
│   ├── ColumnsTable.svelte    (column rows with inline-edit ✏️)
│   └── MetricsTable.svelte    (metric rows with inline-edit ✏️)
└── BulkActionPanel.svelte (reused from current implementation)

Each component follows superset-tools Svelte 5 conventions (runes $state, $derived, $props, Tailwind classes).

Rationale: SS-tools fractal limit (INV_7): modules <400 lines, contract node cyclomatic complexity <10. The current 974-line page violates both. Decomposition into <400-line components is required by ADR-0002.

Alternatives Considered:

• Keep monolithic + extract inline-edit as helper → rejected (still violates INV_7)
• Use SvelteKit layout groups for sub-panels → rejected (over-engineering; this is one page, not multi-route)

Impact: ~5 new .svelte files in frontend/src/routes/datasets/. Existing [id]/+page.svelte remains as fallback but is largely superseded by DatasetPreview.svelte.

---

R6 — Frontend: Inline Edit Interaction

Decision: On ✏️ click, the column/metric row enters isEditing state. A <textarea> replaces the description text. Save button calls PUT /api/datasets/{id}/columns/{col_id}/description. On success, row exits editing state and shows new description. On error, textarea stays open with error message. Cancel reverts to original value.

Rationale: Matches standard inline-edit UX pattern. Svelte 5 $state makes isEditing toggle trivial. Optimistic save rejected for simplicity — descriptions are short text, save latency is negligible.

Alternatives Considered:

• Modal with single textarea → rejected (too heavy for one field)
• Click-to-edit with blur-to-save → rejected (ambiguous "cancel" path)

Impact: ColumnsTable.svelte and MetricsTable.svelte each have isEditingRowId state + textarea logic. ~30 lines of template logic per table.

---

R7 — Frontend: WebSocket Subscription

Decision: +page.svelte subscribes to dataset.updated WebSocket events in onMount. On receiving event with matching env_id, calls loadDatasets() to refresh the list and stats. No visual feedback needed — list updates in-place (Svelte reactivity).

Rationale: Pushes state updates without user action. Existing WebSocket connection is reused (no new connection). Svelte reactivity handles re-render.

Alternatives Considered:

• SSE / EventSource → rejected (WebSocket already connected for task logs)
• Separate WebSocket channel → rejected (overhead)

Impact: ~10 lines in +page.svelte. Reuses wsStore or equivalent from existing WebSocket infrastructure.

---

R8 — i18n: New Keys

Decision: Add 11 new keys to frontend/src/lib/i18n/locales/{en,ru}/datasets.json: all, without_mapping, with_mapping, linked_to_dashboards, metrics, metric_count, no_selection, save_description, add_description, columns, detail_empty. No structural changes to i18n system.

Rationale: Existing i18n uses flat JSON key files loaded by locale. Adding keys is additive. No new dependencies.

Impact: +11 lines per locale file. Components reference via $t.datasets.all, etc.

---

R9 — Backend: override_columns=false Guarantee

Decision: In line-edit endpoint handlers, when calling SupersetClient.update_dataset(), always append ?override_columns=false to the PUT URL. This ensures Superset preserves all columns/metrics not mentioned in the payload.

Rationale: Superset's default is override_columns=false (safe), but explicit parameter ensures no accidental override if default changes. Risk: with override_columns=true, Superset would DELETE any column not present in the PUT payload — catastrophic.

Alternatives Considered:

• Rely on Superset default → rejected (fragile, no explicit guarantee)

Impact: One query parameter addition per PUT call.

---

R10 — Module Placement & File Map

Artifact	File Path	Operation
MetricItem DTO	backend/src/api/routes/datasets.py	ADD class
Stats counts	backend/src/api/routes/datasets.py	MODIFY DatasetsResponse, get_datasets()
Column description endpoint	backend/src/api/routes/datasets.py	ADD route PUT /columns/{col_id}/description
Metric description endpoint	backend/src/api/routes/datasets.py	ADD route PUT /metrics/{metric_id}/description
Metrics extraction	backend/src/core/superset_client/_datasets.py	MODIFY get_dataset_detail()
WebSocket event	backend/src/core/task_manager.py (or equivalent)	MODIFY on_task_complete
StatsBar	frontend/src/routes/datasets/StatsBar.svelte	NEW
DatasetList	frontend/src/routes/datasets/DatasetList.svelte	NEW
DatasetPreview	frontend/src/routes/datasets/DatasetPreview.svelte	NEW
ColumnsTable	frontend/src/routes/datasets/ColumnsTable.svelte	NEW
MetricsTable	frontend/src/routes/datasets/MetricsTable.svelte	NEW
Page orchestrator	frontend/src/routes/datasets/+page.svelte	REWRITE
i18n EN	frontend/src/lib/i18n/locales/en/datasets.json	ADD keys
i18n RU	frontend/src/lib/i18n/locales/ru/datasets.json	ADD keys
Breadcrumb	frontend/src/lib/components/layout/Breadcrumbs.svelte	MODIFY
Sidebar nav	frontend/src/lib/components/layout/sidebarNavigation.js	VERIFY

---

Unresolved / Deferred

• DatasetItem.metric_count in list view: RESOLVED — metric_count added to DatasetItem DTO as required field. Backend must include metric count in list response (fetched from Superset dataset metadata during enrichment).
• Accessibility (ARIA): Deferred to Polish phase. Components should use semantic HTML and aria-busy/aria-live where applicable. Minimum a11y requirements added to spec: Stats tiles as buttons with aria-pressed, slide-over traps focus and closes on Escape, inline edit errors announced via aria-live, skeleton loaders use aria-busy.
• Large-scale (200+ datasets): Server-side filtering (?filter=) adopted for current scope. At 200+, pagination works normally — no client-side filtering needed.
• Performance: Detail panel target <500ms applies to render after API response. Loading state must appear within 100ms. Optional cache for previously opened dataset details deferred to implementation.