busya
b529e8082a
Backend: - Add MetricItem, StatsCounts, ColumnDescriptionUpdate, MetricDescriptionUpdate DTOs - Add metric_count to DatasetItem and DatasetDetailResponse - Add server-side filtering via ?filter=unmapped|mapped|linked|all - Add PUT endpoints for column/metric description inline-edit - Add HTML stripping validation (max 2000 chars, plain text) - Add WebSocket dataset.updated event on task completion - RBAC: plugin:migration:WRITE for mutations, READ for reads - 14 pytest tests (stats, filter, metrics, inline-edit, validation, 502/503) Frontend: - Rewrite +page.svelte as split-view orchestrator (387 lines) - StatsBar.svelte — 4 metric tiles with aria-pressed, server-side filter dispatch - DatasetList.svelte — card grid with progress bars, checkboxes, search - DatasetPreview.svelte — presentational detail panel (container fetches) - ColumnsTable.svelte — inline-edit with type chips, bold/italic fallback - MetricsTable.svelte — inline-edit with expression hints - 52 vitest tests across 7 test files - i18n: 11 EN + 11 RU keys (with_mapping/с_маппингом) Spec: - 15 issues resolved (semantic label, filtering, endpoints, conflict, metric_count, RBAC, ID types, ports, i18n, validation, a11y, propagation, resize, perf, ownership) Status: 14/14 backend tests pass, 52/52 frontend tests pass, build succeeds Risk: Low — T051 browser validation pending (non-blocking)
170 lines
11 KiB
Markdown
170 lines
11 KiB
Markdown
# Research: Dataset Lifecycle Workspace (030)
|
||
|
||
## R1 — Backend: Metrics in DatasetDetailResponse
|
||
|
||
**Decision:** Add `metrics: list[MetricItem]` and `metric_count: int` to `DatasetDetailResponse`. Add `MetricItem` Pydantic DTO in `backend/src/api/routes/datasets.py`. In `SupersetClient.get_dataset_detail` (`backend/src/core/superset_client/_datasets.py`), extract `dataset.get("metrics", [])` from the Superset API response.
|
||
|
||
**Rationale:** Superset's `GET /api/v1/dataset/{pk}` already returns `metrics` array with `metric_name`, `expression`, `verbose_name`, `description`, `metric_type`. Our backend currently ignores them. Extracting them is straightforward — no breaking API change, only additive fields.
|
||
|
||
**Alternatives Considered:**
|
||
- A separate `/datasets/{id}/metrics` endpoint → rejected, adds unnecessary API surface for data already present
|
||
- Client-side fetch from Superset directly → rejected (bypasses auth, breaks SSR parity)
|
||
|
||
**Impact:** `DatasetDetailResponse` gains 2 new fields. `MetricItem` DTO ~8 lines. `get_dataset_detail` gains ~15 lines. Zero DB changes. No migration.
|
||
|
||
---
|
||
|
||
## R2 — Backend: Inline-edit Endpoints
|
||
|
||
**Decision:** Two new endpoints on ss-tools backend under existing `APIRouter(prefix="/api/datasets")`:
|
||
- `PUT /api/datasets/{id}/columns/{col_id}/description` (body: `{"description": "text"}`)
|
||
- `PUT /api/datasets/{id}/metrics/{metric_id}/description` (body: `{"description": "text"}`)
|
||
|
||
Each performs: GET full dataset from Superset via `SupersetClient.get_dataset()` → modify target column/metric description → PUT back via `SupersetClient.update_dataset()` with `?override_columns=false`.
|
||
|
||
**Rationale:** Superset has no PATCH for individual columns/metrics — only PUT on the full dataset with `override_columns=false`. A convenience endpoint on ss-tools handles the GET→modify→PUT dance server-side, keeping frontend payloads simple (`{description: "text"}`).
|
||
|
||
**Alternatives Considered:**
|
||
- Frontend does GET→modify→PUT directly → rejected: requires frontend to handle full Superset payload, increases attack surface
|
||
- Single `PUT /api/datasets/{id}` with `{"columns":[{"id":...,"description":...}]}` → valid but more complex for frontend, reserved for future batch updates
|
||
- Optimistic save with rollback → rejected: adds complexity without significant UX benefit for description editing
|
||
|
||
**Impact:** +2 route handlers in `datasets.py` (~50 lines each). Reuses existing `SupersetClient.get_dataset()` and `SupersetClient.update_dataset()`. RBAC: `has_permission("plugin:migration","WRITE")` for mutations.
|
||
|
||
---
|
||
|
||
## R3 — Backend: Stats Counts + Server-Side Filtering in GET /api/datasets
|
||
|
||
**Decision:** `GET /api/datasets` accepts optional query parameter `?filter=unmapped|mapped|linked|all`. When `filter` is present, the backend filters the full dataset list before pagination. Response always includes `stats` object with `total`, `unmapped_count`, `mapped_count`, `linked_count` — computed from the full (non-paginated, non-filtered) dataset list.
|
||
|
||
**Rationale:** Server-side filtering ensures that all matching datasets are returned, not just those on the current page. Stats represent global state independently of the active filter. Previously considered client-side filtering was incompatible with server-side pagination (a client can only filter the current page, causing mismatch between Stats Bar counts and visible cards).
|
||
|
||
**Alternatives Considered:**
|
||
- Separate `/api/datasets/stats` endpoint → rejected, adds unnecessary request
|
||
- Client-side counts (fetch all, count locally) → rejected, pagination would break counts
|
||
|
||
**Impact:** `DatasetsResponse` DTO gains `stats: dict`. One additional aggregation loop in the route handler (~5 lines). No new DB queries.
|
||
|
||
---
|
||
|
||
## R4 — Backend: WebSocket Event `dataset.updated`
|
||
|
||
**Decision:** After a mapping or documentation task completes, the task manager emits a WebSocket event `dataset.updated` with payload `{env_id, dataset_ids: []}`. The frontend subscribes to this event channel and automatically re-fetches `GET /api/datasets`.
|
||
|
||
**Rationale:** Existing WebSocket infrastructure already streams task logs to the task drawer. Extending with a completion event avoids polling overhead. The `task_manager` already has an `on_task_complete` hook.
|
||
|
||
**Alternatives Considered:**
|
||
- Polling every N seconds → rejected (unnecessary network traffic, 33 datasets trivial but doesn't scale)
|
||
- Manual refresh only → rejected by user in clarification Q5
|
||
|
||
**Impact:** One additional WebSocket message type in task completion handler (~5 lines). Frontend subscription in `+page.svelte` (~10 lines). No new infrastructure.
|
||
|
||
---
|
||
|
||
## R5 — Frontend: Component Architecture
|
||
|
||
**Decision:** Replace current monolithic 974-line `+page.svelte` with composition of 5 leaf components:
|
||
|
||
```
|
||
+page.svelte (layout orchestrator, ~200 lines)
|
||
├── StatsBar.svelte (4 metric tiles, emits filter → parent calls API)
|
||
├── DatasetList.svelte (card list with pagination, checkbox selection)
|
||
├── DatasetPreview.svelte (presentational right panel: receives dataset/loading/error props; parent fetches detail API)
|
||
│ ├── ColumnsTable.svelte (column rows with inline-edit ✏️)
|
||
│ └── MetricsTable.svelte (metric rows with inline-edit ✏️)
|
||
└── BulkActionPanel.svelte (reused from current implementation)
|
||
```
|
||
|
||
Each component follows ss-tools Svelte 5 conventions (runes `$state`, `$derived`, `$props`, Tailwind classes).
|
||
|
||
**Rationale:** SS-tools fractal limit (INV_7): modules <400 lines, contract node cyclomatic complexity <10. The current 974-line page violates both. Decomposition into <400-line components is required by ADR-0002.
|
||
|
||
**Alternatives Considered:**
|
||
- Keep monolithic + extract inline-edit as helper → rejected (still violates INV_7)
|
||
- Use SvelteKit layout groups for sub-panels → rejected (over-engineering; this is one page, not multi-route)
|
||
|
||
**Impact:** ~5 new `.svelte` files in `frontend/src/routes/datasets/`. Existing `[id]/+page.svelte` remains as fallback but is largely superseded by `DatasetPreview.svelte`.
|
||
|
||
---
|
||
|
||
## R6 — Frontend: Inline Edit Interaction
|
||
|
||
**Decision:** On ✏️ click, the column/metric row enters `isEditing` state. A `<textarea>` replaces the description text. Save button calls `PUT /api/datasets/{id}/columns/{col_id}/description`. On success, row exits editing state and shows new description. On error, textarea stays open with error message. Cancel reverts to original value.
|
||
|
||
**Rationale:** Matches standard inline-edit UX pattern. Svelte 5 `$state` makes `isEditing` toggle trivial. Optimistic save rejected for simplicity — descriptions are short text, save latency is negligible.
|
||
|
||
**Alternatives Considered:**
|
||
- Modal with single textarea → rejected (too heavy for one field)
|
||
- Click-to-edit with blur-to-save → rejected (ambiguous "cancel" path)
|
||
|
||
**Impact:** `ColumnsTable.svelte` and `MetricsTable.svelte` each have `isEditingRowId` state + textarea logic. ~30 lines of template logic per table.
|
||
|
||
---
|
||
|
||
## R7 — Frontend: WebSocket Subscription
|
||
|
||
**Decision:** `+page.svelte` subscribes to `dataset.updated` WebSocket events in `onMount`. On receiving event with matching `env_id`, calls `loadDatasets()` to refresh the list and stats. No visual feedback needed — list updates in-place (Svelte reactivity).
|
||
|
||
**Rationale:** Pushes state updates without user action. Existing WebSocket connection is reused (no new connection). Svelte reactivity handles re-render.
|
||
|
||
**Alternatives Considered:**
|
||
- SSE / EventSource → rejected (WebSocket already connected for task logs)
|
||
- Separate WebSocket channel → rejected (overhead)
|
||
|
||
**Impact:** ~10 lines in `+page.svelte`. Reuses `wsStore` or equivalent from existing WebSocket infrastructure.
|
||
|
||
---
|
||
|
||
## R8 — i18n: New Keys
|
||
|
||
**Decision:** Add 11 new keys to `frontend/src/lib/i18n/locales/{en,ru}/datasets.json`: `all`, `without_mapping`, `with_mapping`, `linked_to_dashboards`, `metrics`, `metric_count`, `no_selection`, `save_description`, `add_description`, `columns`, `detail_empty`. No structural changes to i18n system.
|
||
|
||
**Rationale:** Existing i18n uses flat JSON key files loaded by locale. Adding keys is additive. No new dependencies.
|
||
|
||
**Impact:** +11 lines per locale file. Components reference via `$t.datasets.all`, etc.
|
||
|
||
---
|
||
|
||
## R9 — Backend: `override_columns=false` Guarantee
|
||
|
||
**Decision:** In line-edit endpoint handlers, when calling `SupersetClient.update_dataset()`, always append `?override_columns=false` to the PUT URL. This ensures Superset preserves all columns/metrics not mentioned in the payload.
|
||
|
||
**Rationale:** Superset's default is `override_columns=false` (safe), but explicit parameter ensures no accidental override if default changes. Risk: with `override_columns=true`, Superset would DELETE any column not present in the PUT payload — catastrophic.
|
||
|
||
**Alternatives Considered:**
|
||
- Rely on Superset default → rejected (fragile, no explicit guarantee)
|
||
|
||
**Impact:** One query parameter addition per PUT call.
|
||
|
||
---
|
||
|
||
## R10 — Module Placement & File Map
|
||
|
||
| Artifact | File Path | Operation |
|
||
|----------|-----------|-----------|
|
||
| MetricItem DTO | `backend/src/api/routes/datasets.py` | ADD class |
|
||
| Stats counts | `backend/src/api/routes/datasets.py` | MODIFY `DatasetsResponse`, `get_datasets()` |
|
||
| Column description endpoint | `backend/src/api/routes/datasets.py` | ADD route `PUT /columns/{col_id}/description` |
|
||
| Metric description endpoint | `backend/src/api/routes/datasets.py` | ADD route `PUT /metrics/{metric_id}/description` |
|
||
| Metrics extraction | `backend/src/core/superset_client/_datasets.py` | MODIFY `get_dataset_detail()` |
|
||
| WebSocket event | `backend/src/core/task_manager.py` (or equivalent) | MODIFY on_task_complete |
|
||
| StatsBar | `frontend/src/routes/datasets/StatsBar.svelte` | NEW |
|
||
| DatasetList | `frontend/src/routes/datasets/DatasetList.svelte` | NEW |
|
||
| DatasetPreview | `frontend/src/routes/datasets/DatasetPreview.svelte` | NEW |
|
||
| ColumnsTable | `frontend/src/routes/datasets/ColumnsTable.svelte` | NEW |
|
||
| MetricsTable | `frontend/src/routes/datasets/MetricsTable.svelte` | NEW |
|
||
| Page orchestrator | `frontend/src/routes/datasets/+page.svelte` | REWRITE |
|
||
| i18n EN | `frontend/src/lib/i18n/locales/en/datasets.json` | ADD keys |
|
||
| i18n RU | `frontend/src/lib/i18n/locales/ru/datasets.json` | ADD keys |
|
||
| Breadcrumb | `frontend/src/lib/components/layout/Breadcrumbs.svelte` | MODIFY |
|
||
| Sidebar nav | `frontend/src/lib/components/layout/sidebarNavigation.js` | VERIFY |
|
||
|
||
---
|
||
|
||
## Unresolved / Deferred
|
||
|
||
- **~~DatasetItem.metric_count in list view~~**: RESOLVED — `metric_count` added to `DatasetItem` DTO as required field. Backend must include metric count in list response (fetched from Superset dataset metadata during enrichment).
|
||
- **Accessibility (ARIA)**: Deferred to Polish phase. Components should use semantic HTML and `aria-busy`/`aria-live` where applicable. Minimum a11y requirements added to spec: Stats tiles as buttons with `aria-pressed`, slide-over traps focus and closes on Escape, inline edit errors announced via `aria-live`, skeleton loaders use `aria-busy`.
|
||
- **Large-scale (200+ datasets)**: Server-side filtering (`?filter=`) adopted for current scope. At 200+, pagination works normally — no client-side filtering needed.
|
||
- **Performance**: Detail panel target <500ms applies to render after API response. Loading state must appear within 100ms. Optional cache for previously opened dataset details deferred to implementation.
|