docs: amend constitution to v1.6.0 (add 'Everything is a Plugin' principle) and refactor 010 plan

2026-01-07 18:36:38 +03:00
parent 297b29986d
commit a3dc1fb2b9
3 changed files with 178 additions and 7 deletions
--- a/specs/010-refactor-cli-to-web/plan.md
+++ b/specs/010-refactor-cli-to-web/plan.md
@@ -0,0 +1,101 @@
+# Implementation Plan: Refactor CLI Scripts to Web Application
+
+**Branch**: `010-refactor-cli-to-web` | **Date**: 2026-01-07 | **Spec**: [specs/010-refactor-cli-to-web/spec.md](specs/010-refactor-cli-to-web/spec.md)
+**Input**: Feature specification from `/specs/010-refactor-cli-to-web/spec.md`
+
+**Note**: This template is filled in by the `/speckit.plan` command. See `.specify/templates/commands/plan.md` for the execution workflow.
+
+## Summary
+
+This project aims to integrate the functionality of standalone CLI tools (`search_script.py`, `run_mapper.py`, `debug_db_api.py`, `get_dataset_structure.py`) into the existing Web Application (FastAPI + SvelteKit). This involves implementing new **Plugins** (`SearchPlugin`, `MapperPlugin`, `DebugPlugin`) to handle dataset searching, column mapping, and system diagnostics. The legacy CLI scripts will be removed after their functionality is successfully ported.
+
+## Technical Context
+
+<!--
+  ACTION REQUIRED: Replace the content in this section with the technical details
+  for the project. The structure here is presented in advisory capacity to guide
+  the iteration process.
+-->
+
+**Language/Version**: Python 3.9+ (Backend), Node.js 18+ (Frontend)
+**Primary Dependencies**: FastAPI, SvelteKit, Tailwind CSS, Pydantic, SQLAlchemy, `superset_tool` (internal lib)
+**Storage**: SQLite (for job history/results, connection configs), Filesystem (for temporary file uploads)
+**Testing**: pytest (Backend), vitest (Frontend - if applicable)
+**Target Platform**: Linux server (Dockerized environment)
+**Project Type**: Web Application (Backend + Frontend)
+**Performance Goals**: Search operations should handle large metadata sets without blocking the main thread (async/background tasks).
+**Constraints**: Must reuse existing `superset_tool` library. Must integrate with existing Authentication and TaskManager systems.
+**Scale/Scope**: 3 new tools (Search, Mapper, Debug), ~5-7 new API endpoints, ~3-4 new frontend pages.
+
+## Constitution Check
+
+*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
+
+1.  **Causal Validity**: Contracts (API/Data Models) will be defined before implementation.
+2.  **Immutability**: Architecture layers (`superset_tool` vs `backend`) will be respected.
+3.  **Semantic Format**: All new code will follow `[DEF]` syntax.
+4.  **Fractal Complexity**: New tools will be modularized as Plugins (`SearchPlugin`, `MapperPlugin`, `DebugPlugin`).
+
+## Project Structure
+
+### Documentation (this feature)
+
+```text
+specs/010-refactor-cli-to-web/
+├── plan.md              # This file (/speckit.plan command output)
+├── research.md          # Phase 0 output (/speckit.plan command)
+├── data-model.md        # Phase 1 output (/speckit.plan command)
+├── quickstart.md        # Phase 1 output (/speckit.plan command)
+├── contracts/           # Phase 1 output (/speckit.plan command)
+└── tasks.md             # Phase 2 output (/speckit.tasks command - NOT created by /speckit.plan)
+```
+
+### Source Code (repository root)
+<!--
+  ACTION REQUIRED: Replace the placeholder tree below with the concrete layout
+  for this feature. Delete unused options and expand the chosen structure with
+  real paths (e.g., apps/admin, packages/something). The delivered plan must
+  not include Option labels.
+-->
+
+```text
+backend/
+├── src/
+│   ├── api/
+│   │   └── routes/
+│   │       └── connections.py # New: Connection config endpoints
+│   ├── models/
+│   │   └── connection.py      # New: DB model for saved connections
+│   └── plugins/
+│       ├── search.py          # New: SearchPlugin
+│       ├── mapper.py          # New: MapperPlugin
+│       └── debug.py           # New: DebugPlugin
+└── tests/
+
+frontend/
+├── src/
+│   ├── components/
+│   │   ├── tools/             # New: UI components for tools
+│   │   │   ├── SearchTool.svelte
+│   │   │   ├── MapperTool.svelte
+│   │   │   └── DebugTool.svelte
+│   ├── routes/
+│   │   ├── tools/             # New: Pages
+│   │   │   ├── search/
+│   │   │   ├── mapper/
+│   │   │   └── debug/
+│   │   └── settings/
+│   │       └── connections/   # New: Connection management
+│   └── services/
+│       └── toolsService.js    # New: API client
+```
+
+**Structure Decision**: We will extend the existing `backend/` and `frontend/` structure. The CLI logic will be refactored into `plugins/` in the backend, executed via the generic Task API. The frontend will gain a new "Tools" section that interacts with these plugins.
+
+## Complexity Tracking
+
+> **Fill ONLY if Constitution Check has violations that must be justified**
+
+| Violation | Why Needed | Simpler Alternative Rejected Because |
+|-----------|------------|-------------------------------------|
+| N/A | | |
--- a/specs/010-refactor-cli-to-web/tasks.md
+++ b/specs/010-refactor-cli-to-web/tasks.md
@@ -0,0 +1,68 @@
+# Tasks: Refactor CLI Scripts to Web Application
+
+**Feature**: Refactor CLI Scripts to Web Application
+**Status**: Pending
+**Spec**: [specs/010-refactor-cli-to-web/spec.md](specs/010-refactor-cli-to-web/spec.md)
+
+## Phase 1: Core & Infrastructure
+*Goal: Enhance TaskManager to support returning results and set up Connection management.*
+
+- [ ] T001 Create `backend/src/models/connection.py` with `ConnectionConfig` model (ensure secure storage of credentials)
+- [ ] T002 Update `backend/src/core/task_manager/models.py` to add `result: Optional[Dict[str, Any]]` field to `Task` model
+- [ ] T003 Update `backend/src/core/task_manager/manager.py` to capture `plugin.execute` return value and store it in `task.result`
+- [ ] T004 [P] Create `frontend/src/services/toolsService.js` for generic Task API communication (run task, poll status, get result)
+- [ ] T005 [P] Create `frontend/src/services/connectionService.js` for Connection API
+- [ ] T006 Create `frontend/src/components/tools/` directory structure
+
+## Phase 2: User Story 1 - Search Datasets
+*Goal: Enable users to search datasets via the Web UI using a Plugin.*
+
+- [ ] T007 [US1] Implement `SearchPlugin` in `backend/src/plugins/search.py` (port logic from `search_script.py`, return results)
+- [ ] T008 [US1] Create `frontend/src/components/tools/SearchTool.svelte` component (triggers SearchPlugin, displays results)
+- [ ] T009 [US1] Create `frontend/src/routes/tools/search/+page.svelte` page
+
+## Phase 3: User Story 2 - Map Dataset Columns
+*Goal: Enable users to map columns using saved connections via a Plugin.*
+
+- [ ] T010 [US2] Create API endpoints for Connections (`GET`, `POST`, `DELETE`) in `backend/src/api/routes/connections.py`
+- [ ] T011 [US2] Register `connections` router in `backend/src/api/routes/__init__.py`
+- [ ] T012 [US2] Implement `MapperPlugin` in `backend/src/plugins/mapper.py` (port logic from `run_mapper.py`, support `connection_id`)
+- [ ] T013 [US2] Create `frontend/src/components/tools/ConnectionForm.svelte` and `ConnectionList.svelte`
+- [ ] T014 [US2] Create `frontend/src/routes/settings/connections/+page.svelte` for managing connections
+- [ ] T015 [US2] Create `frontend/src/components/tools/MapperTool.svelte` component
+- [ ] T016 [US2] Create `frontend/src/routes/tools/mapper/+page.svelte` page
+
+## Phase 4: User Story 3 - System Debugging
+*Goal: Enable users to run system diagnostics via a Plugin.*
+
+- [ ] T017 [US3] Implement `DebugPlugin` in `backend/src/plugins/debug.py` (port logic from `debug_db_api.py`, `get_dataset_structure.py`)
+- [ ] T018 [US3] Create `frontend/src/components/tools/DebugTool.svelte` component
+- [ ] T019 [US3] Create `frontend/src/routes/tools/debug/+page.svelte` page
+
+## Phase 5: Cleanup & Polish
+*Goal: Remove legacy scripts and ensure system stability.*
+
+- [ ] T020 Verify all new tools are functional (Manual Test)
+- [ ] T021 Verify existing Backup functionality fully covers legacy `backup_script.py` capabilities
+- [ ] T022 Remove legacy script `search_script.py`
+- [ ] T023 Remove legacy script `run_mapper.py`
+- [ ] T024 Remove legacy script `debug_db_api.py`
+- [ ] T025 Remove legacy script `get_dataset_structure.py`
+- [ ] T026 Remove legacy script `backup_script.py`
+- [ ] T027 Update main navigation in `frontend/src/components/Navbar.svelte` to include "Tools" dropdown
+
+## Dependencies
+
+- Phase 2 (Search) depends on Phase 1 (Task Result support)
+- Phase 3 (Mapper) depends on Phase 1 (Connection Model)
+- Phase 4 (Debug) depends on Phase 1 (Task Result support)
+- Phase 5 (Cleanup) depends on Phase 2, 3, 4 completion
+
+## Implementation Strategy
+
+1.  **Core**: Upgrade TaskManager first to support return values.
+2.  **Search**: Implement SearchPlugin as the first "Result-returning" plugin.
+3.  **Connections**: Build connection management.
+4.  **Mapper**: Implement MapperPlugin using Connections.
+5.  **Debug**: Implement DebugPlugin.
+6.  **Cleanup**: Verify and delete.