diff --git a/.specify/memory/constitution.md b/.specify/memory/constitution.md index 56a089a..bc213d9 100644 --- a/.specify/memory/constitution.md +++ b/.specify/memory/constitution.md @@ -1,14 +1,13 @@ # Semantic Code Generation Constitution @@ -47,6 +46,9 @@ To maintain semantic coherence and avoid "Attention Sink" issues: This ensures every vector embedding remains sharp and focused. +### VII. Everything is a Plugin +All functional extensions, tools, or major features must be implemented as modular Plugins inheriting from `PluginBase`. Logic should not reside in standalone services or scripts unless strictly necessary for core infrastructure. This ensures a unified execution model via the `TaskManager`, consistent logging, and modularity. + ## File Structure Standards ### Python Modules @@ -96,4 +98,4 @@ This Constitution establishes the "Semantic Code Generation Protocol" as the sup - **Review**: Code reviews must verify that implementation matches the preceding contracts and that no "naked code" exists outside of semantic anchors. - **Compliance**: Failure to adhere to the `[DEF]` / `[/DEF]` structure (including matching closing tags) constitutes a build failure. -**Version**: 1.5.0 | **Ratified**: 2025-12-19 | **Last Amended**: 2025-12-27 +**Version**: 1.6.0 | **Ratified**: 2025-12-19 | **Last Amended**: 2026-01-07 diff --git a/specs/010-refactor-cli-to-web/plan.md b/specs/010-refactor-cli-to-web/plan.md new file mode 100644 index 0000000..054a0b7 --- /dev/null +++ b/specs/010-refactor-cli-to-web/plan.md @@ -0,0 +1,101 @@ +# Implementation Plan: Refactor CLI Scripts to Web Application + +**Branch**: `010-refactor-cli-to-web` | **Date**: 2026-01-07 | **Spec**: [specs/010-refactor-cli-to-web/spec.md](specs/010-refactor-cli-to-web/spec.md) +**Input**: Feature specification from `/specs/010-refactor-cli-to-web/spec.md` + +**Note**: This template is filled in by the `/speckit.plan` command. See `.specify/templates/commands/plan.md` for the execution workflow. + +## Summary + +This project aims to integrate the functionality of standalone CLI tools (`search_script.py`, `run_mapper.py`, `debug_db_api.py`, `get_dataset_structure.py`) into the existing Web Application (FastAPI + SvelteKit). This involves implementing new **Plugins** (`SearchPlugin`, `MapperPlugin`, `DebugPlugin`) to handle dataset searching, column mapping, and system diagnostics. The legacy CLI scripts will be removed after their functionality is successfully ported. + +## Technical Context + + + +**Language/Version**: Python 3.9+ (Backend), Node.js 18+ (Frontend) +**Primary Dependencies**: FastAPI, SvelteKit, Tailwind CSS, Pydantic, SQLAlchemy, `superset_tool` (internal lib) +**Storage**: SQLite (for job history/results, connection configs), Filesystem (for temporary file uploads) +**Testing**: pytest (Backend), vitest (Frontend - if applicable) +**Target Platform**: Linux server (Dockerized environment) +**Project Type**: Web Application (Backend + Frontend) +**Performance Goals**: Search operations should handle large metadata sets without blocking the main thread (async/background tasks). +**Constraints**: Must reuse existing `superset_tool` library. Must integrate with existing Authentication and TaskManager systems. +**Scale/Scope**: 3 new tools (Search, Mapper, Debug), ~5-7 new API endpoints, ~3-4 new frontend pages. + +## Constitution Check + +*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.* + +1. **Causal Validity**: Contracts (API/Data Models) will be defined before implementation. +2. **Immutability**: Architecture layers (`superset_tool` vs `backend`) will be respected. +3. **Semantic Format**: All new code will follow `[DEF]` syntax. +4. **Fractal Complexity**: New tools will be modularized as Plugins (`SearchPlugin`, `MapperPlugin`, `DebugPlugin`). + +## Project Structure + +### Documentation (this feature) + +```text +specs/010-refactor-cli-to-web/ +├── plan.md # This file (/speckit.plan command output) +├── research.md # Phase 0 output (/speckit.plan command) +├── data-model.md # Phase 1 output (/speckit.plan command) +├── quickstart.md # Phase 1 output (/speckit.plan command) +├── contracts/ # Phase 1 output (/speckit.plan command) +└── tasks.md # Phase 2 output (/speckit.tasks command - NOT created by /speckit.plan) +``` + +### Source Code (repository root) + + +```text +backend/ +├── src/ +│ ├── api/ +│ │ └── routes/ +│ │ └── connections.py # New: Connection config endpoints +│ ├── models/ +│ │ └── connection.py # New: DB model for saved connections +│ └── plugins/ +│ ├── search.py # New: SearchPlugin +│ ├── mapper.py # New: MapperPlugin +│ └── debug.py # New: DebugPlugin +└── tests/ + +frontend/ +├── src/ +│ ├── components/ +│ │ ├── tools/ # New: UI components for tools +│ │ │ ├── SearchTool.svelte +│ │ │ ├── MapperTool.svelte +│ │ │ └── DebugTool.svelte +│ ├── routes/ +│ │ ├── tools/ # New: Pages +│ │ │ ├── search/ +│ │ │ ├── mapper/ +│ │ │ └── debug/ +│ │ └── settings/ +│ │ └── connections/ # New: Connection management +│ └── services/ +│ └── toolsService.js # New: API client +``` + +**Structure Decision**: We will extend the existing `backend/` and `frontend/` structure. The CLI logic will be refactored into `plugins/` in the backend, executed via the generic Task API. The frontend will gain a new "Tools" section that interacts with these plugins. + +## Complexity Tracking + +> **Fill ONLY if Constitution Check has violations that must be justified** + +| Violation | Why Needed | Simpler Alternative Rejected Because | +|-----------|------------|-------------------------------------| +| N/A | | | diff --git a/specs/010-refactor-cli-to-web/tasks.md b/specs/010-refactor-cli-to-web/tasks.md new file mode 100644 index 0000000..b44bdc7 --- /dev/null +++ b/specs/010-refactor-cli-to-web/tasks.md @@ -0,0 +1,68 @@ +# Tasks: Refactor CLI Scripts to Web Application + +**Feature**: Refactor CLI Scripts to Web Application +**Status**: Pending +**Spec**: [specs/010-refactor-cli-to-web/spec.md](specs/010-refactor-cli-to-web/spec.md) + +## Phase 1: Core & Infrastructure +*Goal: Enhance TaskManager to support returning results and set up Connection management.* + +- [ ] T001 Create `backend/src/models/connection.py` with `ConnectionConfig` model (ensure secure storage of credentials) +- [ ] T002 Update `backend/src/core/task_manager/models.py` to add `result: Optional[Dict[str, Any]]` field to `Task` model +- [ ] T003 Update `backend/src/core/task_manager/manager.py` to capture `plugin.execute` return value and store it in `task.result` +- [ ] T004 [P] Create `frontend/src/services/toolsService.js` for generic Task API communication (run task, poll status, get result) +- [ ] T005 [P] Create `frontend/src/services/connectionService.js` for Connection API +- [ ] T006 Create `frontend/src/components/tools/` directory structure + +## Phase 2: User Story 1 - Search Datasets +*Goal: Enable users to search datasets via the Web UI using a Plugin.* + +- [ ] T007 [US1] Implement `SearchPlugin` in `backend/src/plugins/search.py` (port logic from `search_script.py`, return results) +- [ ] T008 [US1] Create `frontend/src/components/tools/SearchTool.svelte` component (triggers SearchPlugin, displays results) +- [ ] T009 [US1] Create `frontend/src/routes/tools/search/+page.svelte` page + +## Phase 3: User Story 2 - Map Dataset Columns +*Goal: Enable users to map columns using saved connections via a Plugin.* + +- [ ] T010 [US2] Create API endpoints for Connections (`GET`, `POST`, `DELETE`) in `backend/src/api/routes/connections.py` +- [ ] T011 [US2] Register `connections` router in `backend/src/api/routes/__init__.py` +- [ ] T012 [US2] Implement `MapperPlugin` in `backend/src/plugins/mapper.py` (port logic from `run_mapper.py`, support `connection_id`) +- [ ] T013 [US2] Create `frontend/src/components/tools/ConnectionForm.svelte` and `ConnectionList.svelte` +- [ ] T014 [US2] Create `frontend/src/routes/settings/connections/+page.svelte` for managing connections +- [ ] T015 [US2] Create `frontend/src/components/tools/MapperTool.svelte` component +- [ ] T016 [US2] Create `frontend/src/routes/tools/mapper/+page.svelte` page + +## Phase 4: User Story 3 - System Debugging +*Goal: Enable users to run system diagnostics via a Plugin.* + +- [ ] T017 [US3] Implement `DebugPlugin` in `backend/src/plugins/debug.py` (port logic from `debug_db_api.py`, `get_dataset_structure.py`) +- [ ] T018 [US3] Create `frontend/src/components/tools/DebugTool.svelte` component +- [ ] T019 [US3] Create `frontend/src/routes/tools/debug/+page.svelte` page + +## Phase 5: Cleanup & Polish +*Goal: Remove legacy scripts and ensure system stability.* + +- [ ] T020 Verify all new tools are functional (Manual Test) +- [ ] T021 Verify existing Backup functionality fully covers legacy `backup_script.py` capabilities +- [ ] T022 Remove legacy script `search_script.py` +- [ ] T023 Remove legacy script `run_mapper.py` +- [ ] T024 Remove legacy script `debug_db_api.py` +- [ ] T025 Remove legacy script `get_dataset_structure.py` +- [ ] T026 Remove legacy script `backup_script.py` +- [ ] T027 Update main navigation in `frontend/src/components/Navbar.svelte` to include "Tools" dropdown + +## Dependencies + +- Phase 2 (Search) depends on Phase 1 (Task Result support) +- Phase 3 (Mapper) depends on Phase 1 (Connection Model) +- Phase 4 (Debug) depends on Phase 1 (Task Result support) +- Phase 5 (Cleanup) depends on Phase 2, 3, 4 completion + +## Implementation Strategy + +1. **Core**: Upgrade TaskManager first to support return values. +2. **Search**: Implement SearchPlugin as the first "Result-returning" plugin. +3. **Connections**: Build connection management. +4. **Mapper**: Implement MapperPlugin using Connections. +5. **Debug**: Implement DebugPlugin. +6. **Cleanup**: Verify and delete. \ No newline at end of file