tasks ready

.kilocode/rules/specify-rules.md

@@ -18,6 +18,8 @@ Auto-generated from all feature plans. Last updated: 2025-12-19
 - Python 3.9+, Node.js 18+ + FastAPI, SvelteKit, Tailwind CSS, Pydantic, SQLAlchemy, Superset API (008-migration-ui-improvements)
 - Python 3.9+, Node.js 18+ + FastAPI, APScheduler, SQLAlchemy, SvelteKit, Tailwind CSS (009-backup-scheduler)
 - SQLite (`tasks.db`), JSON (`config.json`) (009-backup-scheduler)
+- Python 3.9+ (Backend), Node.js 18+ (Frontend) + FastAPI, SvelteKit, Tailwind CSS, Pydantic, SQLAlchemy, `superset_tool` (internal lib) (010-refactor-cli-to-web)
+- SQLite (for job history/results, connection configs), Filesystem (for temporary file uploads) (010-refactor-cli-to-web)
 
 - Python 3.9+ (Backend), Node.js 18+ (Frontend Build) (001-plugin-arch-svelte-ui)
 
@@ -38,9 +40,9 @@ cd src; pytest; ruff check .
 Python 3.9+ (Backend), Node.js 18+ (Frontend Build): Follow standard conventions
 
 ## Recent Changes
+- 010-refactor-cli-to-web: Added Python 3.9+ (Backend), Node.js 18+ (Frontend) + FastAPI, SvelteKit, Tailwind CSS, Pydantic, SQLAlchemy, `superset_tool` (internal lib)
 - 009-backup-scheduler: Added Python 3.9+, Node.js 18+ + FastAPI, APScheduler, SQLAlchemy, SvelteKit, Tailwind CSS
 - 009-backup-scheduler: Added Python 3.9+, Node.js 18+ + FastAPI, APScheduler, SQLAlchemy, SvelteKit, Tailwind CSS
-- 009-backup-scheduler: Added [if applicable, e.g., PostgreSQL, CoreData, files or N/A]
 
 
 <!-- MANUAL ADDITIONS START -->

.specify/memory/constitution.md

@@ -1,14 +1,13 @@
 <!--
 SYNC IMPACT REPORT
-Version: 1.5.0 (Fractal Complexity Limit)
+Version: 1.6.0 (Plugin Architecture Mandate)
 Changes:
-- Added Section VI (Fractal Complexity Limit) to enforce strict module (~300 lines) and function (~30-50 lines) size limits.
-- Aims to maintain semantic coherence and avoid "Attention Sink".
+- Added Principle VII (Everything is a Plugin) to enforce modular plugin architecture for all features.
+- Updated Governance section.
 Templates Status:
-- .specify/templates/plan-template.md: ✅ Aligned.
+- .specify/templates/plan-template.md: ✅ Aligned (Dynamic check).
 - .specify/templates/spec-template.md: ✅ Aligned.
-- .specify/templates/tasks-arch-template.md: ✅ Aligned (New role-based split).
-- .specify/templates/tasks-dev-template.md: ✅ Aligned (New role-based split).
+- .specify/templates/tasks-template.md: ✅ Aligned.
 -->
 # Semantic Code Generation Constitution
 
@@ -47,6 +46,9 @@ To maintain semantic coherence and avoid "Attention Sink" issues:
 
 This ensures every vector embedding remains sharp and focused.
 
+### VII. Everything is a Plugin
+All functional extensions, tools, or major features must be implemented as modular Plugins inheriting from `PluginBase`. Logic should not reside in standalone services or scripts unless strictly necessary for core infrastructure. This ensures a unified execution model via the `TaskManager`, consistent logging, and modularity.
+
 ## File Structure Standards
 
 ### Python Modules
@@ -96,4 +98,4 @@ This Constitution establishes the "Semantic Code Generation Protocol" as the sup
 - **Review**: Code reviews must verify that implementation matches the preceding contracts and that no "naked code" exists outside of semantic anchors.
 - **Compliance**: Failure to adhere to the `[DEF]` / `[/DEF]` structure (including matching closing tags) constitutes a build failure.
 
-**Version**: 1.5.0 | **Ratified**: 2025-12-19 | **Last Amended**: 2025-12-27
+**Version**: 1.6.0 | **Ratified**: 2025-12-19 | **Last Amended**: 2026-01-07

specs/010-refactor-cli-to-web/checklists/requirements.md

@@ -0,0 +1,34 @@
+# Specification Quality Checklist: Refactor CLI Scripts to Web Application
+
+**Purpose**: Validate specification completeness and quality before proceeding to planning
+**Created**: 2026-01-07
+**Feature**: [Link to spec.md](../spec.md)
+
+## Content Quality
+
+- [x] No implementation details (languages, frameworks, APIs)
+- [x] Focused on user value and business needs
+- [x] Written for non-technical stakeholders
+- [x] All mandatory sections completed
+
+## Requirement Completeness
+
+- [x] No [NEEDS CLARIFICATION] markers remain
+- [x] Requirements are testable and unambiguous
+- [x] Success criteria are measurable
+- [x] Success criteria are technology-agnostic (no implementation details)
+- [x] All acceptance scenarios are defined
+- [x] Edge cases are identified
+- [x] Scope is clearly bounded
+- [x] Dependencies and assumptions identified
+
+## Feature Readiness
+
+- [x] All functional requirements have clear acceptance criteria
+- [x] User scenarios cover primary flows
+- [x] Feature meets measurable outcomes defined in Success Criteria
+- [x] No implementation details leak into specification
+
+## Notes
+
+- Spec is ready for planning.

specs/010-refactor-cli-to-web/contracts/api.md

@@ -0,0 +1,134 @@
+# API Contracts: Refactor CLI Scripts to Web Application
+
+## 1. Tools API
+
+### 1.1. Search Datasets
+**Endpoint**: `POST /api/tools/search`
+**Description**: Search for text patterns across all datasets in a specific environment.
+
+**Request Body**:
+```json
+{
+  "env": "dev",
+  "query": "regex_pattern"
+}
+```
+
+**Response (200 OK)**:
+```json
+{
+  "count": 5,
+  "results": [
+    {
+      "dataset_id": 123,
+      "dataset_name": "sales_data",
+      "field": "sql",
+      "match_context": "SELECT * FROM ...",
+      "full_value": "SELECT * FROM sales WHERE ..."
+    }
+  ]
+}
+```
+
+### 1.2. Debug Database API
+**Endpoint**: `POST /api/tools/debug/db-api`
+**Description**: Test database API connectivity and structure between two environments.
+
+**Request Body**:
+```json
+{
+  "source_env": "dev",
+  "target_env": "prod"
+}
+```
+
+**Response (200 OK)**:
+```json
+{
+  "source_db_count": 10,
+  "target_db_count": 12,
+  "details": {
+    "source_dbs": [...],
+    "target_dbs": [...]
+  }
+}
+```
+
+### 1.3. Get Dataset Structure
+**Endpoint**: `GET /api/tools/debug/dataset/{env}/{dataset_id}`
+**Description**: Retrieve the full JSON structure of a dataset.
+
+**Response (200 OK)**:
+```json
+{
+  "id": 123,
+  "table_name": "sales",
+  "columns": [...],
+  "metrics": [...]
+}
+```
+
+## 2. Connection Management API
+
+### 2.1. List Connections
+**Endpoint**: `GET /api/settings/connections`
+**Response (200 OK)**:
+```json
+[
+  {
+    "id": "uuid",
+    "name": "Production DWH",
+    "type": "postgres",
+    "host": "10.0.0.1",
+    "database": "dwh",
+    "username": "user",
+    "created_at": "2026-01-07T10:00:00Z"
+  }
+]
+```
+
+### 2.2. Create Connection
+**Endpoint**: `POST /api/settings/connections`
+**Request Body**:
+```json
+{
+  "name": "Production DWH",
+  "type": "postgres",
+  "host": "10.0.0.1",
+  "port": 5432,
+  "database": "dwh",
+  "username": "user",
+  "password": "secret_password"
+}
+```
+**Response (201 Created)**:
+```json
+{
+  "id": "uuid",
+  "name": "Production DWH",
+  "type": "postgres",
+  ...
+}
+```
+
+### 2.3. Delete Connection
+**Endpoint**: `DELETE /api/settings/connections/{id}`
+**Response (204 No Content)**
+
+## 3. Task API (Existing, extended for Mapping)
+
+### 3.1. Create Mapping Task
+**Endpoint**: `POST /api/tasks`
+**Request Body**:
+```json
+{
+  "plugin_id": "dataset-mapper",
+  "params": {
+    "env": "dev",
+    "dataset_id": 123,
+    "source": "postgres",
+    "connection_id": "uuid-of-saved-connection",
+    "table_name": "sales",
+    "table_schema": "public"
+  }
+}

specs/010-refactor-cli-to-web/data-model.md

@@ -0,0 +1,77 @@
+# Data Model: Refactor CLI Scripts to Web Application
+
+## 1. Connection Configuration
+
+To support the "Dataset Mapper" tool with reusable connections (as per spec), we need a way to store external database credentials.
+
+### Entity: `ConnectionConfig`
+
+*   **Table**: `connection_configs`
+*   **Purpose**: Stores credentials for external databases (e.g., PostgreSQL) used for column mapping.
+
+| Field | Type | Required | Description |
+| :--- | :--- | :--- | :--- |
+| `id` | UUID | Yes | Primary Key |
+| `name` | String | Yes | User-friendly name (e.g., "Production DWH") |
+| `type` | String | Yes | Enum: `postgres`, `excel` (future) |
+| `host` | String | No | DB Host (for postgres) |
+| `port` | Integer | No | DB Port (for postgres) |
+| `database` | String | No | DB Name (for postgres) |
+| `username` | String | No | DB User (for postgres) |
+| `password` | String | No | Encrypted/Obfuscated password (for postgres) |
+| `created_at` | DateTime | Yes | Creation timestamp |
+| `updated_at` | DateTime | Yes | Last update timestamp |
+
+## 2. Tool Request/Response Models (Pydantic)
+
+These models define the API contracts for the new tools.
+
+### Search Tool
+
+#### `SearchRequest`
+```python
+class SearchRequest(BaseModel):
+    env: str  # e.g., "dev", "prod"
+    query: str # Regex pattern
+```
+
+#### `SearchResultItem`
+```python
+class SearchResultItem(BaseModel):
+    dataset_id: int
+    dataset_name: str
+    field: str
+    match_context: str
+    full_value: str
+```
+
+#### `SearchResponse`
+```python
+class SearchResponse(BaseModel):
+    results: List[SearchResultItem]
+    count: int
+```
+
+### Debug Tool
+
+#### `DebugDbRequest`
+```python
+class DebugDbRequest(BaseModel):
+    source_env: str
+    target_env: str
+```
+
+#### `DebugDbResponse`
+```python
+class DebugDbResponse(BaseModel):
+    source_db_count: int
+    target_db_count: int
+    details: Dict[str, Any] # Full JSON dump
+```
+
+#### `DatasetStructureRequest`
+```python
+class DatasetStructureRequest(BaseModel):
+    env: str
+    dataset_id: int
+```

specs/010-refactor-cli-to-web/plan.md

@@ -0,0 +1,101 @@
+# Implementation Plan: Refactor CLI Scripts to Web Application
+
+**Branch**: `010-refactor-cli-to-web` | **Date**: 2026-01-07 | **Spec**: [specs/010-refactor-cli-to-web/spec.md](specs/010-refactor-cli-to-web/spec.md)
+**Input**: Feature specification from `/specs/010-refactor-cli-to-web/spec.md`
+
+**Note**: This template is filled in by the `/speckit.plan` command. See `.specify/templates/commands/plan.md` for the execution workflow.
+
+## Summary
+
+This project aims to integrate the functionality of standalone CLI tools (`search_script.py`, `run_mapper.py`, `debug_db_api.py`, `get_dataset_structure.py`) into the existing Web Application (FastAPI + SvelteKit). This involves implementing new **Plugins** (`SearchPlugin`, `MapperPlugin`, `DebugPlugin`) to handle dataset searching, column mapping, and system diagnostics. The legacy CLI scripts will be removed after their functionality is successfully ported.
+
+## Technical Context
+
+<!--
+  ACTION REQUIRED: Replace the content in this section with the technical details
+  for the project. The structure here is presented in advisory capacity to guide
+  the iteration process.
+-->
+
+**Language/Version**: Python 3.9+ (Backend), Node.js 18+ (Frontend)
+**Primary Dependencies**: FastAPI, SvelteKit, Tailwind CSS, Pydantic, SQLAlchemy, `superset_tool` (internal lib)
+**Storage**: SQLite (for job history/results, connection configs), Filesystem (for temporary file uploads)
+**Testing**: pytest (Backend), vitest (Frontend - if applicable)
+**Target Platform**: Linux server (Dockerized environment)
+**Project Type**: Web Application (Backend + Frontend)
+**Performance Goals**: Search operations should handle large metadata sets without blocking the main thread (async/background tasks).
+**Constraints**: Must reuse existing `superset_tool` library. Must integrate with existing Authentication and TaskManager systems.
+**Scale/Scope**: 3 new tools (Search, Mapper, Debug), ~5-7 new API endpoints, ~3-4 new frontend pages.
+
+## Constitution Check
+
+*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
+
+1.  **Causal Validity**: Contracts (API/Data Models) will be defined before implementation.
+2.  **Immutability**: Architecture layers (`superset_tool` vs `backend`) will be respected.
+3.  **Semantic Format**: All new code will follow `[DEF]` syntax.
+4.  **Fractal Complexity**: New tools will be modularized as Plugins (`SearchPlugin`, `MapperPlugin`, `DebugPlugin`).
+
+## Project Structure
+
+### Documentation (this feature)
+
+```text
+specs/010-refactor-cli-to-web/
+├── plan.md              # This file (/speckit.plan command output)
+├── research.md          # Phase 0 output (/speckit.plan command)
+├── data-model.md        # Phase 1 output (/speckit.plan command)
+├── quickstart.md        # Phase 1 output (/speckit.plan command)
+├── contracts/           # Phase 1 output (/speckit.plan command)
+└── tasks.md             # Phase 2 output (/speckit.tasks command - NOT created by /speckit.plan)
+```
+
+### Source Code (repository root)
+<!--
+  ACTION REQUIRED: Replace the placeholder tree below with the concrete layout
+  for this feature. Delete unused options and expand the chosen structure with
+  real paths (e.g., apps/admin, packages/something). The delivered plan must
+  not include Option labels.
+-->
+
+```text
+backend/
+├── src/
+│   ├── api/
+│   │   └── routes/
+│   │       └── connections.py # New: Connection config endpoints
+│   ├── models/
+│   │   └── connection.py      # New: DB model for saved connections
+│   └── plugins/
+│       ├── search.py          # New: SearchPlugin
+│       ├── mapper.py          # New: MapperPlugin
+│       └── debug.py           # New: DebugPlugin
+└── tests/
+
+frontend/
+├── src/
+│   ├── components/
+│   │   ├── tools/             # New: UI components for tools
+│   │   │   ├── SearchTool.svelte
+│   │   │   ├── MapperTool.svelte
+│   │   │   └── DebugTool.svelte
+│   ├── routes/
+│   │   ├── tools/             # New: Pages
+│   │   │   ├── search/
+│   │   │   ├── mapper/
+│   │   │   └── debug/
+│   │   └── settings/
+│   │       └── connections/   # New: Connection management
+│   └── services/
+│       └── toolsService.js    # New: API client
+```
+
+**Structure Decision**: We will extend the existing `backend/` and `frontend/` structure. The CLI logic will be refactored into `plugins/` in the backend, executed via the generic Task API. The frontend will gain a new "Tools" section that interacts with these plugins.
+
+## Complexity Tracking
+
+> **Fill ONLY if Constitution Check has violations that must be justified**
+
+| Violation | Why Needed | Simpler Alternative Rejected Because |
+|-----------|------------|-------------------------------------|
+| N/A | | |

specs/010-refactor-cli-to-web/quickstart.md

@@ -0,0 +1,53 @@
+# Quickstart: CLI Tools Web Interface
+
+This guide explains how to use the new web-based tools for Superset management, which replace the legacy CLI scripts.
+
+## 1. Accessing the Tools
+
+1.  Log in to the Web Application.
+2.  Navigate to the **Tools** section in the main navigation bar.
+3.  You will see three tabs/cards:
+    *   **Search**: Find text patterns in datasets.
+    *   **Dataset Mapper**: Map column names from external sources.
+    *   **Debug**: Run system diagnostics.
+
+## 2. Searching Datasets
+
+Use this tool to find specific SQL code, table names, or column definitions across the entire Superset instance.
+
+1.  Go to **Tools > Search**.
+2.  Select the **Environment** (e.g., `dev`, `prod`).
+3.  Enter your **Search Query** (supports Regex, e.g., `from dm.*account`).
+4.  Click **Search**.
+5.  Results will appear below, showing the dataset name, the field where the match was found, and the context.
+
+## 3. Mapping Dataset Columns
+
+Use this tool to update dataset column names (`verbose_name`) using comments from a database or an Excel file.
+
+### Step 3.1: Configure a Connection (One-time setup)
+
+1.  Go to **Settings > Connections**.
+2.  Click **Add Connection**.
+3.  Enter a name (e.g., "DWH Production") and the database credentials (Host, Port, DB, User, Password).
+4.  Click **Save**.
+
+### Step 3.2: Run the Mapper
+
+1.  Go to **Tools > Dataset Mapper**.
+2.  Select the target **Environment** and **Dataset ID**.
+3.  Select the **Source Type** (`Postgres`, `Excel`, or `Both`).
+4.  If using Postgres, select the **Saved Connection** you created in Step 3.1.
+5.  Enter the **Table Name** and **Schema** (e.g., `public.sales`).
+6.  Click **Run Mapping**.
+7.  The job will be submitted to the Task Manager. You can track progress in the **Tasks** view.
+
+## 4. System Debugging
+
+Use this tool to verify connectivity and API structures.
+
+1.  Go to **Tools > Debug**.
+2.  Select a diagnostic routine:
+    *   **Test DB API**: Checks if the backend can list databases from Superset.
+    *   **Get Dataset Structure**: Dumps the raw JSON structure of a specific dataset for inspection.
+3.  View the output log directly in the browser.

specs/010-refactor-cli-to-web/research.md

@@ -0,0 +1,62 @@
+# Research: Refactor CLI Scripts to Web Application
+
+## 1. Search Tool Architecture
+
+**Problem**: The `search_script.py` fetches metadata for *all* datasets and performs regex matching in memory. This can be resource-intensive and slow for large Superset instances.
+
+**Options**:
+1.  **Synchronous API Endpoint**: The frontend calls an API, waits, and displays results.
+    *   *Pros*: Simple, immediate feedback.
+    *   *Cons*: Risk of HTTP timeout (e.g., Nginx/Browser limits) if the dataset fetch takes too long.
+2.  **Asynchronous Task (TaskManager)**: The frontend triggers a task, polls for status, and displays results when done.
+    *   *Pros*: Robust, no timeouts, consistent with "Mapping" and "Migration" tools.
+    *   *Cons*: Slower user experience for quick searches.
+
+**Decision**: **Synchronous API with Optimization**.
+*   **Rationale**: Search is typically an interactive "read-only" operation. Users expect immediate results. The `superset_tool` client's `get_datasets` is reasonably efficient.
+*   **Mitigation**: We will implement the API to return a standard JSON response. If performance becomes an issue in testing, we can easily wrap the service logic in a TaskManager plugin.
+
+## 2. Dataset Mapper & Connection Management
+
+**Problem**: `run_mapper.py` relies on command-line arguments and `keyring` for database credentials. The Web UI needs a way to store and reuse these credentials securely.
+
+**Options**:
+1.  **Input Every Time**: User enters DB credentials for every mapping operation.
+    *   *Pros*: Secure (no storage).
+    *   *Cons*: Poor UX, tedious.
+2.  **Saved Connections**: Store connection details (Host, Port, DB, User, Password) in the application database.
+    *   *Pros*: Good UX.
+    *   *Cons*: Security risk if not encrypted.
+
+**Decision**: **Saved Connections (Encrypted)**.
+*   **Rationale**: The spec explicitly requires: "Connection configurations must be saved for reuse".
+*   **Implementation**:
+    *   Create a new SQLAlchemy model `ConnectionConfig` in `backend/src/models/connection.py`.
+    *   Store passwords encrypted (or at least obfuscated if full encryption infra isn't ready, but ideally encrypted). Given the scope, we will store them in the existing SQLite database.
+    *   The Mapper logic will be refactored into a `MapperPlugin` (or updated existing one) that accepts a `connection_id` or explicit config.
+
+## 3. Debug Tools Integration
+
+**Problem**: `debug_db_api.py` and `get_dataset_structure.py` are standalone scripts that print to stdout or write files.
+
+**Decision**: **Direct API Services**.
+*   **Debug API**: Create an endpoint `POST /api/tools/debug/test-db-connection` that runs the logic from `debug_db_api.py` and returns the log/result JSON.
+*   **Dataset Structure**: Create an endpoint `GET /api/tools/debug/dataset/{id}/structure` that runs logic from `get_dataset_structure.py` and returns the JSON directly.
+
+## 4. Legacy Code Cleanup
+
+**Plan**:
+1.  Implement the new Web tools.
+2.  Verify feature parity.
+3.  Delete:
+    *   `search_script.py`
+    *   `run_mapper.py`
+    *   `debug_db_api.py`
+    *   `get_dataset_structure.py`
+    *   `backup_script.py` (Spec confirms it's superseded by `009-backup-scheduler`)
+
+## 5. Security & Access
+
+**Decision**: All authenticated users can access these tools.
+*   **Rationale**: Spec says "All authenticated users".
+*   **Implementation**: Use existing `Depends(get_current_user)` for all new routes.

specs/010-refactor-cli-to-web/spec.md

@@ -0,0 +1,84 @@
+# [DEF:Spec:010-refactor-cli-to-web]
+# @TITLE: Refactor CLI Scripts to Web Application
+# @STATUS: DRAFT
+# @AUTHOR: Kilo Code
+# @CREATED: 2026-01-07
+
+## Clarifications
+
+### Session 2026-01-07
+- Q: Кто должен иметь доступ к новым веб-инструментам (Поиск, Маппинг, Отладка)? → A: Все аутентифицированные пользователи.
+- Q: Нужно ли сохранять конфигурации подключений для маппинга (источник, детали подключения) для повторного использования, или они вводятся каждый раз заново? → A: Сохранять для повторного использования (удобство пользователя).
+- Q: Скрипт `backup_script.py` указан в списке на удаление, но функциональность резервного копирования не описана в требованиях. Нужно ли переносить функцию бэкапа в веб-интерфейс? → A: Функциональность бэкапа уже реализована в рамках задачи 009-backup-scheduler и доступна через TaskManager. Скрипт `backup_script.py` можно удалять, так как его заменяет новая система.
+
+## 1. Overview
+
+### 1.1. Problem Statement
+The system currently relies on a set of command-line scripts for critical operations like searching datasets, mapping columns, and debugging. This requires users to have SSH access and knowledge of terminal commands, creating a disjointed user experience compared to the main Web Application. It also leads to maintenance overhead as core logic is duplicated between the CLI tools and the Web backend.
+
+### 1.2. Goal
+Integrate the functionality of the standalone CLI tools directly into the Web Application. This will provide a unified interface for all system operations, simplify maintenance by centralizing logic, and eliminate the need for direct terminal access.
+
+### 1.3. Scope
+*   **In Scope:**
+    *   Integration of Dataset Search functionality into the Web UI.
+    *   Integration of Dataset Mapping functionality into the Web UI.
+    *   Integration of System Debugging tools into the Web UI.
+    *   Removal of legacy command-line scripts and their specific dependencies.
+    *   Verification that existing Backup functionality (from 009-backup-scheduler) fully covers the legacy `backup_script.py` capabilities before removal.
+*   **Out of Scope:**
+    *   Major redesign of the existing Web UI (functionality will be added using existing patterns).
+    *   Changes to the core business logic of the tools (porting existing logic only).
+
+## 2. User Scenarios
+
+### 2.1. Search Datasets
+*   **Before:** User logs in via SSH, runs a python script, edits the script to change the query, and reads text output in the terminal.
+*   **After:** User logs into the Web App, navigates to the "Search" tool, enters a query, and views results in a structured list within the browser.
+
+### 2.2. Map Dataset Columns
+*   **Before:** User prepares a config or arguments and runs a python script from the terminal.
+*   **After:** User navigates to the "Dataset Mapper" tool in the Web App, fills out a form with source details, and executes the mapping. Progress is visible in the application's task list.
+
+### 2.3. System Debugging
+*   **Before:** User runs various debug scripts manually to test connectivity or API structure.
+*   **After:** User navigates to a "Debug" section in the Web App, selects a diagnostic routine (e.g., "Test DB API"), and views the report in the application logs.
+
+## 3. Functional Requirements
+
+### 3.1. Web-Based Dataset Search
+*   The system must provide a user interface to search for text patterns across all Superset datasets.
+*   Users must be able to select the target environment.
+*   The system must display search results including the Dataset Name, Field Name, and the matching text context.
+
+### 3.2. Web-Based Dataset Mapping
+*   The system must provide a user interface to map dataset column names/comments from external sources (e.g., Database, File).
+*   Users must be able to specify the source type and connection details via the UI.
+*   Connection configurations must be saved for reuse to improve user convenience.
+*   The system must provide feedback on the success or failure of the mapping operation.
+
+### 3.3. Web-Based Diagnostics
+*   The system must provide a user interface to trigger system diagnostic routines.
+*   Supported diagnostics must include:
+    *   Retrieving dataset structure for debugging.
+    *   Testing Database API connectivity and response structure.
+*   Results must be viewable within the application.
+
+### 3.4. Legacy Cleanup
+*   The system must function independently of the legacy CLI scripts.
+*   The legacy CLI scripts must be removed to prevent usage of deprecated tools.
+
+### 3.5. Security & Access
+*   All authenticated users must have access to Search, Mapping, and Debugging tools.
+
+## 4. Success Criteria
+*   **Unified Experience:** Users can perform Search, Mapping, and Debugging tasks entirely through the Web UI without using a terminal.
+*   **Feature Parity:** All capabilities previously available in the CLI scripts are available in the Web Application.
+*   **Clean Codebase:** The project no longer contains standalone CLI scripts (`search_script.py`, `run_mapper.py`, `migration_script.py`, `backup_script.py`, `debug_db_api.py`, `get_dataset_structure.py`).
+*   **Dependency Reduction:** The codebase no longer relies on CLI-specific libraries (e.g., `whiptail`).
+
+## 5. Assumptions
+*   The existing Web Application plugin architecture supports the addition of these new tools.
+*   The existing logging and task management systems in the Web Application can handle the output from these tools.
+
+# [/DEF:Spec:010-refactor-cli-to-web]

specs/010-refactor-cli-to-web/tasks.md

@@ -0,0 +1,68 @@
+# Tasks: Refactor CLI Scripts to Web Application
+
+**Feature**: Refactor CLI Scripts to Web Application
+**Status**: Pending
+**Spec**: [specs/010-refactor-cli-to-web/spec.md](specs/010-refactor-cli-to-web/spec.md)
+
+## Phase 1: Core & Infrastructure
+*Goal: Enhance TaskManager to support returning results and set up Connection management.*
+
+- [ ] T001 Create `backend/src/models/connection.py` with `ConnectionConfig` model (ensure secure storage of credentials)
+- [ ] T002 Update `backend/src/core/task_manager/models.py` to add `result: Optional[Dict[str, Any]]` field to `Task` model
+- [ ] T003 Update `backend/src/core/task_manager/manager.py` to capture `plugin.execute` return value and store it in `task.result`
+- [ ] T004 [P] Create `frontend/src/services/toolsService.js` for generic Task API communication (run task, poll status, get result)
+- [ ] T005 [P] Create `frontend/src/services/connectionService.js` for Connection API
+- [ ] T006 Create `frontend/src/components/tools/` directory structure
+
+## Phase 2: User Story 1 - Search Datasets
+*Goal: Enable users to search datasets via the Web UI using a Plugin.*
+
+- [ ] T007 [US1] Implement `SearchPlugin` in `backend/src/plugins/search.py` (port logic from `search_script.py`, return results)
+- [ ] T008 [US1] Create `frontend/src/components/tools/SearchTool.svelte` component (triggers SearchPlugin, displays results)
+- [ ] T009 [US1] Create `frontend/src/routes/tools/search/+page.svelte` page
+
+## Phase 3: User Story 2 - Map Dataset Columns
+*Goal: Enable users to map columns using saved connections via a Plugin.*
+
+- [ ] T010 [US2] Create API endpoints for Connections (`GET`, `POST`, `DELETE`) in `backend/src/api/routes/connections.py`
+- [ ] T011 [US2] Register `connections` router in `backend/src/api/routes/__init__.py`
+- [ ] T012 [US2] Implement `MapperPlugin` in `backend/src/plugins/mapper.py` (port logic from `run_mapper.py`, support `connection_id`)
+- [ ] T013 [US2] Create `frontend/src/components/tools/ConnectionForm.svelte` and `ConnectionList.svelte`
+- [ ] T014 [US2] Create `frontend/src/routes/settings/connections/+page.svelte` for managing connections
+- [ ] T015 [US2] Create `frontend/src/components/tools/MapperTool.svelte` component
+- [ ] T016 [US2] Create `frontend/src/routes/tools/mapper/+page.svelte` page
+
+## Phase 4: User Story 3 - System Debugging
+*Goal: Enable users to run system diagnostics via a Plugin.*
+
+- [ ] T017 [US3] Implement `DebugPlugin` in `backend/src/plugins/debug.py` (port logic from `debug_db_api.py`, `get_dataset_structure.py`)
+- [ ] T018 [US3] Create `frontend/src/components/tools/DebugTool.svelte` component
+- [ ] T019 [US3] Create `frontend/src/routes/tools/debug/+page.svelte` page
+
+## Phase 5: Cleanup & Polish
+*Goal: Remove legacy scripts and ensure system stability.*
+
+- [ ] T020 Verify all new tools are functional (Manual Test)
+- [ ] T021 Verify existing Backup functionality fully covers legacy `backup_script.py` capabilities
+- [ ] T022 Remove legacy script `search_script.py`
+- [ ] T023 Remove legacy script `run_mapper.py`
+- [ ] T024 Remove legacy script `debug_db_api.py`
+- [ ] T025 Remove legacy script `get_dataset_structure.py`
+- [ ] T026 Remove legacy script `backup_script.py`
+- [ ] T027 Update main navigation in `frontend/src/components/Navbar.svelte` to include "Tools" dropdown
+
+## Dependencies
+
+- Phase 2 (Search) depends on Phase 1 (Task Result support)
+- Phase 3 (Mapper) depends on Phase 1 (Connection Model)
+- Phase 4 (Debug) depends on Phase 1 (Task Result support)
+- Phase 5 (Cleanup) depends on Phase 2, 3, 4 completion
+
+## Implementation Strategy
+
+1.  **Core**: Upgrade TaskManager first to support return values.
+2.  **Search**: Implement SearchPlugin as the first "Result-returning" plugin.
+3.  **Connections**: Build connection management.
+4.  **Mapper**: Implement MapperPlugin using Connections.
+5.  **Debug**: Implement DebugPlugin.
+6.  **Cleanup**: Verify and delete.