WIP: Staged all changes

specs/001-plugin-arch-svelte-ui/checklists/requirements.md

@@ -0,0 +1,34 @@
+# Specification Quality Checklist: Plugin Architecture & Svelte Web UI
+
+**Purpose**: Validate specification completeness and quality before proceeding to planning
+**Created**: 2025-12-19
+**Feature**: [Link to spec.md](../spec.md)
+
+## Content Quality
+
+- [x] No implementation details (languages, frameworks, APIs)
+- [x] Focused on user value and business needs
+- [x] Written for non-technical stakeholders
+- [x] All mandatory sections completed
+
+## Requirement Completeness
+
+- [x] No [NEEDS CLARIFICATION] markers remain
+- [x] Requirements are testable and unambiguous
+- [x] Success criteria are measurable
+- [x] Success criteria are technology-agnostic (no implementation details)
+- [x] All acceptance scenarios are defined
+- [x] Edge cases are identified
+- [x] Scope is clearly bounded
+- [x] Dependencies and assumptions identified
+
+## Feature Readiness
+
+- [x] All functional requirements have clear acceptance criteria
+- [x] User scenarios cover primary flows
+- [x] Feature meets measurable outcomes defined in Success Criteria
+- [x] No implementation details leak into specification
+
+## Notes
+
+- Clarification resolved: Deployment context is hosted multi-user service with ADFS login.

specs/001-plugin-arch-svelte-ui/contracts/api.yaml

@@ -0,0 +1,132 @@
+openapi: 3.0.0
+info:
+  title: Superset Tools API
+  version: 1.0.0
+  description: API for managing Superset automation tools and plugins.
+
+paths:
+  /plugins:
+    get:
+      summary: List available plugins
+      operationId: list_plugins
+      responses:
+        '200':
+          description: List of plugins
+          content:
+            application/json:
+              schema:
+                type: array
+                items:
+                  $ref: '#/components/schemas/Plugin'
+
+  /tasks:
+    post:
+      summary: Start a new task
+      operationId: create_task
+      requestBody:
+        required: true
+        content:
+          application/json:
+            schema:
+              type: object
+              required:
+                - plugin_id
+                - params
+              properties:
+                plugin_id:
+                  type: string
+                params:
+                  type: object
+      responses:
+        '201':
+          description: Task created
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/Task'
+
+    get:
+      summary: List recent tasks
+      operationId: list_tasks
+      responses:
+        '200':
+          description: List of tasks
+          content:
+            application/json:
+              schema:
+                type: array
+                items:
+                  $ref: '#/components/schemas/Task'
+
+  /tasks/{task_id}:
+    get:
+      summary: Get task details
+      operationId: get_task
+      parameters:
+        - name: task_id
+          in: path
+          required: true
+          schema:
+            type: string
+            format: uuid
+      responses:
+        '200':
+          description: Task details
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/Task'
+
+  /tasks/{task_id}/logs:
+    get:
+      summary: Stream task logs (WebSocket upgrade)
+      operationId: stream_logs
+      parameters:
+        - name: task_id
+          in: path
+          required: true
+          schema:
+            type: string
+            format: uuid
+      responses:
+        '101':
+          description: Switching Protocols to WebSocket
+
+components:
+  schemas:
+    Plugin:
+      type: object
+      properties:
+        id:
+          type: string
+        name:
+          type: string
+        description:
+          type: string
+        version:
+          type: string
+        schema:
+          type: object
+          description: JSON Schema for input parameters
+        enabled:
+          type: boolean
+
+    Task:
+      type: object
+      properties:
+        id:
+          type: string
+          format: uuid
+        plugin_id:
+          type: string
+        status:
+          type: string
+          enum: [PENDING, RUNNING, SUCCESS, FAILED]
+        started_at:
+          type: string
+          format: date-time
+        finished_at:
+          type: string
+          format: date-time
+        user_id:
+          type: string

specs/001-plugin-arch-svelte-ui/data-model.md

@@ -0,0 +1,51 @@
+# Data Model: Plugin Architecture & Svelte Web UI
+
+## Entities
+
+### Plugin
+Represents a loadable extension module.
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `id` | `str` | Unique identifier (e.g., "backup-tool") |
+| `name` | `str` | Display name (e.g., "Backup Dashboard") |
+| `description` | `str` | Short description of functionality |
+| `version` | `str` | Plugin version string |
+| `schema` | `dict` | JSON Schema for input parameters (generated from Pydantic) |
+| `enabled` | `bool` | Whether the plugin is active |
+
+### Task
+Represents an execution instance of a plugin.
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `id` | `UUID` | Unique execution ID |
+| `plugin_id` | `str` | ID of the plugin being executed |
+| `status` | `Enum` | `PENDING`, `RUNNING`, `SUCCESS`, `FAILED` |
+| `started_at` | `DateTime` | Timestamp when task started |
+| `finished_at` | `DateTime` | Timestamp when task completed (nullable) |
+| `user_id` | `str` | ID of the user who triggered the task |
+| `logs` | `List[LogEntry]` | Structured logs from the execution |
+
+### LogEntry
+Represents a single log line from a task.
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `timestamp` | `DateTime` | Time of log event |
+| `level` | `Enum` | `INFO`, `WARNING`, `ERROR`, `DEBUG` |
+| `message` | `str` | Log content |
+| `context` | `dict` | Additional metadata (optional) |
+
+## State Transitions
+
+### Task Lifecycle
+1. **Created**: Task initialized with input parameters. Status: `PENDING`.
+2. **Started**: Worker picks up task. Status: `RUNNING`.
+3. **Completed**: Execution finishes without exception. Status: `SUCCESS`.
+4. **Failed**: Execution raises unhandled exception. Status: `FAILED`.
+
+## Validation Rules
+
+- **Plugin ID**: Must be alphanumeric, lowercase, hyphens allowed.
+- **Input Parameters**: Must validate against the plugin's `schema`.

specs/001-plugin-arch-svelte-ui/plan.md

@@ -0,0 +1,76 @@
+# Implementation Plan: Plugin Architecture & Svelte Web UI
+
+**Branch**: `001-plugin-arch-svelte-ui` | **Date**: 2025-12-19 | **Spec**: [spec.md](spec.md)
+**Input**: Feature specification from `specs/001-plugin-arch-svelte-ui/spec.md`
+
+## Summary
+
+This feature introduces a dual-layer architecture: a Python backend exposing core tools (Backup, Migration, Search) via API, and a Svelte-based Single Page Application (SPA) for user interaction. It also implements a dynamic plugin system allowing developers to extend functionality by adding Python modules to a specific directory without modifying core code.
+
+## Technical Context
+
+**Language/Version**: Python 3.9+ (Backend), Node.js 18+ (Frontend Build)
+**Primary Dependencies**: 
+- Backend: Flask or FastAPI [NEEDS CLARIFICATION: Choice of web framework], Pydantic (validation), Plugin loader mechanism (importlib)
+- Frontend: Svelte, Vite, TailwindCSS (likely for UI)
+**Storage**: Filesystem (plugins, logs, backups), SQLite (optional, for job history if needed)
+**Testing**: pytest (Backend), vitest/playwright (Frontend)
+**Target Platform**: Windows/Linux (Hosted Service)
+**Project Type**: Web Application (Backend + Frontend)
+**Performance Goals**: UI load < 1s, Log streaming latency < 200ms
+**Constraints**: Must run in a hosted environment with ADFS authentication.
+**Scale/Scope**: ~5-10 concurrent users, Extensible plugin system.
+
+## Constitution Check
+
+*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
+
+- [x] **Causal Validity**: Do all planned modules have defined Contracts (inputs/outputs/invariants) before implementation logic? (Will be enforced in Phase 1)
+- [x] **Immutability**: Are architectural layers and constraints defined in Module Headers? (Will be enforced in Phase 1)
+- [x] **Format Compliance**: Does the plan ensure all code will be wrapped in `[DEF]` anchors? (Will be enforced in Phase 1)
+- [x] **Belief State**: Is logging planned to follow the `Entry` -> `Validation` -> `Action` -> `Coherence` state transition model? (Will be enforced in Phase 1)
+
+## Project Structure
+
+### Documentation (this feature)
+
+```text
+specs/001-plugin-arch-svelte-ui/
+├── plan.md              # This file
+├── research.md          # Phase 0 output
+├── data-model.md        # Phase 1 output
+├── quickstart.md        # Phase 1 output
+├── contracts/           # Phase 1 output
+└── tasks.md             # Phase 2 output
+```
+
+### Source Code (repository root)
+
+```text
+backend/
+├── src/
+│   ├── app.py           # Entry point
+│   ├── api/             # REST API endpoints
+│   ├── core/            # Plugin loader, Task manager
+│   ├── plugins/         # Directory for dynamic plugins
+│   └── services/        # Auth (ADFS), Logging
+└── tests/
+
+frontend/
+├── src/
+│   ├── components/      # Reusable UI components
+│   ├── pages/           # Route views
+│   ├── lib/             # API client, Stores
+│   └── App.svelte
+└── tests/
+
+superset_tool/           # Existing core logic (refactored to be importable by backend)
+```
+
+**Structure Decision**: Adopting a standard "Web Application" structure with separated `backend` and `frontend` directories to maintain clean separation of concerns. The existing `superset_tool` library will be preserved and imported by the backend to execute actual tasks.
+
+## Complexity Tracking
+
+| Violation | Why Needed | Simpler Alternative Rejected Because |
+|-----------|------------|-------------------------------------|
+| N/A | | |

specs/001-plugin-arch-svelte-ui/quickstart.md

@@ -0,0 +1,47 @@
+# Quickstart: Plugin Architecture & Svelte Web UI
+
+## Prerequisites
+- Python 3.9+
+- Node.js 18+
+- npm or pnpm
+
+## Setup
+
+1. **Install Backend Dependencies**:
+   ```bash
+   cd backend
+   python -m venv venv
+   source venv/bin/activate  # or venv\Scripts\activate on Windows
+   pip install -r requirements.txt
+   ```
+
+2. **Install Frontend Dependencies**:
+   ```bash
+   cd frontend
+   npm install
+   ```
+
+## Running the Application
+
+1. **Start Backend Server**:
+   ```bash
+   # From backend/ directory
+   uvicorn src.app:app --reload --port 8000
+   ```
+
+2. **Start Frontend Dev Server**:
+   ```bash
+   # From frontend/ directory
+   npm run dev
+   ```
+
+3. **Access the UI**:
+   Open `http://localhost:5173` in your browser.
+
+## Adding a Plugin
+
+1. Create a new Python file in `backend/src/plugins/` (e.g., `my_plugin.py`).
+2. Define your plugin class inheriting from `PluginBase`.
+3. Implement `execute` and `get_schema` methods.
+4. Restart the backend (or rely on auto-reload).
+5. Your plugin should appear in the Web UI.

specs/001-plugin-arch-svelte-ui/research.md

@@ -0,0 +1,46 @@
+# Research: Plugin Architecture & Svelte Web UI
+
+## Decisions
+
+### 1. Web Framework: FastAPI
+- **Decision**: Use FastAPI for the Python backend.
+- **Rationale**: 
+    - Native support for Pydantic models (crucial for plugin schema validation).
+    - Async support (essential for handling long-running tasks and log streaming via WebSockets/SSE).
+    - Automatic OpenAPI documentation generation (simplifies frontend integration).
+    - High performance and modern ecosystem.
+- **Alternatives Considered**: 
+    - **Flask**: Mature but requires extensions for validation (Marshmallow) and async support is less native. Slower for high-concurrency API calls.
+    - **Django**: Too heavy for this use case; brings unnecessary ORM and template engine overhead.
+
+### 2. Plugin System: `importlib` + Abstract Base Classes (ABC)
+- **Decision**: Use Python's built-in `importlib` for dynamic loading and `abc` for defining the plugin interface.
+- **Rationale**: 
+    - `importlib` provides a standard, secure way to load modules from a path.
+    - ABCs ensure plugins implement required methods (`execute`, `get_schema`) at load time.
+    - Lightweight, no external dependencies required.
+- **Alternatives Considered**: 
+    - **Pluggy**: Used by pytest, powerful but adds complexity and dependency overhead.
+    - **Stevedore**: OpenStack's plugin loader, too complex for this scope.
+
+### 3. Authentication: `authlib` + ADFS (OIDC/SAML)
+- **Decision**: Use `authlib` to handle ADFS authentication via OpenID Connect (OIDC) or SAML.
+- **Rationale**: 
+    - `authlib` is the modern standard for OAuth/OIDC in Python.
+    - Supports integration with FastAPI via middleware.
+    - ADFS is the required identity provider (IdP).
+- **Alternatives Considered**: 
+    - **python-social-auth**: Older, harder to integrate with FastAPI.
+    - **Manual JWT implementation**: Risky and reinvents the wheel; ADFS handles the token issuance.
+
+### 4. Frontend: Svelte + Vite
+- **Decision**: Use Svelte for the UI framework and Vite as the build tool.
+- **Rationale**: 
+    - Svelte's compiler-based approach results in small bundles and high performance.
+    - Reactive model maps well to real-time log updates.
+    - Vite provides a fast development experience and easy integration with backend proxies.
+
+## Unknowns Resolved
+
+- **Deployment Context**: Hosted multi-user service with ADFS.
+- **Plugin Interface**: Will use Pydantic models to define input schemas, allowing the frontend to generate forms dynamically.

specs/001-plugin-arch-svelte-ui/spec.md

@@ -0,0 +1,72 @@
+# Feature Specification: Plugin Architecture & Svelte Web UI
+
+**Feature Branch**: `001-plugin-arch-svelte-ui`
+**Created**: 2025-12-19
+**Status**: Draft
+**Input**: User description: "Я хочу перевести проект на плагинную архитектуру + добавить web-ui на svelte"
+
+## User Scenarios & Testing *(mandatory)*
+
+### User Story 1 - Web Interface for Superset Tools (Priority: P1)
+
+As a user, I want to interact with the Superset tools (Backup, Migration, Search) through a graphical web interface so that I don't have to memorize CLI commands and arguments.
+
+**Why this priority**: drastically improves usability and accessibility of the tools for non-technical users or quick operations.
+
+**Independent Test**: Can be tested by launching the web server and successfully running a "Backup" task from the browser without touching the command line.
+
+**Acceptance Scenarios**:
+
+1. **Given** the web server is running, **When** I navigate to the home page, **Then** I see a dashboard with available tools (Backup, Migration, etc.).
+2. **Given** I am on the Backup tool page, **When** I click "Run Backup", **Then** I see the progress logs in real-time and a success message upon completion.
+3. **Given** I am on the Search tool page, **When** I enter a search term and submit, **Then** I see a list of matching datasets/dashboards displayed in a table.
+
+---
+
+### User Story 2 - Dynamic Plugin System (Priority: P2)
+
+As a developer, I want to add new functionality (e.g., a new migration type or report generator) by simply dropping a file into a `plugins` directory, so that I can extend the tool without modifying the core codebase.
+
+**Why this priority**: Enables scalable development and separation of concerns; allows custom extensions without merge conflicts in core files.
+
+**Independent Test**: Create a simple "Hello World" plugin file, place it in the plugins folder, and verify it appears in the list of available tasks in the CLI/Web UI.
+
+**Acceptance Scenarios**:
+
+1. **Given** a valid plugin file in the `plugins/` directory, **When** the application starts, **Then** the plugin is automatically registered and listed as an available capability.
+2. **Given** a plugin with specific configuration requirements, **When** I select it in the UI, **Then** the UI dynamically generates a form for those parameters.
+3. **Given** an invalid or broken plugin file, **When** the application starts, **Then** the system logs an error but continues to function for other plugins.
+
+---
+
+## Requirements *(mandatory)*
+
+### Functional Requirements
+*All functional requirements are covered by the Acceptance Scenarios in the User Stories section.*
+
+- **FR-001**: System MUST provide a Python-based web server (backend) to expose existing tool functionality via API.
+- **FR-002**: System MUST provide a Single Page Application (SPA) frontend built with Svelte.
+- **FR-003**: System MUST implement a plugin loader that scans a designated directory for Python modules matching a specific interface.
+- **FR-004**: The Web UI MUST communicate with the backend via REST or WebSocket API.
+- **FR-005**: The Web UI MUST display real-time logs/output from running tasks (streaming response).
+- **FR-006**: System MUST support multi-user hosted deployment with authentication via ADFS (Active Directory Federation Services).
+- **FR-007**: The Plugin interface MUST allow defining input parameters (schema) so the UI can auto-generate forms.
+
+### System Invariants (Constitution Check)
+
+- **INV-001**: Core logic (backup/migrate functions) must remain decoupled from the UI layer (can still be imported/used by CLI).
+- **INV-002**: Plugins must not block the main application thread (long-running tasks must be async or threaded).
+
+### Key Entities
+
+- **Plugin**: Represents an extension module with metadata (name, version), input schema, and an execution entry point.
+- **Task**: A specific execution instance of a Plugin or Core tool, having a status (Running, Success, Failed) and logs.
+
+## Success Criteria *(mandatory)*
+
+### Measurable Outcomes
+
+- **SC-001**: A new plugin can be added and recognized by the system without restarting (or with a simple restart) and without code changes to core files.
+- **SC-002**: Users can successfully trigger a Backup and Migration via the Web UI with 100% functional parity to the CLI.
+- **SC-003**: The Web UI loads and becomes interactive in under 1 second on local networks.
+- **SC-004**: Real-time logs in the UI appear with less than 200ms latency from the backend execution.

specs/001-plugin-arch-svelte-ui/tasks.md

@@ -0,0 +1,68 @@
+# Tasks: Plugin Architecture & Svelte Web UI
+
+**Feature**: `001-plugin-arch-svelte-ui`
+**Status**: Planned
+
+## Dependencies
+
+1. **Phase 1 (Setup)**: Must be completed first to establish the environment.
+2. **Phase 2 (Foundational)**: Implements the core Plugin system and Backend infrastructure required by all User Stories.
+3. **Phase 3 (US1)**: Web Interface depends on the Backend API and Plugin system.
+4. **Phase 4 (US2)**: Dynamic Plugin System extends the core infrastructure.
+
+## Parallel Execution Opportunities
+
+- **US1 (Frontend)**: Frontend components (T013-T016) can be developed in parallel with Backend API endpoints (T011-T012) once the API contract is finalized.
+- **US2 (Plugins)**: Plugin development (T019-T020) can proceed independently once the Plugin Interface (T005) is stable.
+
+---
+
+## Phase 1: Setup
+
+**Goal**: Initialize the project structure and development environment for Backend (Python/FastAPI) and Frontend (Svelte/Vite).
+
+- [x] T001 Create backend directory structure (src/api, src/core, src/plugins) in `backend/`
+- [x] T002 Create frontend directory structure using Vite (Svelte template) in `frontend/`
+- [x] T003 Configure Python environment (requirements.txt with FastAPI, Uvicorn, Pydantic) in `backend/requirements.txt`
+- [x] T004 Configure Frontend environment (package.json with TailwindCSS) in `frontend/package.json`
+
+## Phase 2: Foundational (Core Infrastructure)
+
+**Goal**: Implement the core Plugin interface, Task management system, and basic Backend server.
+
+- [x] T005 [P] Define `PluginBase` abstract class and Pydantic models in `backend/src/core/plugin_base.py`
+- [x] T006 [P] Implement `PluginLoader` to scan and load plugins from directory in `backend/src/core/plugin_loader.py`
+- [x] T007 Implement `TaskManager` to handle async task execution and state in `backend/src/core/task_manager.py`
+- [x] T008 [P] Implement `Logger` with WebSocket streaming support in `backend/src/core/logger.py`
+- [x] T009 Create basic FastAPI application entry point with CORS in `backend/src/app.py`
+- [x] T010 [P] Implement ADFS Authentication middleware in `backend/src/api/auth.py`
+
+## Phase 3: User Story 1 - Web Interface (Priority: P1)
+
+**Goal**: Enable users to interact with tools via a web dashboard.
+**Independent Test**: Launch web server, navigate to dashboard, run a dummy task, view logs.
+
+- [x] T011 [US1] Implement REST API endpoints for Plugin listing (`GET /plugins`) in `backend/src/api/routes/plugins.py`
+- [x] T012 [US1] Implement REST API endpoints for Task management (`POST /tasks`, `GET /tasks/{id}`) in `backend/src/api/routes/tasks.py`
+- [x] T013 [P] [US1] Create Svelte store for Plugin and Task state in `frontend/src/lib/stores.js`
+- [x] T014 [P] [US1] Create `Dashboard` page component listing available tools in `frontend/src/pages/Dashboard.svelte`
+- [x] T015 [P] [US1] Create `TaskRunner` component with real-time log viewer (WebSocket) in `frontend/src/components/TaskRunner.svelte`
+- [x] T016 [US1] Integrate Frontend with Backend API using `fetch` client in `frontend/src/lib/api.js`
+
+## Phase 4: User Story 2 - Dynamic Plugin System (Priority: P2)
+
+**Goal**: Allow developers to add new functionality by dropping files.
+**Independent Test**: Add `hello_world.py` to plugins dir, verify it appears in UI.
+
+- [x] T017 [US2] Implement dynamic form generation component based on JSON Schema in `frontend/src/components/DynamicForm.svelte`
+- [x] T018 [US2] Update `PluginLoader` to validate plugin schema on load in `backend/src/core/plugin_loader.py`
+- [x] T019 [P] [US2] Refactor existing `backup_script.py` into a Plugin (`BackupPlugin`) in `backend/src/plugins/backup.py`
+- [x] T020 [P] [US2] Refactor existing `migration_script.py` into a Plugin (`MigrationPlugin`) in `backend/src/plugins/migration.py`
+
+## Final Phase: Polish
+
+**Goal**: Ensure production readiness.
+
+- [x] T021 Add error handling and user notifications (Toasts) in Frontend
+- [x] T022 Write documentation for Plugin Development in `docs/plugin_dev.md`
+- [ ] T023 Final integration test: Run full Backup and Migration flow via UI