busya/ss-tools
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat: implement plugin architecture and application settings with Svelte UI

Browse Source 
- Added plugin base and loader for backend extensibility
- Implemented application settings management with config persistence
- Created Svelte-based frontend with Dashboard and Settings pages
- Added API routes for plugins, tasks, and settings
- Updated documentation and specifications
- Improved project structure and developer tools
This commit is contained in:
busya
2025-12-20 20:48:18 +03:00
parent ce703322c2
commit 2d8cae563f
98 changed files with 7894 additions and 5021 deletions
Download Patch File Download Diff File Expand all files Collapse all files

66
specs/001-plugin-arch-svelte-ui/checklists/requirements.md Normal file → Executable file
View File

@@ -1,34 +1,34 @@
# Specification Quality Checklist: Plugin Architecture & Svelte Web UI
**Purpose**: Validate specification completeness and quality before proceeding to planning
**Created**: 2025-12-19
**Feature**: [Link to spec.md](../spec.md)
## Content Quality
- [x] No implementation details (languages, frameworks, APIs)
- [x] Focused on user value and business needs
- [x] Written for non-technical stakeholders
- [x] All mandatory sections completed
## Requirement Completeness
- [x] No [NEEDS CLARIFICATION] markers remain
- [x] Requirements are testable and unambiguous
- [x] Success criteria are measurable
- [x] Success criteria are technology-agnostic (no implementation details)
- [x] All acceptance scenarios are defined
- [x] Edge cases are identified
- [x] Scope is clearly bounded
- [x] Dependencies and assumptions identified
## Feature Readiness
- [x] All functional requirements have clear acceptance criteria
- [x] User scenarios cover primary flows
- [x] Feature meets measurable outcomes defined in Success Criteria
- [x] No implementation details leak into specification
## Notes
# Specification Quality Checklist: Plugin Architecture & Svelte Web UI
**Purpose**: Validate specification completeness and quality before proceeding to planning
**Created**: 2025-12-19
**Feature**: [Link to spec.md](../spec.md)
## Content Quality
- [x] No implementation details (languages, frameworks, APIs)
- [x] Focused on user value and business needs
- [x] Written for non-technical stakeholders
- [x] All mandatory sections completed
## Requirement Completeness
- [x] No [NEEDS CLARIFICATION] markers remain
- [x] Requirements are testable and unambiguous
- [x] Success criteria are measurable
- [x] Success criteria are technology-agnostic (no implementation details)
- [x] All acceptance scenarios are defined
- [x] Edge cases are identified
- [x] Scope is clearly bounded
- [x] Dependencies and assumptions identified
## Feature Readiness
- [x] All functional requirements have clear acceptance criteria
- [x] User scenarios cover primary flows
- [x] Feature meets measurable outcomes defined in Success Criteria
- [x] No implementation details leak into specification
## Notes
- Clarification resolved: Deployment context is hosted multi-user service with ADFS login.

264
specs/001-plugin-arch-svelte-ui/contracts/api.yaml Normal file → Executable file
View File

@@ -1,132 +1,132 @@
openapi: 3.0.0
info:
title: Superset Tools API
version: 1.0.0
description: API for managing Superset automation tools and plugins.
paths:
/plugins:
get:
summary: List available plugins
operationId: list_plugins
responses:
'200':
description: List of plugins
content:
application/json:
schema:
type: array
items:
$ref: '#/components/schemas/Plugin'
/tasks:
post:
summary: Start a new task
operationId: create_task
requestBody:
required: true
content:
application/json:
schema:
type: object
required:
- plugin_id
- params
properties:
plugin_id:
type: string
params:
type: object
responses:
'201':
description: Task created
content:
application/json:
schema:
$ref: '#/components/schemas/Task'
get:
summary: List recent tasks
operationId: list_tasks
responses:
'200':
description: List of tasks
content:
application/json:
schema:
type: array
items:
$ref: '#/components/schemas/Task'
/tasks/{task_id}:
get:
summary: Get task details
operationId: get_task
parameters:
- name: task_id
in: path
required: true
schema:
type: string
format: uuid
responses:
'200':
description: Task details
content:
application/json:
schema:
$ref: '#/components/schemas/Task'
/tasks/{task_id}/logs:
get:
summary: Stream task logs (WebSocket upgrade)
operationId: stream_logs
parameters:
- name: task_id
in: path
required: true
schema:
type: string
format: uuid
responses:
'101':
description: Switching Protocols to WebSocket
components:
schemas:
Plugin:
type: object
properties:
id:
type: string
name:
type: string
description:
type: string
version:
type: string
schema:
type: object
description: JSON Schema for input parameters
enabled:
type: boolean
Task:
type: object
properties:
id:
type: string
format: uuid
plugin_id:
type: string
status:
type: string
enum: [PENDING, RUNNING, SUCCESS, FAILED]
started_at:
type: string
format: date-time
finished_at:
type: string
format: date-time
user_id:
type: string
openapi: 3.0.0
info:
title: Superset Tools API
version: 1.0.0
description: API for managing Superset automation tools and plugins.
paths:
/plugins:
get:
summary: List available plugins
operationId: list_plugins
responses:
'200':
description: List of plugins
content:
application/json:
schema:
type: array
items:
$ref: '#/components/schemas/Plugin'
/tasks:
post:
summary: Start a new task
operationId: create_task
requestBody:
required: true
content:
application/json:
schema:
type: object
required:
- plugin_id
- params
properties:
plugin_id:
type: string
params:
type: object
responses:
'201':
description: Task created
content:
application/json:
schema:
$ref: '#/components/schemas/Task'
get:
summary: List recent tasks
operationId: list_tasks
responses:
'200':
description: List of tasks
content:
application/json:
schema:
type: array
items:
$ref: '#/components/schemas/Task'
/tasks/{task_id}:
get:
summary: Get task details
operationId: get_task
parameters:
- name: task_id
in: path
required: true
schema:
type: string
format: uuid
responses:
'200':
description: Task details
content:
application/json:
schema:
$ref: '#/components/schemas/Task'
/tasks/{task_id}/logs:
get:
summary: Stream task logs (WebSocket upgrade)
operationId: stream_logs
parameters:
- name: task_id
in: path
required: true
schema:
type: string
format: uuid
responses:
'101':
description: Switching Protocols to WebSocket
components:
schemas:
Plugin:
type: object
properties:
id:
type: string
name:
type: string
description:
type: string
version:
type: string
schema:
type: object
description: JSON Schema for input parameters
enabled:
type: boolean
Task:
type: object
properties:
id:
type: string
format: uuid
plugin_id:
type: string
status:
type: string
enum: [PENDING, RUNNING, SUCCESS, FAILED]
started_at:
type: string
format: date-time
finished_at:
type: string
format: date-time
user_id:
type: string

100
specs/001-plugin-arch-svelte-ui/data-model.md Normal file → Executable file
View File

@@ -1,51 +1,51 @@
# Data Model: Plugin Architecture & Svelte Web UI
## Entities
### Plugin
Represents a loadable extension module.
| Field | Type | Description |
|-------|------|-------------|
| `id` | `str` | Unique identifier (e.g., "backup-tool") |
| `name` | `str` | Display name (e.g., "Backup Dashboard") |
| `description` | `str` | Short description of functionality |
| `version` | `str` | Plugin version string |
| `schema` | `dict` | JSON Schema for input parameters (generated from Pydantic) |
| `enabled` | `bool` | Whether the plugin is active |
### Task
Represents an execution instance of a plugin.
| Field | Type | Description |
|-------|------|-------------|
| `id` | `UUID` | Unique execution ID |
| `plugin_id` | `str` | ID of the plugin being executed |
| `status` | `Enum` | `PENDING`, `RUNNING`, `SUCCESS`, `FAILED` |
| `started_at` | `DateTime` | Timestamp when task started |
| `finished_at` | `DateTime` | Timestamp when task completed (nullable) |
| `user_id` | `str` | ID of the user who triggered the task |
| `logs` | `List[LogEntry]` | Structured logs from the execution |
### LogEntry
Represents a single log line from a task.
| Field | Type | Description |
|-------|------|-------------|
| `timestamp` | `DateTime` | Time of log event |
| `level` | `Enum` | `INFO`, `WARNING`, `ERROR`, `DEBUG` |
| `message` | `str` | Log content |
| `context` | `dict` | Additional metadata (optional) |
## State Transitions
### Task Lifecycle
1. **Created**: Task initialized with input parameters. Status: `PENDING`.
2. **Started**: Worker picks up task. Status: `RUNNING`.
3. **Completed**: Execution finishes without exception. Status: `SUCCESS`.
4. **Failed**: Execution raises unhandled exception. Status: `FAILED`.
## Validation Rules
- **Plugin ID**: Must be alphanumeric, lowercase, hyphens allowed.
# Data Model: Plugin Architecture & Svelte Web UI
## Entities
### Plugin
Represents a loadable extension module.
| Field | Type | Description |
|-------|------|-------------|
| `id` | `str` | Unique identifier (e.g., "backup-tool") |
| `name` | `str` | Display name (e.g., "Backup Dashboard") |
| `description` | `str` | Short description of functionality |
| `version` | `str` | Plugin version string |
| `schema` | `dict` | JSON Schema for input parameters (generated from Pydantic) |
| `enabled` | `bool` | Whether the plugin is active |
### Task
Represents an execution instance of a plugin.
| Field | Type | Description |
|-------|------|-------------|
| `id` | `UUID` | Unique execution ID |
| `plugin_id` | `str` | ID of the plugin being executed |
| `status` | `Enum` | `PENDING`, `RUNNING`, `SUCCESS`, `FAILED` |
| `started_at` | `DateTime` | Timestamp when task started |
| `finished_at` | `DateTime` | Timestamp when task completed (nullable) |
| `user_id` | `str` | ID of the user who triggered the task |
| `logs` | `List[LogEntry]` | Structured logs from the execution |
### LogEntry
Represents a single log line from a task.
| Field | Type | Description |
|-------|------|-------------|
| `timestamp` | `DateTime` | Time of log event |
| `level` | `Enum` | `INFO`, `WARNING`, `ERROR`, `DEBUG` |
| `message` | `str` | Log content |
| `context` | `dict` | Additional metadata (optional) |
## State Transitions
### Task Lifecycle
1. **Created**: Task initialized with input parameters. Status: `PENDING`.
2. **Started**: Worker picks up task. Status: `RUNNING`.
3. **Completed**: Execution finishes without exception. Status: `SUCCESS`.
4. **Failed**: Execution raises unhandled exception. Status: `FAILED`.
## Validation Rules
- **Plugin ID**: Must be alphanumeric, lowercase, hyphens allowed.
- **Input Parameters**: Must validate against the plugin's `schema`.

0
specs/001-plugin-arch-svelte-ui/plan.md Normal file → Executable file
View File

92
specs/001-plugin-arch-svelte-ui/quickstart.md Normal file → Executable file
View File

@@ -1,47 +1,47 @@
# Quickstart: Plugin Architecture & Svelte Web UI
## Prerequisites
- Python 3.9+
- Node.js 18+
- npm or pnpm
## Setup
1. **Install Backend Dependencies**:
```bash
cd backend
python -m venv venv
source venv/bin/activate # or venv\Scripts\activate on Windows
pip install -r requirements.txt
```
2. **Install Frontend Dependencies**:
```bash
cd frontend
npm install
```
## Running the Application
1. **Start Backend Server**:
```bash
# From backend/ directory
uvicorn src.app:app --reload --port 8000
```
2. **Start Frontend Dev Server**:
```bash
# From frontend/ directory
npm run dev
```
3. **Access the UI**:
Open `http://localhost:5173` in your browser.
## Adding a Plugin
1. Create a new Python file in `backend/src/plugins/` (e.g., `my_plugin.py`).
2. Define your plugin class inheriting from `PluginBase`.
3. Implement `execute` and `get_schema` methods.
4. Restart the backend (or rely on auto-reload).
# Quickstart: Plugin Architecture & Svelte Web UI
## Prerequisites
- Python 3.9+
- Node.js 18+
- npm or pnpm
## Setup
1. **Install Backend Dependencies**:
```bash
cd backend
python -m venv venv
source venv/bin/activate # or venv\Scripts\activate on Windows
pip install -r requirements.txt
```
2. **Install Frontend Dependencies**:
```bash
cd frontend
npm install
```
## Running the Application
1. **Start Backend Server**:
```bash
# From backend/ directory
uvicorn src.app:app --reload --port 8000
```
2. **Start Frontend Dev Server**:
```bash
# From frontend/ directory
npm run dev
```
3. **Access the UI**:
Open `http://localhost:5173` in your browser.
## Adding a Plugin
1. Create a new Python file in `backend/src/plugins/` (e.g., `my_plugin.py`).
2. Define your plugin class inheriting from `PluginBase`.
3. Implement `execute` and `get_schema` methods.
4. Restart the backend (or rely on auto-reload).
5. Your plugin should appear in the Web UI.

90
specs/001-plugin-arch-svelte-ui/research.md Normal file → Executable file
View File

@@ -1,46 +1,46 @@
# Research: Plugin Architecture & Svelte Web UI
## Decisions
### 1. Web Framework: FastAPI
- **Decision**: Use FastAPI for the Python backend.
- **Rationale**:
- Native support for Pydantic models (crucial for plugin schema validation).
- Async support (essential for handling long-running tasks and log streaming via WebSockets/SSE).
- Automatic OpenAPI documentation generation (simplifies frontend integration).
- High performance and modern ecosystem.
- **Alternatives Considered**:
- **Flask**: Mature but requires extensions for validation (Marshmallow) and async support is less native. Slower for high-concurrency API calls.
- **Django**: Too heavy for this use case; brings unnecessary ORM and template engine overhead.
### 2. Plugin System: `importlib` + Abstract Base Classes (ABC)
- **Decision**: Use Python's built-in `importlib` for dynamic loading and `abc` for defining the plugin interface.
- **Rationale**:
- `importlib` provides a standard, secure way to load modules from a path.
- ABCs ensure plugins implement required methods (`execute`, `get_schema`) at load time.
- Lightweight, no external dependencies required.
- **Alternatives Considered**:
- **Pluggy**: Used by pytest, powerful but adds complexity and dependency overhead.
- **Stevedore**: OpenStack's plugin loader, too complex for this scope.
### 3. Authentication: `authlib` + ADFS (OIDC/SAML)
- **Decision**: Use `authlib` to handle ADFS authentication via OpenID Connect (OIDC) or SAML.
- **Rationale**:
- `authlib` is the modern standard for OAuth/OIDC in Python.
- Supports integration with FastAPI via middleware.
- ADFS is the required identity provider (IdP).
- **Alternatives Considered**:
- **python-social-auth**: Older, harder to integrate with FastAPI.
- **Manual JWT implementation**: Risky and reinvents the wheel; ADFS handles the token issuance.
### 4. Frontend: Svelte + Vite
- **Decision**: Use Svelte for the UI framework and Vite as the build tool.
- **Rationale**:
- Svelte's compiler-based approach results in small bundles and high performance.
- Reactive model maps well to real-time log updates.
- Vite provides a fast development experience and easy integration with backend proxies.
## Unknowns Resolved
- **Deployment Context**: Hosted multi-user service with ADFS.
# Research: Plugin Architecture & Svelte Web UI
## Decisions
### 1. Web Framework: FastAPI
- **Decision**: Use FastAPI for the Python backend.
- **Rationale**:
- Native support for Pydantic models (crucial for plugin schema validation).
- Async support (essential for handling long-running tasks and log streaming via WebSockets/SSE).
- Automatic OpenAPI documentation generation (simplifies frontend integration).
- High performance and modern ecosystem.
- **Alternatives Considered**:
- **Flask**: Mature but requires extensions for validation (Marshmallow) and async support is less native. Slower for high-concurrency API calls.
- **Django**: Too heavy for this use case; brings unnecessary ORM and template engine overhead.
### 2. Plugin System: `importlib` + Abstract Base Classes (ABC)
- **Decision**: Use Python's built-in `importlib` for dynamic loading and `abc` for defining the plugin interface.
- **Rationale**:
- `importlib` provides a standard, secure way to load modules from a path.
- ABCs ensure plugins implement required methods (`execute`, `get_schema`) at load time.
- Lightweight, no external dependencies required.
- **Alternatives Considered**:
- **Pluggy**: Used by pytest, powerful but adds complexity and dependency overhead.
- **Stevedore**: OpenStack's plugin loader, too complex for this scope.
### 3. Authentication: `authlib` + ADFS (OIDC/SAML)
- **Decision**: Use `authlib` to handle ADFS authentication via OpenID Connect (OIDC) or SAML.
- **Rationale**:
- `authlib` is the modern standard for OAuth/OIDC in Python.
- Supports integration with FastAPI via middleware.
- ADFS is the required identity provider (IdP).
- **Alternatives Considered**:
- **python-social-auth**: Older, harder to integrate with FastAPI.
- **Manual JWT implementation**: Risky and reinvents the wheel; ADFS handles the token issuance.
### 4. Frontend: Svelte + Vite
- **Decision**: Use Svelte for the UI framework and Vite as the build tool.
- **Rationale**:
- Svelte's compiler-based approach results in small bundles and high performance.
- Reactive model maps well to real-time log updates.
- Vite provides a fast development experience and easy integration with backend proxies.
## Unknowns Resolved
- **Deployment Context**: Hosted multi-user service with ADFS.
- **Plugin Interface**: Will use Pydantic models to define input schemas, allowing the frontend to generate forms dynamically.

142
specs/001-plugin-arch-svelte-ui/spec.md Normal file → Executable file
View File

@@ -1,72 +1,72 @@
# Feature Specification: Plugin Architecture & Svelte Web UI
**Feature Branch**: `001-plugin-arch-svelte-ui`
**Created**: 2025-12-19
**Status**: Draft
**Input**: User description: "Я хочу перевести проект на плагинную архитектуру + добавить web-ui на svelte"
## User Scenarios & Testing *(mandatory)*
### User Story 1 - Web Interface for Superset Tools (Priority: P1)
As a user, I want to interact with the Superset tools (Backup, Migration, Search) through a graphical web interface so that I don't have to memorize CLI commands and arguments.
**Why this priority**: drastically improves usability and accessibility of the tools for non-technical users or quick operations.
**Independent Test**: Can be tested by launching the web server and successfully running a "Backup" task from the browser without touching the command line.
**Acceptance Scenarios**:
1. **Given** the web server is running, **When** I navigate to the home page, **Then** I see a dashboard with available tools (Backup, Migration, etc.).
2. **Given** I am on the Backup tool page, **When** I click "Run Backup", **Then** I see the progress logs in real-time and a success message upon completion.
3. **Given** I am on the Search tool page, **When** I enter a search term and submit, **Then** I see a list of matching datasets/dashboards displayed in a table.
---
### User Story 2 - Dynamic Plugin System (Priority: P2)
As a developer, I want to add new functionality (e.g., a new migration type or report generator) by simply dropping a file into a `plugins` directory, so that I can extend the tool without modifying the core codebase.
**Why this priority**: Enables scalable development and separation of concerns; allows custom extensions without merge conflicts in core files.
**Independent Test**: Create a simple "Hello World" plugin file, place it in the plugins folder, and verify it appears in the list of available tasks in the CLI/Web UI.
**Acceptance Scenarios**:
1. **Given** a valid plugin file in the `plugins/` directory, **When** the application starts, **Then** the plugin is automatically registered and listed as an available capability.
2. **Given** a plugin with specific configuration requirements, **When** I select it in the UI, **Then** the UI dynamically generates a form for those parameters.
3. **Given** an invalid or broken plugin file, **When** the application starts, **Then** the system logs an error but continues to function for other plugins.
---
## Requirements *(mandatory)*
### Functional Requirements
*All functional requirements are covered by the Acceptance Scenarios in the User Stories section.*
- **FR-001**: System MUST provide a Python-based web server (backend) to expose existing tool functionality via API.
- **FR-002**: System MUST provide a Single Page Application (SPA) frontend built with Svelte.
- **FR-003**: System MUST implement a plugin loader that scans a designated directory for Python modules matching a specific interface.
- **FR-004**: The Web UI MUST communicate with the backend via REST or WebSocket API.
- **FR-005**: The Web UI MUST display real-time logs/output from running tasks (streaming response).
- **FR-006**: System MUST support multi-user hosted deployment with authentication via ADFS (Active Directory Federation Services).
- **FR-007**: The Plugin interface MUST allow defining input parameters (schema) so the UI can auto-generate forms.
### System Invariants (Constitution Check)
- **INV-001**: Core logic (backup/migrate functions) must remain decoupled from the UI layer (can still be imported/used by CLI).
- **INV-002**: Plugins must not block the main application thread (long-running tasks must be async or threaded).
### Key Entities
- **Plugin**: Represents an extension module with metadata (name, version), input schema, and an execution entry point.
- **Task**: A specific execution instance of a Plugin or Core tool, having a status (Running, Success, Failed) and logs.
## Success Criteria *(mandatory)*
### Measurable Outcomes
- **SC-001**: A new plugin can be added and recognized by the system without restarting (or with a simple restart) and without code changes to core files.
- **SC-002**: Users can successfully trigger a Backup and Migration via the Web UI with 100% functional parity to the CLI.
- **SC-003**: The Web UI loads and becomes interactive in under 1 second on local networks.
# Feature Specification: Plugin Architecture & Svelte Web UI
**Feature Branch**: `001-plugin-arch-svelte-ui`
**Created**: 2025-12-19
**Status**: Draft
**Input**: User description: "Я хочу перевести проект на плагинную архитектуру + добавить web-ui на svelte"
## User Scenarios & Testing *(mandatory)*
### User Story 1 - Web Interface for Superset Tools (Priority: P1)
As a user, I want to interact with the Superset tools (Backup, Migration, Search) through a graphical web interface so that I don't have to memorize CLI commands and arguments.
**Why this priority**: drastically improves usability and accessibility of the tools for non-technical users or quick operations.
**Independent Test**: Can be tested by launching the web server and successfully running a "Backup" task from the browser without touching the command line.
**Acceptance Scenarios**:
1. **Given** the web server is running, **When** I navigate to the home page, **Then** I see a dashboard with available tools (Backup, Migration, etc.).
2. **Given** I am on the Backup tool page, **When** I click "Run Backup", **Then** I see the progress logs in real-time and a success message upon completion.
3. **Given** I am on the Search tool page, **When** I enter a search term and submit, **Then** I see a list of matching datasets/dashboards displayed in a table.
---
### User Story 2 - Dynamic Plugin System (Priority: P2)
As a developer, I want to add new functionality (e.g., a new migration type or report generator) by simply dropping a file into a `plugins` directory, so that I can extend the tool without modifying the core codebase.
**Why this priority**: Enables scalable development and separation of concerns; allows custom extensions without merge conflicts in core files.
**Independent Test**: Create a simple "Hello World" plugin file, place it in the plugins folder, and verify it appears in the list of available tasks in the CLI/Web UI.
**Acceptance Scenarios**:
1. **Given** a valid plugin file in the `plugins/` directory, **When** the application starts, **Then** the plugin is automatically registered and listed as an available capability.
2. **Given** a plugin with specific configuration requirements, **When** I select it in the UI, **Then** the UI dynamically generates a form for those parameters.
3. **Given** an invalid or broken plugin file, **When** the application starts, **Then** the system logs an error but continues to function for other plugins.
---
## Requirements *(mandatory)*
### Functional Requirements
*All functional requirements are covered by the Acceptance Scenarios in the User Stories section.*
- **FR-001**: System MUST provide a Python-based web server (backend) to expose existing tool functionality via API.
- **FR-002**: System MUST provide a Single Page Application (SPA) frontend built with Svelte.
- **FR-003**: System MUST implement a plugin loader that scans a designated directory for Python modules matching a specific interface.
- **FR-004**: The Web UI MUST communicate with the backend via REST or WebSocket API.
- **FR-005**: The Web UI MUST display real-time logs/output from running tasks (streaming response).
- **FR-006**: System MUST support multi-user hosted deployment with authentication via ADFS (Active Directory Federation Services).
- **FR-007**: The Plugin interface MUST allow defining input parameters (schema) so the UI can auto-generate forms.
### System Invariants (Constitution Check)
- **INV-001**: Core logic (backup/migrate functions) must remain decoupled from the UI layer (can still be imported/used by CLI).
- **INV-002**: Plugins must not block the main application thread (long-running tasks must be async or threaded).
### Key Entities
- **Plugin**: Represents an extension module with metadata (name, version), input schema, and an execution entry point.
- **Task**: A specific execution instance of a Plugin or Core tool, having a status (Running, Success, Failed) and logs.
## Success Criteria *(mandatory)*
### Measurable Outcomes
- **SC-001**: A new plugin can be added and recognized by the system without restarting (or with a simple restart) and without code changes to core files.
- **SC-002**: Users can successfully trigger a Backup and Migration via the Web UI with 100% functional parity to the CLI.
- **SC-003**: The Web UI loads and becomes interactive in under 1 second on local networks.
- **SC-004**: Real-time logs in the UI appear with less than 200ms latency from the backend execution.

134
specs/001-plugin-arch-svelte-ui/tasks.md Normal file → Executable file
View File

@@ -1,68 +1,68 @@
# Tasks: Plugin Architecture & Svelte Web UI
**Feature**: `001-plugin-arch-svelte-ui`
**Status**: Planned
## Dependencies
1. **Phase 1 (Setup)**: Must be completed first to establish the environment.
2. **Phase 2 (Foundational)**: Implements the core Plugin system and Backend infrastructure required by all User Stories.
3. **Phase 3 (US1)**: Web Interface depends on the Backend API and Plugin system.
4. **Phase 4 (US2)**: Dynamic Plugin System extends the core infrastructure.
## Parallel Execution Opportunities
- **US1 (Frontend)**: Frontend components (T013-T016) can be developed in parallel with Backend API endpoints (T011-T012) once the API contract is finalized.
- **US2 (Plugins)**: Plugin development (T019-T020) can proceed independently once the Plugin Interface (T005) is stable.
---
## Phase 1: Setup
**Goal**: Initialize the project structure and development environment for Backend (Python/FastAPI) and Frontend (Svelte/Vite).
- [x] T001 Create backend directory structure (src/api, src/core, src/plugins) in `backend/`
- [x] T002 Create frontend directory structure using Vite (Svelte template) in `frontend/`
- [x] T003 Configure Python environment (requirements.txt with FastAPI, Uvicorn, Pydantic) in `backend/requirements.txt`
- [x] T004 Configure Frontend environment (package.json with TailwindCSS) in `frontend/package.json`
## Phase 2: Foundational (Core Infrastructure)
**Goal**: Implement the core Plugin interface, Task management system, and basic Backend server.
- [x] T005 [P] Define `PluginBase` abstract class and Pydantic models in `backend/src/core/plugin_base.py`
- [x] T006 [P] Implement `PluginLoader` to scan and load plugins from directory in `backend/src/core/plugin_loader.py`
- [x] T007 Implement `TaskManager` to handle async task execution and state in `backend/src/core/task_manager.py`
- [x] T008 [P] Implement `Logger` with WebSocket streaming support in `backend/src/core/logger.py`
- [x] T009 Create basic FastAPI application entry point with CORS in `backend/src/app.py`
- [x] T010 [P] Implement ADFS Authentication middleware in `backend/src/api/auth.py`
## Phase 3: User Story 1 - Web Interface (Priority: P1)
**Goal**: Enable users to interact with tools via a web dashboard.
**Independent Test**: Launch web server, navigate to dashboard, run a dummy task, view logs.
- [x] T011 [US1] Implement REST API endpoints for Plugin listing (`GET /plugins`) in `backend/src/api/routes/plugins.py`
- [x] T012 [US1] Implement REST API endpoints for Task management (`POST /tasks`, `GET /tasks/{id}`) in `backend/src/api/routes/tasks.py`
- [x] T013 [P] [US1] Create Svelte store for Plugin and Task state in `frontend/src/lib/stores.js`
- [x] T014 [P] [US1] Create `Dashboard` page component listing available tools in `frontend/src/pages/Dashboard.svelte`
- [x] T015 [P] [US1] Create `TaskRunner` component with real-time log viewer (WebSocket) in `frontend/src/components/TaskRunner.svelte`
- [x] T016 [US1] Integrate Frontend with Backend API using `fetch` client in `frontend/src/lib/api.js`
## Phase 4: User Story 2 - Dynamic Plugin System (Priority: P2)
**Goal**: Allow developers to add new functionality by dropping files.
**Independent Test**: Add `hello_world.py` to plugins dir, verify it appears in UI.
- [x] T017 [US2] Implement dynamic form generation component based on JSON Schema in `frontend/src/components/DynamicForm.svelte`
- [x] T018 [US2] Update `PluginLoader` to validate plugin schema on load in `backend/src/core/plugin_loader.py`
- [x] T019 [P] [US2] Refactor existing `backup_script.py` into a Plugin (`BackupPlugin`) in `backend/src/plugins/backup.py`
- [x] T020 [P] [US2] Refactor existing `migration_script.py` into a Plugin (`MigrationPlugin`) in `backend/src/plugins/migration.py`
## Final Phase: Polish
**Goal**: Ensure production readiness.
- [x] T021 Add error handling and user notifications (Toasts) in Frontend
- [x] T022 Write documentation for Plugin Development in `docs/plugin_dev.md`
# Tasks: Plugin Architecture & Svelte Web UI
**Feature**: `001-plugin-arch-svelte-ui`
**Status**: Planned
## Dependencies
1. **Phase 1 (Setup)**: Must be completed first to establish the environment.
2. **Phase 2 (Foundational)**: Implements the core Plugin system and Backend infrastructure required by all User Stories.
3. **Phase 3 (US1)**: Web Interface depends on the Backend API and Plugin system.
4. **Phase 4 (US2)**: Dynamic Plugin System extends the core infrastructure.
## Parallel Execution Opportunities
- **US1 (Frontend)**: Frontend components (T013-T016) can be developed in parallel with Backend API endpoints (T011-T012) once the API contract is finalized.
- **US2 (Plugins)**: Plugin development (T019-T020) can proceed independently once the Plugin Interface (T005) is stable.
---
## Phase 1: Setup
**Goal**: Initialize the project structure and development environment for Backend (Python/FastAPI) and Frontend (Svelte/Vite).
- [x] T001 Create backend directory structure (src/api, src/core, src/plugins) in `backend/`
- [x] T002 Create frontend directory structure using Vite (Svelte template) in `frontend/`
- [x] T003 Configure Python environment (requirements.txt with FastAPI, Uvicorn, Pydantic) in `backend/requirements.txt`
- [x] T004 Configure Frontend environment (package.json with TailwindCSS) in `frontend/package.json`
## Phase 2: Foundational (Core Infrastructure)
**Goal**: Implement the core Plugin interface, Task management system, and basic Backend server.
- [x] T005 [P] Define `PluginBase` abstract class and Pydantic models in `backend/src/core/plugin_base.py`
- [x] T006 [P] Implement `PluginLoader` to scan and load plugins from directory in `backend/src/core/plugin_loader.py`
- [x] T007 Implement `TaskManager` to handle async task execution and state in `backend/src/core/task_manager.py`
- [x] T008 [P] Implement `Logger` with WebSocket streaming support in `backend/src/core/logger.py`
- [x] T009 Create basic FastAPI application entry point with CORS in `backend/src/app.py`
- [x] T010 [P] Implement ADFS Authentication middleware in `backend/src/api/auth.py`
## Phase 3: User Story 1 - Web Interface (Priority: P1)
**Goal**: Enable users to interact with tools via a web dashboard.
**Independent Test**: Launch web server, navigate to dashboard, run a dummy task, view logs.
- [x] T011 [US1] Implement REST API endpoints for Plugin listing (`GET /plugins`) in `backend/src/api/routes/plugins.py`
- [x] T012 [US1] Implement REST API endpoints for Task management (`POST /tasks`, `GET /tasks/{id}`) in `backend/src/api/routes/tasks.py`
- [x] T013 [P] [US1] Create Svelte store for Plugin and Task state in `frontend/src/lib/stores.js`
- [x] T014 [P] [US1] Create `Dashboard` page component listing available tools in `frontend/src/pages/Dashboard.svelte`
- [x] T015 [P] [US1] Create `TaskRunner` component with real-time log viewer (WebSocket) in `frontend/src/components/TaskRunner.svelte`
- [x] T016 [US1] Integrate Frontend with Backend API using `fetch` client in `frontend/src/lib/api.js`
## Phase 4: User Story 2 - Dynamic Plugin System (Priority: P2)
**Goal**: Allow developers to add new functionality by dropping files.
**Independent Test**: Add `hello_world.py` to plugins dir, verify it appears in UI.
- [x] T017 [US2] Implement dynamic form generation component based on JSON Schema in `frontend/src/components/DynamicForm.svelte`
- [x] T018 [US2] Update `PluginLoader` to validate plugin schema on load in `backend/src/core/plugin_loader.py`
- [x] T019 [P] [US2] Refactor existing `backup_script.py` into a Plugin (`BackupPlugin`) in `backend/src/plugins/backup.py`
- [x] T020 [P] [US2] Refactor existing `migration_script.py` into a Plugin (`MigrationPlugin`) in `backend/src/plugins/migration.py`
## Final Phase: Polish
**Goal**: Ensure production readiness.
- [x] T021 Add error handling and user notifications (Toasts) in Frontend
- [x] T022 Write documentation for Plugin Development in `docs/plugin_dev.md`
- [ ] T023 Final integration test: Run full Backup and Migration flow via UI

34
specs/002-app-settings/checklists/requirements.md Executable file
View File

@@ -0,0 +1,34 @@
# Specification Quality Checklist: Add web application settings mechanism
**Purpose**: Validate specification completeness and quality before proceeding to planning
**Created**: 2025-12-20
**Feature**: [specs/002-app-settings/spec.md](specs/002-app-settings/spec.md)
## Content Quality
- [x] No implementation details (languages, frameworks, APIs)
- [x] Focused on user value and business needs
- [x] Written for non-technical stakeholders
- [x] All mandatory sections completed
## Requirement Completeness
- [x] No [NEEDS CLARIFICATION] markers remain
- [x] Requirements are testable and unambiguous
- [x] Success criteria are measurable
- [x] Success criteria are technology-agnostic (no implementation details)
- [x] All acceptance scenarios are defined
- [x] Edge cases are identified
- [x] Scope is clearly bounded
- [x] Dependencies and assumptions identified
## Feature Readiness
- [x] All functional requirements have clear acceptance criteria
- [x] User scenarios cover primary flows
- [x] Feature meets measurable outcomes defined in Success Criteria
- [x] No implementation details leak into specification
## Notes
- Initial specification covers all requested points with reasonable defaults for authentication and storage validation.

102
specs/002-app-settings/plan.md Executable file
View File

@@ -0,0 +1,102 @@
# Technical Plan: Web Application Settings Mechanism
This plan outlines the implementation of a settings management system for the Superset Tools application, allowing users to configure multiple Superset environments and global application settings (like backup storage) via the web UI.
## 1. Backend Architecture
### 1.1 Data Models (Pydantic)
We will define models in `backend/src/core/config_models.py`:
```python
from pydantic import BaseModel, Field
from typing import List, Optional
class Environment(BaseModel):
id: str
name: str
url: str
username: str
password: str # Will be masked in UI
is_default: bool = False
class GlobalSettings(BaseModel):
backup_path: str
default_environment_id: Optional[str] = None
class AppConfig(BaseModel):
environments: List[Environment] = []
settings: GlobalSettings
```
### 1.2 Configuration Manager
A new class `ConfigManager` in `backend/src/core/config_manager.py` will handle:
- Loading/saving `AppConfig` to `config.json`.
- CRUD operations for environments.
- Updating global settings.
- Validating backup paths and Superset URLs.
### 1.3 API Endpoints
New router `backend/src/api/routes/settings.py`:
- `GET /settings`: Retrieve all settings (masking passwords).
- `PATCH /settings/global`: Update global settings (backup path, etc.).
- `GET /settings/environments`: List all environments.
- `POST /settings/environments`: Add a new environment.
- `PUT /settings/environments/{id}`: Update an environment.
- `DELETE /settings/environments/{id}`: Remove an environment.
- `POST /settings/environments/{id}/test`: Test connection to a specific environment.
### 1.4 Integration
- Update `backend/src/dependencies.py` to provide a singleton `ConfigManager`.
- Refactor `superset_tool/utils/init_clients.py` to fetch environment details from `ConfigManager` instead of hardcoded values.
## 2. Frontend Implementation
### 2.1 Settings Page
- Create `frontend/src/pages/Settings.svelte`.
- Add a "Settings" link to the main navigation (likely in `App.svelte`).
### 2.2 Components
- **EnvironmentList**: Displays a table/list of configured environments with Edit/Delete buttons.
- **EnvironmentForm**: A modal or inline form for adding/editing environments.
- **GlobalSettingsForm**: Form for editing the backup storage path.
### 2.3 API Integration
- Add functions to `frontend/src/lib/api.js` for interacting with the new settings endpoints.
## 3. Workflow Diagram
```mermaid
graph TD
UI[Web UI - Settings Page] --> API[FastAPI Settings Router]
API --> CM[Config Manager]
CM --> JSON[(config.json)]
CM --> SS[Superset Instance] : Test Connection
Plugins[Plugins - Backup/Migration] --> CM : Get Env/Path
```
## 4. Implementation Steps
1. **Backend Core**:
- Create `config_models.py` and `config_manager.py`.
- Implement file-based persistence.
2. **Backend API**:
- Implement `settings.py` router.
- Register router in `app.py`.
3. **Frontend UI**:
- Create `Settings.svelte` and necessary components.
- Implement API calls and state management.
4. **Refactoring**:
- Update `init_clients.py` to use the new configuration system.
- Ensure existing plugins (Backup, Migration) use the configured settings.
5. **Validation**:
- Add path existence/write checks for backup storage.
- Add URL/Connection checks for Superset environments.

77
specs/002-app-settings/spec.md Executable file
View File

@@ -0,0 +1,77 @@
# Feature Specification: Add web application settings mechanism
**Feature Branch**: `002-app-settings`
**Created**: 2025-12-20
**Status**: Draft
**Input**: User description: "давай внесем полноценный механизм настройки веб приложения. Что нужно точно - 1. Интерфейс для добавления enviroments (разные сервера суперсета) 2. Интерфейс для настройки файлового хранилища бекапов"
## User Scenarios & Testing *(mandatory)*
### User Story 1 - Manage Superset Environments (Priority: P1)
As an administrator, I want to add, edit, and remove Superset environment configurations (URL, credentials, name) so that the application can interact with multiple Superset instances.
**Why this priority**: This is the core functionality required for the tool to be useful across different stages (dev/prod) or different Superset clusters.
**Independent Test**: Can be fully tested by adding a new environment, verifying it appears in the list, and then deleting it.
**Acceptance Scenarios**:
1. **Given** the settings page is open, **When** I enter valid Superset connection details and save, **Then** the new environment is added to the list of available targets.
2. **Given** an existing environment, **When** I update its URL and save, **Then** the system uses the new URL for subsequent operations.
3. **Given** an existing environment, **When** I delete it, **Then** it is no longer available for selection in other parts of the application.
---
### User Story 2 - Configure Backup Storage (Priority: P1)
As an administrator, I want to configure the file path or storage location for backups so that I can control where system backups are stored.
**Why this priority**: Essential for the backup plugin to function correctly and for users to manage disk space/storage locations.
**Independent Test**: Can be tested by setting a backup path and verifying that the system validates the path's existence or accessibility.
**Acceptance Scenarios**:
1. **Given** the storage settings section, **When** I provide a valid local or network path, **Then** the system saves this as the default backup location.
2. **Given** an invalid or inaccessible path, **When** I try to save, **Then** the system displays an error message and does not update the setting.
---
### Edge Cases
- **Duplicate Environments**: What happens when a user tries to add an environment with a name that already exists? (System should prevent duplicates).
- **Invalid Credentials**: How does the system handle saving environments with incorrect credentials? (System should ideally validate connection on save).
- **Path Permissions**: How does the system handle a backup path that is valid but the application lacks write permissions for? (System should check write permissions).
## Requirements *(mandatory)*
### Functional Requirements
- **FR-001**: System MUST provide a dedicated settings interface in the web UI.
- **FR-002**: System MUST allow users to create multiple named "Environments" for Superset.
- **FR-003**: Each Environment MUST include: Name, Base URL, and Authentication details (e.g., Username/Password or API Key).
- **FR-004**: System MUST allow setting a global "Backup Storage Path".
- **FR-005**: System MUST persist these settings across application restarts.
- **FR-006**: System MUST validate the Superset URL format before saving.
- **FR-007**: System MUST verify that the Backup Storage Path is writable by the application.
- **FR-008**: System MUST allow selecting a "Default" environment for operations.
### System Invariants (Constitution Check)
- **INV-001**: Sensitive credentials (passwords/keys) MUST NOT be displayed in plain text after being saved.
- **INV-002**: At least one environment MUST be configured for the application to perform Superset-related tasks.
### Key Entities *(include if feature involves data)*
- **Environment**: Represents a Superset instance. Attributes: Unique ID, Name, URL, Credentials, IsDefault flag.
- **AppConfiguration**: Singleton entity representing global settings. Attributes: BackupPath, DefaultEnvironmentID.
## Success Criteria *(mandatory)*
### Measurable Outcomes
- **SC-001**: Users can add a new Superset environment in under 30 seconds.
- **SC-002**: 100% of saved environments are immediately available for use in backup/migration tasks.
- **SC-003**: System prevents saving invalid backup paths 100% of the time.
- **SC-004**: Configuration changes take effect without requiring a manual restart of the backend services.

141
specs/002-app-settings/tasks.md Normal file
View File

@@ -0,0 +1,141 @@
---
description: "Task list for implementing the web application settings mechanism"
---
# Tasks: Web Application Settings Mechanism
**Input**: Design documents from `specs/002-app-settings/`
**Prerequisites**: plan.md (required), spec.md (required for user stories)
**Organization**: Tasks are grouped by user story to enable independent implementation and testing of each story.
## Format: `[ID] [P?] [Story] Description`
- **[P]**: Can run in parallel (different files, no dependencies)
- **[Story]**: Which user story this task belongs to (e.g., US1, US2, US3)
- Include exact file paths in descriptions
## Phase 1: Setup (Shared Infrastructure)
**Purpose**: Project initialization and basic structure
- [x] T001 Create project structure for settings management in `backend/src/core/` and `backend/src/api/routes/`
- [x] T002 [P] Initialize `frontend/src/pages/Settings.svelte` placeholder
---
## Phase 2: Foundational (Blocking Prerequisites)
**Purpose**: Core infrastructure that MUST be complete before ANY user story can be implemented
**⚠️ CRITICAL**: No user story work can begin until this phase is complete
- [x] T003 Implement configuration models in `backend/src/core/config_models.py`
- [x] T004 Implement `ConfigManager` for JSON persistence in `backend/src/core/config_manager.py`
- [x] T005 [P] Update `backend/src/dependencies.py` to provide `ConfigManager` singleton
- [x] T006 [P] Setup API routing for settings in `backend/src/api/routes/settings.py` and register in `backend/src/app.py`
**Checkpoint**: Foundation ready - user story implementation can now begin in parallel
---
## Phase 3: User Story 1 - Manage Superset Environments (Priority: P1) 🎯 MVP
**Goal**: Add, edit, and remove Superset environment configurations (URL, credentials, name) so that the application can interact with multiple Superset instances.
**Independent Test**: Add a new environment, verify it appears in the list, and then delete it.
### Implementation for User Story 1
- [x] T007 [P] [US1] Implement environment CRUD logic in `backend/src/core/config_manager.py`
- [x] T008 [US1] Implement environment API endpoints in `backend/src/api/routes/settings.py`
- [x] T009 [P] [US1] Add environment API methods to `frontend/src/lib/api.js`
- [x] T010 [US1] Implement environment list and form UI in `frontend/src/pages/Settings.svelte`
- [x] T011 [US1] Implement connection test logic in `backend/src/api/routes/settings.py`
**Checkpoint**: At this point, User Story 1 should be fully functional and testable independently
---
## Phase 4: User Story 2 - Configure Backup Storage (Priority: P1)
**Goal**: Configure the file path or storage location for backups so that I can control where system backups are stored.
**Independent Test**: Set a backup path and verify that the system validates the path's existence or accessibility.
### Implementation for User Story 2
- [x] T012 [P] [US2] Implement global settings update logic in `backend/src/core/config_manager.py`
- [x] T013 [US2] Implement global settings API endpoints in `backend/src/api/routes/settings.py`
- [x] T014 [P] [US2] Add global settings API methods to `frontend/src/lib/api.js`
- [x] T015 [US2] Implement backup storage configuration UI in `frontend/src/pages/Settings.svelte`
- [x] T016 [US2] Add path validation and write permission checks in `backend/src/api/routes/settings.py`
**Checkpoint**: At this point, User Stories 1 AND 2 should both work independently
---
## Phase 5: Polish & Cross-Cutting Concerns
**Purpose**: Improvements that affect multiple user stories
- [x] T017 Refactor `superset_tool/utils/init_clients.py` to use `ConfigManager` for environment details
- [x] T018 Update existing plugins (Backup, Migration) to fetch settings from `ConfigManager`
- [x] T019 [P] Add password masking in `backend/src/api/routes/settings.py` and UI
- [x] T020 [P] Add "Settings" link to navigation in `frontend/src/App.svelte`
- [x] T021 [P] Documentation updates for settings mechanism in `docs/`
---
## Dependencies & Execution Order
### Phase Dependencies
- **Setup (Phase 1)**: No dependencies - can start immediately
- **Foundational (Phase 2)**: Depends on Setup completion - BLOCKS all user stories
- **User Stories (Phase 3+)**: All depend on Foundational phase completion
- User stories can then proceed in parallel (if staffed)
- Or sequentially in priority order (P1 → P2 → P3)
- **Polish (Final Phase)**: Depends on all desired user stories being complete
### User Story Dependencies
- **User Story 1 (P1)**: Can start after Foundational (Phase 2) - No dependencies on other stories
- **User Story 2 (P1)**: Can start after Foundational (Phase 2) - Independent of US1
### Parallel Opportunities
- All Setup tasks marked [P] can run in parallel
- All Foundational tasks marked [P] can run in parallel (within Phase 2)
- Once Foundational phase completes, all user stories can start in parallel
- Models and API methods within a story marked [P] can run in parallel
---
## Parallel Example: User Story 1
```bash
# Launch backend and frontend tasks for User Story 1 together:
Task: "Implement environment CRUD logic in backend/src/core/config_manager.py"
Task: "Add environment API methods to frontend/src/lib/api.js"
```
---
## Implementation Strategy
### MVP First (User Story 1 Only)
1. Complete Phase 1: Setup
2. Complete Phase 2: Foundational (CRITICAL - blocks all stories)
3. Complete Phase 3: User Story 1
4. **STOP and VALIDATE**: Test User Story 1 independently
5. Deploy/demo if ready
### Incremental Delivery
1. Complete Setup + Foundational → Foundation ready
2. Add User Story 1 → Test independently → Deploy/Demo (MVP!)
3. Add User Story 2 → Test independently → Deploy/Demo
4. Each story adds value without breaking previous stories