docs ready

specs/009-backup-scheduler/checklists/requirements.md

@@ -0,0 +1,34 @@
+# Specification Quality Checklist: Backup Scheduler & Unified Task UI
+
+**Purpose**: Validate specification completeness and quality before proceeding to planning
+**Created**: 2025-12-30
+**Feature**: [Link to spec.md](../spec.md)
+
+## Content Quality
+
+- [x] No implementation details (languages, frameworks, APIs)
+- [x] Focused on user value and business needs
+- [x] Written for non-technical stakeholders
+- [x] All mandatory sections completed
+
+## Requirement Completeness
+
+- [x] No [NEEDS CLARIFICATION] markers remain
+- [x] Requirements are testable and unambiguous
+- [x] Success criteria are measurable
+- [x] Success criteria are technology-agnostic (no implementation details)
+- [x] All acceptance scenarios are defined
+- [x] Edge cases are identified
+- [x] Scope is clearly bounded
+- [x] Dependencies and assumptions identified
+
+## Feature Readiness
+
+- [x] All functional requirements have clear acceptance criteria
+- [x] User scenarios cover primary flows
+- [x] Feature meets measurable outcomes defined in Success Criteria
+- [x] No implementation details leak into specification
+
+## Notes
+
+- Items marked incomplete require spec updates before `/speckit.clarify` or `/speckit.plan`

specs/009-backup-scheduler/contracts/api.yaml

@@ -0,0 +1,154 @@
+openapi: 3.0.0
+info:
+  title: Backup Scheduler & Task API
+  version: 1.0.0
+paths:
+  /tasks:
+    get:
+      summary: List all tasks
+      parameters:
+        - name: limit
+          in: query
+          schema:
+            type: integer
+            default: 50
+        - name: type
+          in: query
+          schema:
+            type: string
+            enum: [backup, migration]
+        - name: status
+          in: query
+          schema:
+            type: string
+            enum: [running, success, failed, pending]
+      responses:
+        '200':
+          description: List of tasks
+          content:
+            application/json:
+              schema:
+                type: array
+                items:
+                  $ref: '#/components/schemas/Task'
+
+  /tasks/backup:
+    post:
+      summary: Manually trigger a backup
+      requestBody:
+        required: true
+        content:
+          application/json:
+            schema:
+              type: object
+              required:
+                - environment_id
+              properties:
+                environment_id:
+                  type: string
+                  format: uuid
+      responses:
+        '202':
+          description: Backup task started
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/Task'
+
+  /tasks/{id}:
+    get:
+      summary: Get task details
+      parameters:
+        - name: id
+          in: path
+          required: true
+          schema:
+            type: string
+            format: uuid
+      responses:
+        '200':
+          description: Task details
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/Task'
+
+  /environments/{id}/schedule:
+    get:
+      summary: Get backup schedule for environment
+      parameters:
+        - name: id
+          in: path
+          required: true
+          schema:
+            type: string
+            format: uuid
+      responses:
+        '200':
+          description: Schedule configuration
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/Schedule'
+    put:
+      summary: Update backup schedule
+      parameters:
+        - name: id
+          in: path
+          required: true
+          schema:
+            type: string
+            format: uuid
+      requestBody:
+        required: true
+        content:
+          application/json:
+            schema:
+              $ref: '#/components/schemas/Schedule'
+      responses:
+        '200':
+          description: Schedule updated
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/Schedule'
+
+components:
+  schemas:
+    Task:
+      type: object
+      properties:
+        id:
+          type: string
+          format: uuid
+        type:
+          type: string
+          enum: [backup, migration]
+        status:
+          type: string
+          enum: [pending, running, success, failed]
+        environment_id:
+          type: string
+          format: uuid
+        started_at:
+          type: string
+          format: date-time
+        finished_at:
+          type: string
+          format: date-time
+        created_at:
+          type: string
+          format: date-time
+        error:
+          type: string
+        logs:
+          type: string
+
+    Schedule:
+      type: object
+      properties:
+        enabled:
+          type: boolean
+        cron_expression:
+          type: string
+          example: "0 0 * * *"

specs/009-backup-scheduler/data-model.md

@@ -0,0 +1,34 @@
+# Data Model: Backup Scheduler & Unified Task UI
+
+## Entities
+
+### Task
+Represents a background operation (Backup, Migration) managed by the system.
+
+| Field | Type | Description | Constraints |
+|-------|------|-------------|-------------|
+| `id` | UUID | Unique identifier | Primary Key |
+| `type` | String | Type of task | Enum: "backup", "migration" |
+| `status` | String | Current execution state | Enum: "pending", "running", "success", "failed" |
+| `environment_id` | UUID | Target environment (if applicable) | Foreign Key (Environments), Nullable |
+| `started_at` | DateTime | When the task began | Nullable |
+| `finished_at` | DateTime | When the task completed | Nullable |
+| `logs` | Text | Execution logs | |
+| `error` | Text | Error message if failed | Nullable |
+| `created_at` | DateTime | When task was queued | Default: Now |
+
+### Schedule
+Configuration for automatic task execution. Nested within Environment config.
+
+| Field | Type | Description | Constraints |
+|-------|------|-------------|-------------|
+| `environment_id` | UUID | Target environment | Foreign Key (Environments) |
+| `enabled` | Boolean | Is schedule active? | Default: false |
+| `cron_expression` | String | Frequency definition | Valid Cron string (e.g., "0 0 * * *") |
+| `last_run_at` | DateTime | Last execution time | Nullable |
+| `next_run_at` | DateTime | Calculated next run | Nullable |
+
+## Storage Strategy
+
+- **Tasks**: Stored in `tasks.db` (SQLite) via SQLAlchemy.
+- **Schedules**: Stored in `config.json` as part of the Environment model.

specs/009-backup-scheduler/plan.md

@@ -0,0 +1,84 @@
+# Implementation Plan: Backup Scheduler & Unified Task UI
+
+**Branch**: `009-backup-scheduler` | **Date**: 2025-12-30 | **Spec**: [link](spec.md)
+**Input**: Feature specification from `/specs/009-backup-scheduler/spec.md`
+
+**Note**: This template is filled in by the `/speckit.plan` command. See `.specify/templates/commands/plan.md` for the execution workflow.
+
+## Summary
+
+Implement a robust backup scheduling system using `APScheduler` and a unified "Tasks" UI in SvelteKit to manage and monitor all background operations (backups, migrations).
+
+## Technical Context
+
+**Language/Version**: Python 3.9+, Node.js 18+
+**Primary Dependencies**: FastAPI, APScheduler, SQLAlchemy, SvelteKit, Tailwind CSS
+**Storage**: SQLite (`tasks.db`), JSON (`config.json`)
+**Testing**: pytest
+**Target Platform**: Linux server
+**Project Type**: Web application
+**Performance Goals**: UI latency < 200ms, Backup trigger < 1s
+**Constraints**: Minimal resource footprint for background scheduler
+**Scale/Scope**: ~10 environments, ~1000 historical tasks
+
+## Constitution Check
+
+*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
+
+- **Library-First**: N/A (Feature integration)
+- **CLI Interface**: N/A (Web UI focus)
+- **Test-First**: Mandatory for Scheduler logic and API endpoints. PASS.
+- **Integration Testing**: Required for Scheduler -> TaskManager interaction. PASS.
+
+**Result**: PASS
+
+## Project Structure
+
+### Documentation (this feature)
+
+```text
+specs/009-backup-scheduler/
+├── plan.md              # This file (/speckit.plan command output)
+├── research.md          # Phase 0 output (/speckit.plan command)
+├── data-model.md        # Phase 1 output (/speckit.plan command)
+├── quickstart.md        # Phase 1 output (/speckit.plan command)
+├── contracts/           # Phase 1 output (/speckit.plan command)
+└── tasks.md             # Phase 2 output (/speckit.tasks command - NOT created by /speckit.plan)
+```
+
+### Source Code (repository root)
+
+```text
+backend/
+├── src/
+│   ├── api/
+│   │   └── routes/
+│   │       └── tasks.py        # NEW: Task management endpoints
+│   ├── core/
+│   │   ├── scheduler.py        # NEW: APScheduler integration
+│   │   └── task_manager/       # EXISTING: Updates for DB persistence
+│   ├── models/
+│   │   └── task.py             # NEW: SQLAlchemy model
+│   └── services/
+└── tests/
+
+frontend/
+├── src/
+│   ├── components/
+│   │   └── TaskList.svelte     # NEW: Task display component
+│   │   └── TaskLogViewer.svelte # NEW: Detailed log view
+│   ├── routes/
+│   │   └── tasks/              # NEW: Tasks page
+│   │       └── +page.svelte
+│   └── types/
+```
+
+**Structure Decision**: Standard FastAPI + SvelteKit structure.
+
+## Complexity Tracking
+
+> **Fill ONLY if Constitution Check has violations that must be justified**
+
+| Violation | Why Needed | Simpler Alternative Rejected Because |
+|-----------|------------|-------------------------------------|
+| | | |

specs/009-backup-scheduler/quickstart.md

@@ -0,0 +1,28 @@
+# Quickstart: Backup Scheduler & Unified Task UI
+
+## Prerequisites
+- Backend running: `cd backend && uvicorn src.app:app --reload`
+- Frontend running: `cd frontend && npm run dev`
+
+## Usage Guide
+
+### 1. View Tasks
+1. Navigate to the new **Tasks** tab in the main navigation bar.
+2. Observe the list of recent tasks (Backups, Migrations).
+3. Click on any task row to view detailed logs.
+
+### 2. Configure Scheduled Backups
+1. Go to **Settings**.
+2. Edit an existing Environment (or create a new one).
+3. Scroll to the **Backup Schedule** section.
+4. Enable the "Automatic Backups" toggle.
+5. Enter a valid Cron expression (e.g., `*/5 * * * *` for every 5 minutes).
+6. Save the environment.
+7. Wait for the scheduled time and verify a new Backup task appears in the **Tasks** tab.
+
+### 3. Manual Backup Trigger
+1. Go to the **Tasks** tab.
+2. Click the **Run Backup** button (top right).
+3. Select the target environment from the dropdown.
+4. Click **Start**.
+5. Watch the new task appear with "Running" status.

specs/009-backup-scheduler/research.md

@@ -0,0 +1,29 @@
+# Research: Backup Scheduler & Unified Task UI
+
+## Decisions
+
+### 1. Scheduler Implementation
+- **Decision**: Use `APScheduler` (Advanced Python Scheduler) with `BackgroundScheduler`.
+- **Rationale**: `APScheduler` is the industry standard for Python scheduling. It supports Cron-style scheduling (required by FR-001), runs in the background (FR-003), and handles thread management. It integrates well with FastAPI lifecycles.
+- **Alternatives Considered**: 
+    - `cron` (system level): Harder to manage from within the app, requires OS access.
+    - `schedule` library: Simpler but lacks advanced Cron features and persistence robustness.
+    - Custom thread loop: Error-prone and reinvents the wheel.
+
+### 2. Task History Database
+- **Decision**: SQLite (`tasks.db`) accessed via `SQLAlchemy` (AsyncIO).
+- **Rationale**: The spec explicitly requests `tasks.db` (FR-009). SQLAlchemy provides a robust ORM for the `Task` entity. Using AsyncIO ensures non-blocking database operations within the FastAPI event loop, even if the actual backup tasks run in threads.
+- **Alternatives Considered**: 
+    - `JSON` files: Poor performance for filtering/sorting logs (FR-006).
+    - `PostgreSQL`: Overkill for a local tool configuration.
+
+### 3. Concurrency Handling
+- **Decision**: Skip scheduled backups if a backup is already running for the *same* environment. Allow concurrent backups for *different* environments.
+- **Rationale**: Prevents resource contention and potential corruption of the same target.
+- **Implementation**: The `SchedulerService` will check `TaskManager` for active jobs with the same `environment_id` before triggering.
+
+### 4. Frontend Polling vs WebSockets
+- **Decision**: Polling (every 2-5 seconds) for the "Tasks" tab.
+- **Rationale**: Simpler to implement than WebSockets for this scale. The requirement is "near real-time" (SC-002: latency < 5s), which polling satisfies easily.
+- **Alternatives Considered**: 
+    - WebSockets: Better real-time, but higher complexity for connection management and state.

specs/009-backup-scheduler/spec.md

@@ -0,0 +1,115 @@
+# Feature Specification: Backup Scheduler & Unified Task UI
+
+**Feature Branch**: `009-backup-scheduler`
+**Created**: 2025-12-30
+**Status**: Draft
+**Input**: User description: "Я хочу доработать механизм бекапа. Он должен иметь возможность работать по расписанию, задания и их статус должны использовать TaskManager и быть доступны в общем логе. Я думаю нужно вынести все задачи в отдельную вкладку - миграции, бэкапов и прочих задач которые мы в будущем добавим."
+
+## User Scenarios & Testing *(mandatory)*
+
+### User Story 1 - Scheduled Backups (Priority: P1)
+
+As an Administrator, I want to configure automatic backup schedules for my Superset environments so that my data is preserved regularly without manual intervention.
+
+**Why this priority**: Automation is the core request. It ensures data safety and reduces manual toil.
+
+**Independent Test**: Configure a schedule (e.g., every minute for testing), wait, and verify a backup task is created and executed automatically.
+
+**Acceptance Scenarios**:
+
+1. **Given** an environment configuration, **When** I enable scheduled backups with a specific interval (e.g., daily), **Then** the system automatically triggers a backup task at the specified time.
+2. **Given** a scheduled backup runs, **When** it completes, **Then** a new backup archive is present in the storage and a success log is recorded.
+
+---
+
+### User Story 2 - Unified Task Management UI (Priority: P1)
+
+As an Administrator, I want a dedicated "Tasks" tab where I can see and manage all background operations (backups, migrations) in one place.
+
+**Why this priority**: Centralizes visibility and control, improving usability as the number of background tasks grows.
+
+**Independent Test**: Navigate to the new "Tasks" tab and verify it lists both manual and scheduled tasks with their current status.
+
+**Acceptance Scenarios**:
+
+1. **Given** the application is open, **When** I click the "Tasks" tab, **Then** I see a list of recent tasks including their type (Backup, Migration), status, and timestamp.
+2. **Given** a running task, **When** I view the Tasks tab, **Then** I see the task status update in real-time (or near real-time).
+
+---
+
+### User Story 3 - Manual Backup Trigger (Priority: P2)
+
+As an Administrator, I want to manually trigger a backup from the Tasks UI immediately, for example, before a major change.
+
+**Why this priority**: Ad-hoc backups are necessary for operational safety before maintenance.
+
+**Independent Test**: Click "Run Backup" in the UI and verify a new task starts immediately.
+
+**Acceptance Scenarios**:
+
+1. **Given** the Tasks tab is open, **When** I select an environment and click "Run Backup", **Then** a new backup task appears in the list with "Running" status.
+
+---
+
+### User Story 4 - Task History & Logs (Priority: P2)
+
+As an Administrator, I want to view the detailed logs of any executed task to troubleshoot failures or verify success.
+
+**Why this priority**: Essential for debugging and auditability.
+
+**Independent Test**: Click on a completed task and verify the log output is displayed.
+
+**Acceptance Scenarios**:
+
+1. **Given** a list of tasks, **When** I click on a "Failed" task, **Then** I can see the error logs explaining why it failed.
+2. **Given** a "Success" task, **When** I view logs, **Then** I see the execution steps confirmation.
+
+### Edge Cases
+
+- **Concurrent Backups**: What happens if a scheduled backup triggers while a manual backup is already running for the same environment? (System should likely queue or skip).
+- **Storage Full**: How does the system handle backup failures due to insufficient disk space? (Should fail gracefully and log error).
+- **Superset Offline**: What happens if the Superset environment is unreachable when a backup is triggered? (Task fails with connection error).
+
+### Assumptions
+
+- The backend server is running continuously to process scheduled tasks.
+- Users have configured valid credentials for Superset environments.
+- There is sufficient storage space for backup archives.
+
+## Requirements *(mandatory)*
+
+### Functional Requirements
+
+- **FR-001**: The system MUST allow users to configure a backup schedule using **Cron expressions** for each defined Superset environment via the **Settings** page.
+- **FR-002**: The system MUST persist schedule configurations nested within the Environment configuration in `config.json`.
+- **FR-003**: The system MUST include a `SchedulerService` running in a background thread that triggers backup tasks via the `TaskManager`.
+- **FR-004**: The system MUST provide a dedicated "Tasks" page in the frontend.
+- **FR-005**: The "Tasks" page MUST display a unified list of all `TaskManager` jobs, including Migrations and Backups.
+- **FR-006**: The "Tasks" page MUST allow users to filter tasks by status (Running, Success, Failed) and type.
+- **FR-007**: The system MUST allow users to manually trigger a backup for a specific environment from the "Tasks" page.
+- **FR-008**: All backup operations (scheduled or manual) MUST be executed as `TaskManager` tasks and generate standard logs.
+- **FR-009**: The system MUST retain a history of task executions in a dedicated SQLite database (`tasks.db`) for long-term review.
+- **FR-010**: The system MUST automatically clean up task history older than 30 days to prevent unbounded database growth.
+
+### Key Entities
+
+- **Task**: Represents a unit of work (Backup, Migration) managed by TaskManager. Attributes: ID, Type, Status, StartedAt, FinishedAt, Logs.
+- **Schedule**: Configuration for when to run a task. Attributes: EnvironmentID, Frequency, NextRunTime, Enabled.
+
+## Success Criteria *(mandatory)*
+
+### Measurable Outcomes
+
+- **SC-001**: Users can configure a backup schedule that persists and triggers automatically within 1 minute of the target time.
+- **SC-002**: The "Tasks" UI displays the status of running tasks with a latency of no more than 5 seconds.
+- **SC-003**: 100% of triggered backups (manual or scheduled) are recorded in the TaskManager history.
+- **SC-004**: Users can access logs for any task executed in the last 7 days (or configured retention period).
+
+## Clarifications
+
+### Session 2025-12-30
+- Q: Where should the backup schedule configuration UI be located? → A: In the **Settings** tab, inside each Environment's edit form.
+- Q: How should schedule configurations be persisted? → A: Add a `Schedule` model in `config_models.py` and nest it under `Environment`.
+- Q: What format should be used for defining schedule frequency? → A: Cron-style strings (e.g., "0 0 * * *").
+- Q: How should the scheduling mechanism be implemented? → A: Create a dedicated `SchedulerService` in `backend/src/core/scheduler.py` that runs in a background thread.
+- Q: Where should task history be stored for long-term retention? → A: Add a `tasks.db` SQLite database using SQLAlchemy.

specs/009-backup-scheduler/tasks.md

@@ -0,0 +1,51 @@
+# Tasks: Backup Scheduler & Unified Task UI
+
+## Phase 1: Setup
+- [ ] T001 Initialize SQLite database `tasks.db` and SQLAlchemy engine in `backend/src/core/database.py`
+- [ ] T002 Create SQLAlchemy model for `TaskRecord` in `backend/src/models/task.py`
+- [ ] T003 Update `backend/src/core/config_models.py` to include `Schedule` and update `Environment` model
+- [ ] T004 Create database migrations or initialization script for `tasks.db`
+
+## Phase 2: Foundational
+- [ ] T005 [P] Implement `TaskPersistence` layer in `backend/src/core/task_manager/persistence.py`
+- [ ] T006 Update `TaskManager` in `backend/src/core/task_manager/manager.py` to use persistence for all jobs
+- [ ] T007 Implement `SchedulerService` using `APScheduler` in `backend/src/core/scheduler.py`
+- [ ] T008 Integrate `SchedulerService` into main FastAPI application startup in `backend/src/app.py`
+
+## Phase 3: [US1] Scheduled Backups
+- [ ] T009 [US1] Implement schedule loading and registration logic in `SchedulerService`
+- [ ] T010 [US1] Update `Environment` settings API to handle `backup_schedule` updates in `backend/src/api/routes/environments.py`
+- [ ] T011 [P] [US1] Add schedule configuration fields to Environment edit form in `frontend/src/components/EnvSelector.svelte` (or appropriate component)
+- [ ] T012 [US1] Implement validation for Cron expressions in backend and frontend
+
+## Phase 4: [US2] Unified Task Management UI
+- [ ] T013 [US2] Implement `/api/tasks` endpoint to list and filter tasks in `backend/src/api/routes/tasks.py`
+- [ ] T014 [US2] Create new Tasks page in `frontend/src/routes/tasks/+page.svelte`
+- [ ] T015 [P] [US2] Implement `TaskList` component in `frontend/src/components/TaskList.svelte`
+- [ ] T016 [US2] Add "Tasks" link to main navigation in `frontend/src/components/Navbar.svelte`
+
+## Phase 5: [US3] Manual Backup Trigger
+- [ ] T017 [US3] Implement `/api/tasks/backup` POST endpoint in `backend/src/api/routes/tasks.py`
+- [ ] T018 [US3] Add "Run Backup" button and environment selection to Tasks page in `frontend/src/routes/tasks/+page.svelte`
+
+## Phase 6: [US4] Task History & Logs
+- [ ] T019 [US4] Implement `/api/tasks/{task_id}` GET endpoint for detailed task info and logs in `backend/src/api/routes/tasks.py`
+- [ ] T020 [US4] Implement `TaskLogViewer` component in `frontend/src/components/TaskLogViewer.svelte`
+- [ ] T021 [US4] Integrate log viewer into TaskList or as a separate modal/page
+
+## Final Phase: Polish & Cross-cutting concerns
+- [ ] T022 Implement task cleanup/retention policy (e.g., delete tasks older than 30 days)
+- [ ] T023 Add real-time updates for task status using WebSockets (optional/refinement)
+- [ ] T024 Ensure consistent error handling and logging across scheduler and task manager
+
+## Dependencies
+- US1 depends on Phase 1 & 2
+- US2 depends on Phase 1 & 2
+- US3 depends on US2
+- US4 depends on US2
+
+## Implementation Strategy
+1. **Infrastructure First**: Setup database and basic task persistence.
+2. **Backend Logic**: Implement scheduler and update task manager.
+3. **API & UI**: Build the unified tasks view.
+4. **Feature Integration**: Add scheduling UI and manual triggers.