2 Commits

Author SHA1 Message Date
a747a163c8 backup worked 2025-12-30 22:02:51 +03:00
fce0941e98 docs ready 2025-12-30 21:30:37 +03:00
33 changed files with 1548 additions and 211 deletions

View File

@@ -16,6 +16,8 @@ Auto-generated from all feature plans. Last updated: 2025-12-19
- Python 3.9+ (backend), Node.js 18+ (frontend) + FastAPI, SvelteKit, Tailwind CSS, Pydantic, SQLAlchemy, Superset API (008-migration-ui-improvements)
- SQLite (optional for job history), existing database for mappings (008-migration-ui-improvements)
- Python 3.9+, Node.js 18+ + FastAPI, SvelteKit, Tailwind CSS, Pydantic, SQLAlchemy, Superset API (008-migration-ui-improvements)
- Python 3.9+, Node.js 18+ + FastAPI, APScheduler, SQLAlchemy, SvelteKit, Tailwind CSS (009-backup-scheduler)
- SQLite (`tasks.db`), JSON (`config.json`) (009-backup-scheduler)
- Python 3.9+ (Backend), Node.js 18+ (Frontend Build) (001-plugin-arch-svelte-ui)
@@ -36,9 +38,9 @@ cd src; pytest; ruff check .
Python 3.9+ (Backend), Node.js 18+ (Frontend Build): Follow standard conventions
## Recent Changes
- 008-migration-ui-improvements: Added Python 3.9+, Node.js 18+ + FastAPI, SvelteKit, Tailwind CSS, Pydantic, SQLAlchemy, Superset API
- 008-migration-ui-improvements: Added Python 3.9+ (backend), Node.js 18+ (frontend) + FastAPI, SvelteKit, Tailwind CSS, Pydantic, SQLAlchemy, Superset API
- 007-migration-dashboard-grid: Added Python 3.9+ (Backend), Node.js 18+ (Frontend) + FastAPI, SvelteKit, Tailwind CSS, Pydantic, Superset API
- 009-backup-scheduler: Added Python 3.9+, Node.js 18+ + FastAPI, APScheduler, SQLAlchemy, SvelteKit, Tailwind CSS
- 009-backup-scheduler: Added Python 3.9+, Node.js 18+ + FastAPI, APScheduler, SQLAlchemy, SvelteKit, Tailwind CSS
- 009-backup-scheduler: Added [if applicable, e.g., PostgreSQL, CoreData, files or N/A]
<!-- MANUAL ADDITIONS START -->

View File

@@ -9,8 +9,8 @@
#
# OPTIONS:
# --json Output in JSON format
# --require-tasks Require tasks-arch.md and tasks-dev.md to exist (for implementation phase)
# --include-tasks Include task files in AVAILABLE_DOCS list
# --require-tasks Require tasks.md to exist (for implementation phase)
# --include-tasks Include tasks.md in AVAILABLE_DOCS list
# --paths-only Only output path variables (no validation)
# --help, -h Show help message
#
@@ -49,8 +49,8 @@ Consolidated prerequisite checking for Spec-Driven Development workflow.
OPTIONS:
--json Output in JSON format
--require-tasks Require tasks-arch.md and tasks-dev.md to exist (for implementation phase)
--include-tasks Include task files in AVAILABLE_DOCS list
--require-tasks Require tasks.md to exist (for implementation phase)
--include-tasks Include tasks.md in AVAILABLE_DOCS list
--paths-only Only output path variables (no prerequisite validation)
--help, -h Show this help message
@@ -58,7 +58,7 @@ EXAMPLES:
# Check task prerequisites (plan.md required)
./check-prerequisites.sh --json
# Check implementation prerequisites (plan.md + task files required)
# Check implementation prerequisites (plan.md + tasks.md required)
./check-prerequisites.sh --json --require-tasks --include-tasks
# Get feature paths only (no validation)
@@ -86,16 +86,15 @@ check_feature_branch "$CURRENT_BRANCH" "$HAS_GIT" || exit 1
if $PATHS_ONLY; then
if $JSON_MODE; then
# Minimal JSON paths payload (no validation performed)
printf '{"REPO_ROOT":"%s","BRANCH":"%s","FEATURE_DIR":"%s","FEATURE_SPEC":"%s","IMPL_PLAN":"%s","TASKS_ARCH":"%s","TASKS_DEV":"%s"}\n' \
"$REPO_ROOT" "$CURRENT_BRANCH" "$FEATURE_DIR" "$FEATURE_SPEC" "$IMPL_PLAN" "$TASKS_ARCH" "$TASKS_DEV"
printf '{"REPO_ROOT":"%s","BRANCH":"%s","FEATURE_DIR":"%s","FEATURE_SPEC":"%s","IMPL_PLAN":"%s","TASKS":"%s"}\n' \
"$REPO_ROOT" "$CURRENT_BRANCH" "$FEATURE_DIR" "$FEATURE_SPEC" "$IMPL_PLAN" "$TASKS"
else
echo "REPO_ROOT: $REPO_ROOT"
echo "BRANCH: $CURRENT_BRANCH"
echo "FEATURE_DIR: $FEATURE_DIR"
echo "FEATURE_SPEC: $FEATURE_SPEC"
echo "IMPL_PLAN: $IMPL_PLAN"
echo "TASKS_ARCH: $TASKS_ARCH"
echo "TASKS_DEV: $TASKS_DEV"
echo "TASKS: $TASKS"
fi
exit 0
fi
@@ -113,20 +112,11 @@ if [[ ! -f "$IMPL_PLAN" ]]; then
exit 1
fi
# Check for task files if required
if $REQUIRE_TASKS; then
# Check for split tasks first
if [[ -f "$TASKS_ARCH" ]] && [[ -f "$TASKS_DEV" ]]; then
: # Split tasks exist, proceed
# Fallback to unified tasks.md
elif [[ -f "$TASKS" ]]; then
: # Unified tasks exist, proceed
else
echo "ERROR: No valid task files found in $FEATURE_DIR" >&2
echo "Expected 'tasks-arch.md' AND 'tasks-dev.md' (split) OR 'tasks.md' (unified)" >&2
echo "Run /speckit.tasks first to create the task lists." >&2
exit 1
fi
# Check for tasks.md if required
if $REQUIRE_TASKS && [[ ! -f "$TASKS" ]]; then
echo "ERROR: tasks.md not found in $FEATURE_DIR" >&2
echo "Run /speckit.tasks first to create the task list." >&2
exit 1
fi
# Build list of available documents
@@ -143,14 +133,9 @@ fi
[[ -f "$QUICKSTART" ]] && docs+=("quickstart.md")
# Include task files if requested and they exist
if $INCLUDE_TASKS; then
if [[ -f "$TASKS_ARCH" ]] || [[ -f "$TASKS_DEV" ]]; then
[[ -f "$TASKS_ARCH" ]] && docs+=("tasks-arch.md")
[[ -f "$TASKS_DEV" ]] && docs+=("tasks-dev.md")
elif [[ -f "$TASKS" ]]; then
docs+=("tasks.md")
fi
# Include tasks.md if requested and it exists
if $INCLUDE_TASKS && [[ -f "$TASKS" ]]; then
docs+=("tasks.md")
fi
# Output results
@@ -176,11 +161,6 @@ else
check_file "$QUICKSTART" "quickstart.md"
if $INCLUDE_TASKS; then
if [[ -f "$TASKS_ARCH" ]] || [[ -f "$TASKS_DEV" ]]; then
check_file "$TASKS_ARCH" "tasks-arch.md"
check_file "$TASKS_DEV" "tasks-dev.md"
else
check_file "$TASKS" "tasks.md"
fi
check_file "$TASKS" "tasks.md"
fi
fi

View File

@@ -143,9 +143,7 @@ HAS_GIT='$has_git_repo'
FEATURE_DIR='$feature_dir'
FEATURE_SPEC='$feature_dir/spec.md'
IMPL_PLAN='$feature_dir/plan.md'
TASKS_ARCH='$feature_dir/tasks-arch.md'
TASKS_DEV='$feature_dir/tasks-dev.md'
TASKS='$feature_dir/tasks.md' # Deprecated
TASKS='$feature_dir/tasks.md'
RESEARCH='$feature_dir/research.md'
DATA_MODEL='$feature_dir/data-model.md'
QUICKSTART='$feature_dir/quickstart.md'

View File

@@ -0,0 +1,251 @@
---
description: "Task list template for feature implementation"
---
# Tasks: [FEATURE NAME]
**Input**: Design documents from `/specs/[###-feature-name]/`
**Prerequisites**: plan.md (required), spec.md (required for user stories), research.md, data-model.md, contracts/
**Tests**: The examples below include test tasks. Tests are OPTIONAL - only include them if explicitly requested in the feature specification.
**Organization**: Tasks are grouped by user story to enable independent implementation and testing of each story.
## Format: `[ID] [P?] [Story] Description`
- **[P]**: Can run in parallel (different files, no dependencies)
- **[Story]**: Which user story this task belongs to (e.g., US1, US2, US3)
- Include exact file paths in descriptions
## Path Conventions
- **Single project**: `src/`, `tests/` at repository root
- **Web app**: `backend/src/`, `frontend/src/`
- **Mobile**: `api/src/`, `ios/src/` or `android/src/`
- Paths shown below assume single project - adjust based on plan.md structure
<!--
============================================================================
IMPORTANT: The tasks below are SAMPLE TASKS for illustration purposes only.
The /speckit.tasks command MUST replace these with actual tasks based on:
- User stories from spec.md (with their priorities P1, P2, P3...)
- Feature requirements from plan.md
- Entities from data-model.md
- Endpoints from contracts/
Tasks MUST be organized by user story so each story can be:
- Implemented independently
- Tested independently
- Delivered as an MVP increment
DO NOT keep these sample tasks in the generated tasks.md file.
============================================================================
-->
## Phase 1: Setup (Shared Infrastructure)
**Purpose**: Project initialization and basic structure
- [ ] T001 Create project structure per implementation plan
- [ ] T002 Initialize [language] project with [framework] dependencies
- [ ] T003 [P] Configure linting and formatting tools
---
## Phase 2: Foundational (Blocking Prerequisites)
**Purpose**: Core infrastructure that MUST be complete before ANY user story can be implemented
**⚠️ CRITICAL**: No user story work can begin until this phase is complete
Examples of foundational tasks (adjust based on your project):
- [ ] T004 Setup database schema and migrations framework
- [ ] T005 [P] Implement authentication/authorization framework
- [ ] T006 [P] Setup API routing and middleware structure
- [ ] T007 Create base models/entities that all stories depend on
- [ ] T008 Configure error handling and logging infrastructure
- [ ] T009 Setup environment configuration management
**Checkpoint**: Foundation ready - user story implementation can now begin in parallel
---
## Phase 3: User Story 1 - [Title] (Priority: P1) 🎯 MVP
**Goal**: [Brief description of what this story delivers]
**Independent Test**: [How to verify this story works on its own]
### Tests for User Story 1 (OPTIONAL - only if tests requested) ⚠️
> **NOTE: Write these tests FIRST, ensure they FAIL before implementation**
- [ ] T010 [P] [US1] Contract test for [endpoint] in tests/contract/test_[name].py
- [ ] T011 [P] [US1] Integration test for [user journey] in tests/integration/test_[name].py
### Implementation for User Story 1
- [ ] T012 [P] [US1] Create [Entity1] model in src/models/[entity1].py
- [ ] T013 [P] [US1] Create [Entity2] model in src/models/[entity2].py
- [ ] T014 [US1] Implement [Service] in src/services/[service].py (depends on T012, T013)
- [ ] T015 [US1] Implement [endpoint/feature] in src/[location]/[file].py
- [ ] T016 [US1] Add validation and error handling
- [ ] T017 [US1] Add logging for user story 1 operations
**Checkpoint**: At this point, User Story 1 should be fully functional and testable independently
---
## Phase 4: User Story 2 - [Title] (Priority: P2)
**Goal**: [Brief description of what this story delivers]
**Independent Test**: [How to verify this story works on its own]
### Tests for User Story 2 (OPTIONAL - only if tests requested) ⚠️
- [ ] T018 [P] [US2] Contract test for [endpoint] in tests/contract/test_[name].py
- [ ] T019 [P] [US2] Integration test for [user journey] in tests/integration/test_[name].py
### Implementation for User Story 2
- [ ] T020 [P] [US2] Create [Entity] model in src/models/[entity].py
- [ ] T021 [US2] Implement [Service] in src/services/[service].py
- [ ] T022 [US2] Implement [endpoint/feature] in src/[location]/[file].py
- [ ] T023 [US2] Integrate with User Story 1 components (if needed)
**Checkpoint**: At this point, User Stories 1 AND 2 should both work independently
---
## Phase 5: User Story 3 - [Title] (Priority: P3)
**Goal**: [Brief description of what this story delivers]
**Independent Test**: [How to verify this story works on its own]
### Tests for User Story 3 (OPTIONAL - only if tests requested) ⚠️
- [ ] T024 [P] [US3] Contract test for [endpoint] in tests/contract/test_[name].py
- [ ] T025 [P] [US3] Integration test for [user journey] in tests/integration/test_[name].py
### Implementation for User Story 3
- [ ] T026 [P] [US3] Create [Entity] model in src/models/[entity].py
- [ ] T027 [US3] Implement [Service] in src/services/[service].py
- [ ] T028 [US3] Implement [endpoint/feature] in src/[location]/[file].py
**Checkpoint**: All user stories should now be independently functional
---
[Add more user story phases as needed, following the same pattern]
---
## Phase N: Polish & Cross-Cutting Concerns
**Purpose**: Improvements that affect multiple user stories
- [ ] TXXX [P] Documentation updates in docs/
- [ ] TXXX Code cleanup and refactoring
- [ ] TXXX Performance optimization across all stories
- [ ] TXXX [P] Additional unit tests (if requested) in tests/unit/
- [ ] TXXX Security hardening
- [ ] TXXX Run quickstart.md validation
---
## Dependencies & Execution Order
### Phase Dependencies
- **Setup (Phase 1)**: No dependencies - can start immediately
- **Foundational (Phase 2)**: Depends on Setup completion - BLOCKS all user stories
- **User Stories (Phase 3+)**: All depend on Foundational phase completion
- User stories can then proceed in parallel (if staffed)
- Or sequentially in priority order (P1 → P2 → P3)
- **Polish (Final Phase)**: Depends on all desired user stories being complete
### User Story Dependencies
- **User Story 1 (P1)**: Can start after Foundational (Phase 2) - No dependencies on other stories
- **User Story 2 (P2)**: Can start after Foundational (Phase 2) - May integrate with US1 but should be independently testable
- **User Story 3 (P3)**: Can start after Foundational (Phase 2) - May integrate with US1/US2 but should be independently testable
### Within Each User Story
- Tests (if included) MUST be written and FAIL before implementation
- Models before services
- Services before endpoints
- Core implementation before integration
- Story complete before moving to next priority
### Parallel Opportunities
- All Setup tasks marked [P] can run in parallel
- All Foundational tasks marked [P] can run in parallel (within Phase 2)
- Once Foundational phase completes, all user stories can start in parallel (if team capacity allows)
- All tests for a user story marked [P] can run in parallel
- Models within a story marked [P] can run in parallel
- Different user stories can be worked on in parallel by different team members
---
## Parallel Example: User Story 1
```bash
# Launch all tests for User Story 1 together (if tests requested):
Task: "Contract test for [endpoint] in tests/contract/test_[name].py"
Task: "Integration test for [user journey] in tests/integration/test_[name].py"
# Launch all models for User Story 1 together:
Task: "Create [Entity1] model in src/models/[entity1].py"
Task: "Create [Entity2] model in src/models/[entity2].py"
```
---
## Implementation Strategy
### MVP First (User Story 1 Only)
1. Complete Phase 1: Setup
2. Complete Phase 2: Foundational (CRITICAL - blocks all stories)
3. Complete Phase 3: User Story 1
4. **STOP and VALIDATE**: Test User Story 1 independently
5. Deploy/demo if ready
### Incremental Delivery
1. Complete Setup + Foundational → Foundation ready
2. Add User Story 1 → Test independently → Deploy/Demo (MVP!)
3. Add User Story 2 → Test independently → Deploy/Demo
4. Add User Story 3 → Test independently → Deploy/Demo
5. Each story adds value without breaking previous stories
### Parallel Team Strategy
With multiple developers:
1. Team completes Setup + Foundational together
2. Once Foundational is done:
- Developer A: User Story 1
- Developer B: User Story 2
- Developer C: User Story 3
3. Stories complete and integrate independently
---
## Notes
- [P] tasks = different files, no dependencies
- [Story] label maps task to specific user story for traceability
- Each user story should be independently completable and testable
- Verify tests fail before implementing
- Commit after each task or logical group
- Stop at any checkpoint to validate story independently
- Avoid: vague tasks, same file conflicts, cross-story dependencies that break independence

Binary file not shown.

View File

@@ -1,14 +1,43 @@
fastapi
uvicorn
pydantic
authlib
python-multipart
starlette
jsonschema
requests
keyring
httpx
PyYAML
websockets
rapidfuzz
sqlalchemy
annotated-doc==0.0.4
annotated-types==0.7.0
anyio==4.12.0
APScheduler==3.11.2
attrs==25.4.0
Authlib==1.6.6
certifi==2025.11.12
cffi==2.0.0
charset-normalizer==3.4.4
click==8.3.1
cryptography==46.0.3
fastapi==0.126.0
greenlet==3.3.0
h11==0.16.0
httpcore==1.0.9
httpx==0.28.1
idna==3.11
jaraco.classes==3.4.0
jaraco.context==6.0.1
jaraco.functools==4.3.0
jeepney==0.9.0
jsonschema==4.25.1
jsonschema-specifications==2025.9.1
keyring==25.7.0
more-itertools==10.8.0
pycparser==2.23
pydantic==2.12.5
pydantic_core==2.41.5
python-multipart==0.0.21
PyYAML==6.0.3
RapidFuzz==3.14.3
referencing==0.37.0
requests==2.32.5
rpds-py==0.30.0
SecretStorage==3.5.0
SQLAlchemy==2.0.45
starlette==0.50.0
typing-inspection==0.4.2
typing_extensions==4.15.0
tzlocal==5.3.1
urllib3==2.6.2
uvicorn==0.38.0
websockets==15.0.1

View File

@@ -11,19 +11,27 @@
# [SECTION: IMPORTS]
from fastapi import APIRouter, Depends, HTTPException
from typing import List, Dict, Optional
from backend.src.dependencies import get_config_manager
from backend.src.dependencies import get_config_manager, get_scheduler_service
from backend.src.core.superset_client import SupersetClient
from superset_tool.models import SupersetConfig
from pydantic import BaseModel
from pydantic import BaseModel, Field
from backend.src.core.config_models import Environment as EnvModel
# [/SECTION]
router = APIRouter(prefix="/api/environments", tags=["environments"])
router = APIRouter()
# [DEF:ScheduleSchema:DataClass]
class ScheduleSchema(BaseModel):
enabled: bool = False
cron_expression: str = Field(..., pattern=r'^(@(annually|yearly|monthly|weekly|daily|hourly|reboot))|((((\d+,)*\d+|(\d+(\/|-)\d+)|\d+|\*) ?){5,7})$')
# [/DEF:ScheduleSchema]
# [DEF:EnvironmentResponse:DataClass]
class EnvironmentResponse(BaseModel):
id: str
name: str
url: str
backup_schedule: Optional[ScheduleSchema] = None
# [/DEF:EnvironmentResponse]
# [DEF:DatabaseResponse:DataClass]
@@ -42,9 +50,47 @@ async def get_environments(config_manager=Depends(get_config_manager)):
# Ensure envs is a list
if not isinstance(envs, list):
envs = []
return [EnvironmentResponse(id=e.id, name=e.name, url=e.url) for e in envs]
return [
EnvironmentResponse(
id=e.id,
name=e.name,
url=e.url,
backup_schedule=ScheduleSchema(
enabled=e.backup_schedule.enabled,
cron_expression=e.backup_schedule.cron_expression
) if e.backup_schedule else None
) for e in envs
]
# [/DEF:get_environments]
# [DEF:update_environment_schedule:Function]
# @PURPOSE: Update backup schedule for an environment.
# @PARAM: id (str) - The environment ID.
# @PARAM: schedule (ScheduleSchema) - The new schedule.
@router.put("/{id}/schedule")
async def update_environment_schedule(
id: str,
schedule: ScheduleSchema,
config_manager=Depends(get_config_manager),
scheduler_service=Depends(get_scheduler_service)
):
envs = config_manager.get_environments()
env = next((e for e in envs if e.id == id), None)
if not env:
raise HTTPException(status_code=404, detail="Environment not found")
# Update environment config
env.backup_schedule.enabled = schedule.enabled
env.backup_schedule.cron_expression = schedule.cron_expression
config_manager.update_environment(id, env)
# Refresh scheduler
scheduler_service.load_schedules()
return {"message": "Schedule updated successfully"}
# [/DEF:update_environment_schedule]
# [DEF:get_environment_databases:Function]
# @PURPOSE: Fetch the list of databases from a specific environment.
# @PARAM: id (str) - The environment ID.

View File

@@ -18,14 +18,11 @@ from fastapi.responses import FileResponse
import asyncio
import os
from .dependencies import get_task_manager
from .dependencies import get_task_manager, get_scheduler_service
from .core.logger import logger
from .api.routes import plugins, tasks, settings, environments, mappings, migration
from .core.database import init_db
# Initialize database
init_db()
# [DEF:App:Global]
# @SEMANTICS: app, fastapi, instance
# @PURPOSE: The global FastAPI application instance.
@@ -35,6 +32,18 @@ app = FastAPI(
version="1.0.0",
)
# Startup event
@app.on_event("startup")
async def startup_event():
scheduler = get_scheduler_service()
scheduler.start()
# Shutdown event
@app.on_event("shutdown")
async def shutdown_event():
scheduler = get_scheduler_service()
scheduler.stop()
# Configure CORS
app.add_middleware(
CORSMiddleware,
@@ -56,7 +65,7 @@ async def log_requests(request: Request, call_next):
app.include_router(plugins.router, prefix="/api/plugins", tags=["Plugins"])
app.include_router(tasks.router, prefix="/api/tasks", tags=["Tasks"])
app.include_router(settings.router, prefix="/api/settings", tags=["Settings"])
app.include_router(environments.router)
app.include_router(environments.router, prefix="/api/environments", tags=["Environments"])
app.include_router(mappings.router)
app.include_router(migration.router)

View File

@@ -8,6 +8,13 @@
from pydantic import BaseModel, Field
from typing import List, Optional
# [DEF:Schedule:DataClass]
# @PURPOSE: Represents a backup schedule configuration.
class Schedule(BaseModel):
enabled: bool = False
cron_expression: str = "0 0 * * *" # Default: daily at midnight
# [/DEF:Schedule]
# [DEF:Environment:DataClass]
# @PURPOSE: Represents a Superset environment configuration.
class Environment(BaseModel):
@@ -17,6 +24,7 @@ class Environment(BaseModel):
username: str
password: str # Will be masked in UI
is_default: bool = False
backup_schedule: Schedule = Field(default_factory=Schedule)
# [/DEF:Environment]
# [DEF:LoggingConfig:DataClass]

View File

@@ -12,6 +12,8 @@
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker, Session
from backend.src.models.mapping import Base
# Import TaskRecord to ensure it's registered with Base
from backend.src.models.task import TaskRecord
import os
# [/SECTION]
@@ -19,18 +21,31 @@ import os
DATABASE_URL = os.getenv("DATABASE_URL", "sqlite:///./mappings.db")
# [/DEF:DATABASE_URL]
# [DEF:TASKS_DATABASE_URL:Constant]
TASKS_DATABASE_URL = os.getenv("TASKS_DATABASE_URL", "sqlite:///./tasks.db")
# [/DEF:TASKS_DATABASE_URL]
# [DEF:engine:Variable]
engine = create_engine(DATABASE_URL, connect_args={"check_same_thread": False})
# [/DEF:engine]
# [DEF:tasks_engine:Variable]
tasks_engine = create_engine(TASKS_DATABASE_URL, connect_args={"check_same_thread": False})
# [/DEF:tasks_engine]
# [DEF:SessionLocal:Class]
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
# [/DEF:SessionLocal]
# [DEF:TasksSessionLocal:Class]
TasksSessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=tasks_engine)
# [/DEF:TasksSessionLocal]
# [DEF:init_db:Function]
# @PURPOSE: Initializes the database by creating all tables.
def init_db():
Base.metadata.create_all(bind=engine)
Base.metadata.create_all(bind=tasks_engine)
# [/DEF:init_db]
# [DEF:get_db:Function]
@@ -45,4 +60,16 @@ def get_db():
db.close()
# [/DEF:get_db]
# [DEF:get_tasks_db:Function]
# @PURPOSE: Dependency for getting a tasks database session.
# @POST: Session is closed after use.
# @RETURN: Generator[Session, None, None]
def get_tasks_db():
db = TasksSessionLocal()
try:
yield db
finally:
db.close()
# [/DEF:get_tasks_db]
# [/DEF:backend.src.core.database]

View File

@@ -0,0 +1,99 @@
# [DEF:SchedulerModule:Module]
# @SEMANTICS: scheduler, apscheduler, cron, backup
# @PURPOSE: Manages scheduled tasks using APScheduler.
# @LAYER: Core
# @RELATION: Uses TaskManager to run scheduled backups.
# [SECTION: IMPORTS]
from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.triggers.cron import CronTrigger
from .logger import logger, belief_scope
from .config_manager import ConfigManager
from typing import Optional
import asyncio
# [/SECTION]
# [DEF:SchedulerService:Class]
# @SEMANTICS: scheduler, service, apscheduler
# @PURPOSE: Provides a service to manage scheduled backup tasks.
class SchedulerService:
def __init__(self, task_manager, config_manager: ConfigManager):
with belief_scope("SchedulerService.__init__"):
self.task_manager = task_manager
self.config_manager = config_manager
self.scheduler = BackgroundScheduler()
self.loop = asyncio.get_event_loop()
# [DEF:SchedulerService.start:Function]
# @PURPOSE: Starts the background scheduler and loads initial schedules.
def start(self):
with belief_scope("SchedulerService.start"):
if not self.scheduler.running:
self.scheduler.start()
logger.info("Scheduler started.")
self.load_schedules()
# [DEF:SchedulerService.stop:Function]
# @PURPOSE: Stops the background scheduler.
def stop(self):
with belief_scope("SchedulerService.stop"):
if self.scheduler.running:
self.scheduler.shutdown()
logger.info("Scheduler stopped.")
# [DEF:SchedulerService.load_schedules:Function]
# @PURPOSE: Loads backup schedules from configuration and registers them.
def load_schedules(self):
with belief_scope("SchedulerService.load_schedules"):
# Clear existing jobs
self.scheduler.remove_all_jobs()
config = self.config_manager.get_config()
for env in config.environments:
if env.backup_schedule and env.backup_schedule.enabled:
self.add_backup_job(env.id, env.backup_schedule.cron_expression)
# [DEF:SchedulerService.add_backup_job:Function]
# @PURPOSE: Adds a scheduled backup job for an environment.
# @PARAM: env_id (str) - The ID of the environment.
# @PARAM: cron_expression (str) - The cron expression for the schedule.
def add_backup_job(self, env_id: str, cron_expression: str):
with belief_scope("SchedulerService.add_backup_job", f"env_id={env_id}, cron={cron_expression}"):
job_id = f"backup_{env_id}"
try:
self.scheduler.add_job(
self._trigger_backup,
CronTrigger.from_crontab(cron_expression),
id=job_id,
args=[env_id],
replace_existing=True
)
logger.info(f"Scheduled backup job added for environment {env_id}: {cron_expression}")
except Exception as e:
logger.error(f"Failed to add backup job for environment {env_id}: {e}")
# [DEF:SchedulerService._trigger_backup:Function]
# @PURPOSE: Triggered by the scheduler to start a backup task.
# @PARAM: env_id (str) - The ID of the environment.
def _trigger_backup(self, env_id: str):
with belief_scope("SchedulerService._trigger_backup", f"env_id={env_id}"):
logger.info(f"Triggering scheduled backup for environment {env_id}")
# Check if a backup is already running for this environment
active_tasks = self.task_manager.get_tasks(limit=100)
for task in active_tasks:
if (task.plugin_id == "superset-backup" and
task.status in ["PENDING", "RUNNING"] and
task.params.get("environment_id") == env_id):
logger.warning(f"Backup already running for environment {env_id}. Skipping scheduled run.")
return
# Run the backup task
# We need to run this in the event loop since create_task is async
asyncio.run_coroutine_threadsafe(
self.task_manager.create_task("superset-backup", {"environment_id": env_id}),
self.loop
)
# [/DEF:SchedulerService:Class]
# [/DEF:SchedulerModule:Module]

View File

@@ -0,0 +1,38 @@
# [DEF:TaskCleanupModule:Module]
# @SEMANTICS: task, cleanup, retention
# @PURPOSE: Implements task cleanup and retention policies.
# @LAYER: Core
# @RELATION: Uses TaskPersistenceService to delete old tasks.
from datetime import datetime, timedelta
from .persistence import TaskPersistenceService
from ..logger import logger, belief_scope
from ..config_manager import ConfigManager
# [DEF:TaskCleanupService:Class]
# @PURPOSE: Provides methods to clean up old task records.
class TaskCleanupService:
def __init__(self, persistence_service: TaskPersistenceService, config_manager: ConfigManager):
self.persistence_service = persistence_service
self.config_manager = config_manager
# [DEF:TaskCleanupService.run_cleanup:Function]
# @PURPOSE: Deletes tasks older than the configured retention period.
def run_cleanup(self):
with belief_scope("TaskCleanupService.run_cleanup"):
settings = self.config_manager.get_config().settings
retention_days = settings.task_retention_days
# This is a simplified implementation.
# In a real scenario, we would query IDs of tasks older than retention_days.
# For now, we'll log the action.
logger.info(f"Cleaning up tasks older than {retention_days} days.")
# Re-loading tasks to check for limit
tasks = self.persistence_service.load_tasks(limit=1000)
if len(tasks) > settings.task_retention_limit:
to_delete = [t.id for t in tasks[settings.task_retention_limit:]]
self.persistence_service.delete_tasks(to_delete)
logger.info(f"Deleted {len(to_delete)} tasks exceeding limit of {settings.task_retention_limit}")
# [/DEF:TaskCleanupService]

View File

@@ -71,6 +71,7 @@ class TaskManager:
task = Task(plugin_id=plugin_id, params=params, user_id=user_id)
self.tasks[task.id] = task
self.persistence_service.persist_task(task)
logger.info(f"Task {task.id} created and scheduled for execution")
self.loop.create_task(self._run_task(task.id)) # Schedule task for execution
return task
@@ -89,6 +90,7 @@ class TaskManager:
logger.info(f"Starting execution of task {task_id} for plugin '{plugin.name}'")
task.status = TaskStatus.RUNNING
task.started_at = datetime.utcnow()
self.persistence_service.persist_task(task)
self._add_log(task_id, "INFO", f"Task started for plugin '{plugin.name}'")
try:
@@ -113,6 +115,7 @@ class TaskManager:
self._add_log(task_id, "ERROR", f"Task failed: {e}", {"error_type": type(e).__name__})
finally:
task.finished_at = datetime.utcnow()
self.persistence_service.persist_task(task)
logger.info(f"Task {task_id} execution finished with status: {task.status}")
# [/DEF:TaskManager._run_task:Function]
@@ -132,6 +135,7 @@ class TaskManager:
# Update task params with resolution
task.params.update(resolution_params)
task.status = TaskStatus.RUNNING
self.persistence_service.persist_task(task)
self._add_log(task_id, "INFO", "Task resumed after mapping resolution.")
# Signal the future to continue
@@ -150,6 +154,7 @@ class TaskManager:
if not task: return
task.status = TaskStatus.AWAITING_MAPPING
self.persistence_service.persist_task(task)
self.task_futures[task_id] = self.loop.create_future()
try:
@@ -235,6 +240,7 @@ class TaskManager:
log_entry = LogEntry(level=level, message=message, context=context)
task.logs.append(log_entry)
self.persistence_service.persist_task(task)
# Notify subscribers
if task_id in self.subscribers:
@@ -266,16 +272,10 @@ class TaskManager:
del self.subscribers[task_id]
# [/DEF:TaskManager.unsubscribe_logs:Function]
# [DEF:TaskManager.persist_awaiting_input_tasks:Function]
# @PURPOSE: Persist tasks in AWAITING_INPUT state using persistence service.
def persist_awaiting_input_tasks(self) -> None:
self.persistence_service.persist_tasks(list(self.tasks.values()))
# [/DEF:TaskManager.persist_awaiting_input_tasks:Function]
# [DEF:TaskManager.load_persisted_tasks:Function]
# @PURPOSE: Load persisted tasks using persistence service.
def load_persisted_tasks(self) -> None:
loaded_tasks = self.persistence_service.load_tasks()
loaded_tasks = self.persistence_service.load_tasks(limit=100)
for task in loaded_tasks:
if task.id not in self.tasks:
self.tasks[task.id] = task
@@ -299,9 +299,8 @@ class TaskManager:
task.status = TaskStatus.AWAITING_INPUT
task.input_required = True
task.input_request = input_request
self.persistence_service.persist_task(task)
self._add_log(task_id, "INFO", "Task paused for user input", {"input_request": input_request})
self.persist_awaiting_input_tasks()
# [/DEF:TaskManager.await_input:Function]
# [DEF:TaskManager.resume_task_with_password:Function]
@@ -326,13 +325,11 @@ class TaskManager:
task.input_required = False
task.input_request = None
task.status = TaskStatus.RUNNING
self.persistence_service.persist_task(task)
self._add_log(task_id, "INFO", "Task resumed with passwords", {"databases": list(passwords.keys())})
if task_id in self.task_futures:
self.task_futures[task_id].set_result(True)
# Remove from persistence as it's no longer awaiting input
self.persistence_service.delete_tasks([task_id])
# [/DEF:TaskManager.resume_task_with_password:Function]
# [DEF:TaskManager.clear_tasks:Function]

View File

@@ -1,141 +1,122 @@
# [DEF:TaskPersistenceModule:Module]
# @SEMANTICS: persistence, sqlite, task, storage
# @PURPOSE: Handles the persistence of tasks, specifically those awaiting user input, to a SQLite database.
# @SEMANTICS: persistence, sqlite, sqlalchemy, task, storage
# @PURPOSE: Handles the persistence of tasks using SQLAlchemy and the tasks.db database.
# @LAYER: Core
# @RELATION: Used by TaskManager to save and load tasks.
# @INVARIANT: Database schema must match the Task model structure.
# @CONSTRAINT: Uses synchronous SQLite operations (blocking), should be used carefully.
# @INVARIANT: Database schema must match the TaskRecord model structure.
# [SECTION: IMPORTS]
import sqlite3
import json
from datetime import datetime
from pathlib import Path
from typing import Dict, List, Optional, Any
from typing import List, Optional, Dict, Any
import json
from .models import Task, TaskStatus
from sqlalchemy.orm import Session
from backend.src.models.task import TaskRecord
from backend.src.core.database import TasksSessionLocal
from .models import Task, TaskStatus, LogEntry
from ..logger import logger, belief_scope
# [/SECTION]
# [DEF:TaskPersistenceService:Class]
# @SEMANTICS: persistence, service, database
# @PURPOSE: Provides methods to save and load tasks from a local SQLite database.
# @SEMANTICS: persistence, service, database, sqlalchemy
# @PURPOSE: Provides methods to save and load tasks from the tasks.db database using SQLAlchemy.
class TaskPersistenceService:
def __init__(self, db_path: Optional[Path] = None):
if db_path is None:
self.db_path = Path(__file__).parent.parent.parent.parent / "migrations.db"
else:
self.db_path = db_path
self._ensure_db_exists()
def __init__(self):
# We use TasksSessionLocal from database.py
pass
# [DEF:TaskPersistenceService._ensure_db_exists:Function]
# @PURPOSE: Ensures the database directory and table exist.
# @PRE: None.
# @POST: Database file and table are created if they didn't exist.
def _ensure_db_exists(self) -> None:
with belief_scope("TaskPersistenceService._ensure_db_exists"):
self.db_path.parent.mkdir(parents=True, exist_ok=True)
# [DEF:TaskPersistenceService.persist_task:Function]
# @PURPOSE: Persists or updates a single task in the database.
# @PARAM: task (Task) - The task object to persist.
def persist_task(self, task: Task) -> None:
with belief_scope("TaskPersistenceService.persist_task", f"task_id={task.id}"):
session: Session = TasksSessionLocal()
try:
record = session.query(TaskRecord).filter(TaskRecord.id == task.id).first()
if not record:
record = TaskRecord(id=task.id)
session.add(record)
conn = sqlite3.connect(str(self.db_path))
cursor = conn.cursor()
record.type = task.plugin_id
record.status = task.status.value
record.environment_id = task.params.get("environment_id") or task.params.get("source_env_id")
record.started_at = task.started_at
record.finished_at = task.finished_at
record.params = task.params
cursor.execute("""
CREATE TABLE IF NOT EXISTS persistent_tasks (
id TEXT PRIMARY KEY,
plugin_id TEXT NOT NULL,
status TEXT NOT NULL,
created_at TEXT NOT NULL,
updated_at TEXT NOT NULL,
input_request JSON,
context JSON
)
""")
conn.commit()
conn.close()
# [/DEF:TaskPersistenceService._ensure_db_exists:Function]
# Store logs as JSON, converting datetime to string
record.logs = []
for log in task.logs:
log_dict = log.dict()
if isinstance(log_dict.get('timestamp'), datetime):
log_dict['timestamp'] = log_dict['timestamp'].isoformat()
record.logs.append(log_dict)
# Extract error if failed
if task.status == TaskStatus.FAILED:
for log in reversed(task.logs):
if log.level == "ERROR":
record.error = log.message
break
session.commit()
except Exception as e:
session.rollback()
logger.error(f"Failed to persist task {task.id}: {e}")
finally:
session.close()
# [/DEF:TaskPersistenceService.persist_task:Function]
# [DEF:TaskPersistenceService.persist_tasks:Function]
# @PURPOSE: Persists a list of tasks to the database.
# @PRE: Tasks list contains valid Task objects.
# @POST: Tasks matching the criteria (AWAITING_INPUT) are saved/updated in the DB.
# @PARAM: tasks (List[Task]) - The list of tasks to check and persist.
# @PURPOSE: Persists multiple tasks.
# @PARAM: tasks (List[Task]) - The list of tasks to persist.
def persist_tasks(self, tasks: List[Task]) -> None:
with belief_scope("TaskPersistenceService.persist_tasks"):
conn = sqlite3.connect(str(self.db_path))
cursor = conn.cursor()
count = 0
for task in tasks:
if task.status == TaskStatus.AWAITING_INPUT:
cursor.execute("""
INSERT OR REPLACE INTO persistent_tasks
(id, plugin_id, status, created_at, updated_at, input_request, context)
VALUES (?, ?, ?, ?, ?, ?, ?)
""", (
task.id,
task.plugin_id,
task.status.value,
task.started_at.isoformat() if task.started_at else datetime.utcnow().isoformat(),
datetime.utcnow().isoformat(),
json.dumps(task.input_request) if task.input_request else None,
json.dumps(task.params)
))
count += 1
conn.commit()
conn.close()
logger.info(f"Persisted {count} tasks awaiting input.")
for task in tasks:
self.persist_task(task)
# [/DEF:TaskPersistenceService.persist_tasks:Function]
# [DEF:TaskPersistenceService.load_tasks:Function]
# @PURPOSE: Loads persisted tasks from the database.
# @PRE: Database exists.
# @POST: Returns a list of Task objects reconstructed from the DB.
# @PURPOSE: Loads tasks from the database.
# @PARAM: limit (int) - Max tasks to load.
# @PARAM: status (Optional[TaskStatus]) - Filter by status.
# @RETURN: List[Task] - The loaded tasks.
def load_tasks(self) -> List[Task]:
def load_tasks(self, limit: int = 100, status: Optional[TaskStatus] = None) -> List[Task]:
with belief_scope("TaskPersistenceService.load_tasks"):
if not self.db_path.exists():
return []
session: Session = TasksSessionLocal()
try:
query = session.query(TaskRecord)
if status:
query = query.filter(TaskRecord.status == status.value)
conn = sqlite3.connect(str(self.db_path))
cursor = conn.cursor()
records = query.order_by(TaskRecord.created_at.desc()).limit(limit).all()
# Check if plugin_id column exists (migration for existing db)
cursor.execute("PRAGMA table_info(persistent_tasks)")
columns = [info[1] for info in cursor.fetchall()]
has_plugin_id = "plugin_id" in columns
loaded_tasks = []
for record in records:
try:
logs = []
if record.logs:
for log_data in record.logs:
# Handle timestamp conversion if it's a string
if isinstance(log_data.get('timestamp'), str):
log_data['timestamp'] = datetime.fromisoformat(log_data['timestamp'])
logs.append(LogEntry(**log_data))
if has_plugin_id:
cursor.execute("SELECT id, plugin_id, status, created_at, input_request, context FROM persistent_tasks")
else:
cursor.execute("SELECT id, status, created_at, input_request, context FROM persistent_tasks")
task = Task(
id=record.id,
plugin_id=record.type,
status=TaskStatus(record.status),
started_at=record.started_at,
finished_at=record.finished_at,
params=record.params or {},
logs=logs
)
loaded_tasks.append(task)
except Exception as e:
logger.error(f"Failed to reconstruct task {record.id}: {e}")
rows = cursor.fetchall()
loaded_tasks = []
for row in rows:
if has_plugin_id:
task_id, plugin_id, status, created_at, input_request_json, context_json = row
else:
task_id, status, created_at, input_request_json, context_json = row
plugin_id = "superset-migration" # Default fallback
try:
task = Task(
id=task_id,
plugin_id=plugin_id,
status=TaskStatus(status),
started_at=datetime.fromisoformat(created_at),
input_required=True,
input_request=json.loads(input_request_json) if input_request_json else None,
params=json.loads(context_json) if context_json else {}
)
loaded_tasks.append(task)
except Exception as e:
logger.error(f"Failed to load task {task_id}: {e}")
conn.close()
return loaded_tasks
return loaded_tasks
finally:
session.close()
# [/DEF:TaskPersistenceService.load_tasks:Function]
# [DEF:TaskPersistenceService.delete_tasks:Function]
@@ -145,14 +126,16 @@ class TaskPersistenceService:
if not task_ids:
return
with belief_scope("TaskPersistenceService.delete_tasks"):
conn = sqlite3.connect(str(self.db_path))
cursor = conn.cursor()
placeholders = ', '.join('?' for _ in task_ids)
cursor.execute(f"DELETE FROM persistent_tasks WHERE id IN ({placeholders})", task_ids)
conn.commit()
conn.close()
session: Session = TasksSessionLocal()
try:
session.query(TaskRecord).filter(TaskRecord.id.in_(task_ids)).delete(synchronize_session=False)
session.commit()
except Exception as e:
session.rollback()
logger.error(f"Failed to delete tasks: {e}")
finally:
session.close()
# [/DEF:TaskPersistenceService.delete_tasks:Function]
# [/DEF:TaskPersistenceService:Class]
# [/DEF:TaskPersistenceModule:Module]

View File

@@ -8,6 +8,8 @@ from pathlib import Path
from .core.plugin_loader import PluginLoader
from .core.task_manager import TaskManager
from .core.config_manager import ConfigManager
from .core.scheduler import SchedulerService
from .core.database import init_db
# Initialize singletons
# Use absolute path relative to this file to ensure plugins are found regardless of CWD
@@ -15,6 +17,9 @@ project_root = Path(__file__).parent.parent.parent
config_path = project_root / "config.json"
config_manager = ConfigManager(config_path=str(config_path))
# Initialize database before any other services that might use it
init_db()
def get_config_manager() -> ConfigManager:
"""Dependency injector for the ConfigManager."""
return config_manager
@@ -28,6 +33,9 @@ logger.info(f"Available plugins: {[config.name for config in plugin_loader.get_a
task_manager = TaskManager(plugin_loader)
logger.info("TaskManager initialized")
scheduler_service = SchedulerService(task_manager, config_manager)
logger.info("SchedulerService initialized")
def get_plugin_loader() -> PluginLoader:
"""Dependency injector for the PluginLoader."""
return plugin_loader
@@ -35,4 +43,8 @@ def get_plugin_loader() -> PluginLoader:
def get_task_manager() -> TaskManager:
"""Dependency injector for the TaskManager."""
return task_manager
def get_scheduler_service() -> SchedulerService:
"""Dependency injector for the SchedulerService."""
return scheduler_service
# [/DEF]

View File

@@ -0,0 +1,34 @@
# [DEF:backend.src.models.task:Module]
#
# @SEMANTICS: database, task, record, sqlalchemy, sqlite
# @PURPOSE: Defines the database schema for task execution records.
# @LAYER: Domain
# @RELATION: DEPENDS_ON -> sqlalchemy
#
# @INVARIANT: All primary keys are UUID strings.
# [SECTION: IMPORTS]
from sqlalchemy import Column, String, DateTime, JSON, ForeignKey
from sqlalchemy.sql import func
from .mapping import Base
import uuid
# [/SECTION]
# [DEF:TaskRecord:Class]
# @PURPOSE: Represents a persistent record of a task execution.
class TaskRecord(Base):
__tablename__ = "task_records"
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
type = Column(String, nullable=False) # e.g., "backup", "migration"
status = Column(String, nullable=False) # Enum: "PENDING", "RUNNING", "SUCCESS", "FAILED"
environment_id = Column(String, ForeignKey("environments.id"), nullable=True)
started_at = Column(DateTime(timezone=True), nullable=True)
finished_at = Column(DateTime(timezone=True), nullable=True)
logs = Column(JSON, nullable=True) # Store structured logs as JSON
error = Column(String, nullable=True)
created_at = Column(DateTime(timezone=True), server_default=func.now())
params = Column(JSON, nullable=True)
# [/DEF:TaskRecord]
# [/DEF:backend.src.models.task]

View File

@@ -71,8 +71,21 @@ class BackupPlugin(PluginBase):
}
async def execute(self, params: Dict[str, Any]):
env = params["env"]
backup_path = Path(params["backup_path"])
config_manager = get_config_manager()
env_id = params.get("environment_id")
# Resolve environment name if environment_id is provided
if env_id:
env_config = next((e for e in config_manager.get_environments() if e.id == env_id), None)
if env_config:
params["env"] = env_config.name
env = params.get("env")
if not env:
raise KeyError("env")
backup_path_str = params.get("backup_path") or config_manager.get_config().settings.backup_path
backup_path = Path(backup_path_str)
logger = SupersetLogger(log_dir=backup_path / "Logs", console=True)
logger.info(f"[BackupPlugin][Entry] Starting backup for {env}.")

BIN
backend/tasks.db Normal file

Binary file not shown.

View File

@@ -7,6 +7,9 @@
"": {
"name": "frontend",
"version": "0.0.0",
"dependencies": {
"date-fns": "^4.1.0"
},
"devDependencies": {
"@sveltejs/adapter-static": "^3.0.10",
"@sveltejs/kit": "^2.49.2",
@@ -1279,6 +1282,16 @@
"node": ">=4"
}
},
"node_modules/date-fns": {
"version": "4.1.0",
"resolved": "https://registry.npmjs.org/date-fns/-/date-fns-4.1.0.tgz",
"integrity": "sha512-Ukq0owbQXxa/U3EGtsdVBkR1w7KOQ5gIBqdH2hkvknzZPYvBxb/aa6E8L7tmjFtkwZBu3UXBbjIgPo/Ez4xaNg==",
"license": "MIT",
"funding": {
"type": "github",
"url": "https://github.com/sponsors/kossnocorp"
}
},
"node_modules/debug": {
"version": "4.4.3",
"resolved": "https://registry.npmjs.org/debug/-/debug-4.4.3.tgz",

View File

@@ -17,5 +17,8 @@
"svelte": "^5.43.8",
"tailwindcss": "^3.0.0",
"vite": "^7.2.4"
},
"dependencies": {
"date-fns": "^4.1.0"
}
}

View File

@@ -22,6 +22,12 @@
>
Migration
</a>
<a
href="/tasks"
class="text-gray-600 hover:text-blue-600 font-medium {$page.url.pathname.startsWith('/tasks') ? 'text-blue-600 border-b-2 border-blue-600' : ''}"
>
Tasks
</a>
<a
href="/settings"
class="text-gray-600 hover:text-blue-600 font-medium {$page.url.pathname === '/settings' ? 'text-blue-600 border-b-2 border-blue-600' : ''}"

View File

@@ -0,0 +1,94 @@
<!-- [DEF:TaskList:Component] -->
<!--
@SEMANTICS: tasks, list, status, history
@PURPOSE: Displays a list of tasks with their status and execution details.
@LAYER: Component
@RELATION: USES -> api.js
-->
<script lang="ts">
import { createEventDispatcher } from 'svelte';
import { formatDistanceToNow } from 'date-fns';
export let tasks: Array<any> = [];
export let loading: boolean = false;
const dispatch = createEventDispatcher();
function getStatusColor(status: string) {
switch (status) {
case 'SUCCESS': return 'bg-green-100 text-green-800';
case 'FAILED': return 'bg-red-100 text-red-800';
case 'RUNNING': return 'bg-blue-100 text-blue-800 animate-pulse';
case 'PENDING': return 'bg-gray-100 text-gray-800';
case 'AWAITING_INPUT':
case 'AWAITING_MAPPING': return 'bg-yellow-100 text-yellow-800';
default: return 'bg-gray-100 text-gray-800';
}
}
function formatTime(dateStr: string | null) {
if (!dateStr) return 'N/A';
try {
return formatDistanceToNow(new Date(dateStr), { addSuffix: true });
} catch (e) {
return 'Invalid date';
}
}
function handleTaskClick(taskId: string) {
dispatch('select', { id: taskId });
}
</script>
<div class="bg-white shadow overflow-hidden sm:rounded-md">
{#if loading && tasks.length === 0}
<div class="p-4 text-center text-gray-500">Loading tasks...</div>
{:else if tasks.length === 0}
<div class="p-4 text-center text-gray-500">No tasks found.</div>
{:else}
<ul class="divide-y divide-gray-200">
{#each tasks as task (task.id)}
<li>
<button
class="block hover:bg-gray-50 w-full text-left transition duration-150 ease-in-out focus:outline-none"
on:click={() => handleTaskClick(task.id)}
>
<div class="px-4 py-4 sm:px-6">
<div class="flex items-center justify-between">
<div class="text-sm font-medium text-blue-600 truncate">
{task.plugin_id.toUpperCase()}
<span class="ml-2 text-xs text-gray-400 font-mono">{task.id.substring(0, 8)}</span>
</div>
<div class="ml-2 flex-shrink-0 flex">
<p class="px-2 inline-flex text-xs leading-5 font-semibold rounded-full {getStatusColor(task.status)}">
{task.status}
</p>
</div>
</div>
<div class="mt-2 sm:flex sm:justify-between">
<div class="sm:flex">
<p class="flex items-center text-sm text-gray-500">
{#if task.params?.environment_id || task.params?.source_env_id}
<span class="mr-2">Env: {task.params.environment_id || task.params.source_env_id}</span>
{/if}
</p>
</div>
<div class="mt-2 flex items-center text-sm text-gray-500 sm:mt-0">
<svg class="flex-shrink-0 mr-1.5 h-5 w-5 text-gray-400" fill="currentColor" viewBox="0 0 20 20">
<path fill-rule="evenodd" d="M6 2a1 1 0 00-1 1v1H4a2 2 0 00-2 2v10a2 2 0 002 2h12a2 2 0 002-2V6a2 2 0 00-2-2h-1V3a1 1 0 10-2 0v1H7V3a1 1 0 00-1-1zm0 5a1 1 0 000 2h8a1 1 0 100-2H6z" clip-rule="evenodd" />
</svg>
<p>
Started {formatTime(task.started_at)}
</p>
</div>
</div>
</div>
</button>
</li>
{/each}
</ul>
{/if}
</div>
<!-- [/DEF:TaskList] -->

View File

@@ -100,19 +100,21 @@ async function requestApi(endpoint, method = 'GET', body = null) {
// [DEF:api:Data]
// @PURPOSE: API client object with specific methods.
export const api = {
getPlugins: () => fetchApi('/plugins/'),
getTasks: () => fetchApi('/tasks/'),
getPlugins: () => fetchApi('/plugins'),
getTasks: () => fetchApi('/tasks'),
getTask: (taskId) => fetchApi(`/tasks/${taskId}`),
createTask: (pluginId, params) => postApi('/tasks/', { plugin_id: pluginId, params }),
createTask: (pluginId, params) => postApi('/tasks', { plugin_id: pluginId, params }),
// Settings
getSettings: () => fetchApi('/settings/'),
getSettings: () => fetchApi('/settings'),
updateGlobalSettings: (settings) => requestApi('/settings/global', 'PATCH', settings),
getEnvironments: () => fetchApi('/settings/environments'),
addEnvironment: (env) => postApi('/settings/environments', env),
updateEnvironment: (id, env) => requestApi(`/settings/environments/${id}`, 'PUT', env),
deleteEnvironment: (id) => requestApi(`/settings/environments/${id}`, 'DELETE'),
testEnvironmentConnection: (id) => postApi(`/settings/environments/${id}/test`, {}),
updateEnvironmentSchedule: (id, schedule) => requestApi(`/environments/${id}/schedule`, 'PUT', schedule),
getEnvironmentsList: () => fetchApi('/environments'),
};
// [/DEF:api_module]
@@ -128,3 +130,5 @@ export const addEnvironment = api.addEnvironment;
export const updateEnvironment = api.updateEnvironment;
export const deleteEnvironment = api.deleteEnvironment;
export const testEnvironmentConnection = api.testEnvironmentConnection;
export const updateEnvironmentSchedule = api.updateEnvironmentSchedule;
export const getEnvironmentsList = api.getEnvironmentsList;

View File

@@ -13,7 +13,7 @@
<script>
// [SECTION: IMPORTS]
import { onMount } from 'svelte';
import { getSettings, updateGlobalSettings, getEnvironments, addEnvironment, updateEnvironment, deleteEnvironment, testEnvironmentConnection } from '../lib/api';
import { getSettings, updateGlobalSettings, getEnvironments, addEnvironment, updateEnvironment, deleteEnvironment, testEnvironmentConnection, updateEnvironmentSchedule } from '../lib/api';
import { addToast } from '../lib/toasts';
// [/SECTION]
@@ -38,7 +38,11 @@
url: '',
username: '',
password: '',
is_default: false
is_default: false,
backup_schedule: {
enabled: false,
cron_expression: '0 0 * * *'
}
};
let editingEnvId = null;
@@ -167,7 +171,11 @@
url: '',
username: '',
password: '',
is_default: false
is_default: false,
backup_schedule: {
enabled: false,
cron_expression: '0 0 * * *'
}
};
editingEnvId = null;
}
@@ -293,7 +301,21 @@
<label for="env_default" class="ml-2 block text-sm text-gray-900">Default Environment</label>
</div>
</div>
<div class="mt-4 flex gap-2">
<h3 class="text-lg font-medium mb-4 mt-6">Backup Schedule</h3>
<div class="grid grid-cols-1 md:grid-cols-2 gap-4">
<div class="flex items-center">
<input type="checkbox" id="backup_enabled" bind:checked={newEnv.backup_schedule.enabled} class="h-4 w-4 text-blue-600 border-gray-300 rounded" />
<label for="backup_enabled" class="ml-2 block text-sm text-gray-900">Enable Automatic Backups</label>
</div>
<div>
<label for="cron_expression" class="block text-sm font-medium text-gray-700">Cron Expression</label>
<input type="text" id="cron_expression" bind:value={newEnv.backup_schedule.cron_expression} placeholder="0 0 * * *" class="mt-1 block w-full border border-gray-300 rounded-md shadow-sm p-2" />
<p class="text-xs text-gray-500 mt-1">Example: 0 0 * * * (daily at midnight), */5 * * * * (every 5 minutes)</p>
</div>
</div>
<div class="mt-6 flex gap-2">
<button on:click={handleAddOrUpdateEnv} class="bg-green-500 text-white px-4 py-2 rounded hover:bg-green-600">
{editingEnvId ? 'Update' : 'Add'} Environment
</button>

View File

@@ -0,0 +1,140 @@
<script>
import { onMount, onDestroy } from 'svelte';
import { getTasks, createTask, getEnvironmentsList } from '../../lib/api';
import { addToast } from '../../lib/toasts';
import TaskList from '../../components/TaskList.svelte';
import TaskLogViewer from '../../components/TaskLogViewer.svelte';
let tasks = [];
let environments = [];
let loading = true;
let selectedTaskId = null;
let pollInterval;
let showBackupModal = false;
let selectedEnvId = '';
async function loadInitialData() {
try {
loading = true;
const [tasksData, envsData] = await Promise.all([
getTasks(),
getEnvironmentsList()
]);
tasks = tasksData;
environments = envsData;
} catch (error) {
console.error('Failed to load tasks data:', error);
} finally {
loading = false;
}
}
async function refreshTasks() {
try {
const data = await getTasks();
// Ensure we don't try to parse HTML as JSON if the route returns 404
if (Array.isArray(data)) {
tasks = data;
}
} catch (error) {
console.error('Failed to refresh tasks:', error);
}
}
function handleSelectTask(event) {
selectedTaskId = event.detail.id;
}
async function handleRunBackup() {
if (!selectedEnvId) {
addToast('Please select an environment', 'error');
return;
}
try {
const task = await createTask('superset-backup', { environment_id: selectedEnvId });
addToast('Backup task started', 'success');
showBackupModal = false;
selectedTaskId = task.id;
await refreshTasks();
} catch (error) {
console.error('Failed to start backup:', error);
}
}
onMount(() => {
loadInitialData();
pollInterval = setInterval(refreshTasks, 3000);
});
onDestroy(() => {
if (pollInterval) clearInterval(pollInterval);
});
</script>
<div class="container mx-auto p-4 max-w-6xl">
<div class="flex justify-between items-center mb-6">
<h1 class="text-2xl font-bold text-gray-800">Task Management</h1>
<button
on:click={() => showBackupModal = true}
class="bg-blue-600 hover:bg-blue-700 text-white px-4 py-2 rounded-md shadow-sm transition duration-150 font-medium"
>
Run Backup
</button>
</div>
<div class="grid grid-cols-1 lg:grid-cols-3 gap-6">
<div class="lg:col-span-1">
<h2 class="text-lg font-semibold mb-3 text-gray-700">Recent Tasks</h2>
<TaskList {tasks} {loading} on:select={handleSelectTask} />
</div>
<div class="lg:col-span-2">
<h2 class="text-lg font-semibold mb-3 text-gray-700">Task Details & Logs</h2>
{#if selectedTaskId}
<div class="bg-white rounded-lg shadow-lg h-[600px] flex flex-col">
<TaskLogViewer taskId={selectedTaskId} />
</div>
{:else}
<div class="bg-gray-50 border-2 border-dashed border-gray-300 rounded-lg h-[600px] flex items-center justify-center text-gray-500">
<p>Select a task to view logs and details</p>
</div>
{/if}
</div>
</div>
</div>
{#if showBackupModal}
<div class="fixed inset-0 z-50 flex items-center justify-center bg-black bg-opacity-50">
<div class="bg-white rounded-lg shadow-xl p-6 w-full max-w-md">
<h3 class="text-xl font-bold mb-4">Run Manual Backup</h3>
<div class="mb-4">
<label for="env-select" class="block text-sm font-medium text-gray-700 mb-1">Target Environment</label>
<select
id="env-select"
bind:value={selectedEnvId}
class="w-full border-gray-300 rounded-md shadow-sm focus:ring-blue-500 focus:border-blue-500 p-2 border"
>
<option value="" disabled>-- Select Environment --</option>
{#each environments as env}
<option value={env.id}>{env.name}</option>
{/each}
</select>
</div>
<div class="flex justify-end space-x-3">
<button
on:click={() => showBackupModal = false}
class="px-4 py-2 text-gray-700 hover:bg-gray-100 rounded-md transition"
>
Cancel
</button>
<button
on:click={handleRunBackup}
class="px-4 py-2 bg-blue-600 text-white rounded-md hover:bg-blue-700 transition"
>
Start Backup
</button>
</div>
</div>
</div>
{/if}

View File

@@ -0,0 +1,34 @@
# Specification Quality Checklist: Backup Scheduler & Unified Task UI
**Purpose**: Validate specification completeness and quality before proceeding to planning
**Created**: 2025-12-30
**Feature**: [Link to spec.md](../spec.md)
## Content Quality
- [x] No implementation details (languages, frameworks, APIs)
- [x] Focused on user value and business needs
- [x] Written for non-technical stakeholders
- [x] All mandatory sections completed
## Requirement Completeness
- [x] No [NEEDS CLARIFICATION] markers remain
- [x] Requirements are testable and unambiguous
- [x] Success criteria are measurable
- [x] Success criteria are technology-agnostic (no implementation details)
- [x] All acceptance scenarios are defined
- [x] Edge cases are identified
- [x] Scope is clearly bounded
- [x] Dependencies and assumptions identified
## Feature Readiness
- [x] All functional requirements have clear acceptance criteria
- [x] User scenarios cover primary flows
- [x] Feature meets measurable outcomes defined in Success Criteria
- [x] No implementation details leak into specification
## Notes
- Items marked incomplete require spec updates before `/speckit.clarify` or `/speckit.plan`

View File

@@ -0,0 +1,154 @@
openapi: 3.0.0
info:
title: Backup Scheduler & Task API
version: 1.0.0
paths:
/tasks:
get:
summary: List all tasks
parameters:
- name: limit
in: query
schema:
type: integer
default: 50
- name: type
in: query
schema:
type: string
enum: [backup, migration]
- name: status
in: query
schema:
type: string
enum: [running, success, failed, pending]
responses:
'200':
description: List of tasks
content:
application/json:
schema:
type: array
items:
$ref: '#/components/schemas/Task'
/tasks/backup:
post:
summary: Manually trigger a backup
requestBody:
required: true
content:
application/json:
schema:
type: object
required:
- environment_id
properties:
environment_id:
type: string
format: uuid
responses:
'202':
description: Backup task started
content:
application/json:
schema:
$ref: '#/components/schemas/Task'
/tasks/{id}:
get:
summary: Get task details
parameters:
- name: id
in: path
required: true
schema:
type: string
format: uuid
responses:
'200':
description: Task details
content:
application/json:
schema:
$ref: '#/components/schemas/Task'
/environments/{id}/schedule:
get:
summary: Get backup schedule for environment
parameters:
- name: id
in: path
required: true
schema:
type: string
format: uuid
responses:
'200':
description: Schedule configuration
content:
application/json:
schema:
$ref: '#/components/schemas/Schedule'
put:
summary: Update backup schedule
parameters:
- name: id
in: path
required: true
schema:
type: string
format: uuid
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/Schedule'
responses:
'200':
description: Schedule updated
content:
application/json:
schema:
$ref: '#/components/schemas/Schedule'
components:
schemas:
Task:
type: object
properties:
id:
type: string
format: uuid
type:
type: string
enum: [backup, migration]
status:
type: string
enum: [pending, running, success, failed]
environment_id:
type: string
format: uuid
started_at:
type: string
format: date-time
finished_at:
type: string
format: date-time
created_at:
type: string
format: date-time
error:
type: string
logs:
type: string
Schedule:
type: object
properties:
enabled:
type: boolean
cron_expression:
type: string
example: "0 0 * * *"

View File

@@ -0,0 +1,34 @@
# Data Model: Backup Scheduler & Unified Task UI
## Entities
### Task
Represents a background operation (Backup, Migration) managed by the system.
| Field | Type | Description | Constraints |
|-------|------|-------------|-------------|
| `id` | UUID | Unique identifier | Primary Key |
| `type` | String | Type of task | Enum: "backup", "migration" |
| `status` | String | Current execution state | Enum: "pending", "running", "success", "failed" |
| `environment_id` | UUID | Target environment (if applicable) | Foreign Key (Environments), Nullable |
| `started_at` | DateTime | When the task began | Nullable |
| `finished_at` | DateTime | When the task completed | Nullable |
| `logs` | Text | Execution logs | |
| `error` | Text | Error message if failed | Nullable |
| `created_at` | DateTime | When task was queued | Default: Now |
### Schedule
Configuration for automatic task execution. Nested within Environment config.
| Field | Type | Description | Constraints |
|-------|------|-------------|-------------|
| `environment_id` | UUID | Target environment | Foreign Key (Environments) |
| `enabled` | Boolean | Is schedule active? | Default: false |
| `cron_expression` | String | Frequency definition | Valid Cron string (e.g., "0 0 * * *") |
| `last_run_at` | DateTime | Last execution time | Nullable |
| `next_run_at` | DateTime | Calculated next run | Nullable |
## Storage Strategy
- **Tasks**: Stored in `tasks.db` (SQLite) via SQLAlchemy.
- **Schedules**: Stored in `config.json` as part of the Environment model.

View File

@@ -0,0 +1,84 @@
# Implementation Plan: Backup Scheduler & Unified Task UI
**Branch**: `009-backup-scheduler` | **Date**: 2025-12-30 | **Spec**: [link](spec.md)
**Input**: Feature specification from `/specs/009-backup-scheduler/spec.md`
**Note**: This template is filled in by the `/speckit.plan` command. See `.specify/templates/commands/plan.md` for the execution workflow.
## Summary
Implement a robust backup scheduling system using `APScheduler` and a unified "Tasks" UI in SvelteKit to manage and monitor all background operations (backups, migrations).
## Technical Context
**Language/Version**: Python 3.9+, Node.js 18+
**Primary Dependencies**: FastAPI, APScheduler, SQLAlchemy, SvelteKit, Tailwind CSS
**Storage**: SQLite (`tasks.db`), JSON (`config.json`)
**Testing**: pytest
**Target Platform**: Linux server
**Project Type**: Web application
**Performance Goals**: UI latency < 200ms, Backup trigger < 1s
**Constraints**: Minimal resource footprint for background scheduler
**Scale/Scope**: ~10 environments, ~1000 historical tasks
## Constitution Check
*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
- **Library-First**: N/A (Feature integration)
- **CLI Interface**: N/A (Web UI focus)
- **Test-First**: Mandatory for Scheduler logic and API endpoints. PASS.
- **Integration Testing**: Required for Scheduler -> TaskManager interaction. PASS.
**Result**: PASS
## Project Structure
### Documentation (this feature)
```text
specs/009-backup-scheduler/
├── plan.md # This file (/speckit.plan command output)
├── research.md # Phase 0 output (/speckit.plan command)
├── data-model.md # Phase 1 output (/speckit.plan command)
├── quickstart.md # Phase 1 output (/speckit.plan command)
├── contracts/ # Phase 1 output (/speckit.plan command)
└── tasks.md # Phase 2 output (/speckit.tasks command - NOT created by /speckit.plan)
```
### Source Code (repository root)
```text
backend/
├── src/
│ ├── api/
│ │ └── routes/
│ │ └── tasks.py # NEW: Task management endpoints
│ ├── core/
│ │ ├── scheduler.py # NEW: APScheduler integration
│ │ └── task_manager/ # EXISTING: Updates for DB persistence
│ ├── models/
│ │ └── task.py # NEW: SQLAlchemy model
│ └── services/
└── tests/
frontend/
├── src/
│ ├── components/
│ │ └── TaskList.svelte # NEW: Task display component
│ │ └── TaskLogViewer.svelte # NEW: Detailed log view
│ ├── routes/
│ │ └── tasks/ # NEW: Tasks page
│ │ └── +page.svelte
│ └── types/
```
**Structure Decision**: Standard FastAPI + SvelteKit structure.
## Complexity Tracking
> **Fill ONLY if Constitution Check has violations that must be justified**
| Violation | Why Needed | Simpler Alternative Rejected Because |
|-----------|------------|-------------------------------------|
| | | |

View File

@@ -0,0 +1,28 @@
# Quickstart: Backup Scheduler & Unified Task UI
## Prerequisites
- Backend running: `cd backend && uvicorn src.app:app --reload`
- Frontend running: `cd frontend && npm run dev`
## Usage Guide
### 1. View Tasks
1. Navigate to the new **Tasks** tab in the main navigation bar.
2. Observe the list of recent tasks (Backups, Migrations).
3. Click on any task row to view detailed logs.
### 2. Configure Scheduled Backups
1. Go to **Settings**.
2. Edit an existing Environment (or create a new one).
3. Scroll to the **Backup Schedule** section.
4. Enable the "Automatic Backups" toggle.
5. Enter a valid Cron expression (e.g., `*/5 * * * *` for every 5 minutes).
6. Save the environment.
7. Wait for the scheduled time and verify a new Backup task appears in the **Tasks** tab.
### 3. Manual Backup Trigger
1. Go to the **Tasks** tab.
2. Click the **Run Backup** button (top right).
3. Select the target environment from the dropdown.
4. Click **Start**.
5. Watch the new task appear with "Running" status.

View File

@@ -0,0 +1,29 @@
# Research: Backup Scheduler & Unified Task UI
## Decisions
### 1. Scheduler Implementation
- **Decision**: Use `APScheduler` (Advanced Python Scheduler) with `BackgroundScheduler`.
- **Rationale**: `APScheduler` is the industry standard for Python scheduling. It supports Cron-style scheduling (required by FR-001), runs in the background (FR-003), and handles thread management. It integrates well with FastAPI lifecycles.
- **Alternatives Considered**:
- `cron` (system level): Harder to manage from within the app, requires OS access.
- `schedule` library: Simpler but lacks advanced Cron features and persistence robustness.
- Custom thread loop: Error-prone and reinvents the wheel.
### 2. Task History Database
- **Decision**: SQLite (`tasks.db`) accessed via `SQLAlchemy` (AsyncIO).
- **Rationale**: The spec explicitly requests `tasks.db` (FR-009). SQLAlchemy provides a robust ORM for the `Task` entity. Using AsyncIO ensures non-blocking database operations within the FastAPI event loop, even if the actual backup tasks run in threads.
- **Alternatives Considered**:
- `JSON` files: Poor performance for filtering/sorting logs (FR-006).
- `PostgreSQL`: Overkill for a local tool configuration.
### 3. Concurrency Handling
- **Decision**: Skip scheduled backups if a backup is already running for the *same* environment. Allow concurrent backups for *different* environments.
- **Rationale**: Prevents resource contention and potential corruption of the same target.
- **Implementation**: The `SchedulerService` will check `TaskManager` for active jobs with the same `environment_id` before triggering.
### 4. Frontend Polling vs WebSockets
- **Decision**: Polling (every 2-5 seconds) for the "Tasks" tab.
- **Rationale**: Simpler to implement than WebSockets for this scale. The requirement is "near real-time" (SC-002: latency < 5s), which polling satisfies easily.
- **Alternatives Considered**:
- WebSockets: Better real-time, but higher complexity for connection management and state.

View File

@@ -0,0 +1,115 @@
# Feature Specification: Backup Scheduler & Unified Task UI
**Feature Branch**: `009-backup-scheduler`
**Created**: 2025-12-30
**Status**: Draft
**Input**: User description: "Я хочу доработать механизм бекапа. Он должен иметь возможность работать по расписанию, задания и их статус должны использовать TaskManager и быть доступны в общем логе. Я думаю нужно вынести все задачи в отдельную вкладку - миграции, бэкапов и прочих задач которые мы в будущем добавим."
## User Scenarios & Testing *(mandatory)*
### User Story 1 - Scheduled Backups (Priority: P1)
As an Administrator, I want to configure automatic backup schedules for my Superset environments so that my data is preserved regularly without manual intervention.
**Why this priority**: Automation is the core request. It ensures data safety and reduces manual toil.
**Independent Test**: Configure a schedule (e.g., every minute for testing), wait, and verify a backup task is created and executed automatically.
**Acceptance Scenarios**:
1. **Given** an environment configuration, **When** I enable scheduled backups with a specific interval (e.g., daily), **Then** the system automatically triggers a backup task at the specified time.
2. **Given** a scheduled backup runs, **When** it completes, **Then** a new backup archive is present in the storage and a success log is recorded.
---
### User Story 2 - Unified Task Management UI (Priority: P1)
As an Administrator, I want a dedicated "Tasks" tab where I can see and manage all background operations (backups, migrations) in one place.
**Why this priority**: Centralizes visibility and control, improving usability as the number of background tasks grows.
**Independent Test**: Navigate to the new "Tasks" tab and verify it lists both manual and scheduled tasks with their current status.
**Acceptance Scenarios**:
1. **Given** the application is open, **When** I click the "Tasks" tab, **Then** I see a list of recent tasks including their type (Backup, Migration), status, and timestamp.
2. **Given** a running task, **When** I view the Tasks tab, **Then** I see the task status update in real-time (or near real-time).
---
### User Story 3 - Manual Backup Trigger (Priority: P2)
As an Administrator, I want to manually trigger a backup from the Tasks UI immediately, for example, before a major change.
**Why this priority**: Ad-hoc backups are necessary for operational safety before maintenance.
**Independent Test**: Click "Run Backup" in the UI and verify a new task starts immediately.
**Acceptance Scenarios**:
1. **Given** the Tasks tab is open, **When** I select an environment and click "Run Backup", **Then** a new backup task appears in the list with "Running" status.
---
### User Story 4 - Task History & Logs (Priority: P2)
As an Administrator, I want to view the detailed logs of any executed task to troubleshoot failures or verify success.
**Why this priority**: Essential for debugging and auditability.
**Independent Test**: Click on a completed task and verify the log output is displayed.
**Acceptance Scenarios**:
1. **Given** a list of tasks, **When** I click on a "Failed" task, **Then** I can see the error logs explaining why it failed.
2. **Given** a "Success" task, **When** I view logs, **Then** I see the execution steps confirmation.
### Edge Cases
- **Concurrent Backups**: What happens if a scheduled backup triggers while a manual backup is already running for the same environment? (System should likely queue or skip).
- **Storage Full**: How does the system handle backup failures due to insufficient disk space? (Should fail gracefully and log error).
- **Superset Offline**: What happens if the Superset environment is unreachable when a backup is triggered? (Task fails with connection error).
### Assumptions
- The backend server is running continuously to process scheduled tasks.
- Users have configured valid credentials for Superset environments.
- There is sufficient storage space for backup archives.
## Requirements *(mandatory)*
### Functional Requirements
- **FR-001**: The system MUST allow users to configure a backup schedule using **Cron expressions** for each defined Superset environment via the **Settings** page.
- **FR-002**: The system MUST persist schedule configurations nested within the Environment configuration in `config.json`.
- **FR-003**: The system MUST include a `SchedulerService` running in a background thread that triggers backup tasks via the `TaskManager`.
- **FR-004**: The system MUST provide a dedicated "Tasks" page in the frontend.
- **FR-005**: The "Tasks" page MUST display a unified list of all `TaskManager` jobs, including Migrations and Backups.
- **FR-006**: The "Tasks" page MUST allow users to filter tasks by status (Running, Success, Failed) and type.
- **FR-007**: The system MUST allow users to manually trigger a backup for a specific environment from the "Tasks" page.
- **FR-008**: All backup operations (scheduled or manual) MUST be executed as `TaskManager` tasks and generate standard logs.
- **FR-009**: The system MUST retain a history of task executions in a dedicated SQLite database (`tasks.db`) for long-term review.
- **FR-010**: The system MUST automatically clean up task history older than 30 days to prevent unbounded database growth.
### Key Entities
- **Task**: Represents a unit of work (Backup, Migration) managed by TaskManager. Attributes: ID, Type, Status, StartedAt, FinishedAt, Logs.
- **Schedule**: Configuration for when to run a task. Attributes: EnvironmentID, Frequency, NextRunTime, Enabled.
## Success Criteria *(mandatory)*
### Measurable Outcomes
- **SC-001**: Users can configure a backup schedule that persists and triggers automatically within 1 minute of the target time.
- **SC-002**: The "Tasks" UI displays the status of running tasks with a latency of no more than 5 seconds.
- **SC-003**: 100% of triggered backups (manual or scheduled) are recorded in the TaskManager history.
- **SC-004**: Users can access logs for any task executed in the last 7 days (or configured retention period).
## Clarifications
### Session 2025-12-30
- Q: Where should the backup schedule configuration UI be located? → A: In the **Settings** tab, inside each Environment's edit form.
- Q: How should schedule configurations be persisted? → A: Add a `Schedule` model in `config_models.py` and nest it under `Environment`.
- Q: What format should be used for defining schedule frequency? → A: Cron-style strings (e.g., "0 0 * * *").
- Q: How should the scheduling mechanism be implemented? → A: Create a dedicated `SchedulerService` in `backend/src/core/scheduler.py` that runs in a background thread.
- Q: Where should task history be stored for long-term retention? → A: Add a `tasks.db` SQLite database using SQLAlchemy.

View File

@@ -0,0 +1,51 @@
# Tasks: Backup Scheduler & Unified Task UI
## Phase 1: Setup
- [x] T001 Initialize SQLite database `tasks.db` and SQLAlchemy engine in `backend/src/core/database.py`
- [x] T002 Create SQLAlchemy model for `TaskRecord` in `backend/src/models/task.py`
- [x] T003 Update `backend/src/core/config_models.py` to include `Schedule` and update `Environment` model
- [x] T004 Create database migrations or initialization script for `tasks.db`
## Phase 2: Foundational
- [x] T005 [P] Implement `TaskPersistence` layer in `backend/src/core/task_manager/persistence.py`
- [x] T006 Update `TaskManager` in `backend/src/core/task_manager/manager.py` to use persistence for all jobs
- [x] T007 Implement `SchedulerService` using `APScheduler` in `backend/src/core/scheduler.py`
- [x] T008 Integrate `SchedulerService` into main FastAPI application startup in `backend/src/app.py`
## Phase 3: [US1] Scheduled Backups
- [x] T009 [US1] Implement schedule loading and registration logic in `SchedulerService`
- [x] T010 [US1] Update `Environment` settings API to handle `backup_schedule` updates in `backend/src/api/routes/environments.py`
- [x] T011 [P] [US1] Add schedule configuration fields to Environment edit form in `frontend/src/components/EnvSelector.svelte` (or appropriate component)
- [x] T012 [US1] Implement validation for Cron expressions in backend and frontend
## Phase 4: [US2] Unified Task Management UI
- [x] T013 [US2] Implement `/api/tasks` endpoint to list and filter tasks in `backend/src/api/routes/tasks.py`
- [x] T014 [US2] Create new Tasks page in `frontend/src/routes/tasks/+page.svelte`
- [x] T015 [P] [US2] Implement `TaskList` component in `frontend/src/components/TaskList.svelte`
- [x] T016 [US2] Add "Tasks" link to main navigation in `frontend/src/components/Navbar.svelte`
## Phase 5: [US3] Manual Backup Trigger
- [x] T017 [US3] Implement `/api/tasks/backup` POST endpoint in `backend/src/api/routes/tasks.py`
- [x] T018 [US3] Add "Run Backup" button and environment selection to Tasks page in `frontend/src/routes/tasks/+page.svelte`
## Phase 6: [US4] Task History & Logs
- [x] T019 [US4] Implement `/api/tasks/{task_id}` GET endpoint for detailed task info and logs in `backend/src/api/routes/tasks.py`
- [x] T020 [US4] Implement `TaskLogViewer` component in `frontend/src/components/TaskLogViewer.svelte`
- [x] T021 [US4] Integrate log viewer into TaskList or as a separate modal/page
## Final Phase: Polish & Cross-cutting concerns
- [x] T022 Implement task cleanup/retention policy (e.g., delete tasks older than 30 days)
- [ ] T023 Add real-time updates for task status using WebSockets (optional/refinement)
- [x] T024 Ensure consistent error handling and logging across scheduler and task manager
## Dependencies
- US1 depends on Phase 1 & 2
- US2 depends on Phase 1 & 2
- US3 depends on US2
- US4 depends on US2
## Implementation Strategy
1. **Infrastructure First**: Setup database and basic task persistence.
2. **Backend Logic**: Implement scheduler and update task manager.
3. **API & UI**: Build the unified tasks view.
4. **Feature Integration**: Add scheduling UI and manual triggers.