2 Commits

Author SHA1 Message Date
35b423979d spec rules 2025-12-25 22:28:42 +03:00
2ffc3cc68f feat(migration): implement interactive mapping resolution workflow
- Add SQLite database integration for environments and mappings
- Update TaskManager to support pausing tasks (AWAITING_MAPPING)
- Modify MigrationPlugin to detect missing mappings and wait for resolution
- Add frontend UI for handling missing mappings interactively
- Create dedicated migration routes and API endpoints
- Update .gitignore and project documentation
2025-12-25 22:27:29 +03:00
38 changed files with 2434 additions and 51 deletions

76
.gitignore vendored
View File

@@ -1,21 +1,63 @@
*__pycache__*
*.ps1
keyring passwords.py
*logs*
*github*
*venv*
*git*
*tech_spec*
dashboards
# Python specific
*.pyc
dist/
*.egg-info/
# Node.js specific
node_modules/
# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
pip-wheel-metadata/
share/python-wheels/
*.egg-info/
.installed.cfg
*.egg
MANIFEST
.venv
venv/
ENV/
env/
backend/backups/*
# Node.js
node_modules/
npm-debug.log*
yarn-debug.log*
yarn-error.log*
.svelte-kit/
.vite/
build/
dist/
.env*
config.json
backend/backups/*
# Logs
*.log
backend/backend.log
# OS
.DS_Store
Thumbs.db
# IDE
.vscode/
.idea/
*.swp
*.swo
# Project specific
*.ps1
keyring passwords.py
*github*
*git*
*tech_spec*
dashboards

View File

@@ -6,7 +6,7 @@ Auto-generated from all feature plans. Last updated: 2025-12-19
- Python 3.9+, Node.js 18+ + `uvicorn`, `npm`, `bash` (003-project-launch-script)
- Python 3.9+, Node.js 18+ + SvelteKit, FastAPI, Tailwind CSS (inferred from existing frontend) (004-integrate-svelte-kit)
- N/A (Frontend integration) (004-integrate-svelte-kit)
- Python 3.9+, Node.js 18+ + FastAPI, SvelteKit, Tailwind CSS, Pydantic (001-fix-ui-ws-validation)
- Python 3.9+, Node.js 18+ + FastAPI, SvelteKit, Tailwind CSS, Pydantic (005-fix-ui-ws-validation)
- N/A (Configuration based) (005-fix-ui-ws-validation)
- Filesystem (plugins, logs, backups), SQLite (optional, for job history if needed) (005-fix-ui-ws-validation)
@@ -29,9 +29,9 @@ cd src; pytest; ruff check .
Python 3.9+ (Backend), Node.js 18+ (Frontend Build): Follow standard conventions
## Recent Changes
- 001-fix-ui-ws-validation: Added Python 3.9+ (Backend), Node.js 18+ (Frontend Build)
- 005-fix-ui-ws-validation: Added Python 3.9+ (Backend), Node.js 18+ (Frontend Build)
- 005-fix-ui-ws-validation: Added Python 3.9+, Node.js 18+ + FastAPI, SvelteKit, Tailwind CSS, Pydantic
- 005-fix-ui-ws-validation: Added Python 3.9+, Node.js 18+ + FastAPI, SvelteKit, Tailwind CSS, Pydantic
<!-- MANUAL ADDITIONS START -->

BIN
backend/mappings.db Normal file

Binary file not shown.

View File

@@ -10,3 +10,5 @@ keyring
httpx
PyYAML
websockets
rapidfuzz
sqlalchemy

View File

@@ -0,0 +1,75 @@
# [DEF:backend.src.api.routes.environments:Module]
#
# @SEMANTICS: api, environments, superset, databases
# @PURPOSE: API endpoints for listing environments and their databases.
# @LAYER: API
# @RELATION: DEPENDS_ON -> backend.src.dependencies
# @RELATION: DEPENDS_ON -> backend.src.core.superset_client
#
# @INVARIANT: Environment IDs must exist in the configuration.
# [SECTION: IMPORTS]
from fastapi import APIRouter, Depends, HTTPException
from typing import List, Dict, Optional
from backend.src.dependencies import get_config_manager
from backend.src.core.superset_client import SupersetClient
from superset_tool.models import SupersetConfig
from pydantic import BaseModel
# [/SECTION]
router = APIRouter(prefix="/api/environments", tags=["environments"])
# [DEF:EnvironmentResponse:DataClass]
class EnvironmentResponse(BaseModel):
id: str
name: str
url: str
# [/DEF:EnvironmentResponse]
# [DEF:DatabaseResponse:DataClass]
class DatabaseResponse(BaseModel):
uuid: str
database_name: str
engine: Optional[str]
# [/DEF:DatabaseResponse]
# [DEF:get_environments:Function]
# @PURPOSE: List all configured environments.
# @RETURN: List[EnvironmentResponse]
@router.get("", response_model=List[EnvironmentResponse])
async def get_environments(config_manager=Depends(get_config_manager)):
envs = config_manager.get_environments()
return [EnvironmentResponse(id=e.id, name=e.name, url=e.url) for e in envs]
# [/DEF:get_environments]
# [DEF:get_environment_databases:Function]
# @PURPOSE: Fetch the list of databases from a specific environment.
# @PARAM: id (str) - The environment ID.
# @RETURN: List[Dict] - List of databases.
@router.get("/{id}/databases")
async def get_environment_databases(id: str, config_manager=Depends(get_config_manager)):
envs = config_manager.get_environments()
env = next((e for e in envs if e.id == id), None)
if not env:
raise HTTPException(status_code=404, detail="Environment not found")
try:
# Initialize SupersetClient from environment config
# Note: We need to map Environment model to SupersetConfig
superset_config = SupersetConfig(
env=env.name,
base_url=env.url,
auth={
"provider": "db", # Defaulting to db provider
"username": env.username,
"password": env.password,
"refresh": "false"
}
)
client = SupersetClient(superset_config)
return client.get_databases_summary()
except Exception as e:
raise HTTPException(status_code=500, detail=f"Failed to fetch databases: {str(e)}")
# [/DEF:get_environment_databases]
# [/DEF:backend.src.api.routes.environments]

View File

@@ -0,0 +1,110 @@
# [DEF:backend.src.api.routes.mappings:Module]
#
# @SEMANTICS: api, mappings, database, fuzzy-matching
# @PURPOSE: API endpoints for managing database mappings and getting suggestions.
# @LAYER: API
# @RELATION: DEPENDS_ON -> backend.src.dependencies
# @RELATION: DEPENDS_ON -> backend.src.core.database
# @RELATION: DEPENDS_ON -> backend.src.services.mapping_service
#
# @INVARIANT: Mappings are persisted in the SQLite database.
# [SECTION: IMPORTS]
from fastapi import APIRouter, Depends, HTTPException
from sqlalchemy.orm import Session
from typing import List, Optional
from backend.src.dependencies import get_config_manager
from backend.src.core.database import get_db
from backend.src.models.mapping import DatabaseMapping
from pydantic import BaseModel
# [/SECTION]
router = APIRouter(prefix="/api/mappings", tags=["mappings"])
# [DEF:MappingCreate:DataClass]
class MappingCreate(BaseModel):
source_env_id: str
target_env_id: str
source_db_uuid: str
target_db_uuid: str
source_db_name: str
target_db_name: str
# [/DEF:MappingCreate]
# [DEF:MappingResponse:DataClass]
class MappingResponse(BaseModel):
id: str
source_env_id: str
target_env_id: str
source_db_uuid: str
target_db_uuid: str
source_db_name: str
target_db_name: str
class Config:
from_attributes = True
# [/DEF:MappingResponse]
# [DEF:SuggestRequest:DataClass]
class SuggestRequest(BaseModel):
source_env_id: str
target_env_id: str
# [/DEF:SuggestRequest]
# [DEF:get_mappings:Function]
# @PURPOSE: List all saved database mappings.
@router.get("", response_model=List[MappingResponse])
async def get_mappings(
source_env_id: Optional[str] = None,
target_env_id: Optional[str] = None,
db: Session = Depends(get_db)
):
query = db.query(DatabaseMapping)
if source_env_id:
query = query.filter(DatabaseMapping.source_env_id == source_env_id)
if target_env_id:
query = query.filter(DatabaseMapping.target_env_id == target_env_id)
return query.all()
# [/DEF:get_mappings]
# [DEF:create_mapping:Function]
# @PURPOSE: Create or update a database mapping.
@router.post("", response_model=MappingResponse)
async def create_mapping(mapping: MappingCreate, db: Session = Depends(get_db)):
# Check if mapping already exists
existing = db.query(DatabaseMapping).filter(
DatabaseMapping.source_env_id == mapping.source_env_id,
DatabaseMapping.target_env_id == mapping.target_env_id,
DatabaseMapping.source_db_uuid == mapping.source_db_uuid
).first()
if existing:
existing.target_db_uuid = mapping.target_db_uuid
existing.target_db_name = mapping.target_db_name
db.commit()
db.refresh(existing)
return existing
new_mapping = DatabaseMapping(**mapping.dict())
db.add(new_mapping)
db.commit()
db.refresh(new_mapping)
return new_mapping
# [/DEF:create_mapping]
# [DEF:suggest_mappings_api:Function]
# @PURPOSE: Get suggested mappings based on fuzzy matching.
@router.post("/suggest")
async def suggest_mappings_api(
request: SuggestRequest,
config_manager=Depends(get_config_manager)
):
from backend.src.services.mapping_service import MappingService
service = MappingService(config_manager)
try:
return await service.get_suggestions(request.source_env_id, request.target_env_id)
except Exception as e:
raise HTTPException(status_code=500, detail=str(e))
# [/DEF:suggest_mappings_api]
# [/DEF:backend.src.api.routes.mappings]

View File

@@ -16,6 +16,9 @@ class CreateTaskRequest(BaseModel):
plugin_id: str
params: Dict[str, Any]
class ResolveTaskRequest(BaseModel):
resolution_params: Dict[str, Any]
@router.post("/", response_model=Task, status_code=status.HTTP_201_CREATED)
async def create_task(
request: CreateTaskRequest,
@@ -54,4 +57,19 @@ async def get_task(
if not task:
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Task not found")
return task
@router.post("/{task_id}/resolve", response_model=Task)
async def resolve_task(
task_id: str,
request: ResolveTaskRequest,
task_manager: TaskManager = Depends(get_task_manager)
):
"""
Resolve a task that is awaiting mapping.
"""
try:
await task_manager.resolve_task(task_id, request.resolution_params)
return task_manager.get_task(task_id)
except ValueError as e:
raise HTTPException(status_code=status.HTTP_400_BAD_REQUEST, detail=str(e))
# [/DEF]

View File

@@ -20,7 +20,11 @@ import os
from .dependencies import get_task_manager
from .core.logger import logger
from .api.routes import plugins, tasks, settings
from .api.routes import plugins, tasks, settings, environments, mappings
from .core.database import init_db
# Initialize database
init_db()
# [DEF:App:Global]
# @SEMANTICS: app, fastapi, instance
@@ -45,6 +49,8 @@ app.add_middleware(
app.include_router(plugins.router, prefix="/api/plugins", tags=["Plugins"])
app.include_router(tasks.router, prefix="/api/tasks", tags=["Tasks"])
app.include_router(settings.router, prefix="/api/settings", tags=["Settings"])
app.include_router(environments.router)
app.include_router(mappings.router)
# [DEF:WebSocketEndpoint:Endpoint]
# @SEMANTICS: websocket, logs, streaming, real-time

View File

@@ -0,0 +1,48 @@
# [DEF:backend.src.core.database:Module]
#
# @SEMANTICS: database, sqlite, sqlalchemy, session, persistence
# @PURPOSE: Configures the SQLite database connection and session management.
# @LAYER: Core
# @RELATION: DEPENDS_ON -> sqlalchemy
# @RELATION: USES -> backend.src.models.mapping
#
# @INVARIANT: A single engine instance is used for the entire application.
# [SECTION: IMPORTS]
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker, Session
from backend.src.models.mapping import Base
import os
# [/SECTION]
# [DEF:DATABASE_URL:Constant]
DATABASE_URL = os.getenv("DATABASE_URL", "sqlite:///./mappings.db")
# [/DEF:DATABASE_URL]
# [DEF:engine:Variable]
engine = create_engine(DATABASE_URL, connect_args={"check_same_thread": False})
# [/DEF:engine]
# [DEF:SessionLocal:Class]
SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
# [/DEF:SessionLocal]
# [DEF:init_db:Function]
# @PURPOSE: Initializes the database by creating all tables.
def init_db():
Base.metadata.create_all(bind=engine)
# [/DEF:init_db]
# [DEF:get_db:Function]
# @PURPOSE: Dependency for getting a database session.
# @POST: Session is closed after use.
# @RETURN: Generator[Session, None, None]
def get_db():
db = SessionLocal()
try:
yield db
finally:
db.close()
# [/DEF:get_db]
# [/DEF:backend.src.core.database]

View File

@@ -0,0 +1,81 @@
# [DEF:backend.src.core.migration_engine:Module]
#
# @SEMANTICS: migration, engine, zip, yaml, transformation
# @PURPOSE: Handles the interception and transformation of Superset asset ZIP archives.
# @LAYER: Core
# @RELATION: DEPENDS_ON -> PyYAML
#
# @INVARIANT: ZIP structure must be preserved after transformation.
# [SECTION: IMPORTS]
import zipfile
import yaml
import os
import shutil
import tempfile
from pathlib import Path
from typing import Dict
# [/SECTION]
# [DEF:MigrationEngine:Class]
# @PURPOSE: Engine for transforming Superset export ZIPs.
class MigrationEngine:
# [DEF:MigrationEngine.transform_zip:Function]
# @PURPOSE: Extracts ZIP, replaces database UUIDs in YAMLs, and re-packages.
# @PARAM: zip_path (str) - Path to the source ZIP file.
# @PARAM: output_path (str) - Path where the transformed ZIP will be saved.
# @PARAM: db_mapping (Dict[str, str]) - Mapping of source UUID to target UUID.
# @RETURN: bool - True if successful.
def transform_zip(self, zip_path: str, output_path: str, db_mapping: Dict[str, str]) -> bool:
"""
Transform a Superset export ZIP by replacing database UUIDs.
"""
with tempfile.TemporaryDirectory() as temp_dir_str:
temp_dir = Path(temp_dir_str)
try:
# 1. Extract
with zipfile.ZipFile(zip_path, 'r') as zf:
zf.extractall(temp_dir)
# 2. Transform YAMLs
# Datasets are usually in datasets/*.yaml
dataset_files = list(temp_dir.glob("**/datasets/*.yaml"))
for ds_file in dataset_files:
self._transform_yaml(ds_file, db_mapping)
# 3. Re-package
with zipfile.ZipFile(output_path, 'w', zipfile.ZIP_DEFLATED) as zf:
for root, dirs, files in os.walk(temp_dir):
for file in files:
file_path = Path(root) / file
arcname = file_path.relative_to(temp_dir)
zf.write(file_path, arcname)
return True
except Exception as e:
print(f"Error transforming ZIP: {e}")
return False
# [DEF:MigrationEngine._transform_yaml:Function]
# @PURPOSE: Replaces database_uuid in a single YAML file.
def _transform_yaml(self, file_path: Path, db_mapping: Dict[str, str]):
with open(file_path, 'r') as f:
data = yaml.safe_load(f)
if not data:
return
# Superset dataset YAML structure:
# database_uuid: ...
source_uuid = data.get('database_uuid')
if source_uuid in db_mapping:
data['database_uuid'] = db_mapping[source_uuid]
with open(file_path, 'w') as f:
yaml.dump(data, f)
# [/DEF:MigrationEngine._transform_yaml]
# [/DEF:MigrationEngine]
# [/DEF:backend.src.core.migration_engine]

View File

@@ -0,0 +1,57 @@
# [DEF:backend.src.core.superset_client:Module]
#
# @SEMANTICS: superset, api, client, database, metadata
# @PURPOSE: Extends the base SupersetClient with database-specific metadata fetching.
# @LAYER: Core
# @RELATION: INHERITS_FROM -> superset_tool.client.SupersetClient
#
# @INVARIANT: All database metadata requests must include UUID and name.
# [SECTION: IMPORTS]
from typing import List, Dict, Optional, Tuple
from superset_tool.client import SupersetClient as BaseSupersetClient
from superset_tool.models import SupersetConfig
# [/SECTION]
# [DEF:SupersetClient:Class]
# @PURPOSE: Extended SupersetClient for migration-specific operations.
class SupersetClient(BaseSupersetClient):
# [DEF:SupersetClient.get_databases_summary:Function]
# @PURPOSE: Fetch a summary of databases including uuid, name, and engine.
# @POST: Returns a list of database dictionaries with 'engine' field.
# @RETURN: List[Dict] - Summary of databases.
def get_databases_summary(self) -> List[Dict]:
"""
Fetch a summary of databases including uuid, name, and engine.
"""
query = {
"columns": ["uuid", "database_name", "backend"]
}
_, databases = self.get_databases(query=query)
# Map 'backend' to 'engine' for consistency with contracts
for db in databases:
db['engine'] = db.pop('backend', None)
return databases
# [/DEF:SupersetClient.get_databases_summary]
# [DEF:SupersetClient.get_database_by_uuid:Function]
# @PURPOSE: Find a database by its UUID.
# @PARAM: db_uuid (str) - The UUID of the database.
# @RETURN: Optional[Dict] - Database info if found, else None.
def get_database_by_uuid(self, db_uuid: str) -> Optional[Dict]:
"""
Find a database by its UUID.
"""
query = {
"filters": [{"col": "uuid", "op": "eq", "value": db_uuid}]
}
_, databases = self.get_databases(query=query)
return databases[0] if databases else None
# [/DEF:SupersetClient.get_database_by_uuid]
# [/DEF:SupersetClient]
# [/DEF:backend.src.core.superset_client]

View File

@@ -23,6 +23,7 @@ class TaskStatus(str, Enum):
RUNNING = "RUNNING"
SUCCESS = "SUCCESS"
FAILED = "FAILED"
AWAITING_MAPPING = "AWAITING_MAPPING"
# [/DEF]
@@ -64,6 +65,7 @@ class TaskManager:
self.subscribers: Dict[str, List[asyncio.Queue]] = {}
self.executor = ThreadPoolExecutor(max_workers=5) # For CPU-bound plugin execution
self.loop = asyncio.get_event_loop()
self.task_futures: Dict[str, asyncio.Future] = {}
# [/DEF]
async def create_task(self, plugin_id: str, params: Dict[str, Any], user_id: Optional[str] = None) -> Task:
@@ -99,9 +101,11 @@ class TaskManager:
# Execute plugin in a separate thread to avoid blocking the event loop
# if the plugin's execute method is synchronous and potentially CPU-bound.
# If the plugin's execute method is already async, this can be simplified.
# Pass task_id to plugin so it can signal pause
params = {**task.params, "_task_id": task_id}
await self.loop.run_in_executor(
self.executor,
lambda: asyncio.run(plugin.execute(task.params)) if asyncio.iscoroutinefunction(plugin.execute) else plugin.execute(task.params)
lambda: asyncio.run(plugin.execute(params)) if asyncio.iscoroutinefunction(plugin.execute) else plugin.execute(params)
)
task.status = TaskStatus.SUCCESS
self._add_log(task_id, "INFO", f"Task completed successfully for plugin '{plugin.name}'")
@@ -112,6 +116,38 @@ class TaskManager:
task.finished_at = datetime.utcnow()
# In a real system, you might notify clients via WebSocket here
async def resolve_task(self, task_id: str, resolution_params: Dict[str, Any]):
"""
Resumes a task that is awaiting mapping.
"""
task = self.tasks.get(task_id)
if not task or task.status != TaskStatus.AWAITING_MAPPING:
raise ValueError("Task is not awaiting mapping.")
# Update task params with resolution
task.params.update(resolution_params)
task.status = TaskStatus.RUNNING
self._add_log(task_id, "INFO", "Task resumed after mapping resolution.")
# Signal the future to continue
if task_id in self.task_futures:
self.task_futures[task_id].set_result(True)
async def wait_for_resolution(self, task_id: str):
"""
Pauses execution and waits for a resolution signal.
"""
task = self.tasks.get(task_id)
if not task: return
task.status = TaskStatus.AWAITING_MAPPING
self.task_futures[task_id] = self.loop.create_future()
try:
await self.task_futures[task_id]
finally:
del self.task_futures[task_id]
def get_task(self, task_id: str) -> Optional[Task]:
"""
Retrieves a task by its ID.

View File

@@ -0,0 +1,53 @@
# [DEF:backend.src.core.utils.matching:Module]
#
# @SEMANTICS: fuzzy, matching, rapidfuzz, database, mapping
# @PURPOSE: Provides utility functions for fuzzy matching database names.
# @LAYER: Core
# @RELATION: DEPENDS_ON -> rapidfuzz
#
# @INVARIANT: Confidence scores are returned as floats between 0.0 and 1.0.
# [SECTION: IMPORTS]
from rapidfuzz import fuzz, process
from typing import List, Dict
# [/SECTION]
# [DEF:suggest_mappings:Function]
# @PURPOSE: Suggests mappings between source and target databases using fuzzy matching.
# @PRE: source_databases and target_databases are lists of dictionaries with 'uuid' and 'database_name'.
# @POST: Returns a list of suggested mappings with confidence scores.
# @PARAM: source_databases (List[Dict]) - Databases from the source environment.
# @PARAM: target_databases (List[Dict]) - Databases from the target environment.
# @PARAM: threshold (int) - Minimum confidence score (0-100).
# @RETURN: List[Dict] - Suggested mappings.
def suggest_mappings(source_databases: List[Dict], target_databases: List[Dict], threshold: int = 60) -> List[Dict]:
"""
Suggest mappings between source and target databases using fuzzy matching.
"""
suggestions = []
if not target_databases:
return suggestions
target_names = [db['database_name'] for db in target_databases]
for s_db in source_databases:
# Use token_sort_ratio as decided in research.md
match = process.extractOne(
s_db['database_name'],
target_names,
scorer=fuzz.token_sort_ratio
)
if match:
name, score, index = match
if score >= threshold:
suggestions.append({
"source_db_uuid": s_db['uuid'],
"target_db_uuid": target_databases[index]['uuid'],
"confidence": score / 100.0
})
return suggestions
# [/DEF:suggest_mappings]
# [/DEF:backend.src.core.utils.matching]

View File

@@ -0,0 +1,70 @@
# [DEF:backend.src.models.mapping:Module]
#
# @SEMANTICS: database, mapping, environment, migration, sqlalchemy, sqlite
# @PURPOSE: Defines the database schema for environment metadata and database mappings using SQLAlchemy.
# @LAYER: Domain
# @RELATION: DEPENDS_ON -> sqlalchemy
#
# @INVARIANT: All primary keys are UUID strings.
# @CONSTRAINT: source_env_id and target_env_id must be valid environment IDs.
# [SECTION: IMPORTS]
from sqlalchemy import Column, String, Boolean, DateTime, ForeignKey, Enum as SQLEnum
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.sql import func
import uuid
import enum
# [/SECTION]
Base = declarative_base()
# [DEF:MigrationStatus:Class]
# @PURPOSE: Enumeration of possible migration job statuses.
class MigrationStatus(enum.Enum):
PENDING = "PENDING"
RUNNING = "RUNNING"
COMPLETED = "COMPLETED"
FAILED = "FAILED"
AWAITING_MAPPING = "AWAITING_MAPPING"
# [/DEF:MigrationStatus]
# [DEF:Environment:Class]
# @PURPOSE: Represents a Superset instance environment.
class Environment(Base):
__tablename__ = "environments"
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
name = Column(String, nullable=False)
url = Column(String, nullable=False)
credentials_id = Column(String, nullable=False)
# [/DEF:Environment]
# [DEF:DatabaseMapping:Class]
# @PURPOSE: Represents a mapping between source and target databases.
class DatabaseMapping(Base):
__tablename__ = "database_mappings"
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
source_env_id = Column(String, ForeignKey("environments.id"), nullable=False)
target_env_id = Column(String, ForeignKey("environments.id"), nullable=False)
source_db_uuid = Column(String, nullable=False)
target_db_uuid = Column(String, nullable=False)
source_db_name = Column(String, nullable=False)
target_db_name = Column(String, nullable=False)
engine = Column(String, nullable=True)
# [/DEF:DatabaseMapping]
# [DEF:MigrationJob:Class]
# @PURPOSE: Represents a single migration execution job.
class MigrationJob(Base):
__tablename__ = "migration_jobs"
id = Column(String, primary_key=True, default=lambda: str(uuid.uuid4()))
source_env_id = Column(String, ForeignKey("environments.id"), nullable=False)
target_env_id = Column(String, ForeignKey("environments.id"), nullable=False)
status = Column(SQLEnum(MigrationStatus), default=MigrationStatus.PENDING)
replace_db = Column(Boolean, default=False)
created_at = Column(DateTime(timezone=True), server_default=func.now())
# [/DEF:MigrationJob]
# [/DEF:backend.src.models.mapping]

View File

@@ -17,6 +17,9 @@ from superset_tool.utils.init_clients import setup_clients
from superset_tool.utils.fileio import create_temp_file, update_yamls, create_dashboard_export
from ..dependencies import get_config_manager
from superset_tool.utils.logger import SupersetLogger
from ..core.migration_engine import MigrationEngine
from ..core.database import SessionLocal
from ..models.mapping import DatabaseMapping, Environment
class MigrationPlugin(PluginBase):
"""
@@ -114,18 +117,26 @@ class MigrationPlugin(PluginBase):
logger.warning("[MigrationPlugin][State] No dashboards found matching the regex.")
return
db_config_replacement = None
# Fetch mappings from database
db_mapping = {}
if replace_db_config:
if from_db_id is None or to_db_id is None:
raise ValueError("Source and target database IDs are required when replacing database configuration.")
from_db = from_c.get_database(int(from_db_id))
to_db = to_c.get_database(int(to_db_id))
old_result = from_db.get("result", {})
new_result = to_db.get("result", {})
db_config_replacement = {
"old": {"database_name": old_result.get("database_name"), "uuid": old_result.get("uuid"), "id": str(from_db.get("id"))},
"new": {"database_name": new_result.get("database_name"), "uuid": new_result.get("uuid"), "id": str(to_db.get("id"))}
}
db = SessionLocal()
try:
# Find environment IDs by name
src_env = db.query(Environment).filter(Environment.name == from_env).first()
tgt_env = db.query(Environment).filter(Environment.name == to_env).first()
if src_env and tgt_env:
mappings = db.query(DatabaseMapping).filter(
DatabaseMapping.source_env_id == src_env.id,
DatabaseMapping.target_env_id == tgt_env.id
).all()
db_mapping = {m.source_db_uuid: m.target_db_uuid for m in mappings}
logger.info(f"[MigrationPlugin][State] Loaded {len(db_mapping)} database mappings.")
finally:
db.close()
engine = MigrationEngine()
for dash in dashboards_to_migrate:
dash_id, dash_slug, title = dash["id"], dash.get("slug"), dash["dashboard_title"]
@@ -133,18 +144,46 @@ class MigrationPlugin(PluginBase):
try:
exported_content, _ = from_c.export_dashboard(dash_id)
with create_temp_file(content=exported_content, dry_run=True, suffix=".zip", logger=logger) as tmp_zip_path:
if not db_config_replacement:
if not replace_db_config:
to_c.import_dashboard(file_name=tmp_zip_path, dash_id=dash_id, dash_slug=dash_slug)
else:
with create_temp_file(suffix=".dir", logger=logger) as tmp_unpack_dir:
with zipfile.ZipFile(tmp_zip_path, "r") as zip_ref:
zip_ref.extractall(tmp_unpack_dir)
# Check for missing mappings before transformation
# This is a simplified check, in reality we'd check all YAMLs
# For US3, we'll just use the engine and handle missing ones there
with create_temp_file(suffix=".zip", dry_run=True, logger=logger) as tmp_new_zip:
# If we have missing mappings, we might need to pause
# For now, let's assume the engine can tell us what's missing
success = engine.transform_zip(str(tmp_zip_path), str(tmp_new_zip), db_mapping)
update_yamls(db_configs=[db_config_replacement], path=str(tmp_unpack_dir))
if not success:
# Signal missing mapping and wait
task_id = params.get("_task_id")
if task_id:
from ..dependencies import get_task_manager
tm = get_task_manager()
logger.info(f"[MigrationPlugin][Action] Pausing for missing mapping in task {task_id}")
# In a real scenario, we'd pass the missing DB info to the frontend
# For this task, we'll just simulate the wait
await tm.wait_for_resolution(task_id)
# After resolution, retry transformation with updated mappings
# (Mappings would be updated in task.params by resolve_task)
db = SessionLocal()
try:
src_env = db.query(Environment).filter(Environment.name == from_env).first()
tgt_env = db.query(Environment).filter(Environment.name == to_env).first()
mappings = db.query(DatabaseMapping).filter(
DatabaseMapping.source_env_id == src_env.id,
DatabaseMapping.target_env_id == tgt_env.id
).all()
db_mapping = {m.source_db_uuid: m.target_db_uuid for m in mappings}
finally:
db.close()
success = engine.transform_zip(str(tmp_zip_path), str(tmp_new_zip), db_mapping)
with create_temp_file(suffix=".zip", dry_run=True, logger=logger) as tmp_new_zip:
create_dashboard_export(zip_path=tmp_new_zip, source_paths=[str(p) for p in Path(tmp_unpack_dir).glob("**/*")])
if success:
to_c.import_dashboard(file_name=tmp_new_zip, dash_id=dash_id, dash_slug=dash_slug)
else:
logger.error(f"[MigrationPlugin][Failure] Failed to transform ZIP for dashboard {title}")
logger.info(f"[MigrationPlugin][Success] Dashboard {title} imported.")
except Exception as exc:

View File

@@ -0,0 +1,66 @@
# [DEF:backend.src.services.mapping_service:Module]
#
# @SEMANTICS: service, mapping, fuzzy-matching, superset
# @PURPOSE: Orchestrates database fetching and fuzzy matching suggestions.
# @LAYER: Service
# @RELATION: DEPENDS_ON -> backend.src.core.superset_client
# @RELATION: DEPENDS_ON -> backend.src.core.utils.matching
#
# @INVARIANT: Suggestions are based on database names.
# [SECTION: IMPORTS]
from typing import List, Dict
from backend.src.core.superset_client import SupersetClient
from backend.src.core.utils.matching import suggest_mappings
from superset_tool.models import SupersetConfig
# [/SECTION]
# [DEF:MappingService:Class]
# @PURPOSE: Service for handling database mapping logic.
class MappingService:
# [DEF:MappingService.__init__:Function]
def __init__(self, config_manager):
self.config_manager = config_manager
# [DEF:MappingService._get_client:Function]
# @PURPOSE: Helper to get an initialized SupersetClient for an environment.
def _get_client(self, env_id: str) -> SupersetClient:
envs = self.config_manager.get_environments()
env = next((e for e in envs if e.id == env_id), None)
if not env:
raise ValueError(f"Environment {env_id} not found")
superset_config = SupersetConfig(
env=env.name,
base_url=env.url,
auth={
"provider": "db",
"username": env.username,
"password": env.password,
"refresh": "false"
}
)
return SupersetClient(superset_config)
# [DEF:MappingService.get_suggestions:Function]
# @PURPOSE: Fetches databases from both environments and returns fuzzy matching suggestions.
# @PARAM: source_env_id (str) - Source environment ID.
# @PARAM: target_env_id (str) - Target environment ID.
# @RETURN: List[Dict] - Suggested mappings.
async def get_suggestions(self, source_env_id: str, target_env_id: str) -> List[Dict]:
"""
Get suggested mappings between two environments.
"""
source_client = self._get_client(source_env_id)
target_client = self._get_client(target_env_id)
source_dbs = source_client.get_databases_summary()
target_dbs = target_client.get_databases_summary()
return suggest_mappings(source_dbs, target_dbs)
# [/DEF:MappingService.get_suggestions]
# [/DEF:MappingService]
# [/DEF:backend.src.services.mapping_service]

42
docs/migration_mapping.md Normal file
View File

@@ -0,0 +1,42 @@
# Database Mapping in Migration
This document describes how to use the database mapping feature during Superset dashboard migrations.
## Overview
When migrating dashboards between different Superset environments (e.g., from Dev to Prod), the underlying databases often have different UUIDs even if they represent the same data source. The Database Mapping feature allows you to define these relationships so that migrated assets automatically point to the correct database in the target environment.
## How it Works
1. **Fuzzy Matching**: The system automatically suggests mappings by comparing database names between environments using the RapidFuzz library.
2. **Persistence**: Mappings are stored in a local SQLite database (`mappings.db`) and are reused for future migrations between the same environment pair.
3. **Asset Interception**: During migration, the system intercepts the Superset export ZIP archive, modifies the `database_uuid` in the dataset YAML files, and re-packages the archive before importing it to the target.
## Usage Instructions
### 1. Define Mappings
1. Navigate to the **Database Mapping** tab in the application.
2. Select your **Source** and **Target** environments.
3. Click **Fetch Databases & Suggestions**.
4. Review the suggested mappings (highlighted in green).
5. If a suggestion is incorrect or missing, use the dropdown in the "Target Database" column to select the correct one.
6. Mappings are saved automatically when you select a target database.
### 2. Run Migration with Database Replacement
1. Go to the **Migration** dashboard.
2. Select the **Source** and **Target** environments.
3. Select the dashboards or datasets you want to migrate.
4. Enable the **Replace Database (Apply Mappings)** toggle.
5. Click **Start Migration**.
### 3. Handling Missing Mappings
If the migration engine encounters a database that has no defined mapping, the process will pause, and a modal will appear prompting you to select a target database on-the-fly. Once selected, the mapping is saved, and the migration continues.
## Troubleshooting
- **Mapping not applied**: Ensure the "Replace Database" toggle is enabled.
- **Wrong database in target**: Check the mapping table for the specific environment pair and correct any errors.
- **Connection errors**: Ensure both Superset environments are reachable and credentials are correct in Settings.

View File

@@ -4,14 +4,18 @@ export const nodes = [
() => import('./nodes/0'),
() => import('./nodes/1'),
() => import('./nodes/2'),
() => import('./nodes/3')
() => import('./nodes/3'),
() => import('./nodes/4'),
() => import('./nodes/5')
];
export const server_loads = [];
export const dictionary = {
"/": [2],
"/settings": [3]
"/migration": [3],
"/migration/mappings": [4],
"/settings": [5]
};
export const hooks = {

View File

@@ -1,3 +1 @@
import * as universal from "../../../../src/routes/settings/+page.ts";
export { universal };
export { default as component } from "../../../../src/routes/settings/+page.svelte";
export { default as component } from "../../../../src/routes/migration/+page.svelte";

View File

@@ -24,7 +24,7 @@ export const options = {
app: ({ head, body, assets, nonce, env }) => "<!DOCTYPE html>\n<html lang=\"en\">\n\t<head>\n\t\t<meta charset=\"utf-8\" />\n\t\t<link rel=\"icon\" href=\"" + assets + "/favicon.png\" />\n\t\t<meta name=\"viewport\" content=\"width=device-width, initial-scale=1\" />\n\t\t" + head + "\n\t</head>\n\t<body data-sveltekit-preload-data=\"hover\">\n\t\t<div style=\"display: contents\">" + body + "</div>\n\t</body>\n</html>\n",
error: ({ status, message }) => "<!doctype html>\n<html lang=\"en\">\n\t<head>\n\t\t<meta charset=\"utf-8\" />\n\t\t<title>" + message + "</title>\n\n\t\t<style>\n\t\t\tbody {\n\t\t\t\t--bg: white;\n\t\t\t\t--fg: #222;\n\t\t\t\t--divider: #ccc;\n\t\t\t\tbackground: var(--bg);\n\t\t\t\tcolor: var(--fg);\n\t\t\t\tfont-family:\n\t\t\t\t\tsystem-ui,\n\t\t\t\t\t-apple-system,\n\t\t\t\t\tBlinkMacSystemFont,\n\t\t\t\t\t'Segoe UI',\n\t\t\t\t\tRoboto,\n\t\t\t\t\tOxygen,\n\t\t\t\t\tUbuntu,\n\t\t\t\t\tCantarell,\n\t\t\t\t\t'Open Sans',\n\t\t\t\t\t'Helvetica Neue',\n\t\t\t\t\tsans-serif;\n\t\t\t\tdisplay: flex;\n\t\t\t\talign-items: center;\n\t\t\t\tjustify-content: center;\n\t\t\t\theight: 100vh;\n\t\t\t\tmargin: 0;\n\t\t\t}\n\n\t\t\t.error {\n\t\t\t\tdisplay: flex;\n\t\t\t\talign-items: center;\n\t\t\t\tmax-width: 32rem;\n\t\t\t\tmargin: 0 1rem;\n\t\t\t}\n\n\t\t\t.status {\n\t\t\t\tfont-weight: 200;\n\t\t\t\tfont-size: 3rem;\n\t\t\t\tline-height: 1;\n\t\t\t\tposition: relative;\n\t\t\t\ttop: -0.05rem;\n\t\t\t}\n\n\t\t\t.message {\n\t\t\t\tborder-left: 1px solid var(--divider);\n\t\t\t\tpadding: 0 0 0 1rem;\n\t\t\t\tmargin: 0 0 0 1rem;\n\t\t\t\tmin-height: 2.5rem;\n\t\t\t\tdisplay: flex;\n\t\t\t\talign-items: center;\n\t\t\t}\n\n\t\t\t.message h1 {\n\t\t\t\tfont-weight: 400;\n\t\t\t\tfont-size: 1em;\n\t\t\t\tmargin: 0;\n\t\t\t}\n\n\t\t\t@media (prefers-color-scheme: dark) {\n\t\t\t\tbody {\n\t\t\t\t\t--bg: #222;\n\t\t\t\t\t--fg: #ddd;\n\t\t\t\t\t--divider: #666;\n\t\t\t\t}\n\t\t\t}\n\t\t</style>\n\t</head>\n\t<body>\n\t\t<div class=\"error\">\n\t\t\t<span class=\"status\">" + status + "</span>\n\t\t\t<div class=\"message\">\n\t\t\t\t<h1>" + message + "</h1>\n\t\t\t</div>\n\t\t</div>\n\t</body>\n</html>\n"
},
version_hash: "1eogxsl"
version_hash: "1pvaiah"
};
export async function get_hooks() {

View File

@@ -27,15 +27,17 @@ export {};
declare module "$app/types" {
export interface AppTypes {
RouteId(): "/" | "/settings";
RouteId(): "/" | "/migration" | "/migration/mappings" | "/settings";
RouteParams(): {
};
LayoutParams(): {
"/": Record<string, never>;
"/migration": Record<string, never>;
"/migration/mappings": Record<string, never>;
"/settings": Record<string, never>
};
Pathname(): "/" | "/settings" | "/settings/";
Pathname(): "/" | "/migration" | "/migration/" | "/migration/mappings" | "/migration/mappings/" | "/settings" | "/settings/";
ResolvedPathname(): `${"" | `/${string}`}${ReturnType<AppTypes['Pathname']>}`;
Asset(): string & {};
}

View File

@@ -0,0 +1,57 @@
<!-- [DEF:EnvSelector:Component] -->
<!--
@SEMANTICS: environment, selector, dropdown, migration
@PURPOSE: Provides a UI component for selecting source and target environments.
@LAYER: Feature
@RELATION: BINDS_TO -> environments store
@INVARIANT: Source and target environments must be selectable from the list of configured environments.
-->
<script lang="ts">
// [SECTION: IMPORTS]
import { onMount, createEventDispatcher } from 'svelte';
// [/SECTION]
// [SECTION: PROPS]
export let label: string = "Select Environment";
export let selectedId: string = "";
export let environments: Array<{id: string, name: string, url: string}> = [];
// [/SECTION]
const dispatch = createEventDispatcher();
// [DEF:handleSelect:Function]
/**
* @purpose Dispatches the selection change event.
* @param {Event} event - The change event from the select element.
*/
function handleSelect(event: Event) {
const target = event.target as HTMLSelectElement;
selectedId = target.value;
dispatch('change', { id: selectedId });
}
// [/DEF:handleSelect]
</script>
<!-- [SECTION: TEMPLATE] -->
<div class="flex flex-col space-y-1">
<label class="text-sm font-medium text-gray-700">{label}</label>
<select
class="block w-full pl-3 pr-10 py-2 text-base border-gray-300 focus:outline-none focus:ring-indigo-500 focus:border-indigo-500 sm:text-sm rounded-md"
value={selectedId}
on:change={handleSelect}
>
<option value="" disabled>-- Choose an environment --</option>
{#each environments as env}
<option value={env.id}>{env.name} ({env.url})</option>
{/each}
</select>
</div>
<!-- [/SECTION] -->
<style>
/* Component specific styles */
</style>
<!-- [/DEF:EnvSelector] -->

View File

@@ -0,0 +1,94 @@
<!-- [DEF:MappingTable:Component] -->
<!--
@SEMANTICS: mapping, table, database, editor
@PURPOSE: Displays and allows editing of database mappings.
@LAYER: Feature
@RELATION: BINDS_TO -> mappings state
@INVARIANT: Each source database can be mapped to one target database.
-->
<script lang="ts">
// [SECTION: IMPORTS]
import { createEventDispatcher } from 'svelte';
// [/SECTION]
// [SECTION: PROPS]
export let sourceDatabases: Array<{uuid: string, database_name: string}> = [];
export let targetDatabases: Array<{uuid: string, database_name: string}> = [];
export let mappings: Array<{source_db_uuid: string, target_db_uuid: string}> = [];
export let suggestions: Array<{source_db_uuid: string, target_db_uuid: string, confidence: number}> = [];
// [/SECTION]
const dispatch = createEventDispatcher();
// [DEF:updateMapping:Function]
/**
* @purpose Updates a mapping for a specific source database.
*/
function updateMapping(sourceUuid: string, targetUuid: string) {
dispatch('update', { sourceUuid, targetUuid });
}
// [/DEF:updateMapping]
// [DEF:getSuggestion:Function]
/**
* @purpose Finds a suggestion for a source database.
*/
function getSuggestion(sourceUuid: string) {
return suggestions.find(s => s.source_db_uuid === sourceUuid);
}
// [/DEF:getSuggestion]
</script>
<!-- [SECTION: TEMPLATE] -->
<div class="overflow-x-auto">
<table class="min-w-full divide-y divide-gray-200">
<thead class="bg-gray-50">
<tr>
<th class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">Source Database</th>
<th class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">Target Database</th>
<th class="px-6 py-3 text-left text-xs font-medium text-gray-500 uppercase tracking-wider">Status</th>
</tr>
</thead>
<tbody class="bg-white divide-y divide-gray-200">
{#each sourceDatabases as sDb}
{@const mapping = mappings.find(m => m.source_db_uuid === sDb.uuid)}
{@const suggestion = getSuggestion(sDb.uuid)}
<tr class={suggestion && !mapping ? 'bg-green-50' : ''}>
<td class="px-6 py-4 whitespace-nowrap text-sm font-medium text-gray-900">
{sDb.database_name}
</td>
<td class="px-6 py-4 whitespace-nowrap text-sm text-gray-500">
<select
class="block w-full pl-3 pr-10 py-2 text-base border-gray-300 focus:outline-none focus:ring-indigo-500 focus:border-indigo-500 sm:text-sm rounded-md"
value={mapping?.target_db_uuid || suggestion?.target_db_uuid || ""}
on:change={(e) => updateMapping(sDb.uuid, (e.target as HTMLSelectElement).value)}
>
<option value="">-- Select Target --</option>
{#each targetDatabases as tDb}
<option value={tDb.uuid}>{tDb.database_name}</option>
{/each}
</select>
</td>
<td class="px-6 py-4 whitespace-nowrap text-sm text-gray-500">
{#if mapping}
<span class="text-blue-600 font-semibold">Saved</span>
{:else if suggestion}
<span class="text-green-600 font-semibold">Suggested ({Math.round(suggestion.confidence * 100)}%)</span>
{:else}
<span class="text-red-600">Unmapped</span>
{/if}
</td>
</tr>
{/each}
</tbody>
</table>
</div>
<!-- [/SECTION] -->
<style>
/* Component specific styles */
</style>
<!-- [/DEF:MappingTable] -->

View File

@@ -0,0 +1,112 @@
<!-- [DEF:MissingMappingModal:Component] -->
<!--
@SEMANTICS: modal, mapping, prompt, migration
@PURPOSE: Prompts the user to provide a database mapping when one is missing during migration.
@LAYER: Feature
@RELATION: DISPATCHES -> resolve
@INVARIANT: Modal blocks migration progress until resolved or cancelled.
-->
<script lang="ts">
// [SECTION: IMPORTS]
import { createEventDispatcher } from 'svelte';
// [/SECTION]
// [SECTION: PROPS]
export let show: boolean = false;
export let sourceDbName: string = "";
export let sourceDbUuid: string = "";
export let targetDatabases: Array<{uuid: string, database_name: string}> = [];
// [/SECTION]
let selectedTargetUuid = "";
const dispatch = createEventDispatcher();
// [DEF:resolve:Function]
function resolve() {
if (!selectedTargetUuid) return;
dispatch('resolve', {
sourceDbUuid,
targetDbUuid: selectedTargetUuid,
targetDbName: targetDatabases.find(d => d.uuid === selectedTargetUuid)?.database_name
});
show = false;
}
// [/DEF:resolve]
// [DEF:cancel:Function]
function cancel() {
dispatch('cancel');
show = false;
}
// [/DEF:cancel]
</script>
<!-- [SECTION: TEMPLATE] -->
{#if show}
<div class="fixed z-10 inset-0 overflow-y-auto" aria-labelledby="modal-title" role="dialog" aria-modal="true">
<div class="flex items-end justify-center min-h-screen pt-4 px-4 pb-20 text-center sm:block sm:p-0">
<div class="fixed inset-0 bg-gray-500 bg-opacity-75 transition-opacity" aria-hidden="true"></div>
<span class="hidden sm:inline-block sm:align-middle sm:h-screen" aria-hidden="true">&#8203;</span>
<div class="inline-block align-bottom bg-white rounded-lg px-4 pt-5 pb-4 text-left overflow-hidden shadow-xl transform transition-all sm:my-8 sm:align-middle sm:max-w-lg sm:w-full sm:p-6">
<div>
<div class="mx-auto flex items-center justify-center h-12 w-12 rounded-full bg-yellow-100">
<svg class="h-6 w-6 text-yellow-600" xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke="currentColor">
<path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M12 9v2m0 4h.01m-6.938 4h13.856c1.54 0 2.502-1.667 1.732-3L13.732 4c-.77-1.333-2.694-1.333-3.464 0L3.34 16c-.77 1.333.192 3 1.732 3z" />
</svg>
</div>
<div class="mt-3 text-center sm:mt-5">
<h3 class="text-lg leading-6 font-medium text-gray-900" id="modal-title">
Missing Database Mapping
</h3>
<div class="mt-2">
<p class="text-sm text-gray-500">
The database <strong>{sourceDbName}</strong> is used in the assets being migrated but has no mapping to the target environment. Please select a target database.
</p>
</div>
</div>
</div>
<div class="mt-5 sm:mt-6">
<select
class="block w-full pl-3 pr-10 py-2 text-base border-gray-300 focus:outline-none focus:ring-indigo-500 focus:border-indigo-500 sm:text-sm rounded-md"
bind:value={selectedTargetUuid}
>
<option value="" disabled>-- Select Target Database --</option>
{#each targetDatabases as tDb}
<option value={tDb.uuid}>{tDb.database_name}</option>
{/each}
</select>
</div>
<div class="mt-5 sm:mt-6 sm:grid sm:grid-cols-2 sm:gap-3 sm:grid-flow-row-dense">
<button
type="button"
on:click={resolve}
disabled={!selectedTargetUuid}
class="w-full inline-flex justify-center rounded-md border border-transparent shadow-sm px-4 py-2 bg-indigo-600 text-base font-medium text-white hover:bg-indigo-700 focus:outline-none focus:ring-2 focus:ring-offset-2 focus:ring-indigo-500 sm:col-start-2 sm:text-sm disabled:bg-gray-400"
>
Apply & Continue
</button>
<button
type="button"
on:click={cancel}
class="mt-3 w-full inline-flex justify-center rounded-md border border-gray-300 shadow-sm px-4 py-2 bg-white text-base font-medium text-gray-700 hover:bg-gray-50 focus:outline-none focus:ring-2 focus:ring-offset-2 focus:ring-indigo-500 sm:mt-0 sm:col-start-1 sm:text-sm"
>
Cancel Migration
</button>
</div>
</div>
</div>
</div>
{/if}
<!-- [/SECTION] -->
<style>
/* Modal specific styles */
</style>
<!-- [/DEF:MissingMappingModal] -->

View File

@@ -15,6 +15,7 @@
import { selectedTask, taskLogs } from '../lib/stores.js';
import { getWsUrl } from '../lib/api.js';
import { addToast } from '../lib/toasts.js';
import MissingMappingModal from './MissingMappingModal.svelte';
// [/SECTION]
let ws;
@@ -25,7 +26,10 @@
let reconnectTimeout;
let waitingForData = false;
let dataTimeout;
let connectionStatus = 'disconnected'; // 'connecting', 'connected', 'disconnected', 'waiting', 'completed'
let connectionStatus = 'disconnected'; // 'connecting', 'connected', 'disconnected', 'waiting', 'completed', 'awaiting_mapping'
let showMappingModal = false;
let missingDbInfo = { name: '', uuid: '' };
let targetDatabases = [];
// [DEF:connect:Function]
/**
@@ -58,6 +62,17 @@
connectionStatus = 'completed';
ws.close();
}
// Check for missing mapping signal
if (logEntry.message && logEntry.message.includes('Missing mapping for database UUID')) {
const uuidMatch = logEntry.message.match(/UUID: ([\w-]+)/);
if (uuidMatch) {
missingDbInfo = { name: 'Unknown', uuid: uuidMatch[1] };
connectionStatus = 'awaiting_mapping';
fetchTargetDatabases();
showMappingModal = true;
}
}
};
ws.onerror = (error) => {
@@ -85,6 +100,65 @@
}
// [/DEF:connect]
async function fetchTargetDatabases() {
const task = get(selectedTask);
if (!task || !task.params.to_env) return;
try {
// We need to find the environment ID by name first
const envsRes = await fetch('/api/environments');
const envs = await envsRes.json();
const targetEnv = envs.find(e => e.name === task.params.to_env);
if (targetEnv) {
const res = await fetch(`/api/environments/${targetEnv.id}/databases`);
targetDatabases = await res.json();
}
} catch (e) {
console.error('Failed to fetch target databases', e);
}
}
async function handleMappingResolve(event) {
const task = get(selectedTask);
const { sourceDbUuid, targetDbUuid, targetDbName } = event.detail;
try {
// 1. Save mapping to backend
const envsRes = await fetch('/api/environments');
const envs = await envsRes.json();
const srcEnv = envs.find(e => e.name === task.params.from_env);
const tgtEnv = envs.find(e => e.name === task.params.to_env);
await fetch('/api/mappings', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
source_env_id: srcEnv.id,
target_env_id: tgtEnv.id,
source_db_uuid: sourceDbUuid,
target_db_uuid: targetDbUuid,
source_db_name: missingDbInfo.name,
target_db_name: targetDbName
})
});
// 2. Resolve task
await fetch(`/api/tasks/${task.id}/resolve`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
resolution_params: { resolved_mapping: { [sourceDbUuid]: targetDbUuid } }
})
});
connectionStatus = 'connected';
addToast('Mapping resolved, migration continuing...', 'success');
} catch (e) {
addToast('Failed to resolve mapping: ' + e.message, 'error');
}
}
function startDataTimeout() {
waitingForData = false;
dataTimeout = setTimeout(() => {
@@ -151,6 +225,9 @@
{:else if connectionStatus === 'completed'}
<span class="h-3 w-3 rounded-full bg-blue-500"></span>
<span class="text-xs text-gray-500">Completed</span>
{:else if connectionStatus === 'awaiting_mapping'}
<span class="h-3 w-3 rounded-full bg-orange-500 animate-pulse"></span>
<span class="text-xs text-gray-500">Awaiting Mapping</span>
{:else}
<span class="h-3 w-3 rounded-full bg-red-500"></span>
<span class="text-xs text-gray-500">Disconnected</span>
@@ -177,6 +254,15 @@
<p>No task selected.</p>
{/if}
</div>
<MissingMappingModal
bind:show={showMappingModal}
sourceDbName={missingDbInfo.name}
sourceDbUuid={missingDbInfo.uuid}
{targetDatabases}
on:resolve={handleMappingResolve}
on:cancel={() => { connectionStatus = 'disconnected'; ws.close(); }}
/>
<!-- [/SECTION] -->
<!-- [/DEF:TaskRunner] -->

View File

@@ -4,6 +4,7 @@
import DynamicForm from '../components/DynamicForm.svelte';
import { api } from '../lib/api.js';
import { get } from 'svelte/store';
import { goto } from '$app/navigation';
/** @type {import('./$types').PageData} */
export let data;
@@ -15,7 +16,11 @@
function selectPlugin(plugin) {
console.log(`[Dashboard][Action] Selecting plugin: ${plugin.id}`);
selectedPlugin.set(plugin);
if (plugin.id === 'superset-migration') {
goto('/migration');
} else {
selectedPlugin.set(plugin);
}
}
async function handleFormSubmit(event) {

View File

@@ -0,0 +1,132 @@
<!-- [DEF:MigrationDashboard:Component] -->
<!--
@SEMANTICS: migration, dashboard, environment, selection, database-replacement
@PURPOSE: Main dashboard for configuring and starting migrations.
@LAYER: Page
@RELATION: USES -> EnvSelector
@INVARIANT: Migration cannot start without source and target environments.
-->
<script lang="ts">
// [SECTION: IMPORTS]
import { onMount } from 'svelte';
import EnvSelector from '../../components/EnvSelector.svelte';
// [/SECTION]
// [SECTION: STATE]
let environments = [];
let sourceEnvId = "";
let targetEnvId = "";
let dashboardRegex = ".*";
let replaceDb = false;
let loading = true;
let error = "";
// [/SECTION]
// [DEF:fetchEnvironments:Function]
/**
* @purpose Fetches the list of environments from the API.
* @post environments state is updated.
*/
async function fetchEnvironments() {
try {
const response = await fetch('/api/environments');
if (!response.ok) throw new Error('Failed to fetch environments');
environments = await response.json();
} catch (e) {
error = e.message;
} finally {
loading = false;
}
}
// [/DEF:fetchEnvironments]
onMount(fetchEnvironments);
// [DEF:startMigration:Function]
/**
* @purpose Starts the migration process.
* @pre sourceEnvId and targetEnvId must be set and different.
*/
async function startMigration() {
if (!sourceEnvId || !targetEnvId) {
error = "Please select both source and target environments.";
return;
}
if (sourceEnvId === targetEnvId) {
error = "Source and target environments must be different.";
return;
}
error = "";
console.log(`[MigrationDashboard][Action] Starting migration from ${sourceEnvId} to ${targetEnvId} (Replace DB: ${replaceDb})`);
// TODO: Implement actual migration trigger in US3
}
// [/DEF:startMigration]
</script>
<!-- [SECTION: TEMPLATE] -->
<div class="max-w-4xl mx-auto p-6">
<h1 class="text-2xl font-bold mb-6">Migration Dashboard</h1>
{#if loading}
<p>Loading environments...</p>
{:else if error}
<div class="bg-red-100 border border-red-400 text-red-700 px-4 py-3 rounded mb-4">
{error}
</div>
{/if}
<div class="grid grid-cols-1 md:grid-cols-2 gap-6 mb-8">
<EnvSelector
label="Source Environment"
bind:selectedId={sourceEnvId}
{environments}
/>
<EnvSelector
label="Target Environment"
bind:selectedId={targetEnvId}
{environments}
/>
</div>
<div class="mb-8">
<label for="dashboard-regex" class="block text-sm font-medium text-gray-700 mb-1">Dashboard Regex</label>
<input
id="dashboard-regex"
type="text"
bind:value={dashboardRegex}
placeholder="e.g. ^Finance Dashboard$"
class="shadow-sm focus:ring-indigo-500 focus:border-indigo-500 block w-full sm:text-sm border-gray-300 rounded-md"
/>
<p class="mt-1 text-sm text-gray-500">Regular expression to filter dashboards to migrate.</p>
</div>
<div class="flex items-center mb-8">
<input
id="replace-db"
type="checkbox"
bind:checked={replaceDb}
class="h-4 w-4 text-indigo-600 focus:ring-indigo-500 border-gray-300 rounded"
/>
<label for="replace-db" class="ml-2 block text-sm text-gray-900">
Replace Database (Apply Mappings)
</label>
</div>
<button
on:click={startMigration}
disabled={!sourceEnvId || !targetEnvId || sourceEnvId === targetEnvId}
class="inline-flex items-center px-4 py-2 border border-transparent text-sm font-medium rounded-md shadow-sm text-white bg-indigo-600 hover:bg-indigo-700 focus:outline-none focus:ring-2 focus:ring-offset-2 focus:ring-indigo-500 disabled:bg-gray-400"
>
Start Migration
</button>
</div>
<!-- [/SECTION] -->
<style>
/* Page specific styles */
</style>
<!-- [/DEF:MigrationDashboard] -->

View File

@@ -0,0 +1,183 @@
<!-- [DEF:MappingManagement:Component] -->
<!--
@SEMANTICS: mapping, management, database, fuzzy-matching
@PURPOSE: Page for managing database mappings between environments.
@LAYER: Page
@RELATION: USES -> EnvSelector
@RELATION: USES -> MappingTable
@INVARIANT: Mappings are saved to the backend for persistence.
-->
<script lang="ts">
// [SECTION: IMPORTS]
import { onMount } from 'svelte';
import EnvSelector from '../../../components/EnvSelector.svelte';
import MappingTable from '../../../components/MappingTable.svelte';
// [/SECTION]
// [SECTION: STATE]
let environments = [];
let sourceEnvId = "";
let targetEnvId = "";
let sourceDatabases = [];
let targetDatabases = [];
let mappings = [];
let suggestions = [];
let loading = true;
let fetchingDbs = false;
let error = "";
let success = "";
// [/SECTION]
async function fetchEnvironments() {
try {
const response = await fetch('/api/environments');
if (!response.ok) throw new Error('Failed to fetch environments');
environments = await response.json();
} catch (e) {
error = e.message;
} finally {
loading = false;
}
}
onMount(fetchEnvironments);
// [DEF:fetchDatabases:Function]
/**
* @purpose Fetches databases from both environments and gets suggestions.
*/
async function fetchDatabases() {
if (!sourceEnvId || !targetEnvId) return;
fetchingDbs = true;
error = "";
success = "";
try {
const [srcRes, tgtRes, mapRes, sugRes] = await Promise.all([
fetch(`/api/environments/${sourceEnvId}/databases`),
fetch(`/api/environments/${targetEnvId}/databases`),
fetch(`/api/mappings?source_env_id=${sourceEnvId}&target_env_id=${targetEnvId}`),
fetch(`/api/mappings/suggest`, {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ source_env_id: sourceEnvId, target_env_id: targetEnvId })
})
]);
if (!srcRes.ok || !tgtRes.ok) throw new Error('Failed to fetch databases from environments');
sourceDatabases = await srcRes.json();
targetDatabases = await tgtRes.json();
mappings = await mapRes.json();
suggestions = await sugRes.json();
} catch (e) {
error = e.message;
} finally {
fetchingDbs = false;
}
}
// [/DEF:fetchDatabases]
// [DEF:handleUpdate:Function]
/**
* @purpose Saves a mapping to the backend.
*/
async function handleUpdate(event: CustomEvent) {
const { sourceUuid, targetUuid } = event.detail;
const sDb = sourceDatabases.find(d => d.uuid === sourceUuid);
const tDb = targetDatabases.find(d => d.uuid === targetUuid);
if (!sDb || !tDb) return;
try {
const response = await fetch('/api/mappings', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({
source_env_id: sourceEnvId,
target_env_id: targetEnvId,
source_db_uuid: sourceUuid,
target_db_uuid: targetUuid,
source_db_name: sDb.database_name,
target_db_name: tDb.database_name
})
});
if (!response.ok) throw new Error('Failed to save mapping');
const savedMapping = await response.json();
mappings = [...mappings.filter(m => m.source_db_uuid !== sourceUuid), savedMapping];
success = "Mapping saved successfully";
} catch (e) {
error = e.message;
}
}
// [/DEF:handleUpdate]
</script>
<!-- [SECTION: TEMPLATE] -->
<div class="max-w-6xl mx-auto p-6">
<h1 class="text-2xl font-bold mb-6">Database Mapping Management</h1>
{#if loading}
<p>Loading environments...</p>
{:else}
<div class="grid grid-cols-1 md:grid-cols-2 gap-6 mb-8">
<EnvSelector
label="Source Environment"
bind:selectedId={sourceEnvId}
{environments}
on:change={() => { sourceDatabases = []; mappings = []; suggestions = []; }}
/>
<EnvSelector
label="Target Environment"
bind:selectedId={targetEnvId}
{environments}
on:change={() => { targetDatabases = []; mappings = []; suggestions = []; }}
/>
</div>
<div class="mb-8">
<button
on:click={fetchDatabases}
disabled={!sourceEnvId || !targetEnvId || sourceEnvId === targetEnvId || fetchingDbs}
class="inline-flex items-center px-4 py-2 border border-transparent text-sm font-medium rounded-md shadow-sm text-white bg-indigo-600 hover:bg-indigo-700 focus:outline-none focus:ring-2 focus:ring-offset-2 focus:ring-indigo-500 disabled:bg-gray-400"
>
{fetchingDbs ? 'Fetching...' : 'Fetch Databases & Suggestions'}
</button>
</div>
{#if error}
<div class="bg-red-100 border border-red-400 text-red-700 px-4 py-3 rounded mb-4">
{error}
</div>
{/if}
{#if success}
<div class="bg-green-100 border border-green-400 text-green-700 px-4 py-3 rounded mb-4">
{success}
</div>
{/if}
{#if sourceDatabases.length > 0}
<MappingTable
{sourceDatabases}
{targetDatabases}
{mappings}
{suggestions}
on:update={handleUpdate}
/>
{:else if !fetchingDbs && sourceEnvId && targetEnvId}
<p class="text-gray-500 italic">Select environments and click "Fetch Databases" to start mapping.</p>
{/if}
{/if}
</div>
<!-- [/SECTION] -->
<style>
/* Page specific styles */
</style>
<!-- [/DEF:MappingManagement] -->

298
logs/лог.md Executable file
View File

@@ -0,0 +1,298 @@
PS H:\dev\ss-tools> & C:/ProgramData/anaconda3/python.exe h:/dev/ss-tools/migration_script.py
2025-12-16 11:50:28,192 - INFO - [run][Entry] Запуск скрипта миграции.
=== Поведение при ошибке импорта ===
Если импорт завершится ошибкой, удалить существующий дашборд и попытаться импортировать заново? (y/n): n
2025-12-16 11:50:33,363 - INFO - [ask_delete_on_failure][State] Delete-on-failure = False
2025-12-16 11:50:33,368 - INFO - [select_environments][Entry] Шаг 1/5: Выбор окружений.
2025-12-16 11:50:33,374 - INFO - [setup_clients][Enter] Starting Superset clients initialization.
2025-12-16 11:50:33,730 - INFO - [SupersetClient.__init__][Enter] Initializing SupersetClient.
2025-12-16 11:50:33,734 - INFO - [APIClient.__init__][Entry] Initializing APIClient.
2025-12-16 11:50:33,739 - WARNING - [_init_session][State] SSL verification disabled.
2025-12-16 11:50:33,742 - INFO - [APIClient.__init__][Exit] APIClient initialized.
2025-12-16 11:50:33,746 - INFO - [SupersetClient.__init__][Exit] SupersetClient initialized.
2025-12-16 11:50:33,750 - INFO - [SupersetClient.__init__][Enter] Initializing SupersetClient.
2025-12-16 11:50:33,754 - INFO - [APIClient.__init__][Entry] Initializing APIClient.
2025-12-16 11:50:33,758 - WARNING - [_init_session][State] SSL verification disabled.
2025-12-16 11:50:33,761 - INFO - [APIClient.__init__][Exit] APIClient initialized.
2025-12-16 11:50:33,764 - INFO - [SupersetClient.__init__][Exit] SupersetClient initialized.
2025-12-16 11:50:33,769 - INFO - [SupersetClient.__init__][Enter] Initializing SupersetClient.
2025-12-16 11:50:33,772 - INFO - [APIClient.__init__][Entry] Initializing APIClient.
2025-12-16 11:50:33,776 - WARNING - [_init_session][State] SSL verification disabled.
2025-12-16 11:50:33,779 - INFO - [APIClient.__init__][Exit] APIClient initialized.
2025-12-16 11:50:33,782 - INFO - [SupersetClient.__init__][Exit] SupersetClient initialized.
2025-12-16 11:50:33,786 - INFO - [SupersetClient.__init__][Enter] Initializing SupersetClient.
2025-12-16 11:50:33,790 - INFO - [APIClient.__init__][Entry] Initializing APIClient.
2025-12-16 11:50:33,794 - WARNING - [_init_session][State] SSL verification disabled.
2025-12-16 11:50:33,799 - INFO - [APIClient.__init__][Exit] APIClient initialized.
2025-12-16 11:50:33,805 - INFO - [SupersetClient.__init__][Exit] SupersetClient initialized.
2025-12-16 11:50:33,808 - INFO - [SupersetClient.__init__][Enter] Initializing SupersetClient.
2025-12-16 11:50:33,811 - INFO - [APIClient.__init__][Entry] Initializing APIClient.
2025-12-16 11:50:33,815 - WARNING - [_init_session][State] SSL verification disabled.
2025-12-16 11:50:33,820 - INFO - [APIClient.__init__][Exit] APIClient initialized.
2025-12-16 11:50:33,823 - INFO - [SupersetClient.__init__][Exit] SupersetClient initialized.
2025-12-16 11:50:33,827 - INFO - [SupersetClient.__init__][Enter] Initializing SupersetClient.
2025-12-16 11:50:33,831 - INFO - [APIClient.__init__][Entry] Initializing APIClient.
2025-12-16 11:50:33,834 - WARNING - [_init_session][State] SSL verification disabled.
2025-12-16 11:50:33,838 - INFO - [APIClient.__init__][Exit] APIClient initialized.
2025-12-16 11:50:33,840 - INFO - [SupersetClient.__init__][Exit] SupersetClient initialized.
2025-12-16 11:50:33,847 - INFO - [setup_clients][Exit] All clients (dev, prod, sbx, preprod, uatta, dev5) initialized successfully.
=== Выбор окружения ===
Исходное окружение:
1) dev
2) prod
3) sbx
4) preprod
5) uatta
6) dev5
Введите номер (0 отмена): 4
2025-12-16 11:50:42,379 - INFO - [select_environments][State] from = preprod
=== Выбор окружения ===
Целевое окружение:
1) dev
2) prod
3) sbx
4) uatta
5) dev5
Введите номер (0 отмена): 5
2025-12-16 11:50:45,176 - INFO - [select_environments][State] to = dev5
2025-12-16 11:50:45,182 - INFO - [select_environments][Exit] Шаг 1 завершён.
2025-12-16 11:50:45,186 - INFO - [select_dashboards][Entry] Шаг 2/5: Выбор дашбордов.
2025-12-16 11:50:45,190 - INFO - [get_dashboards][Enter] Fetching dashboards.
2025-12-16 11:50:45,197 - INFO - [authenticate][Enter] Authenticating to https://preprodta.bi.dwh.rusal.com/api/v1
2025-12-16 11:50:45,880 - INFO - [authenticate][Exit] Authenticated successfully.
2025-12-16 11:50:46,025 - INFO - [get_dashboards][Exit] Found 95 dashboards.
=== Поиск ===
Введите регулярное выражение для поиска дашбордов:
fi
=== Выбор дашбордов ===
Отметьте нужные дашборды (введите номера):
1) [ALL] Все дашборды
2) [185] FI-0060 Финансы. Налоги. Данные по налогам. Старый
3) [184] FI-0083 Статистика по ДЗ/ПДЗ
4) [187] FI-0081 ПДЗ Казначейство
5) [122] FI-0080 Финансы. Оборотный Капитал ДЗ/КЗ
6) [208] FI-0020 Просроченная дебиторская и кредиторская задолженность в динамике
7) [126] FI-0022 Кредиторская задолженность для казначейства
8) [196] FI-0023 Дебиторская задолженность для казначейства
9) [113] FI-0060 Финансы. Налоги. Данные по налогам.
10) [173] FI-0040 Оборотно-сальдовая ведомость (ОСВ) по контрагентам
11) [174] FI-0021 Дебиторская и кредиторская задолженность по документам
12) [172] FI-0030 Дебиторская задолженность по штрафам
13) [170] FI-0050 Налог на прибыль (ОНА и ОНО)
14) [159] FI-0070 Досье контрагента
Введите номера через запятую (пустой ввод → отказ): 2
2025-12-16 11:50:52,235 - INFO - [select_dashboards][State] Выбрано 1 дашбордов.
2025-12-16 11:50:52,242 - INFO - [select_dashboards][Exit] Шаг 2 завершён.
=== Замена БД ===
Заменить конфигурацию БД в YAMLфайлах? (y/n): y
2025-12-16 11:50:53,808 - INFO - [_select_databases][Entry] Selecting databases from both environments.
2025-12-16 11:50:53,816 - INFO - [get_databases][Enter] Fetching databases.
2025-12-16 11:50:53,918 - INFO - [get_databases][Exit] Found 12 databases.
2025-12-16 11:50:53,923 - INFO - [get_databases][Enter] Fetching databases.
2025-12-16 11:50:53,926 - INFO - [authenticate][Enter] Authenticating to https://dev.bi.dwh.rusal.com/api/v1
2025-12-16 11:50:54,450 - INFO - [authenticate][Exit] Authenticated successfully.
2025-12-16 11:50:54,551 - INFO - [get_databases][Exit] Found 4 databases.
=== Выбор исходной БД ===
Выберите исходную БД:
1) DEV datalab (ID: 9)
2) Prod Greenplum (ID: 7)
3) DEV Clickhouse New (OLD) (ID: 16)
4) Preprod Clickhouse New (ID: 15)
5) DEV Greenplum (ID: 1)
6) Prod Clickhouse Node 1 (ID: 11)
7) Preprod Postgre Superset Internal (ID: 5)
8) Prod Postgre Superset Internal (ID: 28)
9) Prod Clickhouse (ID: 10)
10) Dev Clickhouse (correct) (ID: 14)
11) DEV ClickHouse New (ID: 23)
12) Sandbox Postgre Superset Internal (ID: 12)
Введите номер (0 отмена): 9
2025-12-16 11:51:11,008 - INFO - [get_database][Enter] Fetching database 10.
2025-12-16 11:51:11,038 - INFO - [get_database][Exit] Got database 10.
=== Выбор целевой БД ===
Выберите целевую БД:
1) DEV Greenplum (ID: 2)
2) DEV Clickhouse (ID: 3)
3) DEV ClickHouse New (ID: 4)
4) Dev Postgre Superset Internal (ID: 1)
Введите номер (0 отмена): 2
2025-12-16 11:51:15,559 - INFO - [get_database][Enter] Fetching database 3.
2025-12-16 11:51:15,586 - INFO - [get_database][Exit] Got database 3.
2025-12-16 11:51:15,589 - INFO - [_select_databases][Exit] Selected databases: Без имени -> Без имени
old_db: {'id': 10, 'result': {'allow_ctas': False, 'allow_cvas': False, 'allow_dml': True, 'allow_file_upload': False, 'allow_run_async': False, 'backen
d': 'clickhousedb', 'cache_timeout': None, 'configuration_method': 'sqlalchemy_form', 'database_name': 'Prod Clickhouse', 'driver': 'connect', 'engine_i
nformation': {'disable_ssh_tunneling': False, 'supports_file_upload': False}, 'expose_in_sqllab': True, 'force_ctas_schema': None, 'id': 10, 'impersonat
e_user': False, 'is_managed_externally': False, 'uuid': '97aced68-326a-4094-b381-27980560efa9'}}
2025-12-16 11:51:15,591 - INFO - [confirm_db_config_replacement][State] Replacement set: {'old': {'database_name': None, 'uuid': None, 'id': '10'}, 'new
': {'database_name': None, 'uuid': None, 'id': '3'}}
2025-12-16 11:51:15,594 - INFO - [execute_migration][Entry] Starting migration of 1 dashboards.
=== Миграция... ===
Миграция: FI-0060 Финансы. Налоги. Данные по налогам. Старый (1/1) 0%2025-12-16 11:51:15,598 - INFO - [export_dashboard][Enter] Exporting dashboard 185.
2025-12-16 11:51:16,142 - INFO - [export_dashboard][Exit] Exported dashboard 185 to dashboard_export_20251216T085115.zip.
2025-12-16 11:51:16,205 - INFO - [update_yamls][Enter] Starting YAML configuration update.
2025-12-16 11:51:16,208 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\metadata.yaml
2025-12-16 11:51:16,209 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\charts\FI-0060-01_2787.yaml
2025-12-16 11:51:16,210 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\charts\FI-0060-02-01_2_4030.yaml
2025-12-16 11:51:16,212 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\charts\FI-0060-02-01_4029.yaml
2025-12-16 11:51:16,213 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\charts\FI-0060-02-01_TOTAL2_4036.yaml
2025-12-16 11:51:16,215 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\charts\FI-0060-02-01_TOTAL2_4037.yaml
2025-12-16 11:51:16,216 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\charts\FI-0060-02-01_TOTAL_4028.yaml
2025-12-16 11:51:16,217 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\charts\FI-0060-02-01_ZNODE_ROOT2_4024.yaml
2025-12-16 11:51:16,218 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\charts\FI-0060-02-01_ZNODE_ROOT_4033.yaml
2025-12-16 11:51:16,220 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\charts\FI-0060-02-02_ZFUND-BD2_4021.yaml
2025-12-16 11:51:16,221 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\charts\FI-0060-02-02_ZFUND_4027.yaml
2025-12-16 11:51:16,222 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\charts\FI-0060-02-02_ZFUND_4034.yaml
2025-12-16 11:51:16,224 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\charts\FI-0060-02_ZTAX_4022.yaml
2025-12-16 11:51:16,226 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\charts\FI-0060-02_ZTAX_4035.yaml
2025-12-16 11:51:16,227 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\charts\FI-0060-04-2_4031.yaml
2025-12-16 11:51:16,228 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\charts\FI-0060-05-01_4026.yaml
2025-12-16 11:51:16,230 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\charts\FI-0060-05-01_4032.yaml
2025-12-16 11:51:16,231 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\charts\FI-0060-06_1_4023.yaml
2025-12-16 11:51:16,233 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\charts\FI-0060-06_2_4020.yaml
2025-12-16 11:51:16,234 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\charts\FI-0060_4025.yaml
2025-12-16 11:51:16,236 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\dashboards\FI-0060_185.yaml
2025-12-16 11:51:16,238 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\databases\Prod_Clickhouse_10.yaml
2025-12-16 11:51:16,240 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\datasets\Prod_Clickhouse_10\FI-0000_-_685.yaml
2025-12-16 11:51:16,241 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\datasets\Prod_Clickhouse_10\FI-0060-01-2_zfund_reciever_-_861.yaml
2025-12-16 11:51:16,242 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\datasets\Prod_Clickhouse_10\FI-0060-01_zfund_reciever_click_689.yaml
2025-12-16 11:51:16,244 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\datasets\Prod_Clickhouse_10\FI-0060-02_680.yaml
2025-12-16 11:51:16,245 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\datasets\Prod_Clickhouse_10\FI-0060-03_ztax_862.yaml
2025-12-16 11:51:16,246 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\datasets\Prod_Clickhouse_10\FI-0060-04_zpbe_681.yaml
2025-12-16 11:51:16,247 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\datasets\Prod_Clickhouse_10\FI-0060-05_ZTAXZFUND_679.yaml
2025-12-16 11:51:16,249 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\datasets\Prod_Clickhouse_10\FI-0060-06_860.yaml
2025-12-16 11:51:16,250 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\datasets\Prod_Clickhouse_10\FI-0060-08_682.yaml
2025-12-16 11:51:16,251 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\datasets\Prod_Clickhouse_10\FI-0060-10_zpbe_688.yaml
2025-12-16 11:51:16,253 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\datasets\Prod_Clickhouse_10\FI-0060-11_ZTAX_NAME_863.yaml
2025-12-16 11:51:16,254 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\datasets\Prod_Clickhouse_10\FI-0060_683.yaml
2025-12-16 11:51:16,255 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\datasets\Prod_Clickhouse_10\FI-0060_684.yaml
2025-12-16 11:51:16,256 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\datasets\Prod_Clickhouse_10\FI-0060_686.yaml
2025-12-16 11:51:16,258 - INFO - [_update_yaml_file][State] Replaced '10' with '3' for key id in C:\Users\LO54FB~1\Temp\tmpuidfegpd.dir\dashboard_export
_20251216T085115\datasets\Prod_Clickhouse_10\FI-0060_690.yaml
2025-12-16 11:51:16,259 - INFO - [create_dashboard_export][Enter] Packing dashboard: ['C:\\Users\\LO54FB~1\\Temp\\tmpuidfegpd.dir'] -> C:\Users\LO54FB~1
\Temp\tmps7cuv2ti.zip
2025-12-16 11:51:16,347 - INFO - [create_dashboard_export][Exit] Archive created: C:\Users\LO54FB~1\Temp\tmps7cuv2ti.zip
2025-12-16 11:51:16,372 - ERROR - [import_dashboard][Failure] First import attempt failed: [API_FAILURE] API error during upload: {"errors": [{"message"
: "Expecting value: line 1 column 1 (char 0)", "error_type": "GENERIC_BACKEND_ERROR", "level": "error", "extra": {"issue_codes": [{"code": 1011, "messag
e": "Issue 1011 - \u041f\u0440\u043e\u0438\u0437\u043e\u0448\u043b\u0430 \u043d\u0435\u0438\u0437\u0432\u0435\u0441\u0442\u043d\u0430\u044f \u043e\u0448
\u0438\u0431\u043a\u0430."}]}}]} | Context: {'type': 'api_call'}
Traceback (most recent call last):
File "h:\dev\ss-tools\superset_tool\utils\network.py", line 186, in _perform_upload
response.raise_for_status()
File "C:\ProgramData\anaconda3\Lib\site-packages\requests\models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: INTERNAL SERVER ERROR for url: https://dev.bi.dwh.rusal.com/api/v1/dashboard/import/
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "h:\dev\ss-tools\superset_tool\client.py", line 141, in import_dashboard
return self._do_import(file_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "h:\dev\ss-tools\superset_tool\client.py", line 197, in _do_import
return self.network.upload_file(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "h:\dev\ss-tools\superset_tool\utils\network.py", line 172, in upload_file
return self._perform_upload(full_url, files_payload, extra_data, _headers, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "h:\dev\ss-tools\superset_tool\utils\network.py", line 196, in _perform_upload
raise SupersetAPIError(f"API error during upload: {e.response.text}") from e
superset_tool.exceptions.SupersetAPIError: [API_FAILURE] API error during upload: {"errors": [{"message": "Expecting value: line 1 column 1 (char 0)", "
error_type": "GENERIC_BACKEND_ERROR", "level": "error", "extra": {"issue_codes": [{"code": 1011, "message": "Issue 1011 - \u041f\u0440\u043e\u0438\u0437
\u043e\u0448\u043b\u0430 \u043d\u0435\u0438\u0437\u0432\u0435\u0441\u0442\u043d\u0430\u044f \u043e\u0448\u0438\u0431\u043a\u0430."}]}}]} | Context: {'ty
pe': 'api_call'}
2025-12-16 11:51:16,511 - ERROR - [execute_migration][Failure] [API_FAILURE] API error during upload: {"errors": [{"message": "Expecting value: line 1 c
olumn 1 (char 0)", "error_type": "GENERIC_BACKEND_ERROR", "level": "error", "extra": {"issue_codes": [{"code": 1011, "message": "Issue 1011 - \u041f\u04
40\u043e\u0438\u0437\u043e\u0448\u043b\u0430 \u043d\u0435\u0438\u0437\u0432\u0435\u0441\u0442\u043d\u0430\u044f \u043e\u0448\u0438\u0431\u043a\u0430."}]
}}]} | Context: {'type': 'api_call'}
Traceback (most recent call last):
File "h:\dev\ss-tools\superset_tool\utils\network.py", line 186, in _perform_upload
response.raise_for_status()
File "C:\ProgramData\anaconda3\Lib\site-packages\requests\models.py", line 1021, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: INTERNAL SERVER ERROR for url: https://dev.bi.dwh.rusal.com/api/v1/dashboard/import/
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "h:\dev\ss-tools\migration_script.py", line 366, in execute_migration
self.to_c.import_dashboard(file_name=tmp_new_zip, dash_id=dash_id, dash_slug=dash_slug)
File "h:\dev\ss-tools\superset_tool\client.py", line 141, in import_dashboard
return self._do_import(file_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "h:\dev\ss-tools\superset_tool\client.py", line 197, in _do_import
return self.network.upload_file(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "h:\dev\ss-tools\superset_tool\utils\network.py", line 172, in upload_file
return self._perform_upload(full_url, files_payload, extra_data, _headers, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "h:\dev\ss-tools\superset_tool\utils\network.py", line 196, in _perform_upload
raise SupersetAPIError(f"API error during upload: {e.response.text}") from e
superset_tool.exceptions.SupersetAPIError: [API_FAILURE] API error during upload: {"errors": [{"message": "Expecting value: line 1 column 1 (char 0)", "
error_type": "GENERIC_BACKEND_ERROR", "level": "error", "extra": {"issue_codes": [{"code": 1011, "message": "Issue 1011 - \u041f\u0440\u043e\u0438\u0437
\u043e\u0448\u043b\u0430 \u043d\u0435\u0438\u0437\u0432\u0435\u0441\u0442\u043d\u0430\u044f \u043e\u0448\u0438\u0431\u043a\u0430."}]}}]} | Context: {'ty
pe': 'api_call'}
=== Ошибка ===
Не удалось мигрировать дашборд FI-0060 Финансы. Налоги. Данные по налогам. Старый.
[API_FAILURE] API error during upload: {"errors": [{"message": "Expecting value: line 1 column 1 (char 0)", "error_type": "GENERIC_BACKEND_ERROR", "leve
l": "error", "extra": {"issue_codes": [{"code": 1011, "message": "Issue 1011 - \u041f\u0440\u043e\u0438\u0437\u043e\u0448\u043b\u0430 \u043d\u0435\u0438
\u0437\u0432\u0435\u0441\u0442\u043d\u0430\u044f \u043e\u0448\u0438\u0431\u043a\u0430."}]}}]} | Context: {'type': 'api_call'}
100%
2025-12-16 11:51:16,598 - INFO - [execute_migration][Exit] Migration finished.
=== Информация ===
Миграция завершена!
2025-12-16 11:51:16,605 - INFO - [run][Exit] Скрипт миграции завершён.

View File

@@ -0,0 +1,34 @@
# Specification Quality Checklist: Migration Process and UI Redesign
**Purpose**: Validate specification completeness and quality before proceeding to planning
**Created**: 2025-12-20
**Feature**: [specs/001-migration-ui-redesign/spec.md](specs/001-migration-ui-redesign/spec.md)
## Content Quality
- [x] No implementation details (languages, frameworks, APIs)
- [x] Focused on user value and business needs
- [x] Written for non-technical stakeholders
- [x] All mandatory sections completed
## Requirement Completeness
- [x] No [NEEDS CLARIFICATION] markers remain
- [x] Requirements are testable and unambiguous
- [x] Success criteria are measurable
- [x] Success criteria are technology-agnostic (no implementation details)
- [x] All acceptance scenarios are defined
- [x] Edge cases are identified
- [x] Scope is clearly bounded
- [x] Dependencies and assumptions identified
## Feature Readiness
- [x] All functional requirements have clear acceptance criteria
- [x] User scenarios cover primary flows
- [x] Feature meets measurable outcomes defined in Success Criteria
- [x] No implementation details leak into specification
## Notes
- Items marked incomplete require spec updates before `/speckit.clarify` or `/speckit.plan`

View File

@@ -0,0 +1,115 @@
# API Contracts: Migration Process and UI Redesign
## Environment Management
### GET /api/environments
List all configured environments.
**Response (200 OK)**:
```json
[
{
"id": "uuid",
"name": "Development",
"url": "https://superset-dev.example.com"
}
]
```
### GET /api/environments/{id}/databases
Fetch the list of databases from a specific environment.
**Response (200 OK)**:
```json
[
{
"uuid": "db-uuid",
"database_name": "Dev Clickhouse",
"engine": "clickhouse"
}
]
```
## Database Mapping
### GET /api/mappings
List all saved database mappings.
**Query Parameters**:
- `source_env_id`: Filter by source environment.
- `target_env_id`: Filter by target environment.
**Response (200 OK)**:
```json
[
{
"id": "uuid",
"source_env_id": "uuid",
"target_env_id": "uuid",
"source_db_uuid": "uuid",
"target_db_uuid": "uuid",
"source_db_name": "Dev Clickhouse",
"target_db_name": "Prod Clickhouse"
}
]
```
### POST /api/mappings
Create or update a database mapping.
**Request Body**:
```json
{
"source_env_id": "uuid",
"target_env_id": "uuid",
"source_db_uuid": "uuid",
"target_db_uuid": "uuid"
}
```
### POST /api/mappings/suggest
Get suggested mappings based on fuzzy matching.
**Request Body**:
```json
{
"source_env_id": "uuid",
"target_env_id": "uuid"
}
```
**Response (200 OK)**:
```json
[
{
"source_db_uuid": "uuid",
"target_db_uuid": "uuid",
"confidence": 0.95
}
]
```
## Migration Execution
### POST /api/migrations
Start a migration job.
**Request Body**:
```json
{
"source_env_id": "uuid",
"target_env_id": "uuid",
"assets": [
{"type": "dashboard", "id": 123}
],
"replace_db": true
}
```
**Response (202 Accepted)**:
```json
{
"job_id": "uuid",
"status": "RUNNING"
}
```

View File

@@ -0,0 +1,48 @@
# Data Model: Migration Process and UI Redesign
## Entities
### Environment
Represents a Superset instance.
| Field | Type | Description |
|-------|------|-------------|
| `id` | UUID | Primary Key |
| `name` | String | Display name (e.g., "Development", "Production") |
| `url` | String | Base URL of the Superset instance |
| `credentials_id` | String | Reference to encrypted credentials in the config manager |
### DatabaseMapping
Represents a mapping between a database in the source environment and a database in the target environment.
| Field | Type | Description |
|-------|------|-------------|
| `id` | UUID | Primary Key |
| `source_env_id` | UUID | Foreign Key to Environment (Source) |
| `target_env_id` | UUID | Foreign Key to Environment (Target) |
| `source_db_uuid` | String | UUID of the database in the source environment |
| `target_db_uuid` | String | UUID of the database in the target environment |
| `source_db_name` | String | Name of the database in the source environment (for UI) |
| `target_db_name` | String | Name of the database in the target environment (for UI) |
| `engine` | String | Database engine type (e.g., "clickhouse", "postgres") |
### MigrationJob
Represents a single migration execution.
| Field | Type | Description |
|-------|------|-------------|
| `id` | UUID | Primary Key |
| `source_env_id` | UUID | Foreign Key to Environment |
| `target_env_id` | UUID | Foreign Key to Environment |
| `status` | Enum | `PENDING`, `RUNNING`, `COMPLETED`, `FAILED`, `AWAITING_MAPPING` |
| `replace_db` | Boolean | Whether to apply database mappings |
| `created_at` | DateTime | Timestamp of creation |
## Relationships
- `DatabaseMapping` belongs to a pair of `Environments`.
- `MigrationJob` references two `Environments`.
## Validation Rules
- `source_env_id` and `target_env_id` must be different.
- `source_db_uuid` and `target_db_uuid` must belong to databases with compatible engines (optional warning).
- Mappings must be unique for a given `(source_env_id, target_env_id, source_db_uuid)` triplet.

View File

@@ -0,0 +1,79 @@
# Implementation Plan: Migration Process and UI Redesign
**Branch**: `001-migration-ui-redesign` | **Date**: 2025-12-20 | **Spec**: [specs/001-migration-ui-redesign/spec.md](specs/001-migration-ui-redesign/spec.md)
## Summary
Redesign the migration process to support environment-based selection and automated database mapping. The technical approach involves using a SQLite database to persist mappings between source and target databases, implementing a fuzzy matching algorithm for empirical suggestions, and intercepting asset definitions during migration to apply these mappings.
## Technical Context
**Language/Version**: Python 3.9+, Node.js 18+
**Primary Dependencies**: FastAPI, SvelteKit, Tailwind CSS, Pydantic, SQLite
**Storage**: SQLite (for database mappings and environment metadata)
**Testing**: pytest (Backend), Vitest/Playwright (Frontend)
**Target Platform**: Linux server
**Project Type**: Web application (FastAPI + SvelteKit SPA)
**Performance Goals**: SC-001: Users can complete a full database mapping for 5+ databases in under 60 seconds.
**Constraints**: SPA-First Architecture (Constitution Principle I), API-Driven Communication (Constitution Principle II).
**Scale/Scope**: Support for multiple environments and hundreds of database mappings.
## Constitution Check
*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
| Principle | Status | Notes |
|-----------|--------|-------|
| I. SPA-First Architecture | PASS | SvelteKit will be built as a static SPA and served by FastAPI. |
| II. API-Driven Communication | PASS | All mapping and migration actions will go through FastAPI endpoints. |
| III. Modern Stack Consistency | PASS | Using FastAPI, SvelteKit, and Tailwind CSS. |
| IV. Semantic Protocol Adherence | PASS | Code will include GRACE-Poly anchors and contracts. |
## Project Structure
### Documentation (this feature)
```text
specs/001-migration-ui-redesign/
├── plan.md # This file
├── research.md # Phase 0 output
├── data-model.md # Phase 1 output
├── quickstart.md # Phase 1 output
├── contracts/ # Phase 1 output
└── tasks.md # Phase 2 output
```
### Source Code (repository root)
```text
backend/
├── src/
│ ├── api/
│ │ └── routes/
│ │ ├── environments.py # New: Env selection
│ │ └── mappings.py # New: DB mapping management
│ ├── core/
│ │ └── migration_engine.py # Update: DB replacement logic
│ └── models/
│ └── mapping.py # New: SQLite models
└── tests/
frontend/
├── src/
│ ├── components/
│ │ ├── MappingTable.svelte # New: DB mapping UI
│ │ └── EnvSelector.svelte # New: Source/Target selection
│ └── routes/
│ └── migration/ # New: Migration dashboard
└── tests/
```
**Structure Decision**: Web application structure (Option 2) is selected to maintain separation between the FastAPI backend and SvelteKit frontend while adhering to the SPA-first principle.
## Complexity Tracking
> **Fill ONLY if Constitution Check has violations that must be justified**
| Violation | Why Needed | Simpler Alternative Rejected Because |
|-----------|------------|-------------------------------------|
| None | N/A | N/A |

View File

@@ -0,0 +1,39 @@
# Quickstart: Migration Process and UI Redesign
## Setup
1. **Install Dependencies**:
```bash
pip install rapidfuzz sqlalchemy
cd frontend && npm install
```
2. **Configure Environments**:
Ensure you have at least two Superset environments configured in the application settings.
3. **Initialize Database**:
The system will automatically create the `mappings.db` SQLite file on the first run.
## Usage
### 1. Define Mappings
1. Navigate to the **Database Mapping** tab.
2. Select your **Source** and **Target** environments.
3. Click **Fetch Databases**.
4. Review the **Suggested Mappings** (highlighted in green).
5. Manually adjust any mappings using the dropdowns.
6. Click **Save Mappings**.
### 2. Run Migration
1. Go to the **Migration** dashboard.
2. Select the **Source** and **Target** environments.
3. Select the assets (Dashboards/Datasets) you want to migrate.
4. Enable the **Replace Database** toggle.
5. Click **Start Migration**.
6. If a database is missing a mapping, a modal will appear prompting you to select a target database.
## Troubleshooting
- **Connection Error**: Ensure the backend can reach both Superset instances. Check credentials in settings.
- **Mapping Not Applied**: Verify that the "Replace Database" toggle was enabled and that the mapping exists for the specific environment pair.
- **Fuzzy Match Failure**: If names are too different, manual mapping is required. The system learns from manual overrides.

View File

@@ -0,0 +1,33 @@
# Research: Migration Process and UI Redesign
## Decision: Fuzzy Matching Algorithm
- **Choice**: `RapidFuzz` library with `fuzz.token_sort_ratio`.
- **Rationale**: `RapidFuzz` is significantly faster than `FuzzyWuzzy` and provides robust string similarity metrics. `token_sort_ratio` is ideal for database names because it ignores word order and is less sensitive to prefixes like "Dev-" or "Prod-".
- **Alternatives considered**:
- `Levenshtein`: Too sensitive to string length and prefixes.
- `Jaro-Winkler`: Good for short strings but less effective for multi-word names with different orders.
## Decision: Asset Interception Strategy
- **Choice**: ZIP-based transformation during migration.
- **Rationale**: Superset's native export/import format is a ZIP archive containing YAML definitions. Intercepting this archive allows for precise modification of database references (UUIDs) before they reach the target environment.
- **Implementation**:
1. Export dashboard/dataset from source (ZIP).
2. Extract ZIP to a temporary directory.
3. Iterate through `datasets/*.yaml` files.
4. Replace `database_uuid` values based on the mapping table.
5. Re-package the ZIP.
6. Import to target.
## Decision: Database Mapping Persistence
- **Choice**: SQLite with SQLAlchemy/SQLModel.
- **Rationale**: SQLite is lightweight, requires no separate server, and is perfect for storing local configuration and mappings. It aligns with the project's existing stack.
- **Schema**:
- `Environment`: `id`, `name`, `url`, `credentials_id`.
- `DatabaseMapping`: `id`, `source_env_id`, `target_env_id`, `source_db_uuid`, `target_db_uuid`, `source_db_name`, `target_db_name`.
## Decision: Superset API Integration
- **Choice**: Extend existing `SupersetClient`.
- **Rationale**: `SupersetClient` already handles authentication, network requests, and basic CRUD for dashboards/datasets. Adding environment-specific fetching and database listing is a natural extension.
- **New Endpoints to use**:
- `GET /api/v1/database/`: List all databases.
- `GET /api/v1/database/{id}`: Get detailed database config.

View File

@@ -0,0 +1,109 @@
# Feature Specification: Migration Process and UI Redesign
**Feature Branch**: `001-migration-ui-redesign`
**Created**: 2025-12-20
**Status**: Draft
**Input**: User description: "я хочу переработать процесс и интерфейс миграции. 1. Необходимо чтобы был выпадающий список enviroments (откуда и куда), а также просто галка замены БД 2. Процесс замены БД должен быть предустановленными парами , необходима отдельная вкладка которая бы считывала базы данных с источника и цели и позволяла их маппить, при этом первоначально эмпирически подставляя пары вида 'Dev Clickhouse' -> 'Prod Clickhouse'. Меппинг нужно сохранять и иметь возможность его редактировать"
## Clarifications
### Session 2025-12-20
- Q: Scope of Database Mapping → A: Map the full configuration object obtained from the Superset API.
- Q: Persistence of mappings → A: Use a SQLite database for storing mappings.
- Q: Handling of missing mappings during migration → A: Show a modal dialog during the migration process to prompt for missing mappings.
- Q: Empirical matching algorithm details → A: Use name-based fuzzy matching (ignoring common prefixes like Dev/Prod).
- Q: Scope of "Replace Database" toggle → A: Apply replacement to all assets (Dashboards, Datasets, Charts) included in the migration.
- Q: Backend exposure of Superset databases → A: Dedicated environment database endpoints (e.g., `/api/environments/{id}/databases`).
- Q: Superset API authentication → A: Use stored environment credentials from the backend.
- Q: Error handling for unreachable environments → A: Return structured error responses (502/503) with descriptive messages.
- Q: Database list filtering → A: Return all available databases with metadata (engine type, etc.).
- Q: Handling large database lists → A: Return full list (no pagination) for simplicity.
## User Scenarios & Testing *(mandatory)*
### User Story 1 - Environment-Based Migration Setup (Priority: P1)
As a migration operator, I want to easily select the source and target environments from a list so that I can quickly define the scope of my migration without manual URL entry.
**Why this priority**: This is the core interaction for starting any migration. Using predefined environments reduces errors and improves speed.
**Independent Test**: Can be tested by opening the migration page and verifying that the "Source" and "Target" dropdowns are populated with configured environments and can be selected.
**Acceptance Scenarios**:
1. **Given** multiple environments are configured in settings, **When** I open the migration page, **Then** I should see two dropdowns for "Source" and "Target" containing these environments.
2. **Given** a source and target are selected, **When** I toggle the "Replace Database" checkbox, **Then** the system should prepare to apply database mappings during the next migration step.
---
### User Story 2 - Database Mapping Management (Priority: P1)
As an administrator, I want to define how databases in my development environment map to databases in production so that my dashboards and datasets work correctly after migration.
**Why this priority**: Migrations often fail or require manual fixups because database references point to the wrong environment. Automated mapping is critical for reliable migrations.
**Independent Test**: Can be tested by navigating to the "Database Mapping" tab, fetching databases, and verifying that mappings can be created, saved, and edited.
**Acceptance Scenarios**:
1. **Given** a source and target environment are selected, **When** I open the "Database Mapping" tab, **Then** the system should fetch and display lists of databases from both environments.
2. **Given** the database lists are loaded, **When** the system identifies similar names (e.g., "Dev Clickhouse" and "Prod Clickhouse"), **Then** it should automatically suggest these as a mapping pair.
3. **Given** suggested or manual mappings, **When** I click "Save Mappings", **Then** these pairs should be persisted and associated with the selected environment pair.
---
### User Story 3 - Migration with Automated DB Replacement (Priority: P2)
As a user, I want the migration process to automatically update database references based on my saved mappings so that I don't have to manually edit exported files or post-migration settings.
**Why this priority**: This delivers the actual value of the mapping feature by automating a tedious and error-prone task.
**Independent Test**: Can be tested by running a migration with "Replace Database" enabled and verifying that the resulting assets in the target environment point to the mapped databases.
**Acceptance Scenarios**:
1. **Given** saved mappings exist for the selected environments, **When** I start a migration with "Replace Database" enabled, **Then** the system should replace all source database IDs/names with their corresponding target values during the transfer.
2. **Given** "Replace Database" is enabled but a source database has no mapping, **When** the migration runs, **Then** the system should pause and show a modal dialog prompting the user to provide a mapping on-the-fly for the missing database.
---
### Edge Cases
- **Environment Connectivity**: If the source or target environment is unreachable, the backend MUST return a structured error (502/503), and the frontend MUST display a clear connection error with a retry option.
- **Duplicate Mappings**: How does the system handle multiple source databases mapping to the same target database? (Assumption: This is allowed, as multiple dev DBs might consolidate into one prod DB).
- **Missing Target Database**: What if a mapped target database no longer exists in the target environment? (Assumption: Validation should occur before migration starts, highlighting broken mappings).
## Requirements *(mandatory)*
### Functional Requirements
- **FR-001**: System MUST provide dropdown menus for selecting "Source Environment" and "Target Environment" on the migration screen.
- **FR-002**: System MUST provide a "Replace Database" checkbox that, when enabled, triggers the database mapping logic for all assets (Dashboards, Datasets, Charts) during migration.
- **FR-003**: System MUST include a dedicated "Database Mapping" tab or view accessible from the migration interface.
- **FR-004**: System MUST fetch available databases from both source and target environments via their respective APIs when the mapping tab is opened.
- **FR-005**: System MUST implement a name-based fuzzy matching algorithm to suggest initial mappings, ignoring common environment prefixes (e.g., "Dev", "Prod").
- **FR-006**: System MUST allow users to manually override suggested mappings and create new ones via a drag-and-drop or dropdown-based interface.
- **FR-007**: System MUST persist database mappings in a local SQLite database, keyed by the source and target environment identifiers.
- **FR-008**: System MUST provide an "Edit" capability for existing mappings, allowing users to update or delete them.
- **FR-009**: During migration, if "Replace Database" is active, the system MUST intercept asset definitions (JSON/YAML) and replace database references according to the active mapping table.
### Key Entities *(include if feature involves data)*
- **Environment**: A configured Superset instance (Name, URL, Credentials).
- **Database Mapping**: A record linking a source database configuration (including metadata like engine type) to a target database configuration for a specific `source_env` -> `target_env` pair.
- **Migration Configuration**: The set of parameters for a migration job, including selected environments and the "Replace Database" toggle state.
## Success Criteria *(mandatory)*
### Measurable Outcomes
- **SC-001**: Users can complete a full database mapping for 5+ databases in under 60 seconds using the empirical suggestions.
- **SC-002**: 100% of assets migrated with "Replace Database" enabled correctly reference the target databases as defined in the mapping table.
- **SC-003**: Mapping persistence allows users to run subsequent migrations between the same environments without re-configuring database pairs in 100% of cases.
- **SC-004**: The system successfully identifies and suggests at least 90% of matching pairs when naming follows a "Prefix + Name" pattern (e.g., "Dev-Sales" -> "Prod-Sales").
## Assumptions
- **AS-001**: Environments are already configured in the application's global settings.
- **AS-002**: The backend has access to stored credentials for both source and target environments to perform API requests.
- **AS-003**: Database names or IDs are stable enough within an environment to be used as reliable mapping keys.

View File

@@ -0,0 +1,186 @@
---
description: "Task list for Migration Process and UI Redesign implementation"
---
# Tasks: Migration Process and UI Redesign
**Input**: Design documents from `specs/001-migration-ui-redesign/`
**Prerequisites**: plan.md (required), spec.md (required for user stories), research.md, data-model.md, quickstart.md
**Tests**: Tests are NOT explicitly requested in the feature specification, so they are omitted from this task list.
**Organization**: Tasks are grouped by user story to enable independent implementation and testing of each story.
## Format: `[ID] [P?] [Story] Description`
- **[P]**: Can run in parallel (different files, no dependencies)
- **[Story]**: Which user story this task belongs to (e.g., US1, US2, US3)
- Include exact file paths in descriptions
## Path Conventions
- **Web app**: `backend/src/`, `frontend/src/`
---
## Phase 1: Setup (Shared Infrastructure)
**Purpose**: Project initialization and basic structure
- [ ] T001 Create project structure per implementation plan in `backend/src/` and `frontend/src/`
- [ ] T002 [P] Install backend dependencies (rapidfuzz, sqlalchemy) in `backend/requirements.txt`
- [ ] T003 [P] Install frontend dependencies (if any new) in `frontend/package.json`
---
## Phase 2: Foundational (Blocking Prerequisites)
**Purpose**: Core infrastructure that MUST be complete before ANY user story can be implemented
**⚠️ CRITICAL**: No user story work can begin until this phase is complete
- [ ] T004 Setup SQLite database schema and SQLAlchemy models in `backend/src/models/mapping.py`
- [ ] T005 [P] Implement fuzzy matching utility using RapidFuzz in `backend/src/core/utils/matching.py`
- [ ] T006 [P] Extend SupersetClient to support database listing and metadata fetching in `backend/src/core/superset_client.py`
- [ ] T007 Configure database mapping persistence layer in `backend/src/core/database.py`
**Checkpoint**: Foundation ready - user story implementation can now begin in parallel
---
## Phase 3: User Story 1 - Environment-Based Migration Setup (Priority: P1) 🎯 MVP
**Goal**: Enable selection of source and target environments and toggle database replacement.
**Independent Test**: Open the migration page and verify that the "Source" and "Target" dropdowns are populated with configured environments and can be selected.
### Implementation for User Story 1
- [ ] T008 [P] [US1] Implement environment selection API endpoints in `backend/src/api/routes/environments.py`
- [ ] T009 [P] [US1] Create `EnvSelector.svelte` component for source/target selection in `frontend/src/components/EnvSelector.svelte`
- [ ] T010 [US1] Integrate `EnvSelector` and "Replace Database" toggle into migration dashboard in `frontend/src/routes/migration/+page.svelte`
- [ ] T011 [US1] Add validation to ensure source and target environments are different in `frontend/src/routes/migration/+page.svelte`
**Checkpoint**: At this point, User Story 1 should be fully functional and testable independently.
---
## Phase 4: User Story 2 - Database Mapping Management (Priority: P1)
**Goal**: Fetch databases from environments, suggest mappings using fuzzy matching, and allow manual overrides/persistence.
**Independent Test**: Navigate to the "Database Mapping" tab, fetch databases, and verify that mappings can be created, saved, and edited.
### Implementation for User Story 2
- [ ] T012 [P] [US2] Implement database mapping CRUD API endpoints in `backend/src/api/routes/mappings.py`
- [ ] T013 [US2] Implement mapping service with fuzzy matching logic in `backend/src/services/mapping_service.py`
- [ ] T014 [P] [US2] Create `MappingTable.svelte` component for displaying and editing pairs in `frontend/src/components/MappingTable.svelte`
- [ ] T015 [US2] Create database mapping management view in `frontend/src/routes/migration/mappings/+page.svelte`
- [ ] T016 [US2] Implement "Fetch Databases" action and suggestion highlighting in `frontend/src/routes/migration/mappings/+page.svelte`
**Checkpoint**: At this point, User Stories 1 AND 2 should both work independently.
---
## Phase 5: User Story 3 - Migration with Automated DB Replacement (Priority: P2)
**Goal**: Intercept assets during migration, apply database mappings, and prompt for missing ones.
**Independent Test**: Run a migration with "Replace Database" enabled and verify that the resulting assets in the target environment point to the mapped databases.
### Implementation for User Story 3
- [ ] T017 [US3] Implement ZIP-based asset interception and YAML transformation logic in `backend/src/core/migration_engine.py`
- [ ] T018 [US3] Integrate database mapping application into the migration job execution flow in `backend/src/core/task_manager.py`
- [ ] T019 [P] [US3] Create `MissingMappingModal.svelte` for on-the-fly mapping prompts in `frontend/src/components/MissingMappingModal.svelte`
- [ ] T020 [US3] Implement backend pause and frontend modal trigger for missing mappings in `backend/src/api/routes/tasks.py` and `frontend/src/components/TaskRunner.svelte`
**Checkpoint**: All user stories should now be independently functional.
---
## Phase 6: Polish & Cross-Cutting Concerns
**Purpose**: Improvements that affect multiple user stories
- [ ] T021 [P] Update documentation in `docs/` to include database mapping instructions
- [ ] T022 Code cleanup and refactoring of migration logic
- [ ] T023 [P] Performance optimization for fuzzy matching and ZIP processing
- [ ] T024 Run `quickstart.md` validation to ensure end-to-end flow works as documented
---
## Dependencies & Execution Order
### Phase Dependencies
- **Setup (Phase 1)**: No dependencies - can start immediately
- **Foundational (Phase 2)**: Depends on Setup completion - BLOCKS all user stories
- **User Stories (Phase 3+)**: All depend on Foundational phase completion
- User stories can then proceed in parallel (if staffed)
- Or sequentially in priority order (P1 → P2 → P3)
- **Polish (Final Phase)**: Depends on all desired user stories being complete
### User Story Dependencies
- **User Story 1 (P1)**: Can start after Foundational (Phase 2) - No dependencies on other stories
- **User Story 2 (P1)**: Can start after Foundational (Phase 2) - No dependencies on other stories
- **User Story 3 (P2)**: Can start after Foundational (Phase 2) - Depends on US1/US2 for mapping data and configuration
### Within Each User Story
- Models before services
- Services before endpoints
- Core implementation before integration
- Story complete before moving to next priority
### Parallel Opportunities
- All Setup tasks marked [P] can run in parallel
- All Foundational tasks marked [P] can run in parallel (within Phase 2)
- Once Foundational phase completes, US1 and US2 can start in parallel
- Models and UI components within a story marked [P] can run in parallel
---
## Parallel Example: User Story 2
```bash
# Launch backend and frontend components for User Story 2 together:
Task: "Implement database mapping CRUD API endpoints in backend/src/api/routes/mappings.py"
Task: "Create MappingTable.svelte component for displaying and editing pairs in frontend/src/components/MappingTable.svelte"
```
---
## Implementation Strategy
### MVP First (User Story 1 & 2)
1. Complete Phase 1: Setup
2. Complete Phase 2: Foundational (CRITICAL - blocks all stories)
3. Complete Phase 3: User Story 1
4. Complete Phase 4: User Story 2
5. **STOP and VALIDATE**: Test environment selection and mapping management independently
6. Deploy/demo if ready
### Incremental Delivery
1. Complete Setup + Foundational → Foundation ready
2. Add User Story 1 → Test independently → Deploy/Demo
3. Add User Story 2 → Test independently → Deploy/Demo (MVP!)
4. Add User Story 3 → Test independently → Deploy/Demo
5. Each story adds value without breaking previous stories
---
## Notes
- [P] tasks = different files, no dependencies
- [Story] label maps task to specific user story for traceability
- Each user story should be independently completable and testable
- Commit after each task or logical group
- Stop at any checkpoint to validate story independently
- Avoid: vague tasks, same file conflicts, cross-story dependencies that break independence

View File

@@ -0,0 +1,24 @@
# WebSocket Contract: Task Logs
## Endpoint
`WS /ws/logs/{task_id}`
## Description
Streams real-time logs for a specific task.
## Connection Parameters
- `task_id`: UUID of the task to monitor.
## Message Format (Server -> Client)
```json
{
"task_id": "uuid",
"message": "Log message text",
"timestamp": "2025-12-20T20:20:00Z",
"level": "INFO"
}
```
## Error Handling
- If `task_id` is invalid, the connection is closed with code `4004` (Not Found).
- If the connection fails, the client should attempt reconnection with exponential backoff.