ss-tools/specs/001-migration-ui-redesign/spec.md

Feature Specification: Migration Process and UI Redesign

Feature Branch: 001-migration-ui-redesign
Created: 2025-12-20
Status: Draft
Input: User description: "я хочу переработать процесс и интерфейс миграции. 1. Необходимо чтобы был выпадающий список enviroments (откуда и куда), а также просто галка замены БД 2. Процесс замены БД должен быть предустановленными парами , необходима отдельная вкладка которая бы считывала базы данных с источника и цели и позволяла их маппить, при этом первоначально эмпирически подставляя пары вида 'Dev Clickhouse' -> 'Prod Clickhouse'. Меппинг нужно сохранять и иметь возможность его редактировать"

Clarifications

Session 2025-12-20

• Q: Scope of Database Mapping → A: Map the full configuration object obtained from the Superset API.
• Q: Persistence of mappings → A: Use a SQLite database for storing mappings.
• Q: Handling of missing mappings during migration → A: Show a modal dialog during the migration process to prompt for missing mappings.
• Q: Empirical matching algorithm details → A: Use name-based fuzzy matching (ignoring common prefixes like Dev/Prod).
• Q: Scope of "Replace Database" toggle → A: Apply replacement to all assets (Dashboards, Datasets, Charts) included in the migration.
• Q: Backend exposure of Superset databases → A: Dedicated environment database endpoints (e.g., /api/environments/{id}/databases).
• Q: Superset API authentication → A: Use stored environment credentials from the backend.
• Q: Error handling for unreachable environments → A: Return structured error responses (502/503) with descriptive messages.
• Q: Database list filtering → A: Return all available databases with metadata (engine type, etc.).
• Q: Handling large database lists → A: Return full list (no pagination) for simplicity.

User Scenarios & Testing (mandatory)

User Story 1 - Environment-Based Migration Setup (Priority: P1)

As a migration operator, I want to easily select the source and target environments from a list so that I can quickly define the scope of my migration without manual URL entry.

Why this priority: This is the core interaction for starting any migration. Using predefined environments reduces errors and improves speed.

Independent Test: Can be tested by opening the migration page and verifying that the "Source" and "Target" dropdowns are populated with configured environments and can be selected.

Acceptance Scenarios:

1. Given multiple environments are configured in settings, When I open the migration page, Then I should see two dropdowns for "Source" and "Target" containing these environments.
2. Given a source and target are selected, When I toggle the "Replace Database" checkbox, Then the system should prepare to apply database mappings during the next migration step.

---

User Story 2 - Database Mapping Management (Priority: P1)

As an administrator, I want to define how databases in my development environment map to databases in production so that my dashboards and datasets work correctly after migration.

Why this priority: Migrations often fail or require manual fixups because database references point to the wrong environment. Automated mapping is critical for reliable migrations.

Independent Test: Can be tested by navigating to the "Database Mapping" tab, fetching databases, and verifying that mappings can be created, saved, and edited.

Acceptance Scenarios:

1. Given a source and target environment are selected, When I open the "Database Mapping" tab, Then the system should fetch and display lists of databases from both environments.
2. Given the database lists are loaded, When the system identifies similar names (e.g., "Dev Clickhouse" and "Prod Clickhouse"), Then it should automatically suggest these as a mapping pair.
3. Given suggested or manual mappings, When I click "Save Mappings", Then these pairs should be persisted and associated with the selected environment pair.

---

User Story 3 - Migration with Automated DB Replacement (Priority: P2)

As a user, I want the migration process to automatically update database references based on my saved mappings so that I don't have to manually edit exported files or post-migration settings.

Why this priority: This delivers the actual value of the mapping feature by automating a tedious and error-prone task.

Independent Test: Can be tested by running a migration with "Replace Database" enabled and verifying that the resulting assets in the target environment point to the mapped databases.

Acceptance Scenarios:

1. Given saved mappings exist for the selected environments, When I start a migration with "Replace Database" enabled, Then the system should replace all source database IDs/names with their corresponding target values during the transfer.
2. Given "Replace Database" is enabled but a source database has no mapping, When the migration runs, Then the system should pause and show a modal dialog prompting the user to provide a mapping on-the-fly for the missing database.

---

Edge Cases

• Environment Connectivity: If the source or target environment is unreachable, the backend MUST return a structured error (502/503), and the frontend MUST display a clear connection error with a retry option.
• Duplicate Mappings: How does the system handle multiple source databases mapping to the same target database? (Assumption: This is allowed, as multiple dev DBs might consolidate into one prod DB).
• Missing Target Database: What if a mapped target database no longer exists in the target environment? (Assumption: Validation should occur before migration starts, highlighting broken mappings).

Requirements (mandatory)

Functional Requirements

• FR-001: System MUST provide dropdown menus for selecting "Source Environment" and "Target Environment" on the migration screen.
• FR-002: System MUST provide a "Replace Database" checkbox that, when enabled, triggers the database mapping logic for all assets (Dashboards, Datasets, Charts) during migration.
• FR-003: System MUST include a dedicated "Database Mapping" tab or view accessible from the migration interface.
• FR-004: System MUST fetch available databases from both source and target environments via their respective APIs when the mapping tab is opened.
• FR-005: System MUST implement a name-based fuzzy matching algorithm to suggest initial mappings, ignoring common environment prefixes (e.g., "Dev", "Prod").
• FR-006: System MUST allow users to manually override suggested mappings and create new ones via a drag-and-drop or dropdown-based interface.
• FR-007: System MUST persist database mappings in a local SQLite database, keyed by the source and target environment identifiers.
• FR-008: System MUST provide an "Edit" capability for existing mappings, allowing users to update or delete them.
• FR-009: During migration, if "Replace Database" is active, the system MUST intercept asset definitions (JSON/YAML) and replace database references according to the active mapping table.

Key Entities (include if feature involves data)

• Environment: A configured Superset instance (Name, URL, Credentials).
• Database Mapping: A record linking a source database configuration (including metadata like engine type) to a target database configuration for a specific source_env -> target_env pair.
• Migration Configuration: The set of parameters for a migration job, including selected environments and the "Replace Database" toggle state.

Success Criteria (mandatory)

Measurable Outcomes

• SC-001: Users can complete a full database mapping for 5+ databases in under 60 seconds using the empirical suggestions.
• SC-002: 100% of assets migrated with "Replace Database" enabled correctly reference the target databases as defined in the mapping table.
• SC-003: Mapping persistence allows users to run subsequent migrations between the same environments without re-configuring database pairs in 100% of cases.
• SC-004: The system successfully identifies and suggests at least 90% of matching pairs when naming follows a "Prefix + Name" pattern (e.g., "Dev-Sales" -> "Prod-Sales").

Assumptions

• AS-001: Environments are already configured in the application's global settings.
• AS-002: The backend has access to stored credentials for both source and target environments to perform API requests.
• AS-003: Database names or IDs are stable enough within an environment to be used as reliable mapping keys.