feat(migration): implement interactive mapping resolution workflow
- Add SQLite database integration for environments and mappings - Update TaskManager to support pausing tasks (AWAITING_MAPPING) - Modify MigrationPlugin to detect missing mappings and wait for resolution - Add frontend UI for handling missing mappings interactively - Create dedicated migration routes and API endpoints - Update .gitignore and project documentation
This commit is contained in:
109
specs/001-migration-ui-redesign/spec.md
Normal file
109
specs/001-migration-ui-redesign/spec.md
Normal file
@@ -0,0 +1,109 @@
|
||||
# Feature Specification: Migration Process and UI Redesign
|
||||
|
||||
**Feature Branch**: `001-migration-ui-redesign`
|
||||
**Created**: 2025-12-20
|
||||
**Status**: Draft
|
||||
**Input**: User description: "я хочу переработать процесс и интерфейс миграции. 1. Необходимо чтобы был выпадающий список enviroments (откуда и куда), а также просто галка замены БД 2. Процесс замены БД должен быть предустановленными парами , необходима отдельная вкладка которая бы считывала базы данных с источника и цели и позволяла их маппить, при этом первоначально эмпирически подставляя пары вида 'Dev Clickhouse' -> 'Prod Clickhouse'. Меппинг нужно сохранять и иметь возможность его редактировать"
|
||||
|
||||
## Clarifications
|
||||
|
||||
### Session 2025-12-20
|
||||
- Q: Scope of Database Mapping → A: Map the full configuration object obtained from the Superset API.
|
||||
- Q: Persistence of mappings → A: Use a SQLite database for storing mappings.
|
||||
- Q: Handling of missing mappings during migration → A: Show a modal dialog during the migration process to prompt for missing mappings.
|
||||
- Q: Empirical matching algorithm details → A: Use name-based fuzzy matching (ignoring common prefixes like Dev/Prod).
|
||||
- Q: Scope of "Replace Database" toggle → A: Apply replacement to all assets (Dashboards, Datasets, Charts) included in the migration.
|
||||
- Q: Backend exposure of Superset databases → A: Dedicated environment database endpoints (e.g., `/api/environments/{id}/databases`).
|
||||
- Q: Superset API authentication → A: Use stored environment credentials from the backend.
|
||||
- Q: Error handling for unreachable environments → A: Return structured error responses (502/503) with descriptive messages.
|
||||
- Q: Database list filtering → A: Return all available databases with metadata (engine type, etc.).
|
||||
- Q: Handling large database lists → A: Return full list (no pagination) for simplicity.
|
||||
|
||||
## User Scenarios & Testing *(mandatory)*
|
||||
|
||||
### User Story 1 - Environment-Based Migration Setup (Priority: P1)
|
||||
|
||||
As a migration operator, I want to easily select the source and target environments from a list so that I can quickly define the scope of my migration without manual URL entry.
|
||||
|
||||
**Why this priority**: This is the core interaction for starting any migration. Using predefined environments reduces errors and improves speed.
|
||||
|
||||
**Independent Test**: Can be tested by opening the migration page and verifying that the "Source" and "Target" dropdowns are populated with configured environments and can be selected.
|
||||
|
||||
**Acceptance Scenarios**:
|
||||
|
||||
1. **Given** multiple environments are configured in settings, **When** I open the migration page, **Then** I should see two dropdowns for "Source" and "Target" containing these environments.
|
||||
2. **Given** a source and target are selected, **When** I toggle the "Replace Database" checkbox, **Then** the system should prepare to apply database mappings during the next migration step.
|
||||
|
||||
---
|
||||
|
||||
### User Story 2 - Database Mapping Management (Priority: P1)
|
||||
|
||||
As an administrator, I want to define how databases in my development environment map to databases in production so that my dashboards and datasets work correctly after migration.
|
||||
|
||||
**Why this priority**: Migrations often fail or require manual fixups because database references point to the wrong environment. Automated mapping is critical for reliable migrations.
|
||||
|
||||
**Independent Test**: Can be tested by navigating to the "Database Mapping" tab, fetching databases, and verifying that mappings can be created, saved, and edited.
|
||||
|
||||
**Acceptance Scenarios**:
|
||||
|
||||
1. **Given** a source and target environment are selected, **When** I open the "Database Mapping" tab, **Then** the system should fetch and display lists of databases from both environments.
|
||||
2. **Given** the database lists are loaded, **When** the system identifies similar names (e.g., "Dev Clickhouse" and "Prod Clickhouse"), **Then** it should automatically suggest these as a mapping pair.
|
||||
3. **Given** suggested or manual mappings, **When** I click "Save Mappings", **Then** these pairs should be persisted and associated with the selected environment pair.
|
||||
|
||||
---
|
||||
|
||||
### User Story 3 - Migration with Automated DB Replacement (Priority: P2)
|
||||
|
||||
As a user, I want the migration process to automatically update database references based on my saved mappings so that I don't have to manually edit exported files or post-migration settings.
|
||||
|
||||
**Why this priority**: This delivers the actual value of the mapping feature by automating a tedious and error-prone task.
|
||||
|
||||
**Independent Test**: Can be tested by running a migration with "Replace Database" enabled and verifying that the resulting assets in the target environment point to the mapped databases.
|
||||
|
||||
**Acceptance Scenarios**:
|
||||
|
||||
1. **Given** saved mappings exist for the selected environments, **When** I start a migration with "Replace Database" enabled, **Then** the system should replace all source database IDs/names with their corresponding target values during the transfer.
|
||||
2. **Given** "Replace Database" is enabled but a source database has no mapping, **When** the migration runs, **Then** the system should pause and show a modal dialog prompting the user to provide a mapping on-the-fly for the missing database.
|
||||
|
||||
---
|
||||
|
||||
### Edge Cases
|
||||
|
||||
- **Environment Connectivity**: If the source or target environment is unreachable, the backend MUST return a structured error (502/503), and the frontend MUST display a clear connection error with a retry option.
|
||||
- **Duplicate Mappings**: How does the system handle multiple source databases mapping to the same target database? (Assumption: This is allowed, as multiple dev DBs might consolidate into one prod DB).
|
||||
- **Missing Target Database**: What if a mapped target database no longer exists in the target environment? (Assumption: Validation should occur before migration starts, highlighting broken mappings).
|
||||
|
||||
## Requirements *(mandatory)*
|
||||
|
||||
### Functional Requirements
|
||||
|
||||
- **FR-001**: System MUST provide dropdown menus for selecting "Source Environment" and "Target Environment" on the migration screen.
|
||||
- **FR-002**: System MUST provide a "Replace Database" checkbox that, when enabled, triggers the database mapping logic for all assets (Dashboards, Datasets, Charts) during migration.
|
||||
- **FR-003**: System MUST include a dedicated "Database Mapping" tab or view accessible from the migration interface.
|
||||
- **FR-004**: System MUST fetch available databases from both source and target environments via their respective APIs when the mapping tab is opened.
|
||||
- **FR-005**: System MUST implement a name-based fuzzy matching algorithm to suggest initial mappings, ignoring common environment prefixes (e.g., "Dev", "Prod").
|
||||
- **FR-006**: System MUST allow users to manually override suggested mappings and create new ones via a drag-and-drop or dropdown-based interface.
|
||||
- **FR-007**: System MUST persist database mappings in a local SQLite database, keyed by the source and target environment identifiers.
|
||||
- **FR-008**: System MUST provide an "Edit" capability for existing mappings, allowing users to update or delete them.
|
||||
- **FR-009**: During migration, if "Replace Database" is active, the system MUST intercept asset definitions (JSON/YAML) and replace database references according to the active mapping table.
|
||||
|
||||
### Key Entities *(include if feature involves data)*
|
||||
|
||||
- **Environment**: A configured Superset instance (Name, URL, Credentials).
|
||||
- **Database Mapping**: A record linking a source database configuration (including metadata like engine type) to a target database configuration for a specific `source_env` -> `target_env` pair.
|
||||
- **Migration Configuration**: The set of parameters for a migration job, including selected environments and the "Replace Database" toggle state.
|
||||
|
||||
## Success Criteria *(mandatory)*
|
||||
|
||||
### Measurable Outcomes
|
||||
|
||||
- **SC-001**: Users can complete a full database mapping for 5+ databases in under 60 seconds using the empirical suggestions.
|
||||
- **SC-002**: 100% of assets migrated with "Replace Database" enabled correctly reference the target databases as defined in the mapping table.
|
||||
- **SC-003**: Mapping persistence allows users to run subsequent migrations between the same environments without re-configuring database pairs in 100% of cases.
|
||||
- **SC-004**: The system successfully identifies and suggests at least 90% of matching pairs when naming follows a "Prefix + Name" pattern (e.g., "Dev-Sales" -> "Prod-Sales").
|
||||
|
||||
## Assumptions
|
||||
|
||||
- **AS-001**: Environments are already configured in the application's global settings.
|
||||
- **AS-002**: The backend has access to stored credentials for both source and target environments to perform API requests.
|
||||
- **AS-003**: Database names or IDs are stable enough within an environment to be used as reliable mapping keys.
|
||||
Reference in New Issue
Block a user