busya/ss-tools
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

refactor complete

Browse Source
This commit is contained in:
busya
2026-01-22 17:37:17 +03:00
parent 203ce446f4
commit d99a13d91f
41 changed files with 1789 additions and 3052 deletions
Download Patch File Download Diff File Expand all files Collapse all files

34
specs/012-remove-superset-tool/checklists/requirements.md Normal file
View File

@@ -0,0 +1,34 @@
# Specification Quality Checklist: Backend Refactoring - Remove superset_tool
**Purpose**: Validate specification completeness and quality before proceeding to planning
**Created**: 2026-01-22
**Feature**: [Link to spec.md](../spec.md)
## Content Quality
- [x] No implementation details (languages, frameworks, APIs) - *Refactoring is technical by nature, but focused on structural goals.*
- [x] Focused on user value and business needs
- [x] Written for non-technical stakeholders
- [x] All mandatory sections completed
## Requirement Completeness
- [x] No [NEEDS CLARIFICATION] markers remain
- [x] Requirements are testable and unambiguous
- [x] Success criteria are measurable
- [x] Success criteria are technology-agnostic (no implementation details)
- [x] All acceptance scenarios are defined
- [x] Edge cases are identified
- [x] Scope is clearly bounded
- [x] Dependencies and assumptions identified
## Feature Readiness
- [x] All functional requirements have clear acceptance criteria
- [x] User scenarios cover primary flows
- [x] Feature meets measurable outcomes defined in Success Criteria
- [x] No implementation details leak into specification
## Notes
- Items marked incomplete require spec updates before `/speckit.clarify` or `/speckit.plan`

39
specs/012-remove-superset-tool/contracts/api.md Normal file
View File

@@ -0,0 +1,39 @@
# API Contracts: 012-remove-superset-tool
## SupersetClient Internal API
The `SupersetClient` will provide the following methods for internal backend use:
### `get_dashboards(query: Optional[Dict] = None) -> Tuple[int, List[Dict]]`
- **Purpose**: Fetches paginated dashboards.
- **Contract**:
- `query`: Optional filters and pagination parameters.
- Returns: `(total_count, dashboard_list)`.
### `export_dashboard(dashboard_id: int) -> Tuple[bytes, str]`
- **Purpose**: Exports a dashboard as a ZIP file.
- **Contract**:
- `dashboard_id`: ID of the dashboard.
- Returns: `(zip_content, filename)`.
### `import_dashboard(file_name: Union[str, Path], dash_id: Optional[int] = None, dash_slug: Optional[str] = None) -> Dict`
- **Purpose**: Imports a dashboard from a ZIP file.
- **Contract**:
- `file_name`: Path to ZIP.
- `dash_id`/`dash_slug`: Optional identifiers for delete-retry logic.
- Returns: API response dictionary.
## Configuration Models (Pydantic)
### `Environment`
```python
class Environment(BaseModel):
id: str
name: str
url: str
username: str
password: str
verify_ssl: bool = True
timeout: int = 30
is_default: bool = False
backup_schedule: Schedule = Field(default_factory=Schedule)

38
specs/012-remove-superset-tool/data-model.md Normal file
View File

@@ -0,0 +1,38 @@
# Data Model: 012-remove-superset-tool
## Entities
### 1. Environment (Updated)
- **Source**: `backend/src/core/config_models.py`
- **Purpose**: Represents a Superset environment configuration, now incorporating all fields required for the `SupersetClient`.
- **Fields**:
- `id`: `str` (Unique identifier)
- `name`: `str` (Display name)
- `url`: `str` (Base URL for API)
- `username`: `str`
- `password`: `str`
- `verify_ssl`: `bool` (Default: `True`)
- `timeout`: `int` (Default: `30`)
- `is_default`: `bool` (Default: `False`)
- `backup_schedule`: `Schedule`
- **Validation Rules**:
- `url` must start with `http://` or `https://`.
- `timeout` must be positive.
### 2. API Contract (SupersetClient)
- **Source**: `backend/src/core/superset_client.py`
- **Purpose**: Self-contained client for Superset API interactions.
- **Key Methods**:
- `authenticate()`
- `get_dashboards(query)`
- `get_datasets(query)`
- `get_databases(query)`
- `export_dashboard(dashboard_id)`
- `import_dashboard(file_name, dash_id, dash_slug)`
- `delete_dashboard(dashboard_id)`
- `get_databases_summary()`
- `get_dashboards_summary()`
## State Transitions
- **Authentication**: Unauthenticated -> Authenticated (token stored in memory/client session).
- **Import/Export**: File System <-> Superset API.

69
specs/012-remove-superset-tool/plan.md Normal file
View File

@@ -0,0 +1,69 @@
# Implementation Plan: 012-remove-superset-tool
**Branch**: `012-remove-superset-tool` | **Date**: 2026-01-22 | **Spec**: [spec.md](specs/012-remove-superset-tool/spec.md)
## Summary
The primary requirement is to refactor the backend by removing the redundant `superset_tool` module and merging its essential logic into the `backend` package. This involves migrating the `SupersetClient` logic, utility modules, and configuration models while deprecating CLI-specific interactive logic.
## Technical Context
**Language/Version**: Python 3.9+
**Primary Dependencies**: FastAPI, Pydantic, requests, pyyaml (migrated from superset_tool)
**Storage**: SQLite (tasks.db, migrations.db), Filesystem
**Testing**: pytest
**Target Platform**: Linux server
**Project Type**: Web application (FastAPI backend + SvelteKit frontend)
**Performance Goals**: N/A (Cleanup task)
**Constraints**: Zero references to `superset_tool` in `backend/` source code.
**Scale/Scope**: Refactoring core backend infrastructure.
## Constitution Check
*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
1. **Semantic Protocol Compliance**: All migrated code must use `[DEF]` anchors and `@RELATION` tags as per `semantic_protocol.md`.
2. **Causal Validity**: Contracts (Pydantic models and API surfaces) must be defined before implementation.
3. **Everything is a Plugin**: Ensure that logic previously in `superset_tool` that acts as a tool or extension is integrated within the backend's plugin architecture if applicable.
4. **Fractal Complexity Limit**: Migrated modules must adhere to complexity limits.
## Project Structure
### Documentation (this feature)
```text
specs/012-remove-superset-tool/
├── plan.md # This file
├── research.md # Phase 0 output
├── data-model.md # Phase 1 output
├── quickstart.md # Phase 1 output
├── contracts/ # Phase 1 output
└── tasks.md # Phase 2 output
```
### Source Code (repository root)
```text
backend/
├── src/
│ ├── api/ # API routes
│ ├── core/ # Core logic (Target for migration)
│ │ ├── utils/ # Shared utilities
│ │ ├── config_models.py
│ │ ├── logger.py
│ │ └── superset_client.py
│ ├── models/ # Database models
│ ├── plugins/ # Plugin system
│ └── app.py
└── tests/ # Backend tests
```
**Structure Decision**: Web application structure. The `superset_tool` logic will be absorbed into `backend/src/core/` and its subdirectories.
## Complexity Tracking
> **Fill ONLY if Constitution Check has violations that must be justified**
| Violation | Why Needed | Simpler Alternative Rejected Because |
|-----------|------------|-------------------------------------|
| None | | |

20
specs/012-remove-superset-tool/quickstart.md Normal file
View File

@@ -0,0 +1,20 @@
# Quickstart: 012-remove-superset-tool
## Overview
This feature refactors the backend to remove the `superset_tool` dependency. All Superset-related logic is now centralized in `backend/src/core/superset_client.py`.
## Key Changes
1. **Models**: `Environment` model in `backend/src/core/config_models.py` now includes `verify_ssl` and `timeout`.
2. **Client**: `SupersetClient` in `backend/src/core/superset_client.py` is now self-contained and does not inherit from any external module.
3. **Utilities**: Network and file I/O utilities previously in `superset_tool` are now available in `backend/src/core/utils/`.
## Verification Steps
1. **Run Backend**: Start the FastAPI server and ensure no import errors occur.
```bash
cd backend && .venv/bin/python3 -m uvicorn src.app:app --reload
```
2. **Check Connections**: Verify that existing Superset connections load correctly in the UI/API.
3. **Test Migration**: Trigger a dashboard list refresh or a migration task to confirm the consolidated client works as expected.
## Cleanup
Once verified, the `superset_tool/` directory and root-level CLI scripts (`migration_script.py`, `run_mapper.py`) can be deleted.

32
specs/012-remove-superset-tool/research.md Normal file
View File

@@ -0,0 +1,32 @@
# Research: 012-remove-superset-tool
## Unknowns & Clarifications
### 1. Merging `SupersetConfig` into `Environment`
- **Decision**: Fields from `SupersetConfig` (auth, verify_ssl, timeout) will be merged into the `Environment` model in `backend/src/core/config_models.py`.
- **Rationale**: Centralizes configuration and removes dependency on `superset_tool.models`.
- **Alternatives considered**: Keeping them separate, but that defeats the purpose of the refactoring.
### 2. Utility Migration
- **Decision**: `superset_tool/utils/` will be flattened into `backend/src/core/utils/`.
- **Rationale**: Simplifies the directory structure and aligns with the backend's existing utility pattern.
- **Alternatives considered**: Creating a `backend/src/core/utils/superset/` subdirectory, but flattening is preferred for simplicity.
### 3. CLI Deprecation
- **Decision**: `migration_script.py`, `run_mapper.py`, and `whiptail_fallback.py` will be deprecated and not migrated.
- **Rationale**: The web UI now handles these operations, and maintaining interactive CLI logic in the backend is out of scope.
### 4. Logging Consolidation
- **Decision**: Remove `SupersetLogger` and `belief_scope` from `superset_tool` and use `backend/src/core/logger.py`.
- **Rationale**: Ensures consistent logging across the entire backend application.
## Dependency Analysis
- `backend/requirements.txt` already contains `requests`, `PyYAML`, `pydantic`.
- `superset_tool/requirements.txt` is empty, indicating it likely relied on the same environment or was partially integrated.
- **Action**: Ensure all `superset_tool` imports are replaced with backend-local equivalents.
## Integration Patterns
- `backend/src/core/superset_client.py` currently inherits from `superset_tool.client.SupersetClient`.
- **Action**: Move all methods from `superset_tool/client.py` into `backend/src/core/superset_client.py` and remove the inheritance. Refactor `APIClient` (from `superset_tool/utils/network.py`) to be a utility or private helper within the backend.

78
specs/012-remove-superset-tool/spec.md Normal file
View File

@@ -0,0 +1,78 @@
# Feature Specification: Backend Refactoring - Remove superset_tool
**Feature Branch**: `012-remove-superset-tool`
**Created**: 2026-01-22
**Status**: Draft
**Input**: User description: "нужен рефакторинг бека - я хочу исключить модуль superset_tool, пусть останется только backend"
## Clarifications
### Session 2026-01-22
- Q: How should we reconcile the `SupersetConfig` model from `superset_tool` with the `Environment` model in the backend? → A: Merge all `SupersetConfig` fields (e.g., `verify_ssl`, `timeout`) into the `Environment` model in `backend/src/core/config_models.py`.
- Q: Where should the utility modules from `superset_tool/utils/` be moved within the `backend/src/core/` directory? → A: Flatten them directly into `backend/src/core/utils/` alongside existing utilities.
- Q: Should interactive CLI utilities (whiptail_fallback.py, migration_script.py) be migrated? → A: No, deprecate them and do not migrate CLI-specific interactive UI logic to the backend.
- Q: How should we handle `SupersetLogger` and `belief_scope`? → A: Consolidate all logging into `backend/src/core/logger.py` and remove the redundant `SupersetLogger` class.
## User Scenarios & Testing *(mandatory)*
### User Story 1 - Maintain Core Functionality (Priority: P1)
As a developer, I want to remove the redundant `superset_tool` module and merge its essential logic into the `backend` package so that the codebase is easier to maintain and deploy.
**Why this priority**: This is a critical architectural cleanup. The existence of two overlapping modules (`superset_tool` and `backend/src/core/superset_client.py`) creates confusion and potential bugs.
**Independent Test**: The application (FastAPI backend) should start and perform all Superset-related operations (fetching dashboards, datasets, etc.) without the `superset_tool` directory present in the root.
**Acceptance Scenarios**:
1. **Given** the `superset_tool` directory is deleted, **When** I run the backend server, **Then** it should start without `ModuleNotFoundError`.
2. **Given** the backend is running, **When** I trigger a dashboard list refresh in the UI, **Then** the backend should successfully fetch data from Superset using the migrated client logic.
3. **Given** a migration task, **When** I execute it, **Then** it should successfully use the consolidated `SupersetClient` to interact with the API.
---
### User Story 2 - Unified Configuration and Models (Priority: P2)
As a developer, I want to use a single set of Pydantic models and configuration logic within the `backend` package instead of relying on external `superset_tool.models`.
**Why this priority**: Eliminates duplicate definitions and ensures consistency across the backend.
**Independent Test**: Verify that `backend/src/core/config_models.py` (or equivalent) contains all necessary fields previously defined in `superset_tool/models.py`.
**Acceptance Scenarios**:
1. **Given** a connection configuration in the database, **When** the backend loads it, **Then** it should correctly instantiate the consolidated configuration models.
---
### Edge Cases
- **Deprecated Scripts**: CLI-only scripts like `migration_script.py` and `run_mapper.py` in the root will be deprecated and removed as their logic is now in the web UI.
- **Dependency Conflicts**: How does the system handle potential version conflicts between `backend/requirements.txt` and `superset_tool/requirements.txt`?
- **Environment Variables**: Ensure that any environment variables previously used by `superset_tool` are still correctly loaded by the consolidated client.
## Requirements *(mandatory)*
### Functional Requirements
- **FR-001**: The system MUST NOT depend on the `superset_tool` top-level module.
- **FR-002**: All essential logic from `superset_tool/client.py` MUST be migrated to `backend/src/core/superset_client.py`. Utility modules from `superset_tool/utils/` (network, fileio, dataset_mapper) MUST be migrated to `backend/src/core/utils/`.
- **FR-003**: `backend/src/core/superset_client.py` MUST be refactored to be a self-contained client that doesn't inherit from `superset_tool.client.BaseSupersetClient`.
- **FR-004**: Redundant logging logic (`SupersetLogger`) MUST be removed in favor of `backend/src/core/logger.py`.
- **FR-004**: The backend's `requirements.txt` MUST be updated to include any dependencies previously required by `superset_tool` (e.g., `requests`, `pyyaml`).
- **FR-005**: All imports in `backend/src/` that reference `superset_tool` MUST be updated to reference the new locations within the backend.
### Key Entities *(include if feature involves data)*
- **SupersetClient**: The consolidated class responsible for all interactions with the Superset REST API.
- **Environment**: The unified configuration model for Superset connections, incorporating legacy `SupersetConfig` fields.
## Success Criteria *(mandatory)*
### Measurable Outcomes
- **SC-001**: Zero references to `superset_tool` in the `backend/` source code.
- **SC-002**: 100% of existing Superset integration tests pass after the refactoring.
- **SC-003**: The `superset_tool/` directory can be safely deleted from the repository.
- **SC-004**: Backend startup time remains unaffected or improves slightly due to simplified import structure.

34
specs/012-remove-superset-tool/tasks.md Normal file
View File

@@ -0,0 +1,34 @@
# Tasks: 012-remove-superset-tool
## Phase 1: Setup
- [x] T001 Initialize branch `012-remove-superset-tool`
- [x] T002 Update `backend/requirements.txt` with dependencies from `superset_tool` (requests, pyyaml)
## Phase 2: Foundational (Infrastructure Migration)
- [x] T003 [P] Migrate utility modules from `superset_tool/utils/` to `backend/src/core/utils/`
- [x] T004 [P] Refactor `backend/src/core/logger.py` to incorporate essential logging logic from `superset_tool/utils/logging.py`
- [x] T005 Update `Environment` model in `backend/src/core/config_models.py` with `verify_ssl` and `timeout` fields
## Phase 3: User Story 1 - Maintain Core Functionality [US1]
- [x] T006 [US1] Refactor `backend/src/core/superset_client.py` to be self-contained (remove inheritance from `superset_tool.client.BaseSupersetClient`)
- [x] T007 [US1] Migrate all remaining methods from `superset_tool/client.py` to `backend/src/core/superset_client.py`
- [x] T008 [P] [US1] Update all imports in `backend/src/` to remove references to `superset_tool`
- [x] T009 [US1] Verify backend starts without `superset_tool` directory present
- [x] T010 [US1] Verify dashboard and dataset fetching works via API/UI
## Phase 4: User Story 2 - Unified Configuration and Models [US2]
- [x] T011 [US2] Ensure all Pydantic models in `backend/src/core/config_models.py` cover legacy `superset_tool/models.py` requirements
- [x] T012 [US2] Update database loading logic to correctly populate new `Environment` fields
## Phase 5: Polish & Cleanup
- [x] T013 Remove `superset_tool/` directory
- [x] T014 Remove deprecated root scripts: `migration_script.py`, `run_mapper.py`, `whiptail_fallback.py`
- [x] T015 Run full backend test suite to ensure no regressions
## Dependencies
- Phase 2 must be completed before Phase 3.
- T006 and T007 are prerequisites for T008 and T009.
## Implementation Strategy
- **MVP First**: Focus on T006-T008 to ensure the backend can function independently of the `superset_tool` module.
- **Incremental Delivery**: Migrate utilities and models first to provide the necessary foundation for the client refactoring.