ss-tools/specs/028-llm-datasource-supeset/plan.md

Implementation Plan: LLM Table Translation Service

Branch: 028-llm-datasource-supeset (актуальная: 032-translate-requests-httpx) | Date: 2026-05-08 (актуализация: 2026-06-11) | Spec: spec.md Input: Feature specification from /specs/028-llm-datasource-supeset/spec.md (updated 2026-06-11 — Core + Enhancement complete) Enhancement Phase: ✅ COMPLETE — User Stories 11–12 (Direct DB Insert + Database Connection Settings)

Summary

Implement an LLM-powered table translation service as a new backend plugin (TranslationPlugin) with companion API routes, ORM models, Pydantic schemas, and Svelte frontend components. The service reads rows from a Superset datasource (translation column + context columns), auto-detects source language per row via LLM, sends batches to a configured LLM provider for ALL target languages simultaneously (multi-target in single structured JSON response), with optional per-batch filtered multilingual terminology dictionary context (language-pair-aware). Generates safe PostgreSQL INSERT/UPSERT SQL, submits to Superset via /api/v1/sqllab/execute/ with status polling. Supporting capabilities: multilingual terminology dictionaries (source_language + target_language per entry), configurable preview (1-100 rows), inline correction from any run result with dictionary submission, bulk find-and-replace, scheduled execution via APScheduler (new-key-only with baseline_expired fallback), structured event logging with per-language MetricSnapshot persistence, RBAC-gated access control, and 90-day retention with metric continuity.

Total implemented: ~174 files, ~39 400 LOC across backend (Python/FastAPI) and frontend (Svelte 5/SvelteKit).
Actual complexity distribution: 2× C5, 6× C4, 14× C3, 9× C2, 5× C1.
Plugins: 59 source files + 2 core services + 13 routes + 1 model + 1 schema + 30 frontend source + 62 test files.

Enhancement scope:

• DbExecutor (C3) ✅ — direct DB execution engine with asyncpg + clickhouse-connect pooling
• ConnectionService (C3) ✅ — CRUD for DB connections in Settings (encrypt/decrypt + test connectivity)
• ConnectionModel (C1) ✅ — Pydantic model for GlobalSettings.connections
• ConnectionsTab.svelte (C2) ✅ — Settings UI tab with CRUD + test button
• InsertMethodSelector component (C1) ✅ — insert method choice in job config
• TranslationJob model update ✅ — insert_method, connection_id columns added
• TranslationRun model update ✅ — insert_method, connection_snapshot columns added
• TranslationOrchestrator update ✅ — dispatch to DbExecutor when insert_method=direct_db
• Settings routes update ✅ — +5 connection CRUD endpoints
• New Alembic migration ✅ — connections table + job/run columns

Technical Context

Language/Version: Python 3.13+ (backend), JavaScript/TypeScript (frontend Svelte 5)
Primary Dependencies: FastAPI 0.115+, SQLAlchemy 2.0+, APScheduler 3.x, Pydantic v2 (backend); SvelteKit 2.x, Svelte 5.43+, Vite 7.x, Tailwind CSS 3.x (frontend)
Storage: PostgreSQL 16 (shared with existing superset-tools schema)
Testing: pytest 8.x + pytest-asyncio (backend); vitest 4.x + @testing-library/svelte 5.x (frontend)
Target Platform: Linux server (Docker Compose: backend + frontend + PostgreSQL)
Project Type: web application — FastAPI REST + WebSocket backend, SvelteKit SPA frontend
Performance Goals: Preview of 10 rows within 30s (LLM-dependent); INSERT generation <5s for 1000 rows; schedule trigger precision ±60s; event recording <10s after occurrence
Constraints: RBAC enforcement via access-control matrix; dictionary per-batch filtering (case-insensitive, word-boundary-aware); 90-day detailed data retention with MetricSnapshot persistence; plugin lifecycle compatible with PluginBase; dialect-aware SQL generation (PostgreSQL/Greenplum, ClickHouse supported for MVP); snapshot isolation for in-progress runs; structured JSON LLM output required
Scale/Scope: Tens of thousands of rows per run; dictionaries of any size; 50 concurrent users; 5–10 active translation jobs per deployment

Constitution Check

GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.

Principle	Status	Evidence
I. Plugin Architecture	✅ PASS	Feature implemented as a new plugin inheriting from PluginBase (backend/src/plugins/translate/), consistent with existing llm_analysis, git, storage plugin pattern.
II. API-First Design	✅ PASS	All operations exposed via REST endpoints under /api/translate/ (jobs, dictionaries, runs, schedules, history) and WebSocket for run progress. Follows existing FastAPI route conventions.
III. Test-First	✅ PASS	Acceptance criteria per user story defined in spec; test strategy in research.md covers unit (pytest), integration (API + DB), and component (vitest + Svelte Testing Library) layers.
IV. RBAC & Security	✅ PASS	Granular permissions (translate.job.*, translate.dictionary.*, translate.schedule.manage, translate.history.view) via existing RBAC model. API keys encrypted via EncryptionManager. PII masking for LLM-facing context per existing patterns.
V. Observability & Retention	✅ PASS	Structured Translation Events (FR-046), per-job metrics (FR-047), notification integration (FR-048), 90-day retention with pruning (FR-049). C4+ flows instrumented with belief_scope/reason/reflect/explore markers.
Semantic Protocol Compliance	✅ PASS	All planned modules assigned @COMPLEXITY 1–5 with appropriate tag density per GRACE complexity scale (C1: anchor only, C2: +BRIEF, C3: +RELATION, C4: +PRE/POST/SIDE_EFFECT, C5: +DATA_CONTRACT/INVARIANT/RATIONALE/REJECTED). @RATIONALE/@REJECTED reserved for C5 contracts only. Canonical @RELATION PREDICATE -> TARGET_ID syntax. Anchor format: #region/#endregion for Python, <!-- #region -->/<!-- #endregion --> for Svelte markup.
Fractal Limit (INV_7)	✅ PASS	Planned modules kept under 400 lines each. No planned contract exceeds 150 lines or CC>10. Decomposition strategy documented in contracts/modules.md.

Gate Result: ✅ ALL PASS — no blocking constitutional or semantic conflicts.

Re-check 2026-06-11: Enhancement does not introduce new constitutional conflicts. ConnectionService and DbExecutor are core services per ADR-0004 (not subprocess plugins).

Project Structure

Documentation (this feature)

specs/028-llm-datasource-supeset/
├── plan.md              # This file
├── research.md          # Phase 0 output
├── data-model.md        # Phase 1 output
├── quickstart.md        # Phase 1 output
├── contracts/           # Phase 1 output
│   └── modules.md       # Semantic contract design
├── spec.md              # Feature specification
├── ux_reference.md      # Interaction reference
└── checklists/
    └── requirements.md  # Quality validation

Source Code (repository root — ACTUAL as implemented)

backend/
├── src/
│   ├── api/routes/
│   │   └── translate/             # [ACTUAL] Пакет из 13 файлов (2 296 строк)
│   │       ├── __init__.py        # Инициализация, re-export router
│   │       ├── _router.py         # APIRouter(prefix="/api/translate")
│   │       ├── _job_routes.py     # Job CRUD (285 строк)
│   │       ├── _run_routes.py     # Run execution, cancel, inline edit (205 строк)
│   │       ├── _run_edit_routes.py # Run edit endpoints (200 строк) [NEW]
│   │       ├── _run_list_routes.py # Run list, detail, CSV (359 строк)
│   │       ├── _run_history_routes.py # Run history (133 строки) [NEW]
│   │       ├── _preview_routes.py  # Preview CRUD (194 строки)
│   │       ├── _dictionary_routes.py # Dictionary CRUD + import (391 строка)
│   │       ├── _correction_routes.py # Correction submission (100 строк)
│   │       ├── _schedule_routes.py   # Schedule CRUD (239 строк)
│   │       ├── _metrics_routes.py    # Metrics endpoints (65 строк)
│   │       └── _helpers.py           # Response helpers (70 строк)
│   ├── core/
│   │   ├── scheduler.py           # [REUSE] APScheduler
│   │   └── superset_client/       # [REUSE] Superset API client
│   ├── models/
│   │   └── translate.py           # [NEW] 15 ORM models (401 строка)
│   ├── plugins/
│   │   └── translate/             # [NEW] TranslationPlugin (59 файлов, декомпозирован)
│   │       ├── __init__.py        # Package (2 строки)
│   │       ├── plugin.py          # Plugin skeleton C2 (61 строка)
│   │       ├── orchestrator*.py   # Orchestrator C5 (14 файлов, 1 631 строка) ⭐
│   │       ├── executor*.py       # Executor/Batch/LLM C4 (9 файлов, 2 196 строк) ⭐
│   │       ├── preview*.py        # Preview engine C4 (11 файлов, 1 291 строка) ⭐
│   │       ├── dictionary*.py     # Dictionary CRUD C4 (7 файлов, 921 строка) ⭐
│   │       ├── service*.py        # Service layer C4 (6 файлов, 1 281 строка) ⭐
│   │       ├── scheduler.py       # Schedule management C4 (443 строки)
│   │       ├── sql_generator.py   # Dialect-aware SQL C3 (402 строки)
│   │       ├── superset_executor.py # Superset SQL Lab API C3 (439 строк)
│   │       ├── events.py          # Event log C5 (274 строки)
│   │       ├── metrics.py         # Metrics C3 (192 строки)
│   │       ├── prompt_builder.py  # Context-aware prompt C3 (153 строки)
│   │       ├── _token_budget.py   # Token estimation C2 (395 строк)
│   │       ├── _lang_detect.py    # Language detection C2 (207 строк) [NEW]
│   │       ├── _text_cleaner.py   # Text sanitization C1 (63 строки) [NEW]
│   │       └── _utils.py          # Utilities C1 (225 строк) [UPDATED]
│   │       └── __tests__/         # Plugin-level unit tests (27 файлов, 9 090 строк)
│   ├── schemas/
│   │   └── translate.py           # [NEW] 35+ Pydantic schemas (712 строк)
│   └── services/
│       ├── llm_provider.py        # [REUSE] LLM provider management
│       └── llm_prompt_templates.py # [REUSE] Prompt rendering
└── tests/                         # Integration tests (7 файлов, 2 458 строк)

frontend/
├── src/
│   ├── routes/
│   │   └── translate/             # [NEW] SvelteKit routes
│   │       ├── +page.svelte       # Job list (288 строк)
│   │       ├── [id]/
│   │       │   └── +page.svelte   # Job config + preview + run (269 строк)
│   │       ├── dictionaries/
│   │       │   ├── +page.svelte   # Dictionary list (247 строк)
│   │       │   └── [id]/
│   │       │       └── +page.svelte # Dictionary editor (202 строки)
│   │       └── history/
│   │           └── +page.svelte   # Run history (231 строка)
│   └── lib/
│       ├── components/
│       │   └── translate/         # [NEW] Translation components (14 файлов, 5 017 строк)
│       │       ├── TranslationPreview.svelte       # C4 (564 строки)
│       │       ├── TranslationRunResult.svelte     # C4 (618 строк)
│       │       ├── ConfigTabForm.svelte            # C3 (539 строк) [NEW]
│       │       ├── BulkReplaceModal.svelte         # C3 (438 строк)
│       │       ├── CorrectionCell.svelte           # C3 (343 строки)
│       │       ├── RunTabContent.svelte            # C3 (342 строки) [NEW]
│       │       ├── TermCorrectionPopup.svelte      # C3 (345 строк)
│       │       ├── TargetSchemaHint.svelte         # C2 (316 строк) [NEW]
│       │       ├── ScheduleConfig.svelte           # C3 (300 строк)
│       │       ├── BulkCorrectionSidebar.svelte    # C3 (288 строк)
│       │       ├── TranslationRunProgress.svelte   # C4 (277 строк)
│       │       ├── TargetTabForm.svelte            # C3 (224 строки) [NEW]
│       │       ├── TranslationRunGlobalIndicator.svelte # C2 (237 строк)
│       │       └── TranslationMetricsDashboard.svelte   # C2 (186 строк)
│       ├── stores/
│       │   └── translationRun.svelte.ts  # [NEW] rune store (370 строк)
│       └── api/
│           └── translate/              # [NEW] API client (8 файлов, 987 строк)
└── e2e/
    └── tests/
        └── translation.e2e.js     # [NEW] E2E тесты (142 строки)

### Enhancement Phase (NEW — 2026-06-10)

```text
backend/
├── src/
│   ├── core/
│   │   ├── config_models.py       # [UPDATE] DatabaseConnection model, GlobalSettings.connections promoted
│   │   └── db_executor.py         # [NEW] DbExecutor C3 — direct DB execution with native drivers (asyncpg, clickhouse-connect)
│   ├── api/routes/
│   │   └── settings.py            # [UPDATE] Connection CRUD endpoints (+5 endpoints)
│   ├── plugins/translate/
│   │   ├── superset_executor.py   # [UPDATE] Refactor: extract SQL execution interface, keep as impl A
│   │   └── orchestrator.py        # [UPDATE] Dispatch to correct executor based on insert_method
│   ├── models/
│   │   └── translate.py           # [UPDATE] Add insert_method, connection_id, connection_snapshot columns
│   └── schemas/
│       └── translate.py           # [UPDATE] Add insert_method, connection_id to job/run schemas
└── tests/
    ├── test_db_executor.py        # [NEW] Test direct DB execution
    └── test_settings_connections.py # [NEW] Test connection CRUD API

frontend/
├── src/
│   ├── routes/settings/
│   │   └── ConnectionsTab.svelte  # [NEW] C2 — connection list + add/edit form + test button
│   └── lib/components/translate/
│       └── InsertMethodSelector.svelte # [NEW] C1 — insert method dropdown + connection selector

## Semantic Contract Guidance

- All planned contracts follow GRACE-Poly v2.6 protocol with `#region`/`#endregion` anchors (or `[DEF:...]`/`[/DEF:...]` for backward compatibility in docs).
- Python backend uses `# #region`/`# #endregion` comment-anchor syntax.
- Svelte components use `<!-- #region -->`/`<!-- #endregion -->` in markup and `// #region`/`// #endregion` in script blocks.
- Complexity assignments follow strict tag density per GRACE complexity scale (detailed in `contracts/modules.md`).
- C4+ Python modules instrumented with `belief_scope()` + `reason()`/`reflect()`/`explore()` markers.
- C5 contracts carry `@RATIONALE` and `@REJECTED` decision memory (C1-C4 MUST NOT have these tags).
- `@RELATION` targets reference existing contracts where applicable (e.g., `[LLMProviderService:Module]`, `[SchedulerService:Class]`, `[SupersetClient:Module]`, `[NotificationService:Module]`).

## Complexity Tracking

No constitutional violations detected. **UPDATED 2026-05-17**: Complexity скорректирована по факту реализации.

| Contract | Plan (C?) | Actual (C?) | Файл | Примечание |
|----------|:---------:|:-----------:|------|------------|
| `TranslatePlugin:Module` | C3 | **C2** | `plugin.py` | Только регистрация; execute() — NotImplementedError |
| `TranslationOrchestrator` | C5 | **C5** ✅ | `orchestrator.py` | Stateful lifecycle, PRE/POST, multi-step coordination |
| `TranslationEventLog` | C5 | **C5** ✅ | `events.py` | Immutable event log + retention + MetricSnapshot |
| `TranslationExecutor` | C4 | **C4** ✅ | `executor.py` | Batch LLM, caching, parsing, retry |
| `TranslationPreview` | C4 | **C4** ✅ | `preview.py` | Quality gate, approve/edit/reject |
| `DictionaryManager` | C4 | **C4** ✅ | `dictionary.py` | CRUD + import + per-batch filtering |
| `TranslationScheduler` | C4 | **C4** ✅ | `scheduler.py` | APScheduler integration |
| `TranslateJobService` | — | **C4** [NEW] | `service.py` | Job CRUD + InlineCorrection + BulkReplace |
| `TranslateRoutes` | C3 | **C4** 🔴 | `translate/` | Пакет с 13 подмодулями |
| `SQLGenerator` | C4 | **C3** 🟢 | `sql_generator.py` | Pure function, нет I/O |
| `SupersetSqlLabExecutor` | — | **C3** [NEW] | `superset_executor.py` | Superset SQL Lab API клиент |
| `ContextAwarePromptBuilder` | — | **C3** [NEW] | `prompt_builder.py` | Jaccard similarity, priority |
| `InlineCorrectionService` | C4 | **C3** 🟢 | `service.py` | Context capture, origin tracking |
| `BulkFindReplaceService` | C4 | **C3** 🟢 | `service.py` | Regex, preview, atomic apply |
| `TranslationMetrics` | C3 | **C3** ✅ | `metrics.py` | Metrics aggregation |
| `TranslationModels` | C2 | **C2** ✅ | `models/translate.py` | 15 ORM models |
| `TranslateSchemas` | C2 | **C2** ✅ | `schemas/translate.py` | 35+ Pydantic schemas (712 строк) |
| `TranslateApiClient` | C2 | **C2** ✅ | `api/translate/` (пакет) | 8-модульный API client |
| `translateStore` | C3 | **C3** ✅ | `translationRun.svelte.ts` | Svelte 5 rune store (370 строк) |
| `TranslationLanguage` | C1 | **C1** ✅ | `models/translate.py` | DTO |
| `TranslationRunLanguageStats` | C1 | **C1** ✅ | `models/translate.py` | DTO |
| `CorrectionCell` (Svelte) | — | **C3** [NEW] | `CorrectionCell.svelte` | Inline edit + submit |
| `BulkReplaceModal` (Svelte) | — | **C3** [NEW] | `BulkReplaceModal.svelte` | Find & replace modal |
| `ConfigTabForm` (Svelte) | — | **C3** [NEW] | `ConfigTabForm.svelte` | Config form |
| `RunTabContent` (Svelte) | — | **C3** [NEW] | `RunTabContent.svelte` | Run tab |
| `TargetTabForm` (Svelte) | — | **C3** [NEW] | `TargetTabForm.svelte` | Target tab |
| `TargetSchemaHint` (Svelte) | — | **C2** [NEW] | `TargetSchemaHint.svelte` | Schema hint |
| Other Svelte components | C3 | **C3** ✅ | — | Все 14 компонентов |

### Enhancement Phase Complexity — ✅ COMPLETE

| Contract | Plan (C?) | Actual (C?) | Файл | Примечание |
|----------|:---------:|:-----------:|------|------------|
| `DatabaseConnection` (model) | C1 | **C1** ✅ | `core/config_models.py` | Pydantic model for connection config + encrypt/decrypt |
| `DbExecutor` | C3 | **C3** ✅ | `core/db_executor.py` | Direct DB execution with asyncpg + clickhouse-connect pooling |
| `ConnectionService` | C3 | **C3** ✅ | `core/connection_service.py` | CRUD + encrypt/decrypt + test connectivity |
| `ConnectionsTab` (Svelte) | C2 | **C2** ✅ | `routes/settings/ConnectionsTab.svelte` | Settings UI tab with CRUD + test button |
| `InsertMethodSelector` (Svelte) | C1 | **C1** ✅ | `lib/components/translate/InsertMethodSelector.svelte` | Dropdown + connection selector |
| `TranslationJob` (model update) | C2 | **C2** ✅ | `models/translate.py` | Add insert_method, connection_id columns |
| `TranslationRun` (model update) | C2 | **C2** ✅ | `models/translate.py` | Add insert_method, connection_snapshot columns |
| `TranslationOrchestrator` (update) | C5 | **C5** ✅ | `plugins/translate/orchestrator.py` | Dispatch to DbExecutor when insert_method=direct_db |
| `Settings routes` (update) | C3 | **C3** ✅ | `api/routes/settings.py` | +5 connection CRUD endpoints |