busya
6988e63967
- Agent configs (.opencode/agents/) - Backend: alembic, routes, app, utils, scripts - Frontend: package.json, vite, components, e2e infra - Specs: 028-llm-datasource-supeset updates - Docker e2e config and Playwright setup
186 lines
15 KiB
Markdown
186 lines
15 KiB
Markdown
# Implementation Plan: LLM Table Translation Service
|
||
|
||
**Branch**: `028-llm-datasource-supeset` | **Date**: 2026-05-08 | **Spec**: [spec.md](./spec.md)
|
||
**Input**: Feature specification from `/specs/028-llm-datasource-supeset/spec.md` (updated 2026-05-14 with multi-language & correction optimization)
|
||
|
||
## Summary
|
||
|
||
Implement an LLM-powered table translation service as a new backend plugin (`TranslationPlugin`) with companion API routes, ORM models, Pydantic schemas, and Svelte frontend components. The service reads rows from a Superset datasource (translation column + context columns), auto-detects source language per row via LLM, sends batches to a configured LLM provider for ALL target languages simultaneously (multi-target in single structured JSON response), with optional per-batch filtered multilingual terminology dictionary context (language-pair-aware). Generates safe PostgreSQL INSERT/UPSERT SQL, submits to Superset via `/api/v1/sqllab/execute/` with status polling. Supporting capabilities: multilingual terminology dictionaries (source_language + target_language per entry), configurable preview (1-100 rows), inline correction from any run result with dictionary submission, bulk find-and-replace, scheduled execution via APScheduler (new-key-only with baseline_expired fallback), structured event logging with per-language MetricSnapshot persistence, RBAC-gated access control, and 90-day retention with metric continuity.
|
||
|
||
**Total implemented**: 72 files, 29 128 LOC across backend (Python/FastAPI) and frontend (Svelte 5/SvelteKit).
|
||
**Actual complexity distribution**: 2× C5, 6× C4, 12× C3, 7× C2, 4× C1.
|
||
**Note**: Архитектура — service-based, не plugin-based. `TranslatePlugin` — только регистрационный скелет (C2). Бизнес-логика через независимые service-классы.
|
||
|
||
## Technical Context
|
||
|
||
**Language/Version**: Python 3.13+ (backend), JavaScript/TypeScript (frontend Svelte 5)
|
||
**Primary Dependencies**: FastAPI 0.115+, SQLAlchemy 2.0+, APScheduler 3.x, Pydantic v2 (backend); SvelteKit 2.x, Svelte 5.43+, Vite 7.x, Tailwind CSS 3.x (frontend)
|
||
**Storage**: PostgreSQL 16 (shared with existing ss-tools schema)
|
||
**Testing**: pytest 8.x + pytest-asyncio (backend); vitest 4.x + @testing-library/svelte 5.x (frontend)
|
||
**Target Platform**: Linux server (Docker Compose: backend + frontend + PostgreSQL)
|
||
**Project Type**: web application — FastAPI REST + WebSocket backend, SvelteKit SPA frontend
|
||
**Performance Goals**: Preview of 10 rows within 30s (LLM-dependent); INSERT generation <5s for 1000 rows; schedule trigger precision ±60s; event recording <10s after occurrence
|
||
**Constraints**: RBAC enforcement via access-control matrix; dictionary per-batch filtering (case-insensitive, word-boundary-aware); 90-day detailed data retention with MetricSnapshot persistence; plugin lifecycle compatible with `PluginBase`; dialect-aware SQL generation (PostgreSQL/Greenplum, ClickHouse supported for MVP); snapshot isolation for in-progress runs; structured JSON LLM output required
|
||
**Scale/Scope**: Tens of thousands of rows per run; dictionaries of any size; 50 concurrent users; 5–10 active translation jobs per deployment
|
||
|
||
## Constitution Check
|
||
|
||
*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
|
||
|
||
| Principle | Status | Evidence |
|
||
|-----------|--------|----------|
|
||
| **I. Plugin Architecture** | ✅ PASS | Feature implemented as a new plugin inheriting from `PluginBase` (`backend/src/plugins/translate/`), consistent with existing `llm_analysis`, `git`, `storage` plugin pattern. |
|
||
| **II. API-First Design** | ✅ PASS | All operations exposed via REST endpoints under `/api/translate/` (jobs, dictionaries, runs, schedules, history) and WebSocket for run progress. Follows existing FastAPI route conventions. |
|
||
| **III. Test-First** | ✅ PASS | Acceptance criteria per user story defined in spec; test strategy in research.md covers unit (pytest), integration (API + DB), and component (vitest + Svelte Testing Library) layers. |
|
||
| **IV. RBAC & Security** | ✅ PASS | Granular permissions (`translate.job.*`, `translate.dictionary.*`, `translate.schedule.manage`, `translate.history.view`) via existing RBAC model. API keys encrypted via `EncryptionManager`. PII masking for LLM-facing context per existing patterns. |
|
||
| **V. Observability & Retention** | ✅ PASS | Structured Translation Events (FR-046), per-job metrics (FR-047), notification integration (FR-048), 90-day retention with pruning (FR-049). C4+ flows instrumented with `belief_scope`/`reason`/`reflect`/`explore` markers. |
|
||
| **Semantic Protocol Compliance** | ✅ PASS | All planned modules assigned `@COMPLEXITY` 1–5 with appropriate tag density per GRACE complexity scale (C1: anchor only, C2: +BRIEF, C3: +RELATION, C4: +PRE/POST/SIDE_EFFECT, C5: +DATA_CONTRACT/INVARIANT/RATIONALE/REJECTED). `@RATIONALE`/`@REJECTED` reserved for C5 contracts only. Canonical `@RELATION PREDICATE -> TARGET_ID` syntax. Anchor format: `#region`/`#endregion` for Python, `<!-- #region -->`/`<!-- #endregion -->` for Svelte markup. |
|
||
| **Fractal Limit (INV_7)** | ✅ PASS | Planned modules kept under 400 lines each. No planned contract exceeds 150 lines or CC>10. Decomposition strategy documented in contracts/modules.md. |
|
||
|
||
**Gate Result**: ✅ ALL PASS — no blocking constitutional or semantic conflicts.
|
||
|
||
## Project Structure
|
||
|
||
### Documentation (this feature)
|
||
|
||
```text
|
||
specs/028-llm-datasource-supeset/
|
||
├── plan.md # This file
|
||
├── research.md # Phase 0 output
|
||
├── data-model.md # Phase 1 output
|
||
├── quickstart.md # Phase 1 output
|
||
├── contracts/ # Phase 1 output
|
||
│ └── modules.md # Semantic contract design
|
||
├── spec.md # Feature specification
|
||
├── ux_reference.md # Interaction reference
|
||
└── checklists/
|
||
└── requirements.md # Quality validation
|
||
```
|
||
|
||
### Source Code (repository root — ACTUAL as implemented)
|
||
|
||
```text
|
||
backend/
|
||
├── src/
|
||
│ ├── api/routes/
|
||
│ │ └── translate/ # [ACTUAL] Пакет, а не один файл
|
||
│ │ ├── __init__.py # Инициализация, re-export router
|
||
│ │ ├── _router.py # APIRouter(prefix="/api/translate")
|
||
│ │ ├── _job_routes.py # Job CRUD (249 строк)
|
||
│ │ ├── _run_routes.py # Run execution, cancel, inline edit (554 строки)
|
||
│ │ ├── _run_list_routes.py # Run list, detail, CSV download (225 строк)
|
||
│ │ ├── _preview_routes.py # Preview CRUD (194 строки)
|
||
│ │ ├── _dictionary_routes.py # Dictionary CRUD + import (391 строка)
|
||
│ │ ├── _correction_routes.py # Correction submission (100 строк)
|
||
│ │ ├── _schedule_routes.py # Schedule CRUD (239 строк)
|
||
│ │ ├── _metrics_routes.py # Metrics endpoints (65 строк)
|
||
│ │ └── _helpers.py # Response helpers (71 строка)
|
||
│ ├── core/
|
||
│ │ ├── scheduler.py # [REUSE] APScheduler
|
||
│ │ └── superset_client/ # [REUSE] Superset API client
|
||
│ ├── models/
|
||
│ │ └── translate.py # [NEW] 15 ORM models (397 строк)
|
||
│ ├── plugins/
|
||
│ │ └── translate/ # [NEW] TranslationPlugin
|
||
│ │ ├── __init__.py # Package (2 строки)
|
||
│ │ ├── plugin.py # Plugin skeleton C2 (61 строка, регистрация)
|
||
│ │ ├── orchestrator.py # C5 Run lifecycle (1 137 строк) ⭐
|
||
│ │ ├── executor.py # C4 Batch LLM (1 974 строки) ⭐
|
||
│ │ ├── service.py # C4 TranslateJobService + InlineCorrection + BulkReplace (1 052 строки) ⭐
|
||
│ │ ├── preview.py # C4 Preview engine (1 303 строки) ⭐
|
||
│ │ ├── dictionary.py # C4 Dictionary CRUD + filtering (1 007 строк) ⭐
|
||
│ │ ├── scheduler.py # C4 Schedule management (436 строк)
|
||
│ │ ├── sql_generator.py # C3 Dialect-aware SQL (378 строк)
|
||
│ │ ├── superset_executor.py # C3 Superset SQL Lab API (420 строк) [NEW в реализации]
|
||
│ │ ├── prompt_builder.py # C3 Context-aware prompt (153 строки) [NEW]
|
||
│ │ ├── events.py # C5 Event log + pruning (272 строки)
|
||
│ │ ├── metrics.py # C3 Metrics aggregation (192 строки)
|
||
│ │ ├── _token_budget.py # C2 Token estimation (341 строка) [NEW]
|
||
│ │ └── _utils.py # C1 Normalization (33 строки) [NEW]
|
||
│ │ └── __tests__/ # Plugin-level unit tests (8 файлов, 5 645 строк)
|
||
│ ├── schemas/
|
||
│ │ └── translate.py # [NEW] 35+ Pydantic schemas (688 строк)
|
||
│ └── services/
|
||
│ ├── llm_provider.py # [REUSE] LLM provider management
|
||
│ └── llm_prompt_templates.py # [REUSE] Prompt rendering
|
||
└── tests/ # Integration tests (7 файлов, 2 458 строк)
|
||
|
||
frontend/
|
||
├── src/
|
||
│ ├── routes/
|
||
│ │ └── translate/ # [NEW] SvelteKit routes
|
||
│ │ ├── +page.svelte # Job list (279 строк)
|
||
│ │ ├── [id]/
|
||
│ │ │ └── +page.svelte # Job config + preview + run (1 363 строки) ⭐
|
||
│ │ ├── dictionaries/
|
||
│ │ │ ├── +page.svelte # Dictionary list (270 строк)
|
||
│ │ │ └── [id]/
|
||
│ │ │ └── +page.svelte # Dictionary editor (821 строка)
|
||
│ │ └── history/
|
||
│ │ └── +page.svelte # Run history (523 строки)
|
||
│ └── lib/
|
||
│ ├── components/
|
||
│ │ └── translate/ # [NEW] Translation components (13 файлов)
|
||
│ │ ├── TranslationPreview.svelte # C4 (561 строка)
|
||
│ │ ├── TranslationRunResult.svelte # C4 (464 строки)
|
||
│ │ ├── BulkReplaceModal.svelte # C3 (396 строк) [NEW]
|
||
│ │ ├── TranslationRunProgress.svelte # C4 (310 строк)
|
||
│ │ ├── ScheduleConfig.svelte # C3 (286 строк)
|
||
│ │ ├── BulkCorrectionSidebar.svelte # C3 (285 строк)
|
||
│ │ ├── TermCorrectionPopup.svelte # C3 (342 строки)
|
||
│ │ ├── CorrectionCell.svelte # C3 (326 строк) [NEW]
|
||
│ │ ├── TranslationRunGlobalIndicator.svelte # C2 (198 строк) [NEW]
|
||
│ │ ├── TranslationMetricsDashboard.svelte # C2 (184 строки) [NEW]
|
||
│ │ ├── DictionaryEditor.svelte # C3 (на странице)
|
||
│ │ └── DictionaryList.svelte # C3 (на странице)
|
||
│ ├── stores/
|
||
│ │ ├── translate.js # [NEW] rune store (196 строк) [NEW в реализации]
|
||
│ │ └── translationRun.js # [NEW] run progress store
|
||
│ └── api/
|
||
│ └── translate.js # [NEW] API client (664 строки)
|
||
└── e2e/
|
||
└── tests/
|
||
└── translation.e2e.js # [NEW] E2E тесты (142 строки)
|
||
```
|
||
|
||
## Semantic Contract Guidance
|
||
|
||
- All planned contracts follow GRACE-Poly v2.6 protocol with `#region`/`#endregion` anchors (or `[DEF:...]`/`[/DEF:...]` for backward compatibility in docs).
|
||
- Python backend uses `# #region`/`# #endregion` comment-anchor syntax.
|
||
- Svelte components use `<!-- #region -->`/`<!-- #endregion -->` in markup and `// #region`/`// #endregion` in script blocks.
|
||
- Complexity assignments follow strict tag density per GRACE complexity scale (detailed in `contracts/modules.md`).
|
||
- C4+ Python modules instrumented with `belief_scope()` + `reason()`/`reflect()`/`explore()` markers.
|
||
- C5 contracts carry `@RATIONALE` and `@REJECTED` decision memory (C1-C4 MUST NOT have these tags).
|
||
- `@RELATION` targets reference existing contracts where applicable (e.g., `[LLMProviderService:Module]`, `[SchedulerService:Class]`, `[SupersetClient:Module]`, `[NotificationService:Module]`).
|
||
|
||
## Complexity Tracking
|
||
|
||
No constitutional violations detected. **UPDATED 2026-05-17**: Complexity скорректирована по факту реализации.
|
||
|
||
| Contract | Plan (C?) | Actual (C?) | Файл | Примечание |
|
||
|----------|:---------:|:-----------:|------|------------|
|
||
| `TranslatePlugin:Module` | C3 | **C2** | `plugin.py` | Только регистрация; execute() — NotImplementedError |
|
||
| `TranslationOrchestrator` | C5 | **C5** ✅ | `orchestrator.py` | Stateful lifecycle, PRE/POST, multi-step coordination |
|
||
| `TranslationEventLog` | C5 | **C5** ✅ | `events.py` | Immutable event log + retention + MetricSnapshot |
|
||
| `TranslationExecutor` | C4 | **C4** ✅ | `executor.py` | Batch LLM, caching, parsing, retry |
|
||
| `TranslationPreview` | C4 | **C4** ✅ | `preview.py` | Quality gate, approve/edit/reject |
|
||
| `DictionaryManager` | C4 | **C4** ✅ | `dictionary.py` | CRUD + import + per-batch filtering |
|
||
| `TranslationScheduler` | C4 | **C4** ✅ | `scheduler.py` | APScheduler integration |
|
||
| `TranslateJobService` | — | **C4** [NEW] | `service.py` | Job CRUD + InlineCorrection + BulkReplace |
|
||
| `TranslateRoutes` | C3 | **C4** 🔴 | `__init__.py` | Пакет с 11 подмодулями |
|
||
| `SQLGenerator` | C4 | **C3** 🟢 | `sql_generator.py` | Pure function, нет I/O |
|
||
| `SupersetSqlLabExecutor` | — | **C3** [NEW] | `superset_executor.py` | Superset SQL Lab API клиент |
|
||
| `ContextAwarePromptBuilder` | — | **C3** [NEW] | `prompt_builder.py` | Jaccard similarity, priority |
|
||
| `InlineCorrectionService` | C4 | **C3** 🟢 | `service.py` | Context capture, origin tracking |
|
||
| `BulkFindReplaceService` | C4 | **C3** 🟢 | `service.py` | Regex, preview, atomic apply |
|
||
| `TranslationMetrics` | C3 | **C3** ✅ | `metrics.py` | Metrics aggregation |
|
||
| `TranslationModels` | C2 | **C2** ✅ | `models/translate.py` | 15 ORM models |
|
||
| `TranslateSchemas` | C2 | **C2** ✅ | `schemas/translate.py` | 35+ Pydantic schemas |
|
||
| `TranslateApiClient` | C2 | **C2** ✅ | `api/translate.js` | API client |
|
||
| `translateStore` | C3 | **C3** ✅ | `translationRun.js` | Svelte 5 rune store |
|
||
| `TranslationLanguage` | C1 | **C1** ✅ | `models/translate.py` | DTO |
|
||
| `TranslationRunLanguageStats` | C1 | **C1** ✅ | `models/translate.py` | DTO |
|
||
| `CorrectionCell` (Svelte) | — | **C3** [NEW] | `CorrectionCell.svelte` | Inline edit + submit |
|
||
| `BulkReplaceModal` (Svelte) | — | **C3** [NEW] | `BulkReplaceModal.svelte` | Find & replace modal |
|
||
| Other Svelte components | C3 | **C3** ✅ | — | Все 10 компонентов |
|