Files
ss-tools/specs/028-llm-datasource-supeset/data-model.md
2026-05-14 15:27:41 +03:00

435 lines
23 KiB
Markdown

# Data Model: LLM Table Translation Service
**Feature Branch**: `028-llm-datasource-supeset`
**Date**: 2026-05-08 (updated post-review)
## 1. SQLAlchemy ORM Entities (dialect-aware — PostgreSQL/Greenplum, ClickHouse supported)
### TranslationJob (UPDATED — multi-language)
```python
# [DEF:TranslationJob:Class]
# @COMPLEXITY 2
# @PURPOSE Persisted configuration for a translation job (multi-language).
class TranslationJob(Base):
__tablename__ = "translation_jobs"
id = Column(String, primary_key=True, default=generate_uuid)
name = Column(String, nullable=False)
owner_id = Column(String, ForeignKey("users.id"), nullable=False)
# Source configuration
datasource_id = Column(String, nullable=False)
database_dialect = Column(String, nullable=False) # Detected from Superset: postgresql, clickhouse, greenplum
source_table = Column(String, nullable=True) # Optional metadata; datasource may be virtual
translation_col = Column(String, nullable=False)
context_cols = Column(JSON, default=list)
source_key_cols = Column(JSON, nullable=False) # [col_name, ...]
target_key_cols = Column(JSON, nullable=False) # [col_name, ...] — mapped to source_key_cols
# Target configuration
target_table = Column(String, nullable=False)
target_col = Column(String, nullable=False)
upsert_strategy = Column(String, default="insert") # insert | skip_existing | overwrite
# LLM configuration
provider_id = Column(String, ForeignKey("llm_providers.id"), nullable=True)
target_languages = Column(JSON, nullable=False) # [BCP-47, ...] — was target_language str, now list [UPDATED]
source_language = Column(String, nullable=True) # Optional fallback hint; auto-detected per row via LLM [DEPRECATED but preserved for migration]
prompt_template = Column(Text, nullable=True)
batch_size = Column(Integer, default=50)
# Timestamps
created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
updated_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), onupdate=lambda: datetime.now(timezone.utc))
owner = relationship("User")
dictionaries = relationship("TranslationJobDictionary", back_populates="job", cascade="all, delete-orphan")
schedule = relationship("TranslationSchedule", back_populates="job", uselist=False, cascade="all, delete-orphan")
runs = relationship("TranslationRun", back_populates="job")
preview_sessions = relationship("TranslationPreviewSession", back_populates="job")
# [/DEF:TranslationJob:Class]
```
### TranslationJobDictionary (M2M join with priority)
```python
class TranslationJobDictionary(Base):
__tablename__ = "translation_job_dictionaries"
id = Column(String, primary_key=True, default=generate_uuid)
job_id = Column(String, ForeignKey("translation_jobs.id"), nullable=False)
dictionary_id = Column(String, ForeignKey("terminology_dictionaries.id"), nullable=False)
priority = Column(Integer, default=0) # Lower = higher priority
job = relationship("TranslationJob", back_populates="dictionaries")
dictionary = relationship("TerminologyDictionary")
```
### TerminologyDictionary (UPDATED — multilingual)
```python
class TerminologyDictionary(Base):
__tablename__ = "terminology_dictionaries"
id = Column(String, primary_key=True, default=generate_uuid)
name = Column(String, nullable=False) # Organizational label only
# target_language removed — entries carry their own language pairs [UPDATED]
owner_id = Column(String, ForeignKey("users.id"), nullable=False)
created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
updated_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), onupdate=lambda: datetime.now(timezone.utc))
owner = relationship("User")
entries = relationship("DictionaryEntry", back_populates="dictionary", cascade="all, delete-orphan")
```
```
### DictionaryEntry (UPDATED — language-aware + context)
```python
class DictionaryEntry(Base):
__tablename__ = "dictionary_entries"
id = Column(String, primary_key=True, default=generate_uuid)
dictionary_id = Column(String, ForeignKey("terminology_dictionaries.id"), nullable=False)
source_term = Column(String, nullable=False)
source_term_normalized = Column(String, nullable=False) # Lowercase, NFC normalized
source_language = Column(String, nullable=False) # BCP-47 tag [NEW]
target_language = Column(String, nullable=False) # BCP-47 tag [NEW — was on dictionary]
target_term = Column(String, nullable=False)
# Context [NEW section]
context_data = Column(JSON, nullable=True) # Snapshot of source row's context columns { col_name: value, ... }
usage_notes = Column(Text, nullable=True) # Free-form user notes on when this translation applies
has_context = Column(Boolean, default=False) # Fast filter flag for prompt construction
context_source = Column(String, nullable=True) # "auto" | "auto_with_edits" | "manual" | "bulk"
# Origin metadata
origin_run_id = Column(String, ForeignKey("translation_runs.id"), nullable=True)
origin_row_key = Column(JSON, nullable=True)
origin_user_id = Column(String, ForeignKey("users.id"), nullable=True)
origin_source_language = Column(String, nullable=True) # Detected source language at correction time
created_at = Column(DateTime(timezone=True), default=datetime.now(timezone.utc))
dictionary = relationship("TerminologyDictionary", back_populates="entries")
__table_args__ = (
# Unique constraint updated to include language pair
UniqueConstraint("dictionary_id", "source_term_normalized", "source_language", "target_language", name="uq_dict_source_term_lang"),
Index("idx_dict_entry_lang", "source_language", "target_language"),
Index("idx_dict_has_context", "has_context"), # [NEW] — fast filter for context-only queries
)
```
### TranslationSchedule
```python
class TranslationSchedule(Base):
__tablename__ = "translation_schedules"
id = Column(String, primary_key=True, default=generate_uuid)
job_id = Column(String, ForeignKey("translation_jobs.id"), unique=True, nullable=False)
schedule_type = Column(String, nullable=False) # cron | interval | once
cron_expression = Column(String, nullable=True)
interval_seconds = Column(Integer, nullable=True)
run_at = Column(DateTime(timezone=True), nullable=True)
timezone = Column(String, default="UTC") # e.g., "Europe/Moscow"
is_enabled = Column(Boolean, default=True)
concurrency = Column(String, default="skip") # skip | queue
next_run_at = Column(DateTime(timezone=True), nullable=True)
created_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
updated_at = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), onupdate=lambda: datetime.now(timezone.utc))
job = relationship("TranslationJob", back_populates="schedule")
```
### TranslationRun
```python
class TranslationRun(Base):
__tablename__ = "translation_runs"
id = Column(String, primary_key=True, default=generate_uuid)
job_id = Column(String, ForeignKey("translation_jobs.id"), nullable=False)
trigger_type = Column(String, nullable=False) # manual | scheduled
translation_status = Column(String, default="pending") # pending|running|completed|partial|failed|cancelled|skipped
insert_status = Column(String, default="not_started") # not_started|submitted|running|succeeded|failed|skipped
# Statistics
total_rows = Column(Integer, default=0)
translated_rows = Column(Integer, default=0)
failed_rows = Column(Integer, default=0)
skipped_rows = Column(Integer, default=0)
token_count = Column(Integer, default=0)
estimated_cost = Column(Float, default=0.0)
# Hashes for idempotency and audit
config_hash = Column(String, nullable=True)
dict_snapshot_hash = Column(String, nullable=True)
# Snapshots
config_snapshot = Column(JSON, nullable=False)
dict_snapshot = Column(JSON, nullable=True)
prompt_used = Column(Text, nullable=True)
# SQL output
insert_sql = Column(Text, nullable=True)
sql_hash = Column(String, nullable=True)
# Superset execution reference
superset_query_id = Column(String, nullable=True)
superset_database_id = Column(String, nullable=True)
insert_error_type = Column(String, nullable=True)
insert_error_message = Column(Text, nullable=True)
rows_affected = Column(Integer, nullable=True)
# Timestamps
started_at = Column(DateTime(timezone=True), nullable=True)
completed_at = Column(DateTime(timezone=True), nullable=True)
insert_started_at = Column(DateTime(timezone=True), nullable=True)
insert_completed_at = Column(DateTime(timezone=True), nullable=True)
created_at = Column(DateTime(timezone=True), default=datetime.now(timezone.utc))
job = relationship("TranslationJob", back_populates="runs")
records = relationship("TranslationRecord", back_populates="run", cascade="all, delete-orphan")
batches = relationship("TranslationBatch", back_populates="run", cascade="all, delete-orphan")
events = relationship("TranslationEvent", back_populates="run")
language_stats = relationship("TranslationRunLanguageStats", back_populates="run", cascade="all, delete-orphan") # [NEW]
```
### TranslationBatch
```python
class TranslationBatch(Base):
__tablename__ = "translation_batches"
id = Column(String, primary_key=True, default=generate_uuid)
run_id = Column(String, ForeignKey("translation_runs.id"), nullable=False)
batch_index = Column(Integer, nullable=False)
status = Column(String, default="pending") # pending|running|completed|failed
row_count = Column(Integer, default=0)
translated_count = Column(Integer, default=0)
failed_count = Column(Integer, default=0)
skipped_count = Column(Integer, default=0)
token_count = Column(Integer, default=0)
estimated_cost = Column(Float, default=0.0)
latency_ms = Column(Integer, nullable=True)
error_type = Column(String, nullable=True)
error_message = Column(Text, nullable=True)
started_at = Column(DateTime(timezone=True), nullable=True)
completed_at = Column(DateTime(timezone=True), nullable=True)
run = relationship("TranslationRun", back_populates="batches")
records = relationship("TranslationRecord", back_populates="batch")
__table_args__ = (
Index("idx_batch_run_idx", "run_id", "batch_index"),
)
```
### TranslationRecord
```python
class TranslationRecord(Base):
__tablename__ = "translation_records"
id = Column(String, primary_key=True, default=generate_uuid)
run_id = Column(String, ForeignKey("translation_runs.id"), nullable=False)
batch_id = Column(String, ForeignKey("translation_batches.id"), nullable=True)
# Source data
source_text = Column(Text, nullable=True)
context_data = Column(JSON, nullable=True)
key_values = Column(JSON, nullable=False)
key_hash = Column(String, nullable=False) # hash(canonical_json(key_values))
# Translation result
llm_translation = Column(Text, nullable=True)
user_edit = Column(Text, nullable=True)
final_value = Column(Text, nullable=True)
status = Column(String, default="pending") # pending|translated|approved|edited|rejected|failed|skipped
error_message = Column(Text, nullable=True)
run = relationship("TranslationRun", back_populates="records")
batch = relationship("TranslationBatch", back_populates="records")
languages = relationship("TranslationLanguage", back_populates="record", cascade="all, delete-orphan") # [NEW]
__table_args__ = (
Index("idx_record_key_hash", "key_hash"),
Index("idx_record_run_key", "run_id", "key_hash"),
)
```
### TranslationLanguage (NEW — per-language translation storage)
```python
class TranslationLanguage(Base):
"""Stores per-language translation for a TranslationRecord.
One row per (record_id, language_code). This replaces direct language fields on TranslationRecord."""
__tablename__ = "translation_languages"
id = Column(String, primary_key=True, default=generate_uuid)
record_id = Column(String, ForeignKey("translation_records.id"), nullable=False)
language_code = Column(String, nullable=False) # BCP-47 tag, e.g., "ru", "en", "de"
source_language_detected = Column(String, nullable=True) # BCP-47 or "und" — auto-detected per row [NEW]
translated_value = Column(Text, nullable=True)
user_edit = Column(Text, nullable=True)
final_value = Column(Text, nullable=True)
status = Column(String, default="pending") # pending|translated|approved|edited|rejected|failed|skipped
error_message = Column(Text, nullable=True)
created_at = Column(DateTime(timezone=True), default=datetime.now(timezone.utc))
record = relationship("TranslationRecord", back_populates="languages")
__table_args__ = (
UniqueConstraint("record_id", "language_code", name="uq_record_language"),
Index("idx_tl_language", "language_code"),
Index("idx_tl_record_lang", "record_id", "language_code"),
)
```
### TranslationPreviewLanguage (NEW — per-language preview results)
```python
class TranslationPreviewLanguage(Base):
"""Stores per-language translation for a preview record."""
__tablename__ = "translation_preview_languages"
id = Column(String, primary_key=True, default=generate_uuid)
preview_record_id = Column(String, ForeignKey("translation_preview_records.id"), nullable=False)
language_code = Column(String, nullable=False) # BCP-47 tag
source_language_detected = Column(String, nullable=True) # BCP-47 or "und"
translated_value = Column(Text, nullable=True)
user_edit = Column(Text, nullable=True)
final_value = Column(Text, nullable=True)
status = Column(String, default="pending") # pending|approved|edited|rejected
created_at = Column(DateTime(timezone=True), default=datetime.now(timezone.utc))
preview_record = relationship("TranslationPreviewRecord", back_populates="languages")
__table_args__ = (
UniqueConstraint("preview_record_id", "language_code", name="uq_preview_record_language"),
)
```
### TranslationRunLanguageStats (NEW — per-language run statistics)
```python
class TranslationRunLanguageStats(Base):
"""Per-language statistics for a translation run."""
__tablename__ = "translation_run_language_stats"
id = Column(String, primary_key=True, default=generate_uuid)
run_id = Column(String, ForeignKey("translation_runs.id"), nullable=False)
language_code = Column(String, nullable=False) # BCP-47 tag
total_rows = Column(Integer, default=0)
translated_rows = Column(Integer, default=0)
failed_rows = Column(Integer, default=0)
skipped_rows = Column(Integer, default=0)
token_count = Column(Integer, default=0)
estimated_cost = Column(Float, default=0.0)
run = relationship("TranslationRun", back_populates="language_stats")
__table_args__ = (
UniqueConstraint("run_id", "language_code", name="uq_run_language"),
Index("idx_rls_run", "run_id"),
)
```
### TranslationPreviewSession
```python
class TranslationPreviewSession(Base):
__tablename__ = "translation_preview_sessions"
id = Column(String, primary_key=True, default=generate_uuid)
job_id = Column(String, ForeignKey("translation_jobs.id"), nullable=False)
user_id = Column(String, ForeignKey("users.id"), nullable=False)
config_hash = Column(String, nullable=False)
dict_snapshot_hash = Column(String, nullable=True)
sample_size = Column(Integer, default=10)
status = Column(String, default="pending") # pending|accepted|rejected
created_at = Column(DateTime(timezone=True), default=datetime.now(timezone.utc))
accepted_at = Column(DateTime(timezone=True), nullable=True)
expires_at = Column(DateTime(timezone=True), nullable=True)
job = relationship("TranslationJob", back_populates="preview_sessions")
rows = relationship("TranslationPreviewRecord", back_populates="session", cascade="all, delete-orphan")
```
### TranslationPreviewRecord
```python
class TranslationPreviewRecord(Base):
__tablename__ = "translation_preview_records"
id = Column(String, primary_key=True, default=generate_uuid)
session_id = Column(String, ForeignKey("translation_preview_sessions.id"), nullable=False)
source_text = Column(Text, nullable=True)
context_data = Column(JSON, nullable=True)
key_values = Column(JSON, nullable=False)
key_hash = Column(String, nullable=False)
llm_translation = Column(Text, nullable=True)
user_edit = Column(Text, nullable=True)
final_value = Column(Text, nullable=True)
status = Column(String, default="pending") # pending|approved|edited|rejected
session = relationship("TranslationPreviewSession", back_populates="rows")
languages = relationship("TranslationPreviewLanguage", back_populates="preview_record", cascade="all, delete-orphan") # [NEW]
```
### TranslationEvent
```python
class TranslationEvent(Base):
__tablename__ = "translation_events"
id = Column(String, primary_key=True, default=generate_uuid)
run_id = Column(String, ForeignKey("translation_runs.id"), nullable=True) # NULL for pre-run events
job_id = Column(String, ForeignKey("translation_jobs.id"), nullable=False)
event_type = Column(String, nullable=False)
# schedule_triggered|schedule_skipped|schedule_failed|
# run_started|batch_started|batch_completed|batch_failed|
# run_succeeded|run_partial|run_failed|run_cancelled|run_skipped|run_noop|
# insert_submitted|insert_succeeded|insert_failed
timestamp = Column(DateTime(timezone=True), default=datetime.now(timezone.utc), nullable=False)
payload = Column(JSON, default=dict)
run = relationship("TranslationRun", back_populates="events")
__table_args__ = (
Index("idx_event_job_ts", "job_id", "timestamp"),
Index("idx_event_type", "event_type"),
)
```
### MetricSnapshot
```python
class MetricSnapshot(Base):
__tablename__ = "translation_metric_snapshots"
id = Column(String, primary_key=True, default=generate_uuid)
job_id = Column(String, ForeignKey("translation_jobs.id"), nullable=False)
snapshot_date = Column(Date, nullable=False)
cumulative_tokens = Column(Integer, default=0)
cumulative_cost = Column(Float, default=0.0)
per_language_metrics = Column(JSON, nullable=True) # [NEW] { "ru": { "tokens": N, "cost": M }, ... }
covers_events_before = Column(DateTime(timezone=True), nullable=False) # Cutoff for event coverage
total_runs = Column(Integer, default=0)
success_runs = Column(Integer, default=0)
failed_runs = Column(Integer, default=0)
partial_runs = Column(Integer, default=0)
avg_batch_latency_ms = Column(Integer, nullable=True)
created_at = Column(DateTime(timezone=True), default=datetime.now(timezone.utc))
__table_args__ = (
UniqueConstraint("job_id", "snapshot_date", name="uq_metric_snapshot_date"),
)
```
## 2. Pydantic Schemas (API DTOs)
Key changes from original:
- `TranslateJobCreate`/`Update`: added `source_key_cols`, `target_key_cols` (explicit mapping), `source_language` → replaced by `target_languages: list[str]` [UPDATED 2026-05-14]
- `TermCorrectionSubmit`: added `source_term`, `incorrect_target_term`, `corrected_target_term`, `source_language` (auto-filled from detected), `target_language` [UPDATED]
- `ScheduleConfig`: added `timezone`
- `TranslationRunResponse`: split into `translation_status` + `insert_status`, added Superset execution fields, added `language_stats: list[LanguageStatsResponse]` [NEW]
- Added: `TranslationBatchResponse`, `PreviewSessionResponse`, `MetricsSnapshotResponse`
- Added: `TranslationLanguageResponse` [NEW] — wraps language_code, detected_source_language, translated_value, user_edit, final_value, status
- Added: `TranslationPreviewLanguageResponse` [NEW] — same structure for preview records
- Added: `BulkFindReplaceRequest` [NEW] — find_pattern (string, regex flag), replacement_text, target_language
- Added: `InlineCorrectionSubmit` [NEW] — record_id, language_code, source_term, corrected_term, dictionary_id, source_language
- Added: `CorrectionSubmitWithContext` [NEW] — extends InlineCorrectionSubmit with `context_data` (optional JSON override), `usage_notes` (optional text), `keep_context` (boolean, default=true — false means strip auto-captured context)
- Updated: `DictionaryEntryResponse` [UPDATED] — added `context_data`, `usage_notes`, `has_context`, `context_source` fields
- Updated: `BulkFindReplaceRequest` [UPDATED] — added `submit_to_dictionary_with_context` (boolean, default=false), `bulk_usage_notes` (optional text applied to all entries)
## 3. Entity Relationship Summary
```
TranslationJob (1) ──< (N) TranslationJobDictionary >── (1) TerminologyDictionary
TranslationJob (1) ──< (N) TranslationRun
TranslationJob (1) ──< (N) TranslationPreviewSession
TranslationJob (1) ──── (0..1) TranslationSchedule
TranslationRun (1) ──< (N) TranslationRunLanguageStats [NEW]
TranslationRun (1) ──< (N) TranslationBatch
TranslationRun (1) ──< (N) TranslationRecord
TranslationRun (1) ──< (N) TranslationEvent (nullable run_id for pre-run events)
TranslationBatch (1) ──< (N) TranslationRecord
TranslationRecord (1) ──< (N) TranslationLanguage [NEW — replaces direct language fields on Record]
TranslationPreviewSession (1) ──< (N) TranslationPreviewRecord
TranslationPreviewRecord (1) ──< (N) TranslationPreviewLanguage [NEW]
TerminologyDictionary (1) ──< (N) DictionaryEntry
```
All UUID PKs, timezone-aware UTC timestamps, JSON columns for dynamic data, hash columns for efficient comparison.
**Key change [2026-05-14]**: `TranslationRecord` no longer holds `source_text`, `llm_translation`, `user_edit`, `final_value`, `status` directly for language-specific fields. Instead, it holds source-level data (source_text, context_data, key_values, key_hash), and each `TranslationLanguage` entry holds per-language data (translated_value, user_edit, final_value, status, detected_source_language).