ss-tools/specs/028-llm-datasource-supeset/data-model.md

# Data Model: LLM Table Translation Service

**Feature Branch**: `028-llm-datasource-supeset`  
**Date**: 2026-05-17 (updated to match implementation)  
**Note**: Все 15 ORM моделей реализованы в `backend/src/models/translate.py` (397 строк).  
**Note**: Маршруты реализованы как пакет `backend/src/api/routes/translate/` (11 файлов), а не единый `translate.py`, как предполагалось в плане.

## 1. SQLAlchemy ORM Entities (dialect-aware — PostgreSQL/Greenplum, ClickHouse supported)

### TranslationJob (UPDATED — multi-language)

```python
# [DEF:TranslationJob:Class]
# @COMPLEXITY 2
# @PURPOSE Persisted configuration for a translation job (multi-language).
class TranslationJob(Base):
    __tablename__ = "translation_jobs"

    id              = Column(String, primary_key=True, default=generate_uuid)
    name            = Column(String, nullable=False)
    owner_id        = Column(String, ForeignKey("users.id"), nullable=False)
    # Source configuration
    datasource_id   = Column(String, nullable=False)
    database_dialect = Column(String, nullable=False)        # Detected from Superset: postgresql, clickhouse, greenplum
    source_table    = Column(String, nullable=True)          # Optional metadata; datasource may be virtual
    translation_col = Column(String, nullable=False)
    context_cols    = Column(JSON, default=list)
    source_key_cols = Column(JSON, nullable=False)           # [col_name, ...]
    target_key_cols = Column(JSON, nullable=False)           # [col_name, ...] — mapped to source_key_cols
    # Target configuration
    target_table    = Column(String, nullable=False)
    target_col      = Column(String, nullable=False)
    upsert_strategy = Column(String, default="insert")      # insert | skip_existing | overwrite
    # LLM configuration
    provider_id     = Column(String, ForeignKey("llm_providers.id"), nullable=True)
    target_languages = Column(JSON, nullable=False)          # [BCP-47, ...] — was target_language str, now list [UPDATED]
    source_language = Column(String, nullable=True)          # Optional fallback hint; auto-detected per row via LLM [DEPRECATED but preserved for migration]
    prompt_template = Column(Text, nullable=True)
    batch_size      = Column(Integer, default=50)
    # Timestamps
    created_at      = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
    updated_at      = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), onupdate=lambda: datetime.now(timezone.utc))

    owner           = relationship("User")
    dictionaries    = relationship("TranslationJobDictionary", back_populates="job", cascade="all, delete-orphan")
    schedule        = relationship("TranslationSchedule", back_populates="job", uselist=False, cascade="all, delete-orphan")
    runs            = relationship("TranslationRun", back_populates="job")
    preview_sessions = relationship("TranslationPreviewSession", back_populates="job")
# [/DEF:TranslationJob:Class]
```

### TranslationJobDictionary (M2M join with priority)

```python
class TranslationJobDictionary(Base):
    __tablename__ = "translation_job_dictionaries"
    id            = Column(String, primary_key=True, default=generate_uuid)
    job_id        = Column(String, ForeignKey("translation_jobs.id"), nullable=False)
    dictionary_id = Column(String, ForeignKey("terminology_dictionaries.id"), nullable=False)
    priority      = Column(Integer, default=0)              # Lower = higher priority

    job           = relationship("TranslationJob", back_populates="dictionaries")
    dictionary    = relationship("TerminologyDictionary")
```

### TerminologyDictionary (UPDATED — multilingual)

```python
class TerminologyDictionary(Base):
    __tablename__ = "terminology_dictionaries"
    id              = Column(String, primary_key=True, default=generate_uuid)
    name            = Column(String, nullable=False)         # Organizational label only
    # target_language removed — entries carry their own language pairs [UPDATED]
    owner_id        = Column(String, ForeignKey("users.id"), nullable=False)
    created_at      = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
    updated_at      = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), onupdate=lambda: datetime.now(timezone.utc))

    owner           = relationship("User")
    entries         = relationship("DictionaryEntry", back_populates="dictionary", cascade="all, delete-orphan")
```
```

### DictionaryEntry (UPDATED — language-aware + context)

```python
class DictionaryEntry(Base):
    __tablename__ = "dictionary_entries"
    id              = Column(String, primary_key=True, default=generate_uuid)
    dictionary_id   = Column(String, ForeignKey("terminology_dictionaries.id"), nullable=False)
    source_term     = Column(String, nullable=False)
    source_term_normalized = Column(String, nullable=False)  # Lowercase, NFC normalized
    source_language = Column(String, nullable=False)         # BCP-47 tag [NEW]
    target_language = Column(String, nullable=False)         # BCP-47 tag [NEW — was on dictionary]
    target_term     = Column(String, nullable=False)
    # Context [NEW section]
    context_data    = Column(JSON, nullable=True)            # Snapshot of source row's context columns { col_name: value, ... }
    usage_notes     = Column(Text, nullable=True)            # Free-form user notes on when this translation applies
    has_context     = Column(Boolean, default=False)         # Fast filter flag for prompt construction
    context_source  = Column(String, nullable=True)          # "auto" | "auto_with_edits" | "manual" | "bulk"
    # Origin metadata
    origin_run_id   = Column(String, ForeignKey("translation_runs.id"), nullable=True)
    origin_row_key  = Column(JSON, nullable=True)
    origin_user_id  = Column(String, ForeignKey("users.id"), nullable=True)
    origin_source_language = Column(String, nullable=True)   # Detected source language at correction time
    created_at      = Column(DateTime(timezone=True), default=datetime.now(timezone.utc))

    dictionary      = relationship("TerminologyDictionary", back_populates="entries")
    __table_args__ = (
        # Unique constraint updated to include language pair
        UniqueConstraint("dictionary_id", "source_term_normalized", "source_language", "target_language", name="uq_dict_source_term_lang"),
        Index("idx_dict_entry_lang", "source_language", "target_language"),
        Index("idx_dict_has_context", "has_context"),        # [NEW] — fast filter for context-only queries
    )
```


### TranslationSchedule

```python
class TranslationSchedule(Base):
    __tablename__ = "translation_schedules"
    id              = Column(String, primary_key=True, default=generate_uuid)
    job_id          = Column(String, ForeignKey("translation_jobs.id"), unique=True, nullable=False)
    schedule_type   = Column(String, nullable=False)          # cron | interval | once
    cron_expression = Column(String, nullable=True)
    interval_seconds = Column(Integer, nullable=True)
    run_at          = Column(DateTime(timezone=True), nullable=True)
    timezone        = Column(String, default="UTC")          # e.g., "Europe/Moscow"
    is_enabled      = Column(Boolean, default=True)
    concurrency     = Column(String, default="skip")         # skip | queue
    next_run_at     = Column(DateTime(timezone=True), nullable=True)
    created_at      = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc))
    updated_at      = Column(DateTime(timezone=True), default=lambda: datetime.now(timezone.utc), onupdate=lambda: datetime.now(timezone.utc))

    job             = relationship("TranslationJob", back_populates="schedule")
```

### TranslationRun

```python
class TranslationRun(Base):
    __tablename__ = "translation_runs"
    id              = Column(String, primary_key=True, default=generate_uuid)
    job_id          = Column(String, ForeignKey("translation_jobs.id"), nullable=False)
    trigger_type    = Column(String, nullable=False)          # manual | scheduled
    translation_status = Column(String, default="pending")   # pending|running|completed|partial|failed|cancelled|skipped
    insert_status   = Column(String, default="not_started")  # not_started|submitted|running|succeeded|failed|skipped
    # Statistics
    total_rows      = Column(Integer, default=0)
    translated_rows = Column(Integer, default=0)
    failed_rows     = Column(Integer, default=0)
    skipped_rows    = Column(Integer, default=0)
    token_count     = Column(Integer, default=0)
    estimated_cost  = Column(Float, default=0.0)
    # Hashes for idempotency and audit
    config_hash     = Column(String, nullable=True)
    dict_snapshot_hash = Column(String, nullable=True)
    # Snapshots
    config_snapshot = Column(JSON, nullable=False)
    dict_snapshot   = Column(JSON, nullable=True)
    prompt_used     = Column(Text, nullable=True)
    # SQL output
    insert_sql      = Column(Text, nullable=True)
    sql_hash        = Column(String, nullable=True)
    # Superset execution reference
    superset_query_id   = Column(String, nullable=True)
    superset_database_id = Column(String, nullable=True)
    insert_error_type   = Column(String, nullable=True)
    insert_error_message = Column(Text, nullable=True)
    rows_affected       = Column(Integer, nullable=True)
    # Timestamps
    started_at          = Column(DateTime(timezone=True), nullable=True)
    completed_at        = Column(DateTime(timezone=True), nullable=True)
    insert_started_at   = Column(DateTime(timezone=True), nullable=True)
    insert_completed_at = Column(DateTime(timezone=True), nullable=True)
    created_at          = Column(DateTime(timezone=True), default=datetime.now(timezone.utc))

    job             = relationship("TranslationJob", back_populates="runs")
    records         = relationship("TranslationRecord", back_populates="run", cascade="all, delete-orphan")
    batches         = relationship("TranslationBatch", back_populates="run", cascade="all, delete-orphan")
    events          = relationship("TranslationEvent", back_populates="run")
    language_stats  = relationship("TranslationRunLanguageStats", back_populates="run", cascade="all, delete-orphan")  # [NEW]
```

### TranslationBatch

```python
class TranslationBatch(Base):
    __tablename__ = "translation_batches"
    id              = Column(String, primary_key=True, default=generate_uuid)
    run_id          = Column(String, ForeignKey("translation_runs.id"), nullable=False)
    batch_index     = Column(Integer, nullable=False)
    status          = Column(String, default="pending")       # pending|running|completed|failed
    row_count       = Column(Integer, default=0)
    translated_count = Column(Integer, default=0)
    failed_count    = Column(Integer, default=0)
    skipped_count   = Column(Integer, default=0)
    token_count     = Column(Integer, default=0)
    estimated_cost  = Column(Float, default=0.0)
    latency_ms      = Column(Integer, nullable=True)
    error_type      = Column(String, nullable=True)
    error_message   = Column(Text, nullable=True)
    started_at      = Column(DateTime(timezone=True), nullable=True)
    completed_at    = Column(DateTime(timezone=True), nullable=True)

    run             = relationship("TranslationRun", back_populates="batches")
    records         = relationship("TranslationRecord", back_populates="batch")
    __table_args__ = (
        Index("idx_batch_run_idx", "run_id", "batch_index"),
    )
```

### TranslationRecord

```python
class TranslationRecord(Base):
    __tablename__ = "translation_records"
    id              = Column(String, primary_key=True, default=generate_uuid)
    run_id          = Column(String, ForeignKey("translation_runs.id"), nullable=False)
    batch_id        = Column(String, ForeignKey("translation_batches.id"), nullable=True)
    # Source data
    source_text     = Column(Text, nullable=True)
    context_data    = Column(JSON, nullable=True)
    key_values      = Column(JSON, nullable=False)
    key_hash        = Column(String, nullable=False)          # hash(canonical_json(key_values))
    # Translation result
    llm_translation = Column(Text, nullable=True)
    user_edit       = Column(Text, nullable=True)
    final_value     = Column(Text, nullable=True)
    status          = Column(String, default="pending")       # pending|translated|approved|edited|rejected|failed|skipped
    error_message   = Column(Text, nullable=True)

    run             = relationship("TranslationRun", back_populates="records")
    batch           = relationship("TranslationBatch", back_populates="records")
    languages       = relationship("TranslationLanguage", back_populates="record", cascade="all, delete-orphan")  # [NEW]
    __table_args__ = (
        Index("idx_record_key_hash", "key_hash"),
        Index("idx_record_run_key", "run_id", "key_hash"),
    )
```

### TranslationLanguage (NEW — per-language translation storage)

```python
class TranslationLanguage(Base):
    """Stores per-language translation for a TranslationRecord.
    One row per (record_id, language_code). This replaces direct language fields on TranslationRecord."""
    __tablename__ = "translation_languages"
    id                      = Column(String, primary_key=True, default=generate_uuid)
    record_id               = Column(String, ForeignKey("translation_records.id"), nullable=False)
    language_code           = Column(String, nullable=False)         # BCP-47 tag, e.g., "ru", "en", "de"
    source_language_detected = Column(String, nullable=True)         # BCP-47 or "und" — auto-detected per row [NEW]
    translated_value        = Column(Text, nullable=True)
    user_edit               = Column(Text, nullable=True)
    final_value             = Column(Text, nullable=True)
    status                  = Column(String, default="pending")      # pending|translated|approved|edited|rejected|failed|skipped
    error_message           = Column(Text, nullable=True)
    created_at              = Column(DateTime(timezone=True), default=datetime.now(timezone.utc))

    record                  = relationship("TranslationRecord", back_populates="languages")
    __table_args__ = (
        UniqueConstraint("record_id", "language_code", name="uq_record_language"),
        Index("idx_tl_language", "language_code"),
        Index("idx_tl_record_lang", "record_id", "language_code"),
    )
```

### TranslationPreviewLanguage (NEW — per-language preview results)

```python
class TranslationPreviewLanguage(Base):
    """Stores per-language translation for a preview record."""
    __tablename__ = "translation_preview_languages"
    id                      = Column(String, primary_key=True, default=generate_uuid)
    preview_record_id       = Column(String, ForeignKey("translation_preview_records.id"), nullable=False)
    language_code           = Column(String, nullable=False)         # BCP-47 tag
    source_language_detected = Column(String, nullable=True)         # BCP-47 or "und"
    translated_value        = Column(Text, nullable=True)
    user_edit               = Column(Text, nullable=True)
    final_value             = Column(Text, nullable=True)
    status                  = Column(String, default="pending")      # pending|approved|edited|rejected
    created_at              = Column(DateTime(timezone=True), default=datetime.now(timezone.utc))

    preview_record          = relationship("TranslationPreviewRecord", back_populates="languages")
    __table_args__ = (
        UniqueConstraint("preview_record_id", "language_code", name="uq_preview_record_language"),
    )
```

### TranslationRunLanguageStats (NEW — per-language run statistics)

```python
class TranslationRunLanguageStats(Base):
    """Per-language statistics for a translation run."""
    __tablename__ = "translation_run_language_stats"
    id              = Column(String, primary_key=True, default=generate_uuid)
    run_id          = Column(String, ForeignKey("translation_runs.id"), nullable=False)
    language_code   = Column(String, nullable=False)         # BCP-47 tag
    total_rows      = Column(Integer, default=0)
    translated_rows = Column(Integer, default=0)
    failed_rows     = Column(Integer, default=0)
    skipped_rows    = Column(Integer, default=0)
    token_count     = Column(Integer, default=0)
    estimated_cost  = Column(Float, default=0.0)

    run             = relationship("TranslationRun", back_populates="language_stats")
    __table_args__ = (
        UniqueConstraint("run_id", "language_code", name="uq_run_language"),
        Index("idx_rls_run", "run_id"),
    )
```

### TranslationPreviewSession

```python
class TranslationPreviewSession(Base):
    __tablename__ = "translation_preview_sessions"
    id                  = Column(String, primary_key=True, default=generate_uuid)
    job_id              = Column(String, ForeignKey("translation_jobs.id"), nullable=False)
    user_id             = Column(String, ForeignKey("users.id"), nullable=False)
    config_hash         = Column(String, nullable=False)
    dict_snapshot_hash  = Column(String, nullable=True)
    sample_size         = Column(Integer, default=10)
    status              = Column(String, default="pending")   # pending|accepted|rejected
    created_at          = Column(DateTime(timezone=True), default=datetime.now(timezone.utc))
    accepted_at         = Column(DateTime(timezone=True), nullable=True)
    expires_at          = Column(DateTime(timezone=True), nullable=True)

    job             = relationship("TranslationJob", back_populates="preview_sessions")
    rows            = relationship("TranslationPreviewRecord", back_populates="session", cascade="all, delete-orphan")
```

### TranslationPreviewRecord

```python
class TranslationPreviewRecord(Base):
    __tablename__ = "translation_preview_records"
    id              = Column(String, primary_key=True, default=generate_uuid)
    session_id      = Column(String, ForeignKey("translation_preview_sessions.id"), nullable=False)
    source_text     = Column(Text, nullable=True)
    context_data    = Column(JSON, nullable=True)
    key_values      = Column(JSON, nullable=False)
    key_hash        = Column(String, nullable=False)
    llm_translation = Column(Text, nullable=True)
    user_edit       = Column(Text, nullable=True)
    final_value     = Column(Text, nullable=True)
    status          = Column(String, default="pending")       # pending|approved|edited|rejected

    session         = relationship("TranslationPreviewSession", back_populates="rows")
    languages       = relationship("TranslationPreviewLanguage", back_populates="preview_record", cascade="all, delete-orphan")  # [NEW]
```

### TranslationEvent

```python
class TranslationEvent(Base):
    __tablename__ = "translation_events"
    id              = Column(String, primary_key=True, default=generate_uuid)
    run_id          = Column(String, ForeignKey("translation_runs.id"), nullable=True)  # NULL for pre-run events
    job_id          = Column(String, ForeignKey("translation_jobs.id"), nullable=False)
    event_type      = Column(String, nullable=False)
    # schedule_triggered|schedule_skipped|schedule_failed|
    # run_started|batch_started|batch_completed|batch_failed|
    # run_succeeded|run_partial|run_failed|run_cancelled|run_skipped|run_noop|
    # insert_submitted|insert_succeeded|insert_failed
    timestamp       = Column(DateTime(timezone=True), default=datetime.now(timezone.utc), nullable=False)
    payload         = Column(JSON, default=dict)

    run             = relationship("TranslationRun", back_populates="events")
    __table_args__ = (
        Index("idx_event_job_ts", "job_id", "timestamp"),
        Index("idx_event_type", "event_type"),
    )
```

### MetricSnapshot

```python
class MetricSnapshot(Base):
    __tablename__ = "translation_metric_snapshots"
    id              = Column(String, primary_key=True, default=generate_uuid)
    job_id          = Column(String, ForeignKey("translation_jobs.id"), nullable=False)
    snapshot_date   = Column(Date, nullable=False)
    cumulative_tokens = Column(Integer, default=0)
    cumulative_cost   = Column(Float, default=0.0)
    per_language_metrics = Column(JSON, nullable=True)       # [NEW] { "ru": { "tokens": N, "cost": M }, ... }
    covers_events_before = Column(DateTime(timezone=True), nullable=False)  # Cutoff for event coverage
    total_runs        = Column(Integer, default=0)
    success_runs      = Column(Integer, default=0)
    failed_runs       = Column(Integer, default=0)
    partial_runs      = Column(Integer, default=0)
    avg_batch_latency_ms = Column(Integer, nullable=True)
    created_at      = Column(DateTime(timezone=True), default=datetime.now(timezone.utc))

    __table_args__ = (
        UniqueConstraint("job_id", "snapshot_date", name="uq_metric_snapshot_date"),
    )
```

## 2. Pydantic Schemas (API DTOs)

Key changes from original:
- `TranslateJobCreate`/`Update`: added `source_key_cols`, `target_key_cols` (explicit mapping), `source_language` → replaced by `target_languages: list[str]` [UPDATED 2026-05-14]
- `TermCorrectionSubmit`: added `source_term`, `incorrect_target_term`, `corrected_target_term`, `source_language` (auto-filled from detected), `target_language` [UPDATED]
- `ScheduleConfig`: added `timezone`
- `TranslationRunResponse`: split into `translation_status` + `insert_status`, added Superset execution fields, added `language_stats: list[LanguageStatsResponse]` [NEW]
- Added: `TranslationBatchResponse`, `PreviewSessionResponse`, `MetricsSnapshotResponse`
- Added: `TranslationLanguageResponse` [NEW] — wraps language_code, detected_source_language, translated_value, user_edit, final_value, status
- Added: `TranslationPreviewLanguageResponse` [NEW] — same structure for preview records
- Added: `BulkFindReplaceRequest` [NEW] — find_pattern (string, regex flag), replacement_text, target_language
- Added: `InlineCorrectionSubmit` [NEW] — record_id, language_code, source_term, corrected_term, dictionary_id, source_language
- Added: `CorrectionSubmitWithContext` [NEW] — extends InlineCorrectionSubmit with `context_data` (optional JSON override), `usage_notes` (optional text), `keep_context` (boolean, default=true — false means strip auto-captured context)
- Updated: `DictionaryEntryResponse` [UPDATED] — added `context_data`, `usage_notes`, `has_context`, `context_source` fields
- Updated: `BulkFindReplaceRequest` [UPDATED] — added `submit_to_dictionary_with_context` (boolean, default=false), `bulk_usage_notes` (optional text applied to all entries)

## 3. Entity Relationship Summary

```
TranslationJob (1) ──< (N) TranslationJobDictionary >── (1) TerminologyDictionary
TranslationJob (1) ──< (N) TranslationRun
TranslationJob (1) ──< (N) TranslationPreviewSession
TranslationJob (1) ──── (0..1) TranslationSchedule
TranslationRun (1) ──< (N) TranslationRunLanguageStats     [NEW]
TranslationRun (1) ──< (N) TranslationBatch
TranslationRun (1) ──< (N) TranslationRecord
TranslationRun (1) ──< (N) TranslationEvent (nullable run_id for pre-run events)
TranslationBatch (1) ──< (N) TranslationRecord
TranslationRecord (1) ──< (N) TranslationLanguage           [NEW — replaces direct language fields on Record]
TranslationPreviewSession (1) ──< (N) TranslationPreviewRecord
TranslationPreviewRecord (1) ──< (N) TranslationPreviewLanguage  [NEW]
TerminologyDictionary (1) ──< (N) DictionaryEntry
```

All UUID PKs, timezone-aware UTC timestamps, JSON columns for dynamic data, hash columns for efficient comparison.

**Key change [2026-05-14]**: `TranslationRecord` no longer holds `source_text`, `llm_translation`, `user_edit`, `final_value`, `status` directly for language-specific fields. Instead, it holds source-level data (source_text, context_data, key_values, key_hash), and each `TranslationLanguage` entry holds per-language data (translated_value, user_edit, final_value, status, detected_source_language).