feat(translate): add translation cache with source_hash dedup + enforce dictionary post-processing

- Add source_hash column (SHA256 of source_text+source_data+dict_hash+config_hash) to TranslationRecord for cache dedup
- Add Alembic migration b1c2d3e4f5a6 for the new column and composite index (source_hash, status)
- Implement _compute_source_hash / _check_translation_cache in executor.py — before LLM call, check if same source+dict+config combo was already translated successfully
- Implement cache-aware routing in _process_batch: full cache hit → skip LLM, partial hit → route to LLM
- Store source_hash on all new TranslationRecord rows for future cache hits
- Strengthen dictionary prompt instructions from 'use when applicable' to 'MUST use — mandatory'
- Add _enforce_dictionary() post-processing: force-replace dictionary terms in LLM output if LLM ignored them
- Fix CRITICAL: use run.dict_snapshot_hash/config_hash (exist on TranslationRun, not TranslationJob)
- Fix MAJOR: verify cached languages cover ALL target_languages before accepting cache hit
- Fix MAJOR: prevent regex back-reference injection in dictionary enforcement (use lambda)
- Fix: add joinedload to cache lookup to avoid N+1 queries
- Fix: remove redundant single-column index (composite index is sufficient)
- Fix(frontend): parse paginated /translate/dictionaries response correctly (result.items)
This commit is contained in:
2026-05-16 00:42:07 +03:00
parent 8797c28958
commit 9c5f763646
12 changed files with 458 additions and 113 deletions

View File

@@ -132,12 +132,15 @@ class TranslationRecord(Base):
token_count_input = Column(Integer, nullable=True)
token_count_output = Column(Integer, nullable=True)
translation_duration_ms = Column(Integer, nullable=True)
source_hash = Column(String, nullable=True,
comment="SHA256(source_text+source_data+dict_snapshot_hash+config_hash) for cache dedup")
created_at = Column(DateTime, default=lambda: datetime.now(UTC))
languages = relationship("TranslationLanguage", back_populates="record")
__table_args__ = (
Index("ix_translation_records_run_status", "run_id", "status"),
Index("ix_translation_records_source_hash_status", "source_hash", "status"),
)
# #endregion TranslationRecord