feat(translate): replace LLM-based language detection with local lingua-detector

- Add _lang_detect.py module wrapping lingua-language-detector (pure Python, 0.076ms/call)
- Add batch_detect() with detector caching for performance
- Pre-filter rows where detected source language matches target (same-language skip)
- Remove detected_source_language from LLM prompt templates (token savings)
- Use local detection for cached rows (was always 'und')
- Preserve backward compat: LLM detected_source_language as fallback when lingua returns 'und'
- Add 24 unit tests for detection, edge cases, caching, and mapping
This commit is contained in:
2026-05-20 14:31:37 +03:00
parent 81f3b4cc7f
commit c334b1b0ca
8 changed files with 438 additions and 18 deletions

View File

@@ -56,3 +56,4 @@ playwright
tenacity
Pillow
ruff>=0.11.0
lingua-language-detector==2.2.0