Add support for regex-based dictionary entries across the full stack:
Backend:
- DictionaryEntry model: add is_regex column (Boolean, default False)
- DictionaryEntryCRUD: validate regex on add_entry(), compile on creation
- _enforce_dictionary: match by regex pattern when is_regex=True
- dictionary_filter: support is_regex in filter/query
- metrics: include is_regex entries in metrics calculations
- Alembic migration: 20260604_add_is_regex_to_dictionary_entries
- Merge migration: 351afb8f961a (merge is_regex + composite index heads)
Frontend:
- DictionaryDetailModel: add is_regex field to DictionaryEntry interface,
EditForm, and addForm; sync edit/add form state with backend schema
- Add composite index ix_translation_records_run_source_hash
for NOT EXISTS dedup subquery performance + Alembic migration
- Remove duplicate #endregion in orchestrator.py (INV_3)
- Replace hardcoded RU fallback 'Статус' with 'Status'
- Add early return guard to loadMoreRecords()
- Show records summary always when recordsTotal > 0
Backend:
- deduplicate param in GET /runs/{id}/records — NOT EXISTS subquery
with (created_at, id) tiebreaker for one row per source_hash
- source_data and source_hash added to records JSON response
- fix missing status import in _run_history_routes.py (NameError)
- fix tab/space mixup in orchestrator_aggregator.py
- remove duplicated @RELATION edges in metrics.py
- fix #region/#endregion style and @PURPOSE→@BRIEF in metrics.py
Frontend:
- RunTabContent: summary metrics bar (fetchJobMetrics) with HelpTooltips
- RunTabContent: trigger_type badge, duration, cache rate in run rows
- TranslationRunResult: deduplicate=true by default, paginated Load more
- TranslationRunResult: per-language token_count and estimated_cost
- TranslationRunResult: collapsible batch breakdown with timing
- TranslationRunResult: Bulk Replace button in header and records table
- TranslationRunResult: source_data key values shown under source text
i18n: all new keys in EN + RU (load_more_records, showing_records,
sum_*, help_sum_*, trigger_*, batch_*, duration_label, cost_label,
cache_rate, load_more_records)
- Removes source_dialect and target_dialect from TerminologyDictionary model,
schemas, routes, helper serialization, and all tests
- Adds allowed_languages config field (BCP-47 language codes) to GlobalSettings
with validation, consolidated settings response, and API endpoint
- Adds alembic migration f1a2b3c4d5e6 to drop the two columns
1. dict_snapshot_hash now includes entry count + max(updated_at)
per dictionary. Previously only hashed dictionary IDs, meaning
edits to dictionary entries did NOT invalidate the translation
cache. Stale cached translations could be served after editing
dictionary entries. (HIGH severity)
2. _batch_insert.py now uses SQLGenerator.generate_batch() with
500-row chunking instead of a single massive INSERT statement.
Prevents potential SQL size limit issues with large batches or
many target languages. (LOW severity)
3. Fixed same bug in preview_response_parser.py —
compute_dict_snapshot_hash had identical ID-only hash flaw.
Tests: 69/69 translate tests pass.
Backend:
- TranslationRun model: add cache_hits Integer column (default 0)
- TranslationRunResponse schema: add cache_hits field
- _helpers.py/_run_list_routes.py/orchestrator_aggregator.py: include
cache_hits in all run API responses
- _batch_proc.py: count pre_rows served from translation cache per
batch, return cache_hits in batch result
- executor.py: accumulate cache_hits across batches, persist to run
- Alembic migration: dabc9709 — add cache_hits column to translation_runs
Frontend:
- translationRun.svelte.ts: add cacheHits to store state, WS handler,
polling handler
- TranslationRunProgress.svelte: 5-col stats grid with purple Cache card
- TranslationRunGlobalIndicator.svelte: 5-col stats with Cache
- TranslationRunResult.svelte: 5-col detail stats with Cache card
- History page: cache_hits shown in run list row + detail panel
Visual: cache hits shown in purple alongside green/yellow/red metrics
(total/success/failed/skipped). Visible during run + in history.
Tests: backend 69/69 translate tests ✅, frontend 698/698 tests ✅,
frontend build ✅
Bug 1 (secondary): _persist_pre() created TranslationLanguage with
empty final_value for non-source target languages when same_language
flag was set but no cache/approved value existed. This produced
is_original=0 rows with empty text in target table.
Fix: skip TranslationLanguage creation when fv is empty (continue
in the loop instead of falling through to else with empty string).
Bug 2 (diagnostic): No visibility into _classify() routing decisions.
Add logger.reason with pre/llm/total counts per batch.
Scenario trace: ru source + targets [ru,en] + no cache → llm_rows ✓
If same-language partial match hits cache for all targets → pre_rows ✓
If same-language partial match misses cache → llm_rows ✓
If same-language all-match (targets=[ru]) → short-circuit pre_rows ✓
Tests: 69/69 translate tests pass.
Root cause: _classify() immediately skipped (continue) when detected
language matched ANY target language, preventing cache lookup and LLM
translation for remaining targets. E.g., ru source + targets [ru, en]
→ only ru row created, en never processed (8978 ru vs 18 en in DB).
Fix:
1. _classify(): Only short-circuit when ALL targets match detected
language. Partial match → mark _same_language but continue to
cache check and LLM for non-matching targets.
2. _persist_pre(): Unify two code paths into single loop over all
target languages. Same-language targets use source_text as-is;
cached targets use cache value; fallback to approved_translation.
3. SIM102: Merge nested 'if cl: if all(...)' → 'if cl and all(...)'
to keep cyclomatic complexity ≤ 10 (INV_7).
QA: All 69 translate tests pass. 7 scenario traces (A-G) verified.
No regressions in approved_translation, preview edits, or single-
target same-language flows.
Root cause: fetch_available_datasources did not return database_id,
so frontend could not auto-set target_database_id when user selected
a datasource. User then had to manually pick a database on Target Config
tab — and could accidentally select the wrong one.
Backend changes:
- Add 'database_id' field to datasources response (from db_info.get('id'))
- Add _database_name tracking in SupersetSqlLabExecutor with getter
- Add database_name + database_backend to TargetSchemaValidationResponse
- Enrich resolve_database_id logging with database_name and all_keys
Frontend changes:
- In selectDatasource(), auto-set targetDatabaseId from ds.database_id
when present, so the correct target DB is used for schema checks.
- Replace fragile substring check (any(kw in backend for kw in ('clickhouse', 'ch')))
with a proper _backend_normalize mapping + exact comparison.
Fixes misdetection of Greenplum (postgresql) and other backends.
- Add 'clickhousedb' to CLICKHOUSE_DIALECTS so SQL generator uses
backtick quoting and correct INSERT strategy for ClickHouse.
- Normalize _extract_dialect to map 'clickhousedb' -> 'clickhouse'
and 'greenplum' -> 'postgresql'.
OpenSSL 3.x ignores intermediate CA certs in -CAfile (verify code 20)
but correctly builds chains with -CApath (verify code 0).
All three clients now use capath:
- service.py: ssl.create_default_context(capath=...)
- _llm_http.py: verify='/etc/ssl/certs/'
- preview_llm_client.py: verify='/etc/ssl/certs/'
check_llm_certs.py rewritten for clarity.
- ADR-0009: comprehensive SSL certificate management strategy
- _get_ssl_verify() returns SSLContext with cafile= instead of bool True
(httpx deprecates verify=<str>, requests needs system CA, not certifi)
- _get_verify() in translate plugin returns system CA path
- All tests updated for SSLContext return type
- All QA findings C1-C3, H1-H4, M1 incorporated into ADR
preview_llm_client.py and _llm_http.py both use requests.post()
without verify=, causing SSLError when corporate CA not in trust store.
Added _get_verify() helper that reads LLM_SSL_VERIFY env var.
All 5 functions now have proper #region/#endregion contracts:
- _detector_cache_key [C:1] — private helper
- build_detector [C:2] — public API
- get_detector [C:2] — public API with caching
- detect_language [C:2] — public API, @RAISES for safety
- batch_detect [C:3] — public API with @RELATION edges
Clean audit: 0 warnings on _lang_detect.py
- frontend: add to reload job data on param change after goto
- frontend: set existingJob from createJob response for immediate tabs
- frontend: suppress error toasts for schedule 404 (normal state)
- frontend: conditional ScheduleConfig mount to avoid 'new' job loading
- backend: skip preview check for jobs with direct Superset datasource
- backend: add orthogonal test for datasource+no-preview success case
- Extract _fallback_cancel_request for row lock timeout recovery in orchestrator_cancel
- Add flush lock timeout fallback in cancel_run
- Set error_message when all records fail in executor
- Track insert_failed state in scheduler and frontend store
- Migrate TranslationRunProgress from local polling to centralized store
- Fix nginx resolver for Docker variable-based proxy_pass
- Add FRONTEND_HOST_PORT env var to docker-compose
- Sidebar.svelte: defer healthStore.refresh() until feature flags confirm
health_monitor is enabled; skip entirely when disabled
- health.js: remove throw error from catch block — log and return null
instead, preventing 'Uncaught (in promise)' on expected 404
- test_report_audit_immutability.py: fix mock assertions —
audit_service uses logger.reason/reflect/explore, not logger.info
- HealthStore and Sidebar now produce zero network noise when
FEATURES__HEALTH_MONITOR=false
- _compute_source_hash now accepts context_keys parameter
- source_data is filtered to only include context-relevant fields (not row_id, primary keys)
- If no context_columns configured, hash is based on source_text + dict_hash + config_hash
- This ensures identical text in different rows produces a cache hit
- Add source_hash column (SHA256 of source_text+source_data+dict_hash+config_hash) to TranslationRecord for cache dedup
- Add Alembic migration b1c2d3e4f5a6 for the new column and composite index (source_hash, status)
- Implement _compute_source_hash / _check_translation_cache in executor.py — before LLM call, check if same source+dict+config combo was already translated successfully
- Implement cache-aware routing in _process_batch: full cache hit → skip LLM, partial hit → route to LLM
- Store source_hash on all new TranslationRecord rows for future cache hits
- Strengthen dictionary prompt instructions from 'use when applicable' to 'MUST use — mandatory'
- Add _enforce_dictionary() post-processing: force-replace dictionary terms in LLM output if LLM ignored them
- Fix CRITICAL: use run.dict_snapshot_hash/config_hash (exist on TranslationRun, not TranslationJob)
- Fix MAJOR: verify cached languages cover ALL target_languages before accepting cache hit
- Fix MAJOR: prevent regex back-reference injection in dictionary enforcement (use lambda)
- Fix: add joinedload to cache lookup to avoid N+1 queries
- Fix: remove redundant single-column index (composite index is sufficient)
- Fix(frontend): parse paginated /translate/dictionaries response correctly (result.items)
- Add RUN_CANCELLED event logging in orchestrator when executor returns CANCELLED (cancel flag path)
- Skip TranslationLanguage creation for languages matching detected source language (no ru in stats)
- Update tests for new behavior (3 entries instead of 4, no fr entry for source-match)
- Fix clickhouse insert test mocks for .options(joinedload()) chain
- All 208 translate tests pass
- Add _normalize_timestamp_value to key column values in orchestrator.py and executor.py before SQL generation
- Fix test mocks for .options(joinedload()) chain, explicit None attrs for mock job
- Add global translation run progress indicator (TranslationRunGlobalIndicator + store)
- Fix translate page import missing translationRunStore
- All 208 translate tests pass
- Enable response_format (json_object) for all providers including Kilo/OpenRouter
- Skip reasoning_effort for Kilo/OpenRouter (returns 400 — mandatory reasoning)
- Add structured_outputs fallback: retry once without response_format if upstream
provider (e.g. StepFun) rejects it with 'structured_outputs is not supported'
- Handle model refusal field (refusal) instead of treating as empty content
- Fix None-handling guards for message/finish_reason/content fields
- Apply same fixes to both preview.py and executor.py _call_openai_compatible
- Connect orphaned TermCorrectionPopup, BulkReplaceModal, BulkCorrectionSidebar
DeepSeek v4 Flash returns finish_reason=length when reasoning uses
all output tokens. Partial row extraction via regex recovers complete
rows from truncated JSON instead of failing with 400 error.
Also: extra_body reasoning_effort for Kilo/OpenRouter proxied providers
Added disable_reasoning toggle that works via:
1. API parameter: reasoning_effort: 'none' (DeepSeek, Qwen)
2. System prompt instruction: 'Respond directly with ONLY the JSON.
Do NOT include any reasoning, thinking, or chain-of-thought'
(works for ALL models universally)
Backend: model column, schema, preview+executor payload
Frontend: checkbox in LLM settings, i18n en+ru
New _token_budget.py calculates safe batch_size and max_tokens based on source text length, target languages, dictionary size, and model context window (64K DeepSeek v4 Flash). preview.py: auto-reduces sample_size when texts are long. executor.py: per-batch dynamic max_tokens. 12 new tests.
finish_reason=length from DeepSeek v4 Flash — model runs out of
output tokens due to reasoning_content (CoT) + multi-language (4 langs).
8192 tokens ensures complete JSON response for preview and execution.
Preview + executor now log finish_reason, message_keys, and raw
response structure when LLM returns empty content, enabling
diagnosis of DeepSeek API issues (empty content with finish_reason=length)
