When max_images could not be detected (e.g. Kilo gateway doesn't
support OpenAI image format, probe returned 0), chunking was
disabled entirely and all screenshots sent in a single request.
Nvidia NeMo still enforces an 8-image limit regardless of the
gateway, causing 400 errors.
Fix: default to 8 images per chunk when max_images is 0 or None,
so chunking always applies for multi-tab dashboards.
Root cause of stuck 'running' runs: the plugin's execute() method
only processed dashboard_ids[0] and returned. The remaining N-1
dashboards were never processed, and _update_run_status was called
after each single dashboard without checking dashboard_count.
Fixes:
- execute() now loops over ALL dashboard_ids and processes each
via _execute_path_a or _execute_path_b
- _update_run_status is called once AFTER the loop finishes
- _update_run_status now checks that len(records) >= dashboard_count
before marking the run as finished (prevents premature completion)
- Per-dashboard _update_run_status calls removed from _execute_path_a
and _execute_path_b (the parent loop owns it now)
- Test updated for new batch return format {dashboards: [...], total: N}
When the backend restarts (deploy, crash, reload), the in-memory
TaskManager loses its queue. ValidationRun records left 'running'
in the DB would hang forever, blocking re-runs.
Fix: during lifespan startup, query all ValidationRun with
status='running' and force-stop them as FAIL with a clear summary.
This prevents the 'already has a running run' error after restarts.
- Extract _optimize_images() helper to eliminate duplicate
optimization code (was duplicated between initial pass and
quality-reduction fallback)
- Move quality-reduction estimate to only apply when NOT chunking
(each chunk fits the image limit by definition)
- Fix _merge_chunk_results to return 'chunk_count' instead of 'chunks'
for consistency with plugin.py
- Simplify plugin.py: analysis.get('chunk_count', 1) instead of
fallback chain
- Document that Kilo API gateway is incompatible with AsyncOpenAI
image format (probe returns 0)
Delete deprecated code that has been fully replaced by v2:
- backend: validation.py routes, validation_run_service.py
- frontend: validation.js API module, validation/ pages
Add binary-search probe endpoint POST /providers/{id}/probe-max-images
that discovers the per-request image limit by sending incrementally
more 1x1 JPEGs via the provider's own API. Result is cached in the
new max_images column on the provider config.
- LLMProviderConfig: add max_images: int | None
- LLMProvider (SQLAlchemy): add max_images column
- Migration ed28d34edde7: clean ADD COLUMN
- LLMProviderService: create/update/set_max_images
- POST /providers/{id}/probe-max-images: binary search + error parsing
- ProviderConfig.svelte: 'Detect' button in edit modal + HelpTooltip
- i18n (en/ru): 11 new keys for probe UI
Fixes:
- Move batch size, upsert strategy, include source reference
from Config tab to Target Config tab (Write Settings section)
- Fix 'Dataset #26' fallback — look up real datasource name
via API when source_table is not saved in job data
- Persist include_source_reference in save payload and
restore from job data on load (was a pre-existing gap)
QA review fixes:
- Update @BRIEF contracts for both tabs
- Fix import indentation in +page.svelte
Other: help tooltips on translate pages, flow hint on list page,
git migration manager components, dashboard hub pages,
migration settings pages
1. Remove get_config_manager() call from translate_run_websocket
(@app.websocket /ws/translate/run/{run_id}) — was not imported
and unused. Also remove unused from sqlalchemy.orm import Session.
2. Fix prop name mismatch: showBulkReplace -> showPageBulkReplace
in RunTabContent. Use () for reactive two-way binding.
3. Add ('DRAFT') for status prop in RunTabContent.
Backend:
- Add /ws/translate/run/{run_id} WebSocket endpoint in app.py
- Streams structured run status every second (total_records,
successful_records, failed_records, progressPct, batch_count, etc.)
- Auto-detects terminal states (COMPLETED, FAILED, CANCELLED) and closes
- Authenticated via JWT/API key token query param
Frontend:
- Add getTranslateRunWsUrl() URL builder in api.js
- Update translationRunStore to use WebSocket instead of log stream
- WebSocket receives structured status JSON, updates store reactively
- Falls back to polling silently if WebSocket fails
- Keep polling as fallback for insert phase
Clean up remaining orphaned references after tab extraction:
- Remove loadColumnsForDatasource, searchDatasources, selectDatasource
etc. from page (moved to ConfigTabForm)
- Add on datasourceId in ConfigTabForm for auto-loading
columns when parent sets datasourceId (loadJob)
- Fix stray braces, remove dead isVirtual function
- Page now 779 lines (down from 1519, -49%)
Remove loadColumnsForDatasource, searchDatasources, selectDatasource,
toggleContextColumn, toggleSourceKeyCol, updateTargetKeyCol, isVirtual
from page — all moved to ConfigTabForm.svelte.
Add on datasourceId in ConfigTabForm to auto-load columns
when parent sets datasourceId (loadJob) or user selects a datasource.
Add comprehensive UX contract header documenting:
- @UX_STATE: all states of the page-level state machine (idle, loading,
configured, saving, validation_error, datasource_unavailable, error)
- @UX_STEP: 5 logically separated steps/tabs (Config, Preview, Target, Run, Schedule)
- @UX_FEEDBACK: toasts, modals, inline progress, schema validation
- @UX_RECOVERY: retry paths for save/preview/run failures
- @UX_REACTIVITY: props, local state, derived, effects, store bindings
- @UX_DEPENDENCY: data flow between datasource selection, env switching,
config validity, and tab availability
- @UX_TEST: test scenarios for the automated Judge Agent
Also remove leftover sourceDialect/targetDialect selects from UI
(now sent as sensible defaults in save payload).
Sub-components (TargetSchemaHint, TranslationPreview, TranslationRunProgress,
ScheduleConfig, BulkReplaceModal) already have UX contracts.
These fields were showing database types (PostgreSQL, ClickHouse)
as human language options for the LLM prompt, which was confusing.
Now sending sensible defaults (source='SQL', target='en') directly
in the save payload without exposing them in the UI.
The backend already has fallbacks:
_llm_call.py: source_language = job.source_dialect or 'SQL'
_batch_proc.py: target_langs = or [job.target_dialect or 'en']
sourceDialect/targetDialect select options contained database types
(PostgreSQL, ClickHouse, MySQL...) while these fields are used as
human language indicators for the LLM prompt:
_llm_call.py: source_language = job.source_dialect or 'SQL'
Changed to use the existing LANGUAGES list (from LANGUAGE_LABELS:
ru, en, de, fr, es, etc.) plus 'SQL' option for source dialect.
Defaults updated: source='SQL' (SQL-like expressions), target='en'.
Bug: loadDatabases() was called at line 374 while environmentId still
held the FIRST environment from initialization (line 259), because
environmentId = j.environment_id was not set until line 391 (inside
the datasource block). Only after that was environmentId correct.
This caused the database dropdown to show databases from a different
environment than the job's actual environment. For example, db_id=2
is DEV Greenplum in 'dev' but Prod Clickhouse in 'preprod'. The user
would select Greenplum from the dev list, but the job would use
preprod where db_id=2 is ClickHouse.
Fix: set environmentId from the job's stored environment BEFORE
calling loadDatabases(), and outside the datasource conditional
so it runs for all existing jobs.
Root cause: fetch_available_datasources did not return database_id,
so frontend could not auto-set target_database_id when user selected
a datasource. User then had to manually pick a database on Target Config
tab — and could accidentally select the wrong one.
Backend changes:
- Add 'database_id' field to datasources response (from db_info.get('id'))
- Add _database_name tracking in SupersetSqlLabExecutor with getter
- Add database_name + database_backend to TargetSchemaValidationResponse
- Enrich resolve_database_id logging with database_name and all_keys
Frontend changes:
- In selectDatasource(), auto-set targetDatabaseId from ds.database_id
when present, so the correct target DB is used for schema checks.
- Replace fragile substring check (any(kw in backend for kw in ('clickhouse', 'ch')))
with a proper _backend_normalize mapping + exact comparison.
Fixes misdetection of Greenplum (postgresql) and other backends.
- Add 'clickhousedb' to CLICKHOUSE_DIALECTS so SQL generator uses
backtick quoting and correct INSERT strategy for ClickHouse.
- Normalize _extract_dialect to map 'clickhousedb' -> 'clickhouse'
and 'greenplum' -> 'postgresql'.
OpenSSL 3.x ignores intermediate CA certs in -CAfile (verify code 20)
but correctly builds chains with -CApath (verify code 0).
All three clients now use capath:
- service.py: ssl.create_default_context(capath=...)
- _llm_http.py: verify='/etc/ssl/certs/'
- preview_llm_client.py: verify='/etc/ssl/certs/'
check_llm_certs.py rewritten for clarity.
Tests all 4 SSL methods with actual project libraries:
- httpx (verify=True/False/SSLContext cafile/cafile+capath)
- requests (verify=True/False/cafile/capath)
- openssl s_client (default/CAfile/CApath/chain bundle)
- certutil (NSS/Chromium)
- certifi vs system CA comparison
Outputs JSON summary with AI diagnosis hint.
Runs from inside Docker container to verify all 4 SSL layers:
1. System CA store (openssl s_client)
2. NSS database (certutil) for Chromium
3. Python httpx (SSLContext with cafile)
4. Python requests (verify with system CA path)
Reads LLM_SSL_VERIFY and CA paths from .env.enterprise-clean
- ADR-0009: comprehensive SSL certificate management strategy
- _get_ssl_verify() returns SSLContext with cafile= instead of bool True
(httpx deprecates verify=<str>, requests needs system CA, not certifi)
- _get_verify() in translate plugin returns system CA path
- All tests updated for SSLContext return type
- All QA findings C1-C3, H1-H4, M1 incorporated into ADR
preview_llm_client.py and _llm_http.py both use requests.post()
without verify=, causing SSLError when corporate CA not in trust store.
Added _get_verify() helper that reads LLM_SSL_VERIFY env var.
C1: fingerprint (SHA256) вместо grep -qF для детекции сертификатов в bundle
— исключает ложные срабатывания и бесконечный рост ca-certificates.crt
C2: hash symlink collision — поддержка .0, .1, .2 суффиксов вместо
перезаписи единственного .0
C3: NSS nickname collision — дедупликация по fingerprint, префикс
директории (llm-/custom-) в nickname, fallback с хешем при коллизии
H1: DER->PEM конвертация — добавлена || проверка ошибки
H2: Удалён install_playwright из entrypoint (Chromium уже baked в Dockerfile)
H3: NSS DB path — sql: префикс, проверка существования и пересоздание
при повреждении
H4: Chicken-and-egg TLS — curl fallback с --insecure при первом скачивании
корп. CA сертификата
M1: nullglob для install_certificates — безопасная итерация по файлам
Introduce a centralized tool registry system for the assistant to manage
executable operations, permissions, and tool catalogs. This refactors
the assistant dispatch logic from a monolithic approach to a modular,
decorator-based registry.
Key changes:
- Implement `AssistantToolRegistry` to handle tool registration,
permission checks, and safe operation identification.
- Add a suite of new assistant tools: backup, commit, branch creation,
deployment, health summary, environment listing, LLM operations,
maintenance, migration, and dashboard searching.
- Refactor `_dispatch.py` and `_llm_planner.py` to utilize the new
registry for intent resolution and tool catalog building.
- Enhance the LLM planner to support more descriptive clarification
responses in Russian when intents are ambiguous.
- Update the frontend to support the new tool-based workflow, including
improved i18n labels for assistant operations and a new step-by-step
wizard for the migration page.
- Add ADR-0008 to document the architectural decision for the tool
registry.
- Removed duplicate #endregion in backend.Dockerfile (P6 critical)
- Added @RATIONALE and @REJECTED to install_ca_to_nss and install_llm_ca_certs
- Changed grep -q to grep -qF in install_llm_ca_certs validation
- install_ca_to_nss() — imports corporate CA certs into Chromium's NSS
database (~/.pki/nssdb), fixing ERR_CERT_AUTHORITY_INVALID in Playwright
- libnss3-tools added to Dockerfile for certutil binary
