sourceDialect/targetDialect select options contained database types
(PostgreSQL, ClickHouse, MySQL...) while these fields are used as
human language indicators for the LLM prompt:
_llm_call.py: source_language = job.source_dialect or 'SQL'
Changed to use the existing LANGUAGES list (from LANGUAGE_LABELS:
ru, en, de, fr, es, etc.) plus 'SQL' option for source dialect.
Defaults updated: source='SQL' (SQL-like expressions), target='en'.
Bug: loadDatabases() was called at line 374 while environmentId still
held the FIRST environment from initialization (line 259), because
environmentId = j.environment_id was not set until line 391 (inside
the datasource block). Only after that was environmentId correct.
This caused the database dropdown to show databases from a different
environment than the job's actual environment. For example, db_id=2
is DEV Greenplum in 'dev' but Prod Clickhouse in 'preprod'. The user
would select Greenplum from the dev list, but the job would use
preprod where db_id=2 is ClickHouse.
Fix: set environmentId from the job's stored environment BEFORE
calling loadDatabases(), and outside the datasource conditional
so it runs for all existing jobs.
Root cause: fetch_available_datasources did not return database_id,
so frontend could not auto-set target_database_id when user selected
a datasource. User then had to manually pick a database on Target Config
tab — and could accidentally select the wrong one.
Backend changes:
- Add 'database_id' field to datasources response (from db_info.get('id'))
- Add _database_name tracking in SupersetSqlLabExecutor with getter
- Add database_name + database_backend to TargetSchemaValidationResponse
- Enrich resolve_database_id logging with database_name and all_keys
Frontend changes:
- In selectDatasource(), auto-set targetDatabaseId from ds.database_id
when present, so the correct target DB is used for schema checks.
- Replace fragile substring check (any(kw in backend for kw in ('clickhouse', 'ch')))
with a proper _backend_normalize mapping + exact comparison.
Fixes misdetection of Greenplum (postgresql) and other backends.
- Add 'clickhousedb' to CLICKHOUSE_DIALECTS so SQL generator uses
backtick quoting and correct INSERT strategy for ClickHouse.
- Normalize _extract_dialect to map 'clickhousedb' -> 'clickhouse'
and 'greenplum' -> 'postgresql'.
OpenSSL 3.x ignores intermediate CA certs in -CAfile (verify code 20)
but correctly builds chains with -CApath (verify code 0).
All three clients now use capath:
- service.py: ssl.create_default_context(capath=...)
- _llm_http.py: verify='/etc/ssl/certs/'
- preview_llm_client.py: verify='/etc/ssl/certs/'
check_llm_certs.py rewritten for clarity.
Tests all 4 SSL methods with actual project libraries:
- httpx (verify=True/False/SSLContext cafile/cafile+capath)
- requests (verify=True/False/cafile/capath)
- openssl s_client (default/CAfile/CApath/chain bundle)
- certutil (NSS/Chromium)
- certifi vs system CA comparison
Outputs JSON summary with AI diagnosis hint.
Runs from inside Docker container to verify all 4 SSL layers:
1. System CA store (openssl s_client)
2. NSS database (certutil) for Chromium
3. Python httpx (SSLContext with cafile)
4. Python requests (verify with system CA path)
Reads LLM_SSL_VERIFY and CA paths from .env.enterprise-clean
- ADR-0009: comprehensive SSL certificate management strategy
- _get_ssl_verify() returns SSLContext with cafile= instead of bool True
(httpx deprecates verify=<str>, requests needs system CA, not certifi)
- _get_verify() in translate plugin returns system CA path
- All tests updated for SSLContext return type
- All QA findings C1-C3, H1-H4, M1 incorporated into ADR
preview_llm_client.py and _llm_http.py both use requests.post()
without verify=, causing SSLError when corporate CA not in trust store.
Added _get_verify() helper that reads LLM_SSL_VERIFY env var.
C1: fingerprint (SHA256) вместо grep -qF для детекции сертификатов в bundle
— исключает ложные срабатывания и бесконечный рост ca-certificates.crt
C2: hash symlink collision — поддержка .0, .1, .2 суффиксов вместо
перезаписи единственного .0
C3: NSS nickname collision — дедупликация по fingerprint, префикс
директории (llm-/custom-) в nickname, fallback с хешем при коллизии
H1: DER->PEM конвертация — добавлена || проверка ошибки
H2: Удалён install_playwright из entrypoint (Chromium уже baked в Dockerfile)
H3: NSS DB path — sql: префикс, проверка существования и пересоздание
при повреждении
H4: Chicken-and-egg TLS — curl fallback с --insecure при первом скачивании
корп. CA сертификата
M1: nullglob для install_certificates — безопасная итерация по файлам
Introduce a centralized tool registry system for the assistant to manage
executable operations, permissions, and tool catalogs. This refactors
the assistant dispatch logic from a monolithic approach to a modular,
decorator-based registry.
Key changes:
- Implement `AssistantToolRegistry` to handle tool registration,
permission checks, and safe operation identification.
- Add a suite of new assistant tools: backup, commit, branch creation,
deployment, health summary, environment listing, LLM operations,
maintenance, migration, and dashboard searching.
- Refactor `_dispatch.py` and `_llm_planner.py` to utilize the new
registry for intent resolution and tool catalog building.
- Enhance the LLM planner to support more descriptive clarification
responses in Russian when intents are ambiguous.
- Update the frontend to support the new tool-based workflow, including
improved i18n labels for assistant operations and a new step-by-step
wizard for the migration page.
- Add ADR-0008 to document the architectural decision for the tool
registry.
- Removed duplicate #endregion in backend.Dockerfile (P6 critical)
- Added @RATIONALE and @REJECTED to install_ca_to_nss and install_llm_ca_certs
- Changed grep -q to grep -qF in install_llm_ca_certs validation
- install_ca_to_nss() — imports corporate CA certs into Chromium's NSS
database (~/.pki/nssdb), fixing ERR_CERT_AUTHORITY_INVALID in Playwright
- libnss3-tools added to Dockerfile for certutil binary
- LLM_SSL_VERIFY env var to disable SSL verification for corporate proxies
- install_llm_ca_certs() entrypoint function — downloads PEM/DER certs
from LLM_CA_CERT_URLS, converts DER→PEM, installs to system CA store
- _format_connection_error() — detailed exception chain logging
- 7 new unit tests for _get_ssl_verify, _format_connection_error, verify= param
- LLM_CA_CERT_URLS and LLM_SSL_VERIFY in .env.enterprise-clean
- Removed duplicate @RELATION DEPENDS_ON -> [EXT:Library:tenacity]
After configure_logger() runs, logger.isEnabledFor(logging.INFO)
returns False despite level=10 (DEBUG). This is a CPython logging
framework anomaly that makes logger.info() silently drop messages.
Workaround: check isEnabledFor first; if False, write JSON directly
to sys.stderr bypassing the logging system entirely.
Without PYTHONUNBUFFERED, when stderr is piped through '2>&1 | sed',
Python fully buffers stderr output. JSON log lines are small and never
fill the buffer, so they never appear in real time (or at all for
sporadic HTTP request logs). Startup logs appeared because the buffer
flushed on process exit during Alembic migration.
Also kept --reload removed as it breaks stderr capture from child process.
- run.sh: removed --reload flag from uvicorn — reload subprocess breaks
the 2>&1 pipe and all JSON logs from superset_tools_app are lost.
Added comment with alternative for hot-reload.
- core/logger.py: reason() and reflect() now use self.info() instead of
self.debug() — per molecular CoT protocol these are INFO-level bonds.
DEBUG is reserved for high-frequency loops only.
Per molecular CoT protocol: REASON and REFLECT are primary reasoning
bonds and must be visible at INFO level in production. DEBUG is
reserved for high-frequency loops.
- reason(): self.debug() → self.info()
- reflect(): self.debug() → self.info()
- explore(): stays at warning() (WARNING level)
- Also removed redundant logger.info() calls in lifespan (now reason()
itself provides visible startup logging)
- Added logger.info() calls before each startup step (encryption key,
Alembic, init_db, admin bootstrap, scheduler, app complete)
- Previously only logger.reason() was used which logs at DEBUG level
and is invisible in production/development at INFO log level
- This fixes 'no backend logs visible' when running via ./run.sh
- Added missing from fastapi import status (global_exception_handler crashed)
- Moved TraceContextMiddleware to outermost (trace_id now available in log_requests)
- Enhanced global_exception_handler with query_params, client host
- Added logger.explore() in WebSocket auth except blocks (was silent pass)
- Fixed middleware registration order
- Added @app.exception_handler(Exception) that logs full traceback via
logger.exception() into superset_tools_app logger (visible in docker logs)
- Previously FastAPI/Starlette wrote unhandled exceptions to uvicorn.error
logger, making 500 errors invisible in docker logs
- Added JSONResponse import for the handler
- Entrypoint: for legacy DBs, add is_admin via raw SQL then stamp head
(instead of running full migration chain which fails on existing tables)
- Fixed admin_api_keys.py Pydantic config (dict → ConfigDict)
- Added tests/test_alembic_migrations.py (PostgreSQL-only integration tests)
- Fixed infinite recursion in _create_table_if_not_exists (QA catch)
The python3 DB detection script exits with code 1 (empty) or 2 (legacy),
but entrypoint has set -e which kills the script on any non-zero exit.
Fixed by using '|| _db_state=$?' pattern to safely capture exit code
without triggering set -e.
Three-way detection in entrypoint:
- alembic_version exists → normal incremental upgrade
- no tables → fresh upgrade from scratch
- tables exist but no alembic_version → stamp head (legacy DB created
by create_all(), mark all migrations as current without running them)
This fixes 'relation already exists' errors when Alembic tries to CREATE
TABLE on databases that already have tables from previous deployments.
The entrypoint was calling init_auth_db.py → init_db() + seed_permissions()
before the FastAPI startup_event (where run_alembic_migrations() lived).
Since the _ensure_*() safety net was removed, seed_permissions() failed
with 'column roles.is_admin does not exist' on databases with old schema.
Now alembic upgrade head runs directly in the entrypoint, ensuring all
migrations are applied before any DB queries.
- Simplified init_db() to only call Base.metadata.create_all() on all
three engines — no more _ensure_*() inline additive migrations
- run_alembic_migrations() now raises on failure (no safety net to
fall back to)
- All schema changes (add/drop/rename columns, data migrations) must
go through Alembic. create_all() handles new tables only.
@REJECTED _ensure_*() inline migrations removed because they:
- Duplicated Alembic logic without versioning
- Created hidden schema drift (columns existed in DB but had no
corresponding Alembic migration)
- Made audits and fresh DB provisioning unreliable
- Added 4 missing columns to _ensure_translation_jobs_columns() inline
migration: target_language_column, target_source_column,
target_source_language_column, disable_reasoning — these were in the
SQLAlchemy model but had no Alembic migration or _ensure_*() coverage
- Added missing api_key, assistant, clean_release model imports to
alembic/env.py so autogenerate can see all tables
- Added alembic==1.18.4 to requirements.txt (was missing, transitive only)
- Added run_alembic_migrations() to startup_event in app.py — runs
before init_db() to apply all pending migrations
- Combined approach: Alembic for complex migrations first, then inline
_ensure_*() additive safety net in init_db()
- This ensures DB schema is always up to date on container startup
without manual intervention
The inline migration _ensure_roles_is_admin_column() was defined but
never called from init_db(). This caused 'column roles.is_admin does
not exist' errors when logging in against databases with an older
schema. Added the missing call to ensure the column is created during
startup.
Migration now sets is_admin=true for existing Admin roles.
has_permission falls back to role.name == 'Admin' check if
is_admin flag is missing (backward compat for pre-migration roles).
Alembic migration 86c7b1d6a710 adds is_admin BOOLEAN DEFAULT FALSE
to the roles table. Fixes login crash after M-3 change:
psycopg2.errors.UndefinedColumn: column roles.is_admin does not exist
Also adds _ensure_roles_is_admin_column() as an additive migration
safety net for environments that don't run alembic automatically.
To apply: cd backend && source .venv/bin/activate && alembic upgrade head
HSTS (Strict-Transport-Security) is now opt-in via FORCE_HTTPS=true.
Without it enabled, no HSTS header is sent — safe for dev environments
and enterprise deployments behind HTTP-only proxies without HTTPS certs.
When FORCE_HTTPS=true, sends 'max-age=31536000; includeSubDomains'.
Enterprise recommendation: set HSTS at nginx/ingress level instead.
.env.example documents the risk: enabling without certs breaks site access.
