fix(ssl+agent): capath for all HTTP clients + isolate gradio import

SSL fix (ADR-0009 Finding 7):
- Replace ssl.create_default_context() with system_ssl_context(capath)
  in client_registry.py, async_network.py, notifications/providers.py,
  git/_base.py. Previous fix (0.5.1) only covered LLM clients; the
  Superset API client still used ssl.create_default_context() which
  loads cafile (flat bundle) where OpenSSL 3.x ignores intermediate CA
  certificates. system_ssl_context() uses capath only (hash symlinks).

Agent fix:
- Extract _check_llm_provider_health + _llm_status from agent/app.py
  into agent/_llm_health.py. The /api/agent/llm-status endpoint was
  importing from agent/app.py which triggers 'import gradio' at module
  level. Backend container does not have gradio installed, causing
  ModuleNotFoundError (500 error) every 30s on health check polling.

Config:
- Add ALLOWED_ORIGINS to docker-compose.yml + docker-compose.enterprise-clean.yml

ADR-0009 updated: Layer 2 table expanded with 4 missing clients,
Finding 5 reconciled (LLM_CA_CERT_URLS restored in 0.5.1), version 0.5.2.

Verified: 1110 unit tests passed, gradio import isolation confirmed.
This commit is contained in:
2026-07-07 12:18:43 +03:00
parent 18b9f2e94e
commit 58d06fb287
10 changed files with 161 additions and 87 deletions

View File

@@ -0,0 +1,96 @@
# backend/src/agent/_llm_health.py
# #region AgentChat.LlmHealth [C:3] [TYPE Module] [SEMANTICS agent-chat,llm,health,status]
# @ingroup AgentChat
# @BRIEF LLM provider health check with in-memory cache (30s TTL).
# Extracted from agent/app.py to avoid importing gradio in backend container.
# @LAYER Infrastructure
# @RELATION CALLED_BY -> [api/routes/agent_status.py]
# @RELATION CALLED_BY -> [agent/app.py]
# @RATIONALE agent/app.py imports gradio at module level (line 28). The backend
# container does not have gradio installed (only the agent container does).
# The /api/agent/llm-status endpoint needs _check_llm_provider_health but must
# not trigger gradio import. This module isolates the health-check logic.
# @POST Returns status string: 'ok' | 'unavailable' | 'timeout' | 'auth_error'.
# @SIDE_EFFECT Makes a probe request to the LLM provider; caches result in module memory.
import time
from typing import Any
import httpx
from langchain_core.messages import HumanMessage
from langchain_openai import ChatOpenAI
from openai import APIConnectionError, APITimeoutError, AuthenticationError, RateLimitError
from src.agent._llm_params import chat_openai_kwargs
from src.core.logger import logger
# ── LLM provider health cache ─────────────────────────────────────────
_llm_status: dict[str, Any] = {
"status": "ok",
"last_error": "",
"last_check_ts": 0.0,
}
_LLM_CHECK_CACHE_TTL = 30 # seconds between health checks
_LLM_LAST_ERROR_KEY = "last_llm_error"
_LLM_LAST_ERROR_TS_KEY = "last_llm_error_ts"
# #region AgentChat.LlmHealth.Check [C:2] [TYPE Function] [SEMANTICS agent-chat,llm,health,check]
# @ingroup AgentChat
# @BRIEF Check LLM provider connectivity with in-memory cache (30s TTL).
# @POST Returns status string: 'ok' | 'unavailable' | 'timeout' | 'auth_error'.
# @SIDE_EFFECT Makes a probe request to the LLM provider; caches result in module memory.
# @RATIONALE Prevents sending every user request into a dead LLM backend.
async def _check_llm_provider_health() -> str:
"""Check LLM provider connectivity. Cached for _LLM_CHECK_CACHE_TTL seconds."""
now = time.time()
if now - _llm_status["last_check_ts"] < _LLM_CHECK_CACHE_TTL:
return _llm_status["status"]
try:
from src.agent.langgraph_setup import _fetch_llm_config as _get_config
config = await _get_config()
if not config or not config.get("configured"):
return _llm_status["status"]
llm = ChatOpenAI(
**chat_openai_kwargs(
model=config.get("default_model", "gpt-4o-mini"),
base_url=config.get("base_url", "https://api.openai.com/v1"),
api_key=config.get("api_key", ""),
max_tokens=1,
)
)
await llm.ainvoke([HumanMessage(content="ping")])
_llm_status["status"] = "ok"
_llm_status["last_error"] = ""
_llm_status["last_check_ts"] = time.time()
return "ok"
except (APIConnectionError, httpx.ConnectError):
_llm_status["status"] = "unavailable"
_llm_status["last_error"] = "Connection refused"
_llm_status["last_check_ts"] = time.time()
return "unavailable"
except (APITimeoutError, httpx.ReadTimeout):
_llm_status["status"] = "timeout"
_llm_status["last_error"] = "Request timed out"
_llm_status["last_check_ts"] = time.time()
return "timeout"
except AuthenticationError:
_llm_status["status"] = "auth_error"
_llm_status["last_error"] = "Invalid API key"
_llm_status["last_check_ts"] = time.time()
return "auth_error"
except Exception as exc:
if "usage_metadata.total_tokens" in str(exc):
_llm_status["status"] = "ok"
_llm_status["last_error"] = ""
_llm_status["last_check_ts"] = time.time()
return "ok"
logger.explore("LLM health check failed", error=str(exc), extra={"src": "AgentChat.LlmHealth.Check"})
return _llm_status["status"] # return cached status on unexpected errors
# #endregion AgentChat.LlmHealth.Check
# #endregion AgentChat.LlmHealth

View File

@@ -40,6 +40,14 @@ from src.agent._confirmation import (
handle_resume,
permission_denied_payload,
)
from src.agent._jwt_decoder import decode_token
from src.agent._llm_health import (
_LLM_CHECK_CACHE_TTL,
_LLM_LAST_ERROR_KEY,
_LLM_LAST_ERROR_TS_KEY,
_check_llm_provider_health,
_llm_status,
)
from src.agent._llm_params import chat_openai_kwargs
from src.agent._persistence import (
extract_user_id,
@@ -58,7 +66,6 @@ from src.agent.tools import (
get_all_tools,
start_tool_retry_event_buffer,
)
from src.agent._jwt_decoder import decode_token
from src.core.cot_logger import seed_trace_id
from src.core.logger import logger
@@ -140,14 +147,9 @@ async def _generate_title_best_effort(conv_id: str, visible_user_text: str) -> N
_user_locks: dict[str, bool] = {}
# ── LLM provider health cache ─────────────────────────────────---
_llm_status: dict[str, Any] = {
"status": "ok",
"last_error": "",
"last_check_ts": 0.0,
}
_LLM_CHECK_CACHE_TTL = 30 # seconds between health checks
_LLM_LAST_ERROR_KEY = "last_llm_error"
_LLM_LAST_ERROR_TS_KEY = "last_llm_error_ts"
# _llm_status, _LLM_CHECK_CACHE_TTL, _LLM_LAST_ERROR_KEY, _LLM_LAST_ERROR_TS_KEY,
# and _check_llm_provider_health() are imported from src.agent._llm_health
# to avoid triggering gradio import in backend container.
# ── File persistence ────────────────────────────────────────────
@@ -214,64 +216,9 @@ _service_jwt_cache: dict[str, str] = {}
# @REJECTED Returning a single response (non-streaming) was rejected — violates FR-003 (streaming mandate).
# #region AgentChat.GradioApp.LlmHealthCheck [C:2] [TYPE Function] [SEMANTICS agent-chat,llm,health]
# @ingroup AgentChat
# @BRIEF Check LLM provider connectivity with in-memory cache (30s TTL).
# @POST Returns status string: 'ok' | 'unavailable' | 'timeout' | 'auth_error'.
# @SIDE_EFFECT Makes a probe request to the LLM provider; caches result in module memory.
# @RATIONALE Prevents sending every user request into an dead LLM backend.
async def _check_llm_provider_health() -> str:
"""Check LLM provider connectivity. Cached for _LLM_CHECK_CACHE_TTL seconds."""
now = time.time()
if now - _llm_status["last_check_ts"] < _LLM_CHECK_CACHE_TTL:
return _llm_status["status"]
# _check_llm_provider_health() moved to src.agent._llm_health
# (avoids gradio import in backend container for /api/agent/llm-status endpoint)
try:
from src.agent.langgraph_setup import _fetch_llm_config as _get_config
config = await _get_config()
if not config or not config.get("configured"):
return _llm_status["status"]
llm = ChatOpenAI(
**chat_openai_kwargs(
model=config.get("default_model", "gpt-4o-mini"),
base_url=config.get("base_url", "https://api.openai.com/v1"),
api_key=config.get("api_key", ""),
max_tokens=1,
)
)
await llm.ainvoke([HumanMessage(content="ping")])
_llm_status["status"] = "ok"
_llm_status["last_error"] = ""
_llm_status["last_check_ts"] = time.time()
return "ok"
except (APIConnectionError, httpx.ConnectError):
_llm_status["status"] = "unavailable"
_llm_status["last_error"] = "Connection refused"
_llm_status["last_check_ts"] = time.time()
return "unavailable"
except (APITimeoutError, httpx.ReadTimeout):
_llm_status["status"] = "timeout"
_llm_status["last_error"] = "Request timed out"
_llm_status["last_check_ts"] = time.time()
return "timeout"
except AuthenticationError:
_llm_status["status"] = "auth_error"
_llm_status["last_error"] = "Invalid API key"
_llm_status["last_check_ts"] = time.time()
return "auth_error"
except Exception as exc:
if "usage_metadata.total_tokens" in str(exc):
_llm_status["status"] = "ok"
_llm_status["last_error"] = ""
_llm_status["last_check_ts"] = time.time()
return "ok"
logger.explore("LLM health check failed", error=str(exc), extra={"src": "AgentChat.GradioApp.LlmHealthCheck"})
return _llm_status["status"] # return cached status on unexpected errors
# #endregion AgentChat.GradioApp.LlmHealthCheck
# #region AgentChat.GradioApp.InjectUIContext [C:2] [TYPE Function] [SEMANTICS agent-chat,context,uicontext]

View File

@@ -16,7 +16,7 @@ router = APIRouter(prefix="/api/agent", tags=["agent-status"])
@router.get("/llm-status")
async def get_llm_status():
"""Get cached LLM provider health status. Probes provider if cache expired."""
from src.agent.app import _check_llm_provider_health, _llm_status
from src.agent._llm_health import _check_llm_provider_health, _llm_status
status = await _check_llm_provider_health()
return {
"status": status,

View File

@@ -24,6 +24,7 @@ from typing import Any
import httpx
from ..logger import belief_scope, logger as app_logger
from ..ssl import system_ssl_context
from .network import (
AuthenticationError,
DashboardNotFoundError,
@@ -66,10 +67,11 @@ class AsyncAPIClient:
self.api_base_url: str = f"{self.base_url}/api/v1"
self.auth = config.get("auth")
self.request_settings = {"verify_ssl": verify_ssl, "timeout": timeout}
# Use system CA store via ssl.create_default_context() instead of bool True.
# httpx defaults to certifi, ignoring SSL_CERT_FILE and corporate CA certs.
# Use system CA store via capath (not ssl.create_default_context() which
# loads cafile + capath; OpenSSL 3.x ignores intermediate CAs in cafile,
# ADR-0009 Finding 7). system_ssl_context() uses capath only.
if verify_ssl is True:
ssl_context: bool | ssl.SSLContext = ssl.create_default_context()
ssl_context: bool | ssl.SSLContext = system_ssl_context()
elif verify_ssl is False:
ssl_context = False
else:

View File

@@ -23,6 +23,7 @@ import ssl
from typing import Any
from ..logger import logger
from ..ssl import system_ssl_context
from .async_network import AsyncAPIClient
@@ -99,10 +100,14 @@ async def get_client(
config = {}
verify_ssl = True
timeout = request_timeout
# Convert bool to SSLContext with system CA store — httpx defaults to certifi,
# ignoring corporate CA certificates installed via update-ca-certificates.
# Convert bool to SSLContext with system CA store via capath.
# ssl.create_default_context() without capath loads BOTH cafile (flat bundle)
# and capath (hash symlinks). OpenSSL 3.x ignores intermediate CA certs in
# cafile (ADR-0009 Finding 7), causing CERTIFICATE_VERIFY_FAILED for chains
# with intermediate CAs (e.g. Root → Policy CA → Issuing CA → Server).
# system_ssl_context() uses capath only, correctly building intermediate chains.
if verify_ssl is True:
ssl_context: bool | ssl.SSLContext = ssl.create_default_context()
ssl_context: bool | ssl.SSLContext = system_ssl_context()
else:
ssl_context = verify_ssl
async_client = AsyncAPIClient(

View File

@@ -25,6 +25,7 @@ import httpx
from src.core.database import SessionLocal
from src.core.logger import belief_scope, logger
from src.core.ssl import system_ssl_context
from src.core.utils.executors import run_blocking
from src.models.config import AppConfigRecord
from src.models.git import GitRepository
@@ -58,10 +59,11 @@ class GitServiceBase:
self._close_lock = threading.Lock()
# Fix 5: Shared httpx.AsyncClient with connection pooling.
# Uses system CA store (ssl.create_default_context) instead of certifi
# to trust corporate CA certificates installed via update-ca-certificates.
# Uses system CA store via capath (ADR-0009 Finding 7) instead of
# ssl.create_default_context() which loads cafile where OpenSSL 3.x
# ignores intermediate CA certificates.
self._http_client = httpx.AsyncClient(
verify=ssl.create_default_context(),
verify=system_ssl_context(),
timeout=30.0,
limits=httpx.Limits(max_keepalive_connections=20, max_connections=100),
)

View File

@@ -29,6 +29,7 @@ import aiosmtplib
import httpx
from src.core.logger import logger
from src.core.ssl import system_ssl_context
# Module-level httpx client, lazily initialized for connection reuse
_http_client: httpx.AsyncClient | None = None
@@ -36,13 +37,14 @@ _http_client: httpx.AsyncClient | None = None
async def _get_http_client() -> httpx.AsyncClient:
"""Get or create the module-level httpx.AsyncClient singleton.
Uses system CA store (ssl.create_default_context) instead of certifi
to trust corporate CA certificates installed via update-ca-certificates.
Uses system CA store via capath (ADR-0009 Finding 7) instead of
ssl.create_default_context() which loads cafile where OpenSSL 3.x
ignores intermediate CA certificates.
"""
global _http_client
if _http_client is None:
_http_client = httpx.AsyncClient(
verify=ssl.create_default_context(),
verify=system_ssl_context(),
timeout=httpx.Timeout(30.0),
)
return _http_client

View File

@@ -42,6 +42,7 @@ services:
TASKS_DATABASE_URL: postgresql+psycopg2://${POSTGRES_USER:-postgres}:${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env.enterprise-clean}@${POSTGRES_HOST:?Set POSTGRES_HOST in .env.enterprise-clean}:${POSTGRES_PORT:-5432}/${POSTGRES_DB:-ss_tools}
AUTH_DATABASE_URL: postgresql+psycopg2://${POSTGRES_USER:-postgres}:${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env.enterprise-clean}@${POSTGRES_HOST:?Set POSTGRES_HOST in .env.enterprise-clean}:${POSTGRES_PORT:-5432}/${POSTGRES_DB:-ss_tools}
BACKEND_PORT: 8000
ALLOWED_ORIGINS: ${ALLOWED_ORIGINS:-}
AUTH_SECRET_KEY: ${AUTH_SECRET_KEY:?Set AUTH_SECRET_KEY in .env.enterprise-clean}
ENCRYPTION_KEY: ${ENCRYPTION_KEY:?ENCRYPTION_KEY обязателен — сгенерируйте и укажите в .env.enterprise-clean}
ENABLE_BELIEF_STATE_LOGGING: ${ENABLE_BELIEF_STATE_LOGGING:-true}

View File

@@ -32,6 +32,7 @@ services:
TASKS_DATABASE_URL: postgresql+psycopg2://postgres:postgres@db:5432/ss_tools
AUTH_DATABASE_URL: postgresql+psycopg2://postgres:postgres@db:5432/ss_tools
BACKEND_PORT: 8000
ALLOWED_ORIGINS: ${ALLOWED_ORIGINS:-}
INITIAL_ADMIN_CREATE: ${INITIAL_ADMIN_CREATE:-false}
INITIAL_ADMIN_USERNAME: ${INITIAL_ADMIN_USERNAME:-admin}
INITIAL_ADMIN_PASSWORD: ${INITIAL_ADMIN_PASSWORD:-}

View File

@@ -10,6 +10,10 @@
# @RELATION CALLS -> [backend/src/core/ssl.py]
# @RELATION CALLS -> [backend/src/plugins/llm_analysis/service.py]
# @RELATION CALLS -> [backend/src/plugins/translate/_llm_async_http.py]
# @RELATION CALLS -> [backend/src/core/utils/client_registry.py]
# @RELATION CALLS -> [backend/src/core/utils/async_network.py]
# @RELATION CALLS -> [backend/src/services/notifications/providers.py]
# @RELATION CALLS -> [backend/src/services/git/_base.py]
# @RELATION CALLS -> [scripts/check_llm_certs.py]
# @RELATION CALLS -> [scripts/diag_container.py]
# @RATIONALE superset-tools operates in corporate environments with internal Certificate Authorities.
@@ -81,12 +85,19 @@ Diagnostic matrix (verified on production server 2026-05-28):
| Library | File | Mechanism | Status |
|---------|------|-----------|--------|
| `httpx` (AsyncOpenAI) | `service.py:LLMClient._get_ssl_verify()` | `ssl.create_default_context(capath="/etc/ssl/certs/")` | ✅ Works (0.1.7) |
| `requests` (`_llm_http.py`) | `_get_verify()` | `"/etc/ssl/certs/"` (string path to dir) | ✅ Works (0.1.7) |
| `requests` (`preview_llm_client.py`) | `_get_verify()` | `"/etc/ssl/certs/"` (string path to dir) | ✅ Works (0.1.7) |
| `httpx` (AsyncOpenAI) | `service.py:LLMClient._get_ssl_verify()` | `system_ssl_context(capath="/etc/ssl/certs/")` | ✅ Fixed (0.5.1) |
| `httpx` (translate) | `_llm_async_http.py:_get_verify()` | `httpx_verify()` `system_ssl_context(capath)` | ✅ Fixed (0.5.1) |
| `httpx` (SupersetClientRegistry) | `client_registry.py:get_client()` | `system_ssl_context(capath="/etc/ssl/certs/")` | ✅ Fixed (0.5.2) |
| `httpx` (AsyncAPIClient) | `async_network.py:__init__()` | `system_ssl_context(capath="/etc/ssl/certs/")` | ✅ Fixed (0.5.2) |
| `httpx` (Notifications) | `notifications/providers.py:_get_http_client()` | `system_ssl_context(capath="/etc/ssl/certs/")` | ✅ Fixed (0.5.2) |
| `httpx` (Git) | `git/_base.py:GitServiceBase.__init__()` | `system_ssl_context(capath="/etc/ssl/certs/")` | ✅ Fixed (0.5.2) |
Key insight: `verify=True` uses certifi, NOT system CA. `verify=<cafile_path>` ignores
intermediate CA in OpenSSL 3.x. `verify=<capath_dir>` works correctly.
`ssl.create_default_context()` without explicit capath loads BOTH cafile and capath
from OpenSSL defaults — the cafile may fail intermediate CA chain-building (Finding 7).
`system_ssl_context(capath="/etc/ssl/certs")` loads ONLY capath (hash symlinks),
correctly building chains with intermediate CAs.
### Layer 3: NSS Database (Playwright Chromium)
@@ -126,6 +137,10 @@ intermediate CA in OpenSSL 3.x. `verify=<capath_dir>` works correctly.
| `docker/frontend.entrypoint.sh` | Installs CA certs from `CERTS_PATH` into Alpine store |
| `docker/agent.entrypoint.sh` | Sources `certs.sh`, installs certs for agent container |
| `backend/src/core/ssl.py` | Centralized SSL helper: `system_ssl_context()`, `httpx_verify()`, `describe_context()` |
| `backend/src/core/utils/client_registry.py` | SupersetClientRegistry: `get_client()``system_ssl_context()` |
| `backend/src/core/utils/async_network.py` | AsyncAPIClient: `__init__()``system_ssl_context()` |
| `backend/src/services/notifications/providers.py` | Notification HTTP: `_get_http_client()``system_ssl_context()` |
| `backend/src/services/git/_base.py` | Git HTTP: `GitServiceBase.__init__()``system_ssl_context()` |
| `plugins/llm_analysis/service.py` | `LLMClient._get_ssl_verify()` → delegates to `core.ssl.httpx_verify()` |
| `plugins/translate/_llm_async_http.py` | `_get_verify()` → delegates to `core.ssl.httpx_verify()` |
| `scripts/check_llm_certs.py` | Full diagnostic: openssl, httpx, requests, NSS, centralized SSL |
@@ -196,11 +211,13 @@ Corporate PKI servers (`pki.rusal.com`) often serve certificates in DER (binary)
`update-ca-certificates` requires PEM. Auto-detection and conversion
(`openssl x509 -inform DER -outform PEM`) is essential.
### Finding 5: Chicken-and-egg TLS bootstrap (HISTORICAL — removed in 0.2.x)
### Finding 5: Chicken-and-egg TLS bootstrap (HISTORICAL — resolved in 0.5.1)
Downloading a CA certificate from a PKI server that uses the same CA causes a TLS
verification loop. Formerly handled by `install_llm_ca_certs()` with `curl --insecure`.
**Removed in 0.2.x**: certificates must be placed in `CERTS_PATH` volume mount by operator,
eliminating the chicken-and-egg download problem.
**Removed in 0.2.x**: certificates must be placed in `CERTS_PATH` volume mount by operator.
**Restored in 0.5.1**: `LLM_CA_CERT_URLS` HTTP download re-added with `--proto '=http'`
policy (plain HTTP only, no TLS chicken-and-egg). Both channels (`LLM_CA_CERT_URLS`
download + `CERTS_PATH` mount) are now active simultaneously.
### Finding 6: NSS DB format
Chromium uses NSS Shared DB in `~/.pki/nssdb/`. The `sql:` prefix is required for
@@ -233,9 +250,9 @@ Fix: pass `input=""` or `echo | openssl s_client ...`.
xz -dc superset-tools-backend.0.2.x.tar.xz | docker load
xz -dc superset-tools-frontend.0.2.x.tar.xz | docker load
# Place corporate CA files in ./certs (LLM_SSL_VERIFY and LLM_CA_CERT_URLS removed)
# Place corporate CA files in ./certs (LLM_SSL_VERIFY removed, LLM_CA_CERT_URLS optional)
ls ./certs/
# RUSAL_ROOT.crt RGM_Issuing.crt etc.
# RUSAL_ROOT.crt RGM_Issuing.crt UC_RUSAL_Policy_CA.crt etc.
docker compose -f docker-compose.enterprise-clean.yml \
--env-file .env.enterprise-clean down
@@ -259,5 +276,6 @@ docker compose -f docker-compose.enterprise-clean.yml \
| 0.1.7 | **capath instead of cafile**: OpenSSL 3.x cafile limitation discovered and fixed. `.crt` extension for downloaded certs. Diagnostic script rewritten. |
| 0.2.x | **Centralized SSL**: `LLM_SSL_VERIFY` and `LLM_CA_CERT_URLS` removed. Shared `docker/certs.sh` for all containers. `backend/src/core/ssl.py` central helper. `CERTS_PATH` volume mount is the only certificate input. Verify=False escape hatch permanently removed. |
| 0.5.1 | **Restored**: `LLM_CA_CERT_URLS` HTTP download as primary cert input channel. `download_llm_ca_certs()` added to `certs.sh`. Downloaded certs saved to `llm/` subdirectory, `install_all_certs` extended to create hash symlinks for both `custom/` and `llm/` dirs. `CERTS_PATH` volume mount kept as secondary channel. Integration test added for HTTP download flow. |
| 0.5.2 | **Fixed capath for ALL HTTP clients**: `ssl.create_default_context()` (no capath, loads cafile+capath) replaced with `system_ssl_context(capath="/etc/ssl/certs")` (capath only) in `client_registry.py`, `async_network.py`, `notifications/providers.py`, `git/_base.py`. Previous fix (0.5.1) only covered LLM clients; Superset API, notification, and git clients still used `ssl.create_default_context()` which fails intermediate CA chain-building in OpenSSL 3.x (Finding 7). Production diagnosis: RUSAL cert chain Root→Policy CA→RGM Issuing→Server requires Policy CA as intermediate; without capath, OpenSSL ignores it in cafile. |
# [/DEF:ADR-0009:ADR]