ss-tools/docs/adr/ADR-0009-ssl-certificate-management.md

[DEF:ADR-0009:ADR]

@STATUS ACTIVE

@PURPOSE Define the strategy for corporate SSL certificate management across all HTTP clients in superset-tools (httpx, requests, openai), covering system CA store, NSS database for Chromium/Playwright. Two input channels: (1) HTTP download from corporate PKI via LLM_CA_CERT_URLS (primary), (2) volume mount via CERTS_PATH (additional). LLM_SSL_VERIFY removed in 0.2.x.

@RELATION DEPENDS_ON -> [ADR-0001:ADR]

@RELATION DEPENDS_ON -> [ADR-0004:ADR]

@RELATION CALLS -> [docker/backend.entrypoint.sh]

@RELATION CALLS -> [docker/certs.sh]

@RELATION CALLS -> [docker/frontend.entrypoint.sh]

@RELATION CALLS -> [docker/agent.entrypoint.sh]

@RELATION CALLS -> [backend/src/core/ssl.py]

@RELATION CALLS -> [backend/src/plugins/llm_analysis/service.py]

@RELATION CALLS -> [backend/src/plugins/translate/_llm_async_http.py]

@RELATION CALLS -> [backend/src/core/utils/client_registry.py]

@RELATION CALLS -> [backend/src/core/utils/async_network.py]

@RELATION CALLS -> [backend/src/services/notifications/providers.py]

@RELATION CALLS -> [backend/src/services/git/_base.py]

@RELATION CALLS -> [scripts/check_llm_certs.py]

@RELATION CALLS -> [scripts/diag_container.py]

@RATIONALE superset-tools operates in corporate environments with internal Certificate Authorities.

Three separate HTTP stacks are used: httpx (AsyncOpenAI in llm_analysis plugin),

requests (sync translate plugin), and Playwright Chromium (dashboard screenshots).

Each has its own CA trust store, requiring different installation strategies.

@REJECTED Centralizing all LLM calls into a single HTTP client — rejected because

llm_analysis uses async httpx+AsyncOpenAI for streaming and multiple concurrent calls,

while translate uses sync requests for simpler request/response. Merging would

require rewriting one or both, creating regression risk with no business value.

Problem

Corporate SSL certificates installed via update-ca-certificates into /etc/ssl/certs/ca-certificates.crt are NOT automatically trusted by:

1. Python requests library — uses bundled certifi CA bundle, not system store
2. Python httpx library — uses certifi by default when verify=True
3. Playwright Chromium — uses NSS Shared DB (~/.pki/nssdb), not OpenSSL store
4. openssl s_client — hangs without stdin (needs echo \| or input="")

This causes SSLError: CERTIFICATE_VERIFY_FAILED and ERR_CERT_AUTHORITY_INVALID even after correct system-wide CA installation.

Discovery Journey

Phase 1: certifi vs system CA (0.1.5)

• requests and httpx use certifi bundle, not /etc/ssl/certs/ca-certificates.crt
• Fix: return system CA path instead of True in all _get_verify() functions
• Implemented in service.py, _llm_http.py, preview_llm_client.py

Phase 2: .pem vs .crt extension (0.1.6)

• update-ca-certificates only processes .crt files, silently ignores .pem
• Downloaded certificates from PKI were saved as .pem, skipped by update-ca-certificates
• Result: ca-certificates.crt was empty (0 certs), hash symlinks existed but bundle was broken
• Fix: save downloaded certs with .crt extension, not .pem

Phase 3: cafile vs capath (0.1.7) — KEY DISCOVERY

• openssl s_client -CAfile /etc/ssl/certs/ca-certificates.crt → code 20 (intermediate CA ignored)
• openssl s_client -CApath /etc/ssl/certs/ → code 0 (chain built correctly)
• ssl.create_default_context(cafile=...) → SSL error
• ssl.create_default_context(capath=...) → HTTP 200
• requests.get(verify="/etc/ssl/certs/ca-certificates.crt") → SSLError
• requests.get(verify="/etc/ssl/certs/") → HTTP 200

Root cause: OpenSSL 3.x treats non-self-signed certificates in -CAfile as trust anchors, NOT as intermediates. -CApath (directory with hash symlinks) correctly builds chains using all certificates found. The intermediate CA certs (Policy CA, RGM Issuing CA) are not self-signed, so they are ignored in -CAfile but correctly used in -CApath.

Diagnostic matrix (verified on production server 2026-05-28):

Method	cafile	capath
openssl s_client	code 20 FAIL	code 0 OK
httpx (SSLContext)	SSLError FAIL	HTTP 200 OK
requests (verify=)	SSLError FAIL	HTTP 200 OK

Solution

Layer 1: System CA Store (OpenSSL)

• Shared docker/certs.sh:install_all_certs() copies .crt/.pem/.cer/.der files from CERTS_PATH volume mount
• DER is auto-converted to PEM via normalize_cert() (openssl x509 -inform DER -outform PEM)
• Server/private files (server.crt, server.key, *.key, *.p12, *.pfx) are excluded from CA import
• Certificates are saved with .crt extension to /usr/local/share/ca-certificates/custom/
• update-ca-certificates --fresh adds them to /etc/ssl/certs/ca-certificates.crt
• Hash symlinks created with collision support: .0, .1, .2 suffixes

Layer 2: Python HTTP Clients

Library	File	Mechanism	Status
httpx (AsyncOpenAI)	service.py:LLMClient._get_ssl_verify()	system_ssl_context(capath="/etc/ssl/certs/")	✅ Fixed (0.5.1)
httpx (translate)	_llm_async_http.py:_get_verify()	httpx_verify() → system_ssl_context(capath)	✅ Fixed (0.5.1)
httpx (SupersetClientRegistry)	client_registry.py:get_client()	system_ssl_context(capath="/etc/ssl/certs/")	✅ Fixed (0.5.2)
httpx (AsyncAPIClient)	async_network.py:__init__()	system_ssl_context(capath="/etc/ssl/certs/")	✅ Fixed (0.5.2)
httpx (Notifications)	notifications/providers.py:_get_http_client()	system_ssl_context(capath="/etc/ssl/certs/")	✅ Fixed (0.5.2)
httpx (Git)	git/_base.py:GitServiceBase.__init__()	system_ssl_context(capath="/etc/ssl/certs/")	✅ Fixed (0.5.2)

Key insight: verify=True uses certifi, NOT system CA. verify=<cafile_path> ignores intermediate CA in OpenSSL 3.x. verify=<capath_dir> works correctly. ssl.create_default_context() without explicit capath loads BOTH cafile and capath from OpenSSL defaults — the cafile may fail intermediate CA chain-building (Finding 7). system_ssl_context(capath="/etc/ssl/certs") loads ONLY capath (hash symlinks), correctly building chains with intermediate CAs.

Layer 3: NSS Database (Playwright Chromium)

• Entrypoint install_ca_to_nss() imports PEM certs into ~/.pki/nssdb/ using certutil
• NSS DB path format: sql:$HOME/.pki/nssdb (SQLite prefix required, DBM not supported by Chromium)
• Nickname format: {dir_prefix}-{filename} (e.g., llm-UC_RUSAL_Policy_CA)
• Dedup by SHA256 fingerprint (not nickname), preventing duplicate imports
• Trust attributes: "C,," (trusted CA for TLS server certs)
• Fixes ERR_CERT_AUTHORITY_INVALID in Playwright dashboard screenshots

Layer 4: LLM_CA_CERT_URLS HTTP Download (0.5.1)

• Env var LLM_CA_CERT_URLS — space-separated list of HTTP URLs to CA certificates
• Downloaded on container startup by certs.sh:download_llm_ca_certs()
• URLs use plain HTTP only — no TLS chicken-and-egg problem
• Supports PEM and DER formats (auto-detected, DER auto-converted to PEM)
• Retry logic: 3 attempts with 2-second delay
• Downloaded certs saved to /usr/local/share/ca-certificates/llm/
• update-ca-certificates --fresh includes them in the system bundle
• Example: LLM_CA_CERT_URLS="http://pki.rusal.com/CertData/UC_RUSAL_Policy_CA.crt http://pki.rusal.com/CertData/UC_RUSAL_RGM_Issuing_CA.crt"

Layer 5: CERTS_PATH Volume Mount

• Env var CERTS_PATH — directory with .crt/.pem/.cer/.der files
• Mounted as read-only volume: ${CERTS_PATH:-./certs}:/opt/certs:ro
• Installed by certs.sh:install_all_certs() after download_llm_ca_certs()
• Server/private files (server.crt, *.key, *.p12, *.pfx) are excluded from CA import
• Shared docker/certs.sh handles installation across all containers (backend, frontend, agent).

Key Files

File	Role
docker/backend.Dockerfile	Installs libnss3-tools (certutil), Playwright Chromium
docker/certs.sh	Shared cert installer: download_llm_ca_certs, install_all_certs, install_ca_to_nss, normalize_cert
docker/backend.entrypoint.sh	Sources certs.sh, calls download_llm_ca_certs → install_all_certs → install_ca_to_nss
docker/frontend.entrypoint.sh	Installs CA certs from CERTS_PATH into Alpine store
docker/agent.entrypoint.sh	Sources certs.sh, installs certs for agent container
backend/src/core/ssl.py	Centralized SSL helper: system_ssl_context(), httpx_verify(), describe_context()
backend/src/core/utils/client_registry.py	SupersetClientRegistry: get_client() → system_ssl_context()
backend/src/core/utils/async_network.py	AsyncAPIClient: __init__() → system_ssl_context()
backend/src/services/notifications/providers.py	Notification HTTP: _get_http_client() → system_ssl_context()
backend/src/services/git/_base.py	Git HTTP: GitServiceBase.__init__() → system_ssl_context()
plugins/llm_analysis/service.py	LLMClient._get_ssl_verify() → delegates to core.ssl.httpx_verify()
plugins/translate/_llm_async_http.py	_get_verify() → delegates to core.ssl.httpx_verify()
scripts/check_llm_certs.py	Full diagnostic: openssl, httpx, requests, NSS, centralized SSL
docker-compose.yml	Mounts ${CERTS_PATH:-./certs}:/opt/certs:ro (LLM env vars removed)
docker-compose.enterprise-clean.yml	Same — CERTS_PATH mount only

How to Test Certificate Installation

On the server, after container restart:

docker cp scripts/check_llm_certs.py superset-tools-backend-1:/tmp/

docker compose -f docker-compose.enterprise-clean.yml \
  --env-file .env.enterprise-clean exec backend \
  python3 /tmp/check_llm_certs.py --target https://lite.ai.rusal.com

Expected diagnostic matrix:

openssl_default: ✅ code 0
openssl_capath:  ✅ code 0
openssl_cafile:  ❌ code 20 (expected — OpenSSL 3.x limitation)
httpx_capath:    ✅ HTTP 200
httpx_cafile:    ❌ SSL error (expected)
requests_capath: ✅ HTTP 200
requests_cafile: ❌ SSL error (expected)

Manually testing chain validity

# 1. Check individual certs
openssl x509 -in /usr/local/share/ca-certificates/custom/RUSAL_ROOT.crt -noout -subject -issuer

# 2. Verify full chain
openssl verify -CAfile /usr/local/share/ca-certificates/custom/RUSAL_ROOT.crt \
  -untrusted /usr/local/share/ca-certificates/custom/UC_RUSAL_Policy_CA.crt \
  /usr/local/share/ca-certificates/custom/UC_RUSAL_RGM_Issuing_CA.crt
# → OK

# 3. Connect with capath
echo | openssl s_client -connect lite.ai.rusal.com:443 \
  -servername lite.ai.rusal.com \
  -CApath /etc/ssl/certs/

# 4. Verify certificate count
grep -c "BEGIN CERTIFICATE" /etc/ssl/certs/ca-certificates.crt

Discovered Findings

Finding 1: certifi vs system CA

requests and httpx use certifi CA bundle by default. Corporate CA certs installed into /etc/ssl/certs/ are invisible to Python HTTP clients unless explicitly pointed to the system bundle.

Finding 2: Three separate HTTP stacks

superset-tools has three LLM HTTP clients (httpx for async, requests for sync, Playwright for screenshots), each with independent CA trust configuration.

Finding 3: .pem extension ignored by update-ca-certificates

On Debian Bookworm (python:3.11-slim), update-ca-certificates only processes .crt files. Files with .pem extension in /usr/local/share/ca-certificates/ are silently skipped. This left ca-certificates.crt empty (0 certs) while hash symlinks existed. Fix: save downloaded certificates with .crt extension.

Finding 4: DER format from corporate PKI

Corporate PKI servers (pki.rusal.com) often serve certificates in DER (binary) format. update-ca-certificates requires PEM. Auto-detection and conversion (openssl x509 -inform DER -outform PEM) is essential.

Finding 5: Chicken-and-egg TLS bootstrap (HISTORICAL — resolved in 0.5.1)

Downloading a CA certificate from a PKI server that uses the same CA causes a TLS verification loop. Formerly handled by install_llm_ca_certs() with curl --insecure. Removed in 0.2.x: certificates must be placed in CERTS_PATH volume mount by operator. Restored in 0.5.1: LLM_CA_CERT_URLS HTTP download re-added with --proto '=http' policy (plain HTTP only, no TLS chicken-and-egg). Both channels (LLM_CA_CERT_URLS download + CERTS_PATH mount) are now active simultaneously.

Finding 6: NSS DB format

Chromium uses NSS Shared DB in ~/.pki/nssdb/. The sql: prefix is required for certutil to use the SQLite format. Without it, certutil defaults to the legacy DBM format which Chromium may not read.

Finding 7: OpenSSL 3.x cafile vs capath (CRITICAL)

OpenSSL 3.x treats non-self-signed certificates in -CAfile as trust anchors, NOT as intermediates. This means intermediate CA certificates (Policy CA, RGM Issuing CA) are ignored when using -CAfile /etc/ssl/certs/ca-certificates.crt, producing verify code 20. Using -CApath /etc/ssl/certs/ (directory with hash symlinks) correctly builds the full chain, producing verify code 0.

This affects ALL Python libraries:

• ssl.create_default_context(cafile=...) → calls OpenSSL's SSL_CTX_load_verify_locations with the file, which exhibits the same limitation
• ssl.create_default_context(capath=...) → works correctly
• requests.get(verify="/path/to/file") → fails (uses cafile internally)
• requests.get(verify="/path/to/dir/") → works (uses capath internally)

Finding 8: openssl s_client hangs without stdin

openssl s_client waits for input after TLS handshake. When run via subprocess.run(cmd, capture_output=True), it hangs until timeout. Fix: pass input="" or echo | openssl s_client ....

Deploy

# On target server:
xz -dc superset-tools-backend.0.2.x.tar.xz | docker load
xz -dc superset-tools-frontend.0.2.x.tar.xz | docker load

# Place corporate CA files in ./certs (LLM_SSL_VERIFY removed, LLM_CA_CERT_URLS optional)
ls ./certs/
# RUSAL_ROOT.crt  RGM_Issuing.crt  UC_RUSAL_Policy_CA.crt  etc.

docker compose -f docker-compose.enterprise-clean.yml \
  --env-file .env.enterprise-clean down

docker compose -f docker-compose.enterprise-clean.yml \
  --env-file .env.enterprise-clean up -d

# Verify
docker cp scripts/check_llm_certs.py superset-tools-backend-1:/tmp/
docker compose -f docker-compose.enterprise-clean.yml \
  --env-file .env.enterprise-clean exec backend \
  python3 /tmp/check_llm_certs.py --target https://lite.ai.rusal.com

Version History

Version	Changes
0.1.5	Initial SSL support: LLM_SSL_VERIFY, _format_connection_error(), CA download via URL, NSS import
0.1.6	QA fixes: fingerprint dedup, hash symlink collision, NSS collision, DER→PEM error handling, chicken-and-egg TLS, nullglob, translate plugin SSL
0.1.7	capath instead of cafile: OpenSSL 3.x cafile limitation discovered and fixed. .crt extension for downloaded certs. Diagnostic script rewritten.
0.2.x	Centralized SSL: LLM_SSL_VERIFY and LLM_CA_CERT_URLS removed. Shared docker/certs.sh for all containers. backend/src/core/ssl.py central helper. CERTS_PATH volume mount is the only certificate input. Verify=False escape hatch permanently removed.
0.5.1	Restored: LLM_CA_CERT_URLS HTTP download as primary cert input channel. download_llm_ca_certs() added to certs.sh. Downloaded certs saved to llm/ subdirectory, install_all_certs extended to create hash symlinks for both custom/ and llm/ dirs. CERTS_PATH volume mount kept as secondary channel. Integration test added for HTTP download flow.
0.5.2	Fixed capath for ALL HTTP clients: ssl.create_default_context() (no capath, loads cafile+capath) replaced with system_ssl_context(capath="/etc/ssl/certs") (capath only) in client_registry.py, async_network.py, notifications/providers.py, git/_base.py. Previous fix (0.5.1) only covered LLM clients; Superset API, notification, and git clients still used ssl.create_default_context() which fails intermediate CA chain-building in OpenSSL 3.x (Finding 7). Production diagnosis: RUSAL cert chain Root→Policy CA→RGM Issuing→Server requires Policy CA as intermediate; without capath, OpenSSL ignores it in cafile.

[/DEF:ADR-0009:ADR]