ss-tools/docs/adr/ADR-0009-ssl-certificate-management.md

[DEF:ADR-0009:ADR]

@STATUS ACTIVE

@PURPOSE Define the strategy for corporate SSL certificate management across all LLM HTTP clients in ss-tools (httpx, requests, openai), covering system CA store, NSS database for Chromium/Playwright, and the LLM_SSL_VERIFY escape hatch.

@RELATION DEPENDS_ON -> [ADR-0001:ADR]

@RELATION DEPENDS_ON -> [ADR-0004:ADR]

@RELATION CALLS -> [docker/backend.entrypoint.sh]

@RELATION CALLS -> [backend/src/plugins/llm_analysis/service.py]

@RELATION CALLS -> [backend/src/plugins/translate/_llm_http.py]

@RELATION CALLS -> [backend/src/plugins/translate/preview_llm_client.py]

@RELATION CALLS -> [scripts/check_llm_certs.py]

@RATIONALE ss-tools operates in corporate environments with internal Certificate Authorities.

Three separate HTTP stacks are used: httpx (AsyncOpenAI in llm_analysis plugin),

requests (sync translate plugin), and Playwright Chromium (dashboard screenshots).

Each has its own CA trust store, requiring different installation strategies.

@REJECTED Centralizing all LLM calls into a single HTTP client — rejected because

llm_analysis uses async httpx+AsyncOpenAI for streaming and multiple concurrent calls,

while translate uses sync requests for simpler request/response. Merging would

require rewriting one or both, creating regression risk with no business value.

Problem

Corporate SSL certificates installed via update-ca-certificates into /etc/ssl/certs/ca-certificates.crt are NOT automatically trusted by:

1. Python requests library — uses bundled certifi CA bundle, not system store
2. Python httpx library — uses certifi by default when verify=True
3. Playwright Chromium — uses NSS Shared DB (~/.pki/nssdb), not OpenSSL store
4. openssl s_client — hangs without stdin (needs echo \| or input="")

This causes SSLError: CERTIFICATE_VERIFY_FAILED and ERR_CERT_AUTHORITY_INVALID even after correct system-wide CA installation.

Discovery Journey

Phase 1: certifi vs system CA (0.1.5)

• requests and httpx use certifi bundle, not /etc/ssl/certs/ca-certificates.crt
• Fix: return system CA path instead of True in all _get_verify() functions
• Implemented in service.py, _llm_http.py, preview_llm_client.py

Phase 2: .pem vs .crt extension (0.1.6)

• update-ca-certificates only processes .crt files, silently ignores .pem
• Downloaded certificates from PKI were saved as .pem, skipped by update-ca-certificates
• Result: ca-certificates.crt was empty (0 certs), hash symlinks existed but bundle was broken
• Fix: save downloaded certs with .crt extension, not .pem

Phase 3: cafile vs capath (0.1.7) — KEY DISCOVERY

• openssl s_client -CAfile /etc/ssl/certs/ca-certificates.crt → code 20 (intermediate CA ignored)
• openssl s_client -CApath /etc/ssl/certs/ → code 0 (chain built correctly)
• ssl.create_default_context(cafile=...) → SSL error
• ssl.create_default_context(capath=...) → HTTP 200
• requests.get(verify="/etc/ssl/certs/ca-certificates.crt") → SSLError
• requests.get(verify="/etc/ssl/certs/") → HTTP 200

Root cause: OpenSSL 3.x treats non-self-signed certificates in -CAfile as trust anchors, NOT as intermediates. -CApath (directory with hash symlinks) correctly builds chains using all certificates found. The intermediate CA certs (Policy CA, RGM Issuing CA) are not self-signed, so they are ignored in -CAfile but correctly used in -CApath.

Diagnostic matrix (verified on production server 2026-05-28):

Method	cafile	capath
openssl s_client	code 20 FAIL	code 0 OK
httpx (SSLContext)	SSLError FAIL	HTTP 200 OK
requests (verify=)	SSLError FAIL	HTTP 200 OK

Solution

Layer 1: System CA Store (OpenSSL)

• Entrypoint install_certificates() copies .crt files from CERTS_PATH volume mount
• Entrypoint install_llm_ca_certs() downloads PEM/DER certificates from LLM_CA_CERT_URLS
• DER is auto-converted to PEM (openssl x509 -inform DER -outform PEM)
• Certificates are saved with .crt extension (.pem is silently ignored by update-ca-certificates)
• update-ca-certificates --fresh adds them to /etc/ssl/certs/ca-certificates.crt
• Fallback: if update-ca-certificates misses certs, they are appended to ca-certificates.crt with SHA256 fingerprint dedup
• Hash symlinks created with collision support: .0, .1, .2 suffixes

Layer 2: Python HTTP Clients

Library	File	Mechanism	Status
httpx (AsyncOpenAI)	service.py:LLMClient._get_ssl_verify()	ssl.create_default_context(capath="/etc/ssl/certs/")	✅ Works (0.1.7)
requests (_llm_http.py)	_get_verify()	"/etc/ssl/certs/" (string path to dir)	✅ Works (0.1.7)
requests (preview_llm_client.py)	_get_verify()	"/etc/ssl/certs/" (string path to dir)	✅ Works (0.1.7)

Key insight: verify=True uses certifi, NOT system CA. verify=<cafile_path> ignores intermediate CA in OpenSSL 3.x. verify=<capath_dir> works correctly.

Layer 3: NSS Database (Playwright Chromium)

• Entrypoint install_ca_to_nss() imports PEM certs into ~/.pki/nssdb/ using certutil
• NSS DB path format: sql:$HOME/.pki/nssdb (SQLite prefix required, DBM not supported by Chromium)
• Nickname format: {dir_prefix}-{filename} (e.g., llm-UC_RUSAL_Policy_CA)
• Dedup by SHA256 fingerprint (not nickname), preventing duplicate imports
• Trust attributes: "C,," (trusted CA for TLS server certs)
• Fixes ERR_CERT_AUTHORITY_INVALID in Playwright dashboard screenshots

Layer 4: LLM_SSL_VERIFY Escape Hatch

• Env var LLM_SSL_VERIFY=false disables SSL verification entirely
• Accepted values for false: false, 0, no, off (case-insensitive)
• Default: enabled (returns capath-based SSLContext or path)
• Used by all three HTTP client implementations
• WARNING: verify=False is for DIAGNOSTIC USE ONLY. Never leave in production.

Key Files

File	Role
docker/backend.Dockerfile	Installs libnss3-tools (certutil), Playwright Chromium
docker/backend.entrypoint.sh	install_certificates, install_llm_ca_certs, install_ca_to_nss
plugins/llm_analysis/service.py	LLMClient._get_ssl_verify() → ssl.create_default_context(capath=...)
plugins/translate/_llm_http.py	_get_verify() → "/etc/ssl/certs/"
plugins/translate/preview_llm_client.py	_get_verify() → "/etc/ssl/certs/"
scripts/check_llm_certs.py	Full diagnostic: openssl, httpx, requests, NSS
docker-compose.yml	Passes LLM_SSL_VERIFY, LLM_CA_CERT_URLS
docker-compose.enterprise-clean.yml	Same as above
.env.enterprise-clean	Default values for both env vars

How to Test Certificate Installation

On the server, after container restart:

docker cp scripts/check_llm_certs.py ss-tools-backend-1:/tmp/

docker compose -f docker-compose.enterprise-clean.yml \
  --env-file .env.enterprise-clean exec backend \
  python3 /tmp/check_llm_certs.py --target https://lite.ai.rusal.com

Expected diagnostic matrix:

openssl_default: ✅ code 0
openssl_capath:  ✅ code 0
openssl_cafile:  ❌ code 20 (expected — OpenSSL 3.x limitation)
httpx_capath:    ✅ HTTP 200
httpx_cafile:    ❌ SSL error (expected)
requests_capath: ✅ HTTP 200
requests_cafile: ❌ SSL error (expected)

Manually testing chain validity

# 1. Check individual certs
openssl x509 -in /usr/local/share/ca-certificates/custom/RUSAL_ROOT.crt -noout -subject -issuer

# 2. Verify full chain
openssl verify -CAfile /usr/local/share/ca-certificates/custom/RUSAL_ROOT.crt \
  -untrusted /usr/local/share/ca-certificates/llm/UC_RUSAL_Policy_CA.crt \
  /usr/local/share/ca-certificates/llm/UC_RUSAL_RGM_Issuing_CA.crt
# → OK

# 3. Connect with capath
echo | openssl s_client -connect lite.ai.rusal.com:443 \
  -servername lite.ai.rusal.com \
  -CApath /etc/ssl/certs/

# 4. Verify certificate count
grep -c "BEGIN CERTIFICATE" /etc/ssl/certs/ca-certificates.crt

Discovered Findings

Finding 1: certifi vs system CA

requests and httpx use certifi CA bundle by default. Corporate CA certs installed into /etc/ssl/certs/ are invisible to Python HTTP clients unless explicitly pointed to the system bundle.

Finding 2: Three separate HTTP stacks

ss-tools has three LLM HTTP clients (httpx for async, requests for sync, Playwright for screenshots), each with independent CA trust configuration.

Finding 3: .pem extension ignored by update-ca-certificates

On Debian Bookworm (python:3.11-slim), update-ca-certificates only processes .crt files. Files with .pem extension in /usr/local/share/ca-certificates/ are silently skipped. This left ca-certificates.crt empty (0 certs) while hash symlinks existed. Fix: save downloaded certificates with .crt extension.

Finding 4: DER format from corporate PKI

Corporate PKI servers (pki.rusal.com) often serve certificates in DER (binary) format. update-ca-certificates requires PEM. Auto-detection and conversion (openssl x509 -inform DER -outform PEM) is essential.

Finding 5: Chicken-and-egg TLS bootstrap

Downloading a CA certificate from a PKI server that uses the same CA causes a TLS verification loop. Entrypoint uses curl --insecure as fallback when initial curl fails with TLS error.

Finding 6: NSS DB format

Chromium uses NSS Shared DB in ~/.pki/nssdb/. The sql: prefix is required for certutil to use the SQLite format. Without it, certutil defaults to the legacy DBM format which Chromium may not read.

Finding 7: OpenSSL 3.x cafile vs capath (CRITICAL)

OpenSSL 3.x treats non-self-signed certificates in -CAfile as trust anchors, NOT as intermediates. This means intermediate CA certificates (Policy CA, RGM Issuing CA) are ignored when using -CAfile /etc/ssl/certs/ca-certificates.crt, producing verify code 20. Using -CApath /etc/ssl/certs/ (directory with hash symlinks) correctly builds the full chain, producing verify code 0.

This affects ALL Python libraries:

• ssl.create_default_context(cafile=...) → calls OpenSSL's SSL_CTX_load_verify_locations with the file, which exhibits the same limitation
• ssl.create_default_context(capath=...) → works correctly
• requests.get(verify="/path/to/file") → fails (uses cafile internally)
• requests.get(verify="/path/to/dir/") → works (uses capath internally)

Finding 8: openssl s_client hangs without stdin

openssl s_client waits for input after TLS handshake. When run via subprocess.run(cmd, capture_output=True), it hangs until timeout. Fix: pass input="" or echo | openssl s_client ....

Deploy

# On target server:
xz -dc ss-tools-backend.0.1.7.tar.xz | docker load
xz -dc ss-tools-frontend.0.1.7.tar.xz | docker load

# Enable capath-based verification (remove LLM_SSL_VERIFY=false)
sed -i '/LLM_SSL_VERIFY=false/d' .env.enterprise-clean

docker compose -f docker-compose.enterprise-clean.yml \
  --env-file .env.enterprise-clean down

docker compose -f docker-compose.enterprise-clean.yml \
  --env-file .env.enterprise-clean up -d

# Verify
docker cp scripts/check_llm_certs.py ss-tools-backend-1:/tmp/
docker compose -f docker-compose.enterprise-clean.yml \
  --env-file .env.enterprise-clean exec backend \
  python3 /tmp/check_llm_certs.py --target https://lite.ai.rusal.com

Version History

Version	Changes
0.1.5	Initial SSL support: LLM_SSL_VERIFY, _format_connection_error(), CA download via URL, NSS import
0.1.6	QA fixes: fingerprint dedup, hash symlink collision, NSS collision, DER→PEM error handling, chicken-and-egg TLS, nullglob, translate plugin SSL
0.1.7	capath instead of cafile: OpenSSL 3.x cafile limitation discovered and fixed. .crt extension for downloaded certs. Diagnostic script rewritten.

[/DEF:ADR-0009:ADR]