diff --git a/docs/adr/ADR-0009-ssl-certificate-management.md b/docs/adr/ADR-0009-ssl-certificate-management.md index 3109e154..cd1c0db8 100644 --- a/docs/adr/ADR-0009-ssl-certificate-management.md +++ b/docs/adr/ADR-0009-ssl-certificate-management.md @@ -7,6 +7,7 @@ # @RELATION CALLS -> [backend/src/plugins/llm_analysis/service.py] # @RELATION CALLS -> [backend/src/plugins/translate/_llm_http.py] # @RELATION CALLS -> [backend/src/plugins/translate/preview_llm_client.py] +# @RELATION CALLS -> [scripts/check_llm_certs.py] # @RATIONALE ss-tools operates in corporate environments with internal Certificate Authorities. # Three separate HTTP stacks are used: httpx (AsyncOpenAI in llm_analysis plugin), # requests (sync translate plugin), and Playwright Chromium (dashboard screenshots). @@ -23,32 +24,71 @@ Corporate SSL certificates installed via `update-ca-certificates` into `/etc/ssl 1. **Python `requests` library** — uses bundled `certifi` CA bundle, not system store 2. **Python `httpx` library** — uses `certifi` by default when `verify=True` 3. **Playwright Chromium** — uses NSS Shared DB (`~/.pki/nssdb`), not OpenSSL store +4. **`openssl s_client`** — hangs without stdin (needs `echo \|` or `input=""`) This causes `SSLError: CERTIFICATE_VERIFY_FAILED` and `ERR_CERT_AUTHORITY_INVALID` even after correct system-wide CA installation. +## Discovery Journey + +### Phase 1: certifi vs system CA (0.1.5) +- `requests` and `httpx` use `certifi` bundle, not `/etc/ssl/certs/ca-certificates.crt` +- Fix: return system CA path instead of `True` in all `_get_verify()` functions +- Implemented in `service.py`, `_llm_http.py`, `preview_llm_client.py` + +### Phase 2: .pem vs .crt extension (0.1.6) +- `update-ca-certificates` only processes `.crt` files, silently ignores `.pem` +- Downloaded certificates from PKI were saved as `.pem`, skipped by update-ca-certificates +- Result: `ca-certificates.crt` was empty (0 certs), hash symlinks existed but bundle was broken +- Fix: save downloaded certs with `.crt` extension, not `.pem` + +### Phase 3: cafile vs capath (0.1.7) — KEY DISCOVERY +- `openssl s_client -CAfile /etc/ssl/certs/ca-certificates.crt` → code 20 (intermediate CA ignored) +- `openssl s_client -CApath /etc/ssl/certs/` → code 0 (chain built correctly) +- `ssl.create_default_context(cafile=...)` → SSL error +- `ssl.create_default_context(capath=...)` → HTTP 200 +- `requests.get(verify="/etc/ssl/certs/ca-certificates.crt")` → SSLError +- `requests.get(verify="/etc/ssl/certs/")` → HTTP 200 + +**Root cause**: OpenSSL 3.x treats non-self-signed certificates in `-CAfile` as trust anchors, +NOT as intermediates. `-CApath` (directory with hash symlinks) correctly builds chains using +all certificates found. The intermediate CA certs (Policy CA, RGM Issuing CA) are not +self-signed, so they are ignored in `-CAfile` but correctly used in `-CApath`. + +Diagnostic matrix (verified on production server 2026-05-28): + +| Method | cafile | capath | +|--------|--------|--------| +| openssl s_client | code 20 FAIL | code 0 OK | +| httpx (SSLContext) | SSLError FAIL | HTTP 200 OK | +| requests (verify=) | SSLError FAIL | HTTP 200 OK | + ## Solution ### Layer 1: System CA Store (OpenSSL) -- Entrypoint `install_certificates()` copies `.crt/.pem` files from `CERTS_PATH` volume mount +- Entrypoint `install_certificates()` copies `.crt` files from `CERTS_PATH` volume mount - Entrypoint `install_llm_ca_certs()` downloads PEM/DER certificates from `LLM_CA_CERT_URLS` - DER is auto-converted to PEM (`openssl x509 -inform DER -outform PEM`) +- Certificates are saved with **`.crt` extension** (`.pem` is silently ignored by `update-ca-certificates`) - `update-ca-certificates --fresh` adds them to `/etc/ssl/certs/ca-certificates.crt` -- **Fallback**: if `update-ca-certificates` misses certs (Debian Bookworm bug with subdirectories), they are appended to `ca-certificates.crt` with SHA256 fingerprint dedup +- **Fallback**: if `update-ca-certificates` misses certs, they are appended to `ca-certificates.crt` with SHA256 fingerprint dedup +- Hash symlinks created with collision support: `.0`, `.1`, `.2` suffixes ### Layer 2: Python HTTP Clients -| Library | File | Mechanism | -|---------|------|-----------| -| `httpx` (AsyncOpenAI) | `service.py:LLMClient._get_ssl_verify()` | Returns `ssl.create_default_context(cafile="/etc/ssl/certs/ca-certificates.crt")` | -| `requests` (`_llm_http.py`) | `_get_verify()` | Returns `"/etc/ssl/certs/ca-certificates.crt"` (string path) | -| `requests` (`preview_llm_client.py`) | `_get_verify()` | Returns `"/etc/ssl/certs/ca-certificates.crt"` (string path) | +| Library | File | Mechanism | Status | +|---------|------|-----------|--------| +| `httpx` (AsyncOpenAI) | `service.py:LLMClient._get_ssl_verify()` | `ssl.create_default_context(capath="/etc/ssl/certs/")` | ✅ Works (0.1.7) | +| `requests` (`_llm_http.py`) | `_get_verify()` | `"/etc/ssl/certs/"` (string path to dir) | ✅ Works (0.1.7) | +| `requests` (`preview_llm_client.py`) | `_get_verify()` | `"/etc/ssl/certs/"` (string path to dir) | ✅ Works (0.1.7) | -Key insight: `verify=True` uses certifi, NOT system CA. All three functions now return the system CA path instead of boolean `True`. +Key insight: `verify=True` uses certifi, NOT system CA. `verify=` ignores +intermediate CA in OpenSSL 3.x. `verify=` works correctly. ### Layer 3: NSS Database (Playwright Chromium) - Entrypoint `install_ca_to_nss()` imports PEM certs into `~/.pki/nssdb/` using `certutil` +- NSS DB path format: `sql:$HOME/.pki/nssdb` (SQLite prefix required, DBM not supported by Chromium) - Nickname format: `{dir_prefix}-{filename}` (e.g., `llm-UC_RUSAL_Policy_CA`) - Dedup by SHA256 fingerprint (not nickname), preventing duplicate imports - Trust attributes: `"C,,"` (trusted CA for TLS server certs) @@ -58,8 +98,9 @@ Key insight: `verify=True` uses certifi, NOT system CA. All three functions now - Env var `LLM_SSL_VERIFY=false` disables SSL verification entirely - Accepted values for false: `false`, `0`, `no`, `off` (case-insensitive) -- Default: enabled (returns system CA path / SSLContext) +- Default: enabled (returns capath-based SSLContext or path) - Used by all three HTTP client implementations +- **WARNING**: `verify=False` is for DIAGNOSTIC USE ONLY. Never leave in production. ## Key Files @@ -67,13 +108,59 @@ Key insight: `verify=True` uses certifi, NOT system CA. All three functions now |------|------| | `docker/backend.Dockerfile` | Installs `libnss3-tools` (certutil), Playwright Chromium | | `docker/backend.entrypoint.sh` | `install_certificates`, `install_llm_ca_certs`, `install_ca_to_nss` | -| `plugins/llm_analysis/service.py` | `LLMClient._get_ssl_verify()` → `ssl.create_default_context(cafile=...)` | -| `plugins/translate/_llm_http.py` | `_get_verify()` → `"/etc/ssl/certs/ca-certificates.crt"` | -| `plugins/translate/preview_llm_client.py` | `_get_verify()` → `"/etc/ssl/certs/ca-certificates.crt"` | +| `plugins/llm_analysis/service.py` | `LLMClient._get_ssl_verify()` → `ssl.create_default_context(capath=...)` | +| `plugins/translate/_llm_http.py` | `_get_verify()` → `"/etc/ssl/certs/"` | +| `plugins/translate/preview_llm_client.py` | `_get_verify()` → `"/etc/ssl/certs/"` | +| `scripts/check_llm_certs.py` | Full diagnostic: openssl, httpx, requests, NSS | | `docker-compose.yml` | Passes `LLM_SSL_VERIFY`, `LLM_CA_CERT_URLS` | | `docker-compose.enterprise-clean.yml` | Same as above | | `.env.enterprise-clean` | Default values for both env vars | +## How to Test Certificate Installation + +### On the server, after container restart: + +```bash +docker cp scripts/check_llm_certs.py ss-tools-backend-1:/tmp/ + +docker compose -f docker-compose.enterprise-clean.yml \ + --env-file .env.enterprise-clean exec backend \ + python3 /tmp/check_llm_certs.py --target https://lite.ai.rusal.com +``` + +Expected diagnostic matrix: + +``` +openssl_default: ✅ code 0 +openssl_capath: ✅ code 0 +openssl_cafile: ❌ code 20 (expected — OpenSSL 3.x limitation) +httpx_capath: ✅ HTTP 200 +httpx_cafile: ❌ SSL error (expected) +requests_capath: ✅ HTTP 200 +requests_cafile: ❌ SSL error (expected) +``` + +## Manually testing chain validity + +```bash +# 1. Check individual certs +openssl x509 -in /usr/local/share/ca-certificates/custom/RUSAL_ROOT.crt -noout -subject -issuer + +# 2. Verify full chain +openssl verify -CAfile /usr/local/share/ca-certificates/custom/RUSAL_ROOT.crt \ + -untrusted /usr/local/share/ca-certificates/llm/UC_RUSAL_Policy_CA.crt \ + /usr/local/share/ca-certificates/llm/UC_RUSAL_RGM_Issuing_CA.crt +# → OK + +# 3. Connect with capath +echo | openssl s_client -connect lite.ai.rusal.com:443 \ + -servername lite.ai.rusal.com \ + -CApath /etc/ssl/certs/ + +# 4. Verify certificate count +grep -c "BEGIN CERTIFICATE" /etc/ssl/certs/ca-certificates.crt +``` + ## Discovered Findings ### Finding 1: certifi vs system CA @@ -82,30 +169,75 @@ Key insight: `verify=True` uses certifi, NOT system CA. All three functions now ### Finding 2: Three separate HTTP stacks ss-tools has three LLM HTTP clients (httpx for async, requests for sync, Playwright for screenshots), each with independent CA trust configuration. -### Finding 3: update-ca-certificates inconsistency -On Debian Bookworm (python:3.11-slim), `update-ca-certificates` occasionally skips symlinks for PEM files in subdirectories (`/usr/local/share/ca-certificates/custom/`, `llm/`), requiring a manual `cat >> ca-certificates.crt` fallback and explicit hash symlink creation. +### Finding 3: .pem extension ignored by update-ca-certificates +On Debian Bookworm (python:3.11-slim), `update-ca-certificates` only processes `.crt` files. +Files with `.pem` extension in `/usr/local/share/ca-certificates/` are silently skipped. +This left `ca-certificates.crt` empty (0 certs) while hash symlinks existed. +Fix: save downloaded certificates with `.crt` extension. ### Finding 4: DER format from corporate PKI -Corporate PKI servers (`pki.rusal.com`) often serve certificates in DER (binary) format. `update-ca-certificates` requires PEM. Auto-detection and conversion (`openssl x509 -inform DER -outform PEM`) is essential. +Corporate PKI servers (`pki.rusal.com`) often serve certificates in DER (binary) format. +`update-ca-certificates` requires PEM. Auto-detection and conversion +(`openssl x509 -inform DER -outform PEM`) is essential. ### Finding 5: Chicken-and-egg TLS bootstrap -Downloading a CA certificate from a PKI server that uses the same CA causes a TLS verification loop. Entrypoint uses `curl --insecure` as fallback when initial `curl` fails with TLS error. +Downloading a CA certificate from a PKI server that uses the same CA causes a TLS +verification loop. Entrypoint uses `curl --insecure` as fallback when initial `curl` +fails with TLS error. ### Finding 6: NSS DB format -Chromium uses NSS Shared DB in `~/.pki/nssdb/`. The `sql:` prefix is required for `certutil` to use the SQLite format. Without it, certutil defaults to the legacy DBM format which Chromium may not read. +Chromium uses NSS Shared DB in `~/.pki/nssdb/`. The `sql:` prefix is required for +`certutil` to use the SQLite format. Without it, certutil defaults to the legacy +DBM format which Chromium may not read. + +### Finding 7: OpenSSL 3.x cafile vs capath (CRITICAL) +OpenSSL 3.x treats non-self-signed certificates in `-CAfile` as trust anchors, +NOT as intermediates. This means intermediate CA certificates (Policy CA, RGM Issuing CA) +are ignored when using `-CAfile /etc/ssl/certs/ca-certificates.crt`, producing +verify code 20. Using `-CApath /etc/ssl/certs/` (directory with hash symlinks) +correctly builds the full chain, producing verify code 0. + +This affects ALL Python libraries: +- `ssl.create_default_context(cafile=...)` → calls OpenSSL's `SSL_CTX_load_verify_locations` + with the file, which exhibits the same limitation +- `ssl.create_default_context(capath=...)` → works correctly +- `requests.get(verify="/path/to/file")` → fails (uses cafile internally) +- `requests.get(verify="/path/to/dir/")` → works (uses capath internally) + +### Finding 8: openssl s_client hangs without stdin +`openssl s_client` waits for input after TLS handshake. When run via +`subprocess.run(cmd, capture_output=True)`, it hangs until timeout. +Fix: pass `input=""` or `echo | openssl s_client ...`. ## Deploy ```bash # On target server: -xz -dc ss-tools-backend.0.1.6.tar.xz | docker load -xz -dc ss-tools-frontend.0.1.6.tar.xz | docker load +xz -dc ss-tools-backend.0.1.7.tar.xz | docker load +xz -dc ss-tools-frontend.0.1.7.tar.xz | docker load + +# Enable capath-based verification (remove LLM_SSL_VERIFY=false) +sed -i '/LLM_SSL_VERIFY=false/d' .env.enterprise-clean docker compose -f docker-compose.enterprise-clean.yml \ --env-file .env.enterprise-clean down docker compose -f docker-compose.enterprise-clean.yml \ --env-file .env.enterprise-clean up -d + +# Verify +docker cp scripts/check_llm_certs.py ss-tools-backend-1:/tmp/ +docker compose -f docker-compose.enterprise-clean.yml \ + --env-file .env.enterprise-clean exec backend \ + python3 /tmp/check_llm_certs.py --target https://lite.ai.rusal.com ``` +## Version History + +| Version | Changes | +|---------|---------| +| 0.1.5 | Initial SSL support: `LLM_SSL_VERIFY`, `_format_connection_error()`, CA download via URL, NSS import | +| 0.1.6 | QA fixes: fingerprint dedup, hash symlink collision, NSS collision, DER→PEM error handling, chicken-and-egg TLS, nullglob, translate plugin SSL | +| 0.1.7 | **capath instead of cafile**: OpenSSL 3.x cafile limitation discovered and fixed. `.crt` extension for downloaded certs. Diagnostic script rewritten. | + # [/DEF:ADR-0009:ADR]