busya/ss-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update ADR-0009 with complete discovery journey

Browse Source 
Added Phase 1-3 discovery timeline (certifi → .pem→.crt → cafile→capath),
OpenSSL 3.x cafile limitation analysis, diagnostic matrix,
test commands, version history. All 8 findings documented.
This commit is contained in:
busya
2026-05-29 14:30:33 +03:00
parent d294e0295b
commit 745c9e7ef2
1 changed files with 151 additions and 19 deletions
Download Patch File Download Diff File Expand all files Collapse all files

170
docs/adr/ADR-0009-ssl-certificate-management.md
View File

@@ -7,6 +7,7 @@
# @RELATION CALLS -> [backend/src/plugins/llm_analysis/service.py]
# @RELATION CALLS -> [backend/src/plugins/translate/_llm_http.py]
# @RELATION CALLS -> [backend/src/plugins/translate/preview_llm_client.py]
# @RELATION CALLS -> [scripts/check_llm_certs.py]
# @RATIONALE ss-tools operates in corporate environments with internal Certificate Authorities.
# Three separate HTTP stacks are used: httpx (AsyncOpenAI in llm_analysis plugin),
# requests (sync translate plugin), and Playwright Chromium (dashboard screenshots).
@@ -23,32 +24,71 @@ Corporate SSL certificates installed via `update-ca-certificates` into `/etc/ssl
1. **Python `requests` library** — uses bundled `certifi` CA bundle, not system store
2. **Python `httpx` library** — uses `certifi` by default when `verify=True`
3. **Playwright Chromium** — uses NSS Shared DB (`~/.pki/nssdb`), not OpenSSL store
4. **`openssl s_client`** — hangs without stdin (needs `echo \|` or `input=""`)
This causes `SSLError: CERTIFICATE_VERIFY_FAILED` and `ERR_CERT_AUTHORITY_INVALID` even after correct system-wide CA installation.
## Discovery Journey
### Phase 1: certifi vs system CA (0.1.5)
- `requests` and `httpx` use `certifi` bundle, not `/etc/ssl/certs/ca-certificates.crt`
- Fix: return system CA path instead of `True` in all `_get_verify()` functions
- Implemented in `service.py`, `_llm_http.py`, `preview_llm_client.py`
### Phase 2: .pem vs .crt extension (0.1.6)
- `update-ca-certificates` only processes `.crt` files, silently ignores `.pem`
- Downloaded certificates from PKI were saved as `.pem`, skipped by update-ca-certificates
- Result: `ca-certificates.crt` was empty (0 certs), hash symlinks existed but bundle was broken
- Fix: save downloaded certs with `.crt` extension, not `.pem`
### Phase 3: cafile vs capath (0.1.7) — KEY DISCOVERY
- `openssl s_client -CAfile /etc/ssl/certs/ca-certificates.crt` → code 20 (intermediate CA ignored)
- `openssl s_client -CApath /etc/ssl/certs/` → code 0 (chain built correctly)
- `ssl.create_default_context(cafile=...)` → SSL error
- `ssl.create_default_context(capath=...)` → HTTP 200
- `requests.get(verify="/etc/ssl/certs/ca-certificates.crt")` → SSLError
- `requests.get(verify="/etc/ssl/certs/")` → HTTP 200
**Root cause**: OpenSSL 3.x treats non-self-signed certificates in `-CAfile` as trust anchors,
NOT as intermediates. `-CApath` (directory with hash symlinks) correctly builds chains using
all certificates found. The intermediate CA certs (Policy CA, RGM Issuing CA) are not
self-signed, so they are ignored in `-CAfile` but correctly used in `-CApath`.
Diagnostic matrix (verified on production server 2026-05-28):
| Method | cafile | capath |
|--------|--------|--------|
| openssl s_client | code 20 FAIL | code 0 OK |
| httpx (SSLContext) | SSLError FAIL | HTTP 200 OK |
| requests (verify=) | SSLError FAIL | HTTP 200 OK |
## Solution
### Layer 1: System CA Store (OpenSSL)
- Entrypoint `install_certificates()` copies `.crt/.pem` files from `CERTS_PATH` volume mount
- Entrypoint `install_certificates()` copies `.crt` files from `CERTS_PATH` volume mount
- Entrypoint `install_llm_ca_certs()` downloads PEM/DER certificates from `LLM_CA_CERT_URLS`
- DER is auto-converted to PEM (`openssl x509 -inform DER -outform PEM`)
- Certificates are saved with **`.crt` extension** (`.pem` is silently ignored by `update-ca-certificates`)
- `update-ca-certificates --fresh` adds them to `/etc/ssl/certs/ca-certificates.crt`
- **Fallback**: if `update-ca-certificates` misses certs (Debian Bookworm bug with subdirectories), they are appended to `ca-certificates.crt` with SHA256 fingerprint dedup
- **Fallback**: if `update-ca-certificates` misses certs, they are appended to `ca-certificates.crt` with SHA256 fingerprint dedup
- Hash symlinks created with collision support: `.0`, `.1`, `.2` suffixes
### Layer 2: Python HTTP Clients
| Library | File | Mechanism |
|---------|------|-----------|
| `httpx` (AsyncOpenAI) | `service.py:LLMClient._get_ssl_verify()` | Returns `ssl.create_default_context(cafile="/etc/ssl/certs/ca-certificates.crt")` |
| `requests` (`_llm_http.py`) | `_get_verify()` | Returns `"/etc/ssl/certs/ca-certificates.crt"` (string path) |
| `requests` (`preview_llm_client.py`) | `_get_verify()` | Returns `"/etc/ssl/certs/ca-certificates.crt"` (string path) |
| Library | File | Mechanism | Status |
|---------|------|-----------|--------|
| `httpx` (AsyncOpenAI) | `service.py:LLMClient._get_ssl_verify()` | `ssl.create_default_context(capath="/etc/ssl/certs/")` | ✅ Works (0.1.7) |
| `requests` (`_llm_http.py`) | `_get_verify()` | `"/etc/ssl/certs/"` (string path to dir) | ✅ Works (0.1.7) |
| `requests` (`preview_llm_client.py`) | `_get_verify()` | `"/etc/ssl/certs/"` (string path to dir) | ✅ Works (0.1.7) |
Key insight: `verify=True` uses certifi, NOT system CA. All three functions now return the system CA path instead of boolean `True`.
Key insight: `verify=True` uses certifi, NOT system CA. `verify=<cafile_path>` ignores
intermediate CA in OpenSSL 3.x. `verify=<capath_dir>` works correctly.
### Layer 3: NSS Database (Playwright Chromium)
- Entrypoint `install_ca_to_nss()` imports PEM certs into `~/.pki/nssdb/` using `certutil`
- NSS DB path format: `sql:$HOME/.pki/nssdb` (SQLite prefix required, DBM not supported by Chromium)
- Nickname format: `{dir_prefix}-{filename}` (e.g., `llm-UC_RUSAL_Policy_CA`)
- Dedup by SHA256 fingerprint (not nickname), preventing duplicate imports
- Trust attributes: `"C,,"` (trusted CA for TLS server certs)
@@ -58,8 +98,9 @@ Key insight: `verify=True` uses certifi, NOT system CA. All three functions now
- Env var `LLM_SSL_VERIFY=false` disables SSL verification entirely
- Accepted values for false: `false`, `0`, `no`, `off` (case-insensitive)
- Default: enabled (returns system CA path / SSLContext)
- Default: enabled (returns capath-based SSLContext or path)
- Used by all three HTTP client implementations
- **WARNING**: `verify=False` is for DIAGNOSTIC USE ONLY. Never leave in production.
## Key Files
@@ -67,13 +108,59 @@ Key insight: `verify=True` uses certifi, NOT system CA. All three functions now
|------|------|
| `docker/backend.Dockerfile` | Installs `libnss3-tools` (certutil), Playwright Chromium |
| `docker/backend.entrypoint.sh` | `install_certificates`, `install_llm_ca_certs`, `install_ca_to_nss` |
| `plugins/llm_analysis/service.py` | `LLMClient._get_ssl_verify()` → `ssl.create_default_context(cafile=...)` |
| `plugins/translate/_llm_http.py` | `_get_verify()` → `"/etc/ssl/certs/ca-certificates.crt"` |
| `plugins/translate/preview_llm_client.py` | `_get_verify()` → `"/etc/ssl/certs/ca-certificates.crt"` |
| `plugins/llm_analysis/service.py` | `LLMClient._get_ssl_verify()` → `ssl.create_default_context(capath=...)` |
| `plugins/translate/_llm_http.py` | `_get_verify()` → `"/etc/ssl/certs/"` |
| `plugins/translate/preview_llm_client.py` | `_get_verify()` → `"/etc/ssl/certs/"` |
| `scripts/check_llm_certs.py` | Full diagnostic: openssl, httpx, requests, NSS |
| `docker-compose.yml` | Passes `LLM_SSL_VERIFY`, `LLM_CA_CERT_URLS` |
| `docker-compose.enterprise-clean.yml` | Same as above |
| `.env.enterprise-clean` | Default values for both env vars |
## How to Test Certificate Installation
### On the server, after container restart:
```bash
docker cp scripts/check_llm_certs.py ss-tools-backend-1:/tmp/
docker compose -f docker-compose.enterprise-clean.yml \
--env-file .env.enterprise-clean exec backend \
python3 /tmp/check_llm_certs.py --target https://lite.ai.rusal.com
```
Expected diagnostic matrix:
```
openssl_default: ✅ code 0
openssl_capath: ✅ code 0
openssl_cafile: ❌ code 20 (expected — OpenSSL 3.x limitation)
httpx_capath: ✅ HTTP 200
httpx_cafile: ❌ SSL error (expected)
requests_capath: ✅ HTTP 200
requests_cafile: ❌ SSL error (expected)
```
## Manually testing chain validity
```bash
# 1. Check individual certs
openssl x509 -in /usr/local/share/ca-certificates/custom/RUSAL_ROOT.crt -noout -subject -issuer
# 2. Verify full chain
openssl verify -CAfile /usr/local/share/ca-certificates/custom/RUSAL_ROOT.crt \
-untrusted /usr/local/share/ca-certificates/llm/UC_RUSAL_Policy_CA.crt \
/usr/local/share/ca-certificates/llm/UC_RUSAL_RGM_Issuing_CA.crt
# → OK
# 3. Connect with capath
echo | openssl s_client -connect lite.ai.rusal.com:443 \
-servername lite.ai.rusal.com \
-CApath /etc/ssl/certs/
# 4. Verify certificate count
grep -c "BEGIN CERTIFICATE" /etc/ssl/certs/ca-certificates.crt
```
## Discovered Findings
### Finding 1: certifi vs system CA
@@ -82,30 +169,75 @@ Key insight: `verify=True` uses certifi, NOT system CA. All three functions now
### Finding 2: Three separate HTTP stacks
ss-tools has three LLM HTTP clients (httpx for async, requests for sync, Playwright for screenshots), each with independent CA trust configuration.
### Finding 3: update-ca-certificates inconsistency
On Debian Bookworm (python:3.11-slim), `update-ca-certificates` occasionally skips symlinks for PEM files in subdirectories (`/usr/local/share/ca-certificates/custom/`, `llm/`), requiring a manual `cat >> ca-certificates.crt` fallback and explicit hash symlink creation.
### Finding 3: .pem extension ignored by update-ca-certificates
On Debian Bookworm (python:3.11-slim), `update-ca-certificates` only processes `.crt` files.
Files with `.pem` extension in `/usr/local/share/ca-certificates/` are silently skipped.
This left `ca-certificates.crt` empty (0 certs) while hash symlinks existed.
Fix: save downloaded certificates with `.crt` extension.
### Finding 4: DER format from corporate PKI
Corporate PKI servers (`pki.rusal.com`) often serve certificates in DER (binary) format. `update-ca-certificates` requires PEM. Auto-detection and conversion (`openssl x509 -inform DER -outform PEM`) is essential.
Corporate PKI servers (`pki.rusal.com`) often serve certificates in DER (binary) format.
`update-ca-certificates` requires PEM. Auto-detection and conversion
(`openssl x509 -inform DER -outform PEM`) is essential.
### Finding 5: Chicken-and-egg TLS bootstrap
Downloading a CA certificate from a PKI server that uses the same CA causes a TLS verification loop. Entrypoint uses `curl --insecure` as fallback when initial `curl` fails with TLS error.
Downloading a CA certificate from a PKI server that uses the same CA causes a TLS
verification loop. Entrypoint uses `curl --insecure` as fallback when initial `curl`
fails with TLS error.
### Finding 6: NSS DB format
Chromium uses NSS Shared DB in `~/.pki/nssdb/`. The `sql:` prefix is required for `certutil` to use the SQLite format. Without it, certutil defaults to the legacy DBM format which Chromium may not read.
Chromium uses NSS Shared DB in `~/.pki/nssdb/`. The `sql:` prefix is required for
`certutil` to use the SQLite format. Without it, certutil defaults to the legacy
DBM format which Chromium may not read.
### Finding 7: OpenSSL 3.x cafile vs capath (CRITICAL)
OpenSSL 3.x treats non-self-signed certificates in `-CAfile` as trust anchors,
NOT as intermediates. This means intermediate CA certificates (Policy CA, RGM Issuing CA)
are ignored when using `-CAfile /etc/ssl/certs/ca-certificates.crt`, producing
verify code 20. Using `-CApath /etc/ssl/certs/` (directory with hash symlinks)
correctly builds the full chain, producing verify code 0.
This affects ALL Python libraries:
- `ssl.create_default_context(cafile=...)` → calls OpenSSL's `SSL_CTX_load_verify_locations`
with the file, which exhibits the same limitation
- `ssl.create_default_context(capath=...)` → works correctly
- `requests.get(verify="/path/to/file")` → fails (uses cafile internally)
- `requests.get(verify="/path/to/dir/")` → works (uses capath internally)
### Finding 8: openssl s_client hangs without stdin
`openssl s_client` waits for input after TLS handshake. When run via
`subprocess.run(cmd, capture_output=True)`, it hangs until timeout.
Fix: pass `input=""` or `echo | openssl s_client ...`.
## Deploy
```bash
# On target server:
xz -dc ss-tools-backend.0.1.6.tar.xz | docker load
xz -dc ss-tools-frontend.0.1.6.tar.xz | docker load
xz -dc ss-tools-backend.0.1.7.tar.xz | docker load
xz -dc ss-tools-frontend.0.1.7.tar.xz | docker load
# Enable capath-based verification (remove LLM_SSL_VERIFY=false)
sed -i '/LLM_SSL_VERIFY=false/d' .env.enterprise-clean
docker compose -f docker-compose.enterprise-clean.yml \
--env-file .env.enterprise-clean down
docker compose -f docker-compose.enterprise-clean.yml \
--env-file .env.enterprise-clean up -d
# Verify
docker cp scripts/check_llm_certs.py ss-tools-backend-1:/tmp/
docker compose -f docker-compose.enterprise-clean.yml \
--env-file .env.enterprise-clean exec backend \
python3 /tmp/check_llm_certs.py --target https://lite.ai.rusal.com
```
## Version History
| Version | Changes |
|---------|---------|
| 0.1.5 | Initial SSL support: `LLM_SSL_VERIFY`, `_format_connection_error()`, CA download via URL, NSS import |
| 0.1.6 | QA fixes: fingerprint dedup, hash symlink collision, NSS collision, DER→PEM error handling, chicken-and-egg TLS, nullglob, translate plugin SSL |
| 0.1.7 | **capath instead of cafile**: OpenSSL 3.x cafile limitation discovered and fixed. `.crt` extension for downloaded certs. Diagnostic script rewritten. |
# [/DEF:ADR-0009:ADR]