ss-tools/docs/adr/ADR-0009-ssl-certificate-management.md

# [DEF:ADR-0009:ADR]
# @STATUS ACTIVE
# @PURPOSE Define the strategy for corporate SSL certificate management across all LLM HTTP clients in ss-tools (httpx, requests, openai), covering system CA store, NSS database for Chromium/Playwright, and the LLM_SSL_VERIFY escape hatch.
# @RELATION DEPENDS_ON -> [ADR-0001:ADR]
# @RELATION DEPENDS_ON -> [ADR-0004:ADR]
# @RELATION CALLS -> [docker/backend.entrypoint.sh]
# @RELATION CALLS -> [backend/src/plugins/llm_analysis/service.py]
# @RELATION CALLS -> [backend/src/plugins/translate/_llm_http.py]
# @RELATION CALLS -> [backend/src/plugins/translate/preview_llm_client.py]
# @RELATION CALLS -> [scripts/check_llm_certs.py]
# @RATIONALE ss-tools operates in corporate environments with internal Certificate Authorities.
#   Three separate HTTP stacks are used: httpx (AsyncOpenAI in llm_analysis plugin),
#   requests (sync translate plugin), and Playwright Chromium (dashboard screenshots).
#   Each has its own CA trust store, requiring different installation strategies.
# @REJECTED Centralizing all LLM calls into a single HTTP client — rejected because
#   llm_analysis uses async httpx+AsyncOpenAI for streaming and multiple concurrent calls,
#   while translate uses sync requests for simpler request/response. Merging would
#   require rewriting one or both, creating regression risk with no business value.

## Problem

Corporate SSL certificates installed via `update-ca-certificates` into `/etc/ssl/certs/ca-certificates.crt` are NOT automatically trusted by:

1. **Python `requests` library** — uses bundled `certifi` CA bundle, not system store
2. **Python `httpx` library** — uses `certifi` by default when `verify=True`
3. **Playwright Chromium** — uses NSS Shared DB (`~/.pki/nssdb`), not OpenSSL store
4. **`openssl s_client`** — hangs without stdin (needs `echo \|` or `input=""`)

This causes `SSLError: CERTIFICATE_VERIFY_FAILED` and `ERR_CERT_AUTHORITY_INVALID` even after correct system-wide CA installation.

## Discovery Journey

### Phase 1: certifi vs system CA (0.1.5)
- `requests` and `httpx` use `certifi` bundle, not `/etc/ssl/certs/ca-certificates.crt`
- Fix: return system CA path instead of `True` in all `_get_verify()` functions
- Implemented in `service.py`, `_llm_http.py`, `preview_llm_client.py`

### Phase 2: .pem vs .crt extension (0.1.6)
- `update-ca-certificates` only processes `.crt` files, silently ignores `.pem`
- Downloaded certificates from PKI were saved as `.pem`, skipped by update-ca-certificates
- Result: `ca-certificates.crt` was empty (0 certs), hash symlinks existed but bundle was broken
- Fix: save downloaded certs with `.crt` extension, not `.pem`

### Phase 3: cafile vs capath (0.1.7) — KEY DISCOVERY
- `openssl s_client -CAfile /etc/ssl/certs/ca-certificates.crt` → code 20 (intermediate CA ignored)
- `openssl s_client -CApath /etc/ssl/certs/` → code 0 (chain built correctly)
- `ssl.create_default_context(cafile=...)` → SSL error
- `ssl.create_default_context(capath=...)` → HTTP 200
- `requests.get(verify="/etc/ssl/certs/ca-certificates.crt")` → SSLError
- `requests.get(verify="/etc/ssl/certs/")` → HTTP 200

**Root cause**: OpenSSL 3.x treats non-self-signed certificates in `-CAfile` as trust anchors,
NOT as intermediates. `-CApath` (directory with hash symlinks) correctly builds chains using
all certificates found. The intermediate CA certs (Policy CA, RGM Issuing CA) are not
self-signed, so they are ignored in `-CAfile` but correctly used in `-CApath`.

Diagnostic matrix (verified on production server 2026-05-28):

| Method | cafile | capath |
|--------|--------|--------|
| openssl s_client | code 20 FAIL | code 0 OK |
| httpx (SSLContext) | SSLError FAIL | HTTP 200 OK |
| requests (verify=) | SSLError FAIL | HTTP 200 OK |

## Solution

### Layer 1: System CA Store (OpenSSL)

- Entrypoint `install_certificates()` copies `.crt` files from `CERTS_PATH` volume mount
- Entrypoint `install_llm_ca_certs()` downloads PEM/DER certificates from `LLM_CA_CERT_URLS`
- DER is auto-converted to PEM (`openssl x509 -inform DER -outform PEM`)
- Certificates are saved with **`.crt` extension** (`.pem` is silently ignored by `update-ca-certificates`)
- `update-ca-certificates --fresh` adds them to `/etc/ssl/certs/ca-certificates.crt`
- **Fallback**: if `update-ca-certificates` misses certs, they are appended to `ca-certificates.crt` with SHA256 fingerprint dedup
- Hash symlinks created with collision support: `.0`, `.1`, `.2` suffixes

### Layer 2: Python HTTP Clients

| Library | File | Mechanism | Status |
|---------|------|-----------|--------|
| `httpx` (AsyncOpenAI) | `service.py:LLMClient._get_ssl_verify()` | `ssl.create_default_context(capath="/etc/ssl/certs/")` | ✅ Works (0.1.7) |
| `requests` (`_llm_http.py`) | `_get_verify()` | `"/etc/ssl/certs/"` (string path to dir) | ✅ Works (0.1.7) |
| `requests` (`preview_llm_client.py`) | `_get_verify()` | `"/etc/ssl/certs/"` (string path to dir) | ✅ Works (0.1.7) |

Key insight: `verify=True` uses certifi, NOT system CA. `verify=<cafile_path>` ignores
intermediate CA in OpenSSL 3.x. `verify=<capath_dir>` works correctly.

### Layer 3: NSS Database (Playwright Chromium)

- Entrypoint `install_ca_to_nss()` imports PEM certs into `~/.pki/nssdb/` using `certutil`
- NSS DB path format: `sql:$HOME/.pki/nssdb` (SQLite prefix required, DBM not supported by Chromium)
- Nickname format: `{dir_prefix}-{filename}` (e.g., `llm-UC_RUSAL_Policy_CA`)
- Dedup by SHA256 fingerprint (not nickname), preventing duplicate imports
- Trust attributes: `"C,,"` (trusted CA for TLS server certs)
- Fixes `ERR_CERT_AUTHORITY_INVALID` in Playwright dashboard screenshots

### Layer 4: LLM_SSL_VERIFY Escape Hatch

- Env var `LLM_SSL_VERIFY=false` disables SSL verification entirely
- Accepted values for false: `false`, `0`, `no`, `off` (case-insensitive)
- Default: enabled (returns capath-based SSLContext or path)
- Used by all three HTTP client implementations
- **WARNING**: `verify=False` is for DIAGNOSTIC USE ONLY. Never leave in production.

## Key Files

| File | Role |
|------|------|
| `docker/backend.Dockerfile` | Installs `libnss3-tools` (certutil), Playwright Chromium |
| `docker/backend.entrypoint.sh` | `install_certificates`, `install_llm_ca_certs`, `install_ca_to_nss` |
| `plugins/llm_analysis/service.py` | `LLMClient._get_ssl_verify()` → `ssl.create_default_context(capath=...)` |
| `plugins/translate/_llm_http.py` | `_get_verify()` → `"/etc/ssl/certs/"` |
| `plugins/translate/preview_llm_client.py` | `_get_verify()` → `"/etc/ssl/certs/"` |
| `scripts/check_llm_certs.py` | Full diagnostic: openssl, httpx, requests, NSS |
| `docker-compose.yml` | Passes `LLM_SSL_VERIFY`, `LLM_CA_CERT_URLS` |
| `docker-compose.enterprise-clean.yml` | Same as above |
| `.env.enterprise-clean` | Default values for both env vars |

## How to Test Certificate Installation

### On the server, after container restart:

```bash
docker cp scripts/check_llm_certs.py ss-tools-backend-1:/tmp/

docker compose -f docker-compose.enterprise-clean.yml \
  --env-file .env.enterprise-clean exec backend \
  python3 /tmp/check_llm_certs.py --target https://lite.ai.rusal.com
```

Expected diagnostic matrix:

```
openssl_default: ✅ code 0
openssl_capath:  ✅ code 0
openssl_cafile:  ❌ code 20 (expected — OpenSSL 3.x limitation)
httpx_capath:    ✅ HTTP 200
httpx_cafile:    ❌ SSL error (expected)
requests_capath: ✅ HTTP 200
requests_cafile: ❌ SSL error (expected)
```

## Manually testing chain validity

```bash
# 1. Check individual certs
openssl x509 -in /usr/local/share/ca-certificates/custom/RUSAL_ROOT.crt -noout -subject -issuer

# 2. Verify full chain
openssl verify -CAfile /usr/local/share/ca-certificates/custom/RUSAL_ROOT.crt \
  -untrusted /usr/local/share/ca-certificates/llm/UC_RUSAL_Policy_CA.crt \
  /usr/local/share/ca-certificates/llm/UC_RUSAL_RGM_Issuing_CA.crt
# → OK

# 3. Connect with capath
echo | openssl s_client -connect lite.ai.rusal.com:443 \
  -servername lite.ai.rusal.com \
  -CApath /etc/ssl/certs/

# 4. Verify certificate count
grep -c "BEGIN CERTIFICATE" /etc/ssl/certs/ca-certificates.crt
```

## Discovered Findings

### Finding 1: certifi vs system CA
`requests` and `httpx` use `certifi` CA bundle by default. Corporate CA certs installed into `/etc/ssl/certs/` are invisible to Python HTTP clients unless explicitly pointed to the system bundle.

### Finding 2: Three separate HTTP stacks
ss-tools has three LLM HTTP clients (httpx for async, requests for sync, Playwright for screenshots), each with independent CA trust configuration.

### Finding 3: .pem extension ignored by update-ca-certificates
On Debian Bookworm (python:3.11-slim), `update-ca-certificates` only processes `.crt` files.
Files with `.pem` extension in `/usr/local/share/ca-certificates/` are silently skipped.
This left `ca-certificates.crt` empty (0 certs) while hash symlinks existed.
Fix: save downloaded certificates with `.crt` extension.

### Finding 4: DER format from corporate PKI
Corporate PKI servers (`pki.rusal.com`) often serve certificates in DER (binary) format.
`update-ca-certificates` requires PEM. Auto-detection and conversion
(`openssl x509 -inform DER -outform PEM`) is essential.

### Finding 5: Chicken-and-egg TLS bootstrap
Downloading a CA certificate from a PKI server that uses the same CA causes a TLS
verification loop. Entrypoint uses `curl --insecure` as fallback when initial `curl`
fails with TLS error.

### Finding 6: NSS DB format
Chromium uses NSS Shared DB in `~/.pki/nssdb/`. The `sql:` prefix is required for
`certutil` to use the SQLite format. Without it, certutil defaults to the legacy
DBM format which Chromium may not read.

### Finding 7: OpenSSL 3.x cafile vs capath (CRITICAL)
OpenSSL 3.x treats non-self-signed certificates in `-CAfile` as trust anchors,
NOT as intermediates. This means intermediate CA certificates (Policy CA, RGM Issuing CA)
are ignored when using `-CAfile /etc/ssl/certs/ca-certificates.crt`, producing
verify code 20. Using `-CApath /etc/ssl/certs/` (directory with hash symlinks)
correctly builds the full chain, producing verify code 0.

This affects ALL Python libraries:
- `ssl.create_default_context(cafile=...)` → calls OpenSSL's `SSL_CTX_load_verify_locations`
  with the file, which exhibits the same limitation
- `ssl.create_default_context(capath=...)` → works correctly
- `requests.get(verify="/path/to/file")` → fails (uses cafile internally)
- `requests.get(verify="/path/to/dir/")` → works (uses capath internally)

### Finding 8: openssl s_client hangs without stdin
`openssl s_client` waits for input after TLS handshake. When run via
`subprocess.run(cmd, capture_output=True)`, it hangs until timeout.
Fix: pass `input=""` or `echo | openssl s_client ...`.

## Deploy

```bash
# On target server:
xz -dc ss-tools-backend.0.1.7.tar.xz | docker load
xz -dc ss-tools-frontend.0.1.7.tar.xz | docker load

# Enable capath-based verification (remove LLM_SSL_VERIFY=false)
sed -i '/LLM_SSL_VERIFY=false/d' .env.enterprise-clean

docker compose -f docker-compose.enterprise-clean.yml \
  --env-file .env.enterprise-clean down

docker compose -f docker-compose.enterprise-clean.yml \
  --env-file .env.enterprise-clean up -d

# Verify
docker cp scripts/check_llm_certs.py ss-tools-backend-1:/tmp/
docker compose -f docker-compose.enterprise-clean.yml \
  --env-file .env.enterprise-clean exec backend \
  python3 /tmp/check_llm_certs.py --target https://lite.ai.rusal.com
```

## Version History

| Version | Changes |
|---------|---------|
| 0.1.5 | Initial SSL support: `LLM_SSL_VERIFY`, `_format_connection_error()`, CA download via URL, NSS import |
| 0.1.6 | QA fixes: fingerprint dedup, hash symlink collision, NSS collision, DER→PEM error handling, chicken-and-egg TLS, nullglob, translate plugin SSL |
| 0.1.7 | **capath instead of cafile**: OpenSSL 3.x cafile limitation discovered and fixed. `.crt` extension for downloaded certs. Diagnostic script rewritten. |

# [/DEF:ADR-0009:ADR]