ss-tools/docs/adr/ADR-0009-ssl-certificate-management.md

# [DEF:Doc.Adr.ADR0009:ADR]
# @STATUS ACCEPTED
# @PURPOSE Define the strategy for corporate SSL certificate management across all HTTP clients in superset-tools (httpx, requests, openai), covering system CA store, NSS database for Chromium/Playwright. Two input channels: (1) HTTP download from corporate PKI via LLM_CA_CERT_URLS (primary), (2) volume mount via CERTS_PATH (additional). LLM_SSL_VERIFY removed in 0.2.x.
# @RELATION DEPENDS_ON -> [Doc.Adr.ADR0001:ADR]
# @RELATION DEPENDS_ON -> [Doc.Adr.ADR0016:ADR]
# @RELATION CALLS -> [docker/backend.entrypoint.sh]
# @RELATION CALLS -> [docker/certs.sh]
# @RELATION CALLS -> [docker/frontend.entrypoint.sh]
# @RELATION CALLS -> [docker/agent.entrypoint.sh]
# @RELATION CALLS -> [backend/src/core/ssl.py]
# @RELATION CALLS -> [backend/src/plugins/llm_analysis/service.py]
# @RELATION CALLS -> [backend/src/plugins/translate/_llm_async_http.py]
# @RELATION CALLS -> [backend/src/core/utils/client_registry.py]
# @RELATION CALLS -> [backend/src/core/utils/async_network.py]
# @RELATION CALLS -> [backend/src/services/notifications/providers.py]
# @RELATION CALLS -> [backend/src/services/git/_base.py]
# @RELATION CALLS -> [scripts/check_llm_certs.py]
# @RELATION CALLS -> [scripts/diag_container.py]
# @RATIONALE superset-tools operates in corporate environments with internal Certificate Authorities.
#   Three separate HTTP stacks are used: httpx (AsyncOpenAI in llm_analysis plugin),
#   requests (sync translate plugin), and Playwright Chromium (dashboard screenshots).
#   Each has its own CA trust store, requiring different installation strategies.
# @REJECTED Centralizing all LLM calls into a single HTTP client — rejected because
#   llm_analysis uses async httpx+AsyncOpenAI for streaming and multiple concurrent calls,
#   while translate uses sync requests for simpler request/response. Merging would
#   require rewriting one or both, creating regression risk with no business value.

## Problem

Corporate SSL certificates installed via `update-ca-certificates` into `/etc/ssl/certs/ca-certificates.crt` are NOT automatically trusted by:

1. **Python `requests` library** — uses bundled `certifi` CA bundle, not system store
2. **Python `httpx` library** — uses `certifi` by default when `verify=True`
3. **Playwright Chromium** — uses NSS Shared DB (`~/.pki/nssdb`), not OpenSSL store
4. **`openssl s_client`** — hangs without stdin (needs `echo \|` or `input=""`)

This causes `SSLError: CERTIFICATE_VERIFY_FAILED` and `ERR_CERT_AUTHORITY_INVALID` even after correct system-wide CA installation.

## Discovery Journey

### Phase 1: certifi vs system CA (0.1.5)
- `requests` and `httpx` use `certifi` bundle, not `/etc/ssl/certs/ca-certificates.crt`
- Fix: return system CA path instead of `True` in all `_get_verify()` functions
- Implemented in `service.py`, `_llm_http.py`, `preview_llm_client.py`

### Phase 2: .pem vs .crt extension (0.1.6)
- `update-ca-certificates` only processes `.crt` files, silently ignores `.pem`
- Downloaded certificates from PKI were saved as `.pem`, skipped by update-ca-certificates
- Result: `ca-certificates.crt` was empty (0 certs), hash symlinks existed but bundle was broken
- Fix: save downloaded certs with `.crt` extension, not `.pem`

### Phase 3: cafile vs capath (0.1.7) — KEY DISCOVERY
- `openssl s_client -CAfile /etc/ssl/certs/ca-certificates.crt` → code 20 (intermediate CA ignored)
- `openssl s_client -CApath /etc/ssl/certs/` → code 0 (chain built correctly)
- `ssl.create_default_context(cafile=...)` → SSL error
- `ssl.create_default_context(capath=...)` → HTTP 200
- `requests.get(verify="/etc/ssl/certs/ca-certificates.crt")` → SSLError
- `requests.get(verify="/etc/ssl/certs/")` → HTTP 200

**Root cause**: OpenSSL 3.x treats non-self-signed certificates in `-CAfile` as trust anchors,
NOT as intermediates. `-CApath` (directory with hash symlinks) correctly builds chains using
all certificates found. The intermediate CA certs (Policy CA, RGM Issuing CA) are not
self-signed, so they are ignored in `-CAfile` but correctly used in `-CApath`.

Diagnostic matrix (verified on production server 2026-05-28):

| Method | cafile | capath |
|--------|--------|--------|
| openssl s_client | code 20 FAIL | code 0 OK |
| httpx (SSLContext) | SSLError FAIL | HTTP 200 OK |
| requests (verify=) | SSLError FAIL | HTTP 200 OK |

## Solution

### Layer 1: System CA Store (OpenSSL)

- Shared `docker/certs.sh:install_all_certs()` copies `.crt`/`.pem`/`.cer`/`.der` files from `CERTS_PATH` volume mount
- DER is auto-converted to PEM via `normalize_cert()` (`openssl x509 -inform DER -outform PEM`)
- Server/private files (`server.crt`, `server.key`, `*.key`, `*.p12`, `*.pfx`) are excluded from CA import
- Certificates are saved with **`.crt` extension** to `/usr/local/share/ca-certificates/custom/`
- `update-ca-certificates --fresh` adds them to `/etc/ssl/certs/ca-certificates.crt`
- Hash symlinks created with collision support: `.0`, `.1`, `.2` suffixes

### Layer 2: Python HTTP Clients

| Library | File | Mechanism | Status |
|---------|------|-----------|--------|
| `httpx` (AsyncOpenAI) | `service.py:LLMClient._get_ssl_verify()` | `system_ssl_context(capath="/etc/ssl/certs/")` | ✅ Fixed (0.5.1) |
| `httpx` (translate) | `_llm_async_http.py:_get_verify()` | `httpx_verify()` → `system_ssl_context(capath)` | ✅ Fixed (0.5.1) |
| `httpx` (SupersetClientRegistry) | `client_registry.py:get_client()` | `system_ssl_context(capath="/etc/ssl/certs/")` | ✅ Fixed (0.5.2) |
| `httpx` (AsyncAPIClient) | `async_network.py:__init__()` | `system_ssl_context(capath="/etc/ssl/certs/")` | ✅ Fixed (0.5.2) |
| `httpx` (Notifications) | `notifications/providers.py:_get_http_client()` | `system_ssl_context(capath="/etc/ssl/certs/")` | ✅ Fixed (0.5.2) |
| `httpx` (Git) | `git/_base.py:GitServiceBase.__init__()` | `system_ssl_context(capath="/etc/ssl/certs/")` | ✅ Fixed (0.5.2) |

Key insight: `verify=True` uses certifi, NOT system CA. `verify=<cafile_path>` ignores
intermediate CA in OpenSSL 3.x. `verify=<capath_dir>` works correctly.
`ssl.create_default_context()` without explicit capath loads BOTH cafile and capath
from OpenSSL defaults — the cafile may fail intermediate CA chain-building (Finding 7).
`system_ssl_context(capath="/etc/ssl/certs")` loads ONLY capath (hash symlinks),
correctly building chains with intermediate CAs.

### Layer 3: NSS Database (Playwright Chromium)

- Entrypoint `install_ca_to_nss()` imports PEM certs into `~/.pki/nssdb/` using `certutil`
- NSS DB path format: `sql:$HOME/.pki/nssdb` (SQLite prefix required, DBM not supported by Chromium)
- Nickname format: `{dir_prefix}-{filename}` (e.g., `llm-UC_RUSAL_Policy_CA`)
- Dedup by SHA256 fingerprint (not nickname), preventing duplicate imports
- Trust attributes: `"C,,"` (trusted CA for TLS server certs)
- Fixes `ERR_CERT_AUTHORITY_INVALID` in Playwright dashboard screenshots

### Layer 4: LLM_CA_CERT_URLS HTTP Download (0.5.1)

- Env var `LLM_CA_CERT_URLS` — space-separated list of HTTP URLs to CA certificates
- Downloaded on container startup by `certs.sh:download_llm_ca_certs()`
- URLs use **plain HTTP only** — no TLS chicken-and-egg problem
- Supports PEM and DER formats (auto-detected, DER auto-converted to PEM)
- Retry logic: 3 attempts with 2-second delay
- Downloaded certs saved to `/usr/local/share/ca-certificates/llm/`
- `update-ca-certificates --fresh` includes them in the system bundle
- Example: `LLM_CA_CERT_URLS="http://pki.rusal.com/CertData/UC_RUSAL_Policy_CA.crt http://pki.rusal.com/CertData/UC_RUSAL_RGM_Issuing_CA.crt"`

### Layer 5: CERTS_PATH Volume Mount

- Env var `CERTS_PATH` — directory with `.crt`/`.pem`/`.cer`/`.der` files
- Mounted as read-only volume: `${CERTS_PATH:-./certs}:/opt/certs:ro`
- Installed by `certs.sh:install_all_certs()` **after** `download_llm_ca_certs()`
- Server/private files (server.crt, *.key, *.p12, *.pfx) are excluded from CA import
- Shared `docker/certs.sh` handles installation across all containers (backend, frontend, agent).

## Key Files

| File | Role |
|------|------|
| `docker/backend.Dockerfile` | Installs `libnss3-tools` (certutil), Playwright Chromium |
| `docker/certs.sh` | Shared cert installer: `download_llm_ca_certs`, `install_all_certs`, `install_ca_to_nss`, `normalize_cert` |
| `docker/backend.entrypoint.sh` | Sources `certs.sh`, calls `download_llm_ca_certs` → `install_all_certs` → `install_ca_to_nss` |
| `docker/frontend.entrypoint.sh` | Installs CA certs from `CERTS_PATH` into Alpine store |
| `docker/agent.entrypoint.sh` | Sources `certs.sh`, installs certs for agent container |
| `backend/src/core/ssl.py` | Centralized SSL helper: `system_ssl_context()`, `httpx_verify()`, `describe_context()` |
| `backend/src/core/utils/client_registry.py` | SupersetClientRegistry: `get_client()` → `system_ssl_context()` |
| `backend/src/core/utils/async_network.py` | AsyncAPIClient: `__init__()` → `system_ssl_context()` |
| `backend/src/services/notifications/providers.py` | Notification HTTP: `_get_http_client()` → `system_ssl_context()` |
| `backend/src/services/git/_base.py` | Git HTTP: `GitServiceBase.__init__()` → `system_ssl_context()` |
| `plugins/llm_analysis/service.py` | `LLMClient._get_ssl_verify()` → delegates to `core.ssl.httpx_verify()` |
| `plugins/translate/_llm_async_http.py` | `_get_verify()` → delegates to `core.ssl.httpx_verify()` |
| `scripts/check_llm_certs.py` | Full diagnostic: openssl, httpx, requests, NSS, centralized SSL |
| `docker-compose.yml` | Mounts `${CERTS_PATH:-./certs}:/opt/certs:ro` (LLM env vars removed) |
| `docker-compose.enterprise-clean.yml` | Same — `CERTS_PATH` mount only |

## How to Test Certificate Installation

### On the server, after container restart:

```bash
docker cp scripts/check_llm_certs.py superset-tools-backend-1:/tmp/

docker compose -f docker-compose.enterprise-clean.yml \
  --env-file .env.enterprise-clean exec backend \
  python3 /tmp/check_llm_certs.py --target https://lite.ai.rusal.com
```

Expected diagnostic matrix:

```
openssl_default: ✅ code 0
openssl_capath:  ✅ code 0
openssl_cafile:  ❌ code 20 (expected — OpenSSL 3.x limitation)
httpx_capath:    ✅ HTTP 200
httpx_cafile:    ❌ SSL error (expected)
requests_capath: ✅ HTTP 200
requests_cafile: ❌ SSL error (expected)
```

## Manually testing chain validity

```bash
# 1. Check individual certs
openssl x509 -in /usr/local/share/ca-certificates/custom/RUSAL_ROOT.crt -noout -subject -issuer

# 2. Verify full chain
openssl verify -CAfile /usr/local/share/ca-certificates/custom/RUSAL_ROOT.crt \
  -untrusted /usr/local/share/ca-certificates/custom/UC_RUSAL_Policy_CA.crt \
  /usr/local/share/ca-certificates/custom/UC_RUSAL_RGM_Issuing_CA.crt
# → OK

# 3. Connect with capath
echo | openssl s_client -connect lite.ai.rusal.com:443 \
  -servername lite.ai.rusal.com \
  -CApath /etc/ssl/certs/

# 4. Verify certificate count
grep -c "BEGIN CERTIFICATE" /etc/ssl/certs/ca-certificates.crt
```

## Discovered Findings

### Finding 1: certifi vs system CA
`requests` and `httpx` use `certifi` CA bundle by default. Corporate CA certs installed into `/etc/ssl/certs/` are invisible to Python HTTP clients unless explicitly pointed to the system bundle.

### Finding 2: Three separate HTTP stacks
superset-tools has three LLM HTTP clients (httpx for async, requests for sync, Playwright for screenshots), each with independent CA trust configuration.

### Finding 3: .pem extension ignored by update-ca-certificates
On Debian Bookworm (python:3.11-slim), `update-ca-certificates` only processes `.crt` files.
Files with `.pem` extension in `/usr/local/share/ca-certificates/` are silently skipped.
This left `ca-certificates.crt` empty (0 certs) while hash symlinks existed.
Fix: save downloaded certificates with `.crt` extension.

### Finding 4: DER format from corporate PKI
Corporate PKI servers (`pki.rusal.com`) often serve certificates in DER (binary) format.
`update-ca-certificates` requires PEM. Auto-detection and conversion
(`openssl x509 -inform DER -outform PEM`) is essential.

### Finding 5: Chicken-and-egg TLS bootstrap (HISTORICAL — resolved in 0.5.1)
Downloading a CA certificate from a PKI server that uses the same CA causes a TLS
verification loop. Formerly handled by `install_llm_ca_certs()` with `curl --insecure`.
**Removed in 0.2.x**: certificates must be placed in `CERTS_PATH` volume mount by operator.
**Restored in 0.5.1**: `LLM_CA_CERT_URLS` HTTP download re-added with `--proto '=http'`
policy (plain HTTP only, no TLS chicken-and-egg). Both channels (`LLM_CA_CERT_URLS`
download + `CERTS_PATH` mount) are now active simultaneously.

### Finding 6: NSS DB format
Chromium uses NSS Shared DB in `~/.pki/nssdb/`. The `sql:` prefix is required for
`certutil` to use the SQLite format. Without it, certutil defaults to the legacy
DBM format which Chromium may not read.

### Finding 7: OpenSSL 3.x cafile vs capath (CRITICAL)
OpenSSL 3.x treats non-self-signed certificates in `-CAfile` as trust anchors,
NOT as intermediates. This means intermediate CA certificates (Policy CA, RGM Issuing CA)
are ignored when using `-CAfile /etc/ssl/certs/ca-certificates.crt`, producing
verify code 20. Using `-CApath /etc/ssl/certs/` (directory with hash symlinks)
correctly builds the full chain, producing verify code 0.

This affects ALL Python libraries:
- `ssl.create_default_context(cafile=...)` → calls OpenSSL's `SSL_CTX_load_verify_locations`
  with the file, which exhibits the same limitation
- `ssl.create_default_context(capath=...)` → works correctly
- `requests.get(verify="/path/to/file")` → fails (uses cafile internally)
- `requests.get(verify="/path/to/dir/")` → works (uses capath internally)

### Finding 8: openssl s_client hangs without stdin
`openssl s_client` waits for input after TLS handshake. When run via
`subprocess.run(cmd, capture_output=True)`, it hangs until timeout.
Fix: pass `input=""` or `echo | openssl s_client ...`.

## Deploy

```bash
# On target server:
xz -dc superset-tools-backend.0.2.x.tar.xz | docker load
xz -dc superset-tools-frontend.0.2.x.tar.xz | docker load

# Place corporate CA files in ./certs (LLM_SSL_VERIFY removed, LLM_CA_CERT_URLS optional)
ls ./certs/
# RUSAL_ROOT.crt  RGM_Issuing.crt  UC_RUSAL_Policy_CA.crt  etc.

docker compose -f docker-compose.enterprise-clean.yml \
  --env-file .env.enterprise-clean down

docker compose -f docker-compose.enterprise-clean.yml \
  --env-file .env.enterprise-clean up -d

# Verify
docker cp scripts/check_llm_certs.py superset-tools-backend-1:/tmp/
docker compose -f docker-compose.enterprise-clean.yml \
  --env-file .env.enterprise-clean exec backend \
  python3 /tmp/check_llm_certs.py --target https://lite.ai.rusal.com
```

## Version History

| Version | Changes |
|---------|---------|
| 0.1.5 | Initial SSL support: `LLM_SSL_VERIFY`, `_format_connection_error()`, CA download via URL, NSS import |
| 0.1.6 | QA fixes: fingerprint dedup, hash symlink collision, NSS collision, DER→PEM error handling, chicken-and-egg TLS, nullglob, translate plugin SSL |
| 0.1.7 | **capath instead of cafile**: OpenSSL 3.x cafile limitation discovered and fixed. `.crt` extension for downloaded certs. Diagnostic script rewritten. |
| 0.2.x | **Centralized SSL**: `LLM_SSL_VERIFY` and `LLM_CA_CERT_URLS` removed. Shared `docker/certs.sh` for all containers. `backend/src/core/ssl.py` central helper. `CERTS_PATH` volume mount is the only certificate input. Verify=False escape hatch permanently removed. |
| 0.5.1 | **Restored**: `LLM_CA_CERT_URLS` HTTP download as primary cert input channel. `download_llm_ca_certs()` added to `certs.sh`. Downloaded certs saved to `llm/` subdirectory, `install_all_certs` extended to create hash symlinks for both `custom/` and `llm/` dirs. `CERTS_PATH` volume mount kept as secondary channel. Integration test added for HTTP download flow. |
| 0.5.2 | **Fixed capath for ALL HTTP clients**: `ssl.create_default_context()` (no capath, loads cafile+capath) replaced with `system_ssl_context(capath="/etc/ssl/certs")` (capath only) in `client_registry.py`, `async_network.py`, `notifications/providers.py`, `git/_base.py`. Previous fix (0.5.1) only covered LLM clients; Superset API, notification, and git clients still used `ssl.create_default_context()` which fails intermediate CA chain-building in OpenSSL 3.x (Finding 7). Production diagnosis: RUSAL cert chain Root→Policy CA→RGM Issuing→Server requires Policy CA as intermediate; without capath, OpenSSL ignores it in cafile. |

# [/DEF:Doc.Adr.ADR0009:ADR]