ss-tools/.kilo/plans/1783320721445-eager-tiger.md

Plan: centralized SSL certificate management for all containers

Goal

Replace scattered LLM-specific SSL environment variables with one centralized certificate/trust mechanism used consistently by backend, frontend, agent, Python HTTP clients, Playwright/Chromium, curl/openssl diagnostics, and release bundles.

User intent:

• Remove LLM_CA_CERT_URLS and LLM_SSL_VERIFY from operator-facing configuration.
• Do not manage LLM TLS separately from other corporate TLS needs.
• Certificates should be mounted/installed once through a single CERTS_PATH / /opt/certs contract.
• All containers should trust the same corporate CA set.
• Runtime clients should use system trust, not custom per-client env toggles.

Current state inventory

ADR

• docs/adr/ADR-0009-ssl-certificate-management.md
  • Correctly identified that OpenSSL 3.x works with capath=/etc/ssl/certs/ and can fail with flat cafile bundles.
  • Still documents LLM_SSL_VERIFY and LLM_CA_CERT_URLS as separate LLM-specific paths.
  • Needs update: centralized CERTS_PATH replaces LLM-specific env vars.

Backend container

• docker/backend.entrypoint.sh
  • install_certificates() already installs *.crt/*.pem from ${CERTS_PATH:-/opt/certs} into /usr/local/share/ca-certificates/custom, then update-ca-certificates --fresh.
  • install_llm_ca_certs() separately uses LLM_CA_CERT_URLS to download DER/PEM CA certs into /usr/local/share/ca-certificates/llm, then creates hash symlinks.
  • install_ca_to_nss() imports custom and llm certs into Chromium NSS DB.
  • Problem: there are two certificate sources (CERTS_PATH and LLM_CA_CERT_URLS) and only the LLM path has robust DER conversion/hash-symlink validation.

Frontend container

• docker/frontend.entrypoint.sh
  • Uses ${CERTS_PATH:-/opt/certs} and installs mounted CA files into Alpine CA store.
  • Skips server.crt and server.key.
  • Does not use LLM_CA_CERT_URLS / LLM_SSL_VERIFY, which is good.

Agent container

• docker/Dockerfile.agent
  • Python slim image currently installs libgl1 libglib2.0-0 libpq5, but not necessarily ca-certificates, openssl, or a startup entrypoint to install /opt/certs CAs.
  • Compose mounts CERTS_PATH into /opt/certs but agent likely does not install those certificates into trust store.
  • If agent makes HTTPS calls to backend/LLM/internal APIs, it must share the same trust installation.

Python clients

• backend/src/plugins/llm_analysis/service.py

  • LLMClient._get_ssl_verify() reads LLM_SSL_VERIFY; if false, returns False; otherwise returns ssl.create_default_context(capath="/etc/ssl/certs").
  • Need central replacement: no LLM-specific toggle; always use centralized trust context.

• backend/src/plugins/translate/_llm_async_http.py

  • _get_verify() reads LLM_SSL_VERIFY; if false, disables TLS verification; otherwise uses ssl.create_default_context(capath="/etc/ssl/certs").
  • Need central replacement.

• Other LLM/provider/test code may use httpx or AsyncOpenAI; all should route through a single helper.

Compose/env examples

Likely references to remove/update:

• docker-compose.yml
• docker-compose.enterprise-clean.yml
• docker-compose.e2e.yml
• build.sh generated bundle compose
• .env.example
• .env.enterprise-clean.example
• backend/.env.example
• docker/.env.agent.example
• .env.current.example, .env.master.example, .env.e2e.example, frontend/.env.example
• scripts/diag_container.py

Current operator-facing variables:

• Keep: CERTS_PATH=./certs
• Remove: LLM_CA_CERT_URLS
• Remove: LLM_SSL_VERIFY

Proposed canonical contract

One certificate source

CERTS_PATH is the only external certificate input.

Host layout:

./certs/
  RUSAL_ROOT.crt
  RGM_Issuing.crt
  lite_ai_issuing.crt
  any-other-corporate-ca.crt
  server.crt        # optional frontend HTTPS server cert, not trusted as CA
  server.key        # optional frontend HTTPS server key, not trusted as CA
  server.p12        # optional frontend HTTPS bundle

Container mount:

volumes:
  - ${CERTS_PATH:-./certs}:/opt/certs:ro

Rules:

• .crt, .pem, .cer, .der under /opt/certs are treated as trust candidates.
• server.crt, server.key, server.p12, *.key, *.p12, *.pfx are not imported as CA trust anchors.
• DER certificates are detected and converted to PEM.
• Every valid CA cert is installed into system CA store.
• Backend additionally imports valid CA certs into NSS DB for Chromium/Playwright.
• No runtime download of certificates from URLs.
• No environment variable disables TLS verification.

One Python SSL helper

Add central helper, e.g. backend/src/core/ssl.py:

• get_ssl_context() -> ssl.SSLContext
  • returns ssl.create_default_context(capath="/etc/ssl/certs") if available
  • fallback to ssl.create_default_context() only if capath missing
• get_httpx_verify() -> ssl.SSLContext
• optional get_requests_verify() -> str | bool
  • if requests is still used, use /etc/ssl/certs/ca-certificates.crt only if unavoidable; prefer no requests-specific LLM client.
• no verify=False code path.
• log only safe diagnostic: SSLContext(capath=/etc/ssl/certs).

Replace local _get_ssl_verify() / _get_verify() functions with the shared helper.

One diagnostics script

Update scripts/diag_container.py:

• Remove LLM_SSL_VERIFY and LLM_CA_CERT_URLS checks.
• Report only:
  • CERTS_PATH value (if set)
  • /opt/certs contents
  • system CA store state
  • hash symlinks
  • NSS DB entries
  • openssl -CApath
  • Python SSLContext(capath)
  • httpx(verify=context)
  • encryption health
• If cert for target fails, say:
  • “Place the issuing/root CA .crt/.cer/.der into CERTS_PATH and restart containers.”

Implementation plan

1. Backend entrypoint: unify certificate installation

File: docker/backend.entrypoint.sh

Changes:

1. Replace/extend install_certificates() to become the single robust installer.
2. Remove install_llm_ca_certs() or leave as unused deprecated internal no-op during one release.
3. Add DER/PEM auto-detection for all certs from /opt/certs:
   • Try openssl x509 -in file -noout as PEM.
   • If fails, try openssl x509 -inform DER -in file -out converted.crt.
4. Copy normalized certs to /usr/local/share/ca-certificates/custom/.
5. Exclude non-CA/server/private files:
   • server.crt, server.key, server.p12, *.key, *.p12, *.pfx.
6. Run update-ca-certificates --fresh once.
7. Validate each installed cert:
   • fingerprint presence in ca-certificates.crt if possible
   • hash symlink exists under /etc/ssl/certs/<hash>.N
   • create collision-safe symlink if missing
8. Import the same normalized certs to NSS DB.
9. Emit clear startup logs:
   • installed count
   • skipped count
   • invalid cert count
   • hash symlink count
   • NSS import count

Expected result:

• Adding lite_ai_issuing.crt to ./certs is enough; no LLM-specific URL env var.

2. Frontend entrypoint: align cert parser

File: docker/frontend.entrypoint.sh

Changes:

1. Keep CERTS_PATH as only input.
2. Accept .crt, .pem, .cer, .der.
3. Convert DER to PEM before update-ca-certificates.
4. Keep skipping server cert/key/bundle files.
5. Log installed/skipped/invalid certs.

Expected result:

• Frontend/nginx Alpine trust store uses the same ./certs content.

3. Agent container: add centralized cert installation

Files:

• docker/Dockerfile.agent
• new docker/agent.entrypoint.sh or reuse a shared cert installer copied into agent image.

Changes:

1. Install system packages:
   • ca-certificates
   • openssl
2. Add entrypoint that runs the same centralized cert installer against /opt/certs before python -m src.agent.run.
3. Ensure compose mounts ${CERTS_PATH:-./certs}:/opt/certs:ro for agent.
4. Use the same skip rules and DER conversion.

Expected result:

• Agent trusts corporate CA certs identically to backend.

4. Optional shared shell library

To avoid three divergent installers, create one shared script:

• docker/certs.sh

Functions:

• install_certs_debian()
• install_certs_alpine()
• normalize_cert_dir()
• install_to_nss()
• create_hash_symlinks()

Then:

• backend entrypoint sources docker/certs.sh
• frontend entrypoint sources docker/certs.sh
• agent entrypoint sources docker/certs.sh

If minimizing churn, duplicate logic initially but prefer shared script for zero drift.

Recommended: shared docker/certs.sh.

5. Central Python SSL helper

New file:

• backend/src/core/ssl.py

API:

def get_system_ssl_context() -> ssl.SSLContext:
    ...

def describe_ssl_context(ctx: ssl.SSLContext) -> str:
    ...

Update callers:

• backend/src/plugins/llm_analysis/service.py
  • remove LLM_SSL_VERIFY logic
  • use get_system_ssl_context()
• backend/src/plugins/translate/_llm_async_http.py
  • remove LLM_SSL_VERIFY logic
  • use get_system_ssl_context()
• search all LLM_SSL_VERIFY occurrences and remove from runtime code.

Complete files list needing changes (runtime + tests + docs):

File	Action
backend/src/core/ssl.py	NEW — centralized SSL helper
backend/src/plugins/llm_analysis/service.py	Remove _get_ssl_verify, delegate to core.ssl
backend/src/plugins/translate/_llm_async_http.py	Remove _get_verify, delegate to core.ssl
docker/backend.entrypoint.sh	Remove install_llm_ca_certs, merge DER/PEM logic into unified installer
docker/frontend.entrypoint.sh	Add DER conversion, align with unified logic
docker/Dockerfile.agent	Add ca-certificates, openssl
docker/agent.entrypoint.sh	NEW — agent entrypoint with cert install
docker/certs.sh	NEW — shared cert installer (optional, refactor step)
docker-compose.yml	Remove LLM_SSL_VERIFY, LLM_CA_CERT_URLS; add CERTS_PATH mount
docker-compose.enterprise-clean.yml	Remove LLM_SSL_VERIFY, LLM_CA_CERT_URLS; add agent CERTS_PATH mount
docker-compose.e2e.yml	Add CERTS_PATH
build.sh	Update generated compose
.env.example	Remove LLM_SSL_VERIFY, LLM_CA_CERT_URLS; enhance CERTS_PATH comments
.env.enterprise-clean.example	Same
.env.current.example	Same
.env.master.example	Same
backend/.env.example	Same
docker/.env.agent.example	Same
scripts/diag_container.py	Remove LLM_* refs, add /opt/certs inventory
scripts/check_llm_certs.py	Remove LLM_SSL_VERIFY section (or deprecate file)
docs/adr/ADR-0009-ssl-certificate-management.md	Replace LLM_SSL_VERIFY + LLM_CA_CERT_URLS with centralized CERTS_PATH
README.md	Update cert section
backend/tests/plugins/test_llm_analysis_service.py	Update tests for centralized ssl helper
backend/tests/plugins/translate/test_llm_async_http.py	Same
backend/tests/integration/test_superset_tls_custom_ca.py	Same

Policy:

• There is no verify=False env escape hatch.
• If operators need temporary bypass for manual debugging, they can use curl/openssl outside app; app remains secure-by-default.

6. Compose/env cleanup

Files:

• docker-compose.yml
• docker-compose.enterprise-clean.yml
• docker-compose.e2e.yml
• build.sh generated compose
• .env.example
• .env.enterprise-clean.example
• backend/.env.example
• docker/.env.agent.example
• other .env.*.example

Changes:

1. Remove LLM_CA_CERT_URLS from all compose env blocks and examples.
2. Remove LLM_SSL_VERIFY from all compose env blocks and examples.
3. Keep one variable:

CERTS_PATH=./certs

4. Add comments:

# Put all corporate root/intermediate CA certificates here.
# Applies to backend, frontend, and agent containers.
# Accepted trust files: *.crt, *.pem, *.cer, *.der
# Do not put private keys here except server.key/server.p12 used by frontend TLS.
CERTS_PATH=./certs

5. Bundle generated compose must mount CERTS_PATH into all containers:
   • backend
   • frontend
   • agent

7. Diagnostics update

File: scripts/diag_container.py

Changes:

1. Remove LLM_SSL_VERIFY and LLM_CA_CERT_URLS reporting.
2. Add /opt/certs inventory:
   • list recognized trust candidates
   • list skipped server/private files
   • list invalid files
3. Add NSS DB diagnostics if certutil is installed.
4. Fix OpenSSL output classification:
   • if return code is 0 but no verify code parsed, print raw verify line excerpt.
5. Summary should say:

If CApath/httpx failures:
  -> put the issuing/root CA for target into CERTS_PATH (./certs)
  -> restart affected containers
  -> rerun this diagnostic

8. ADR update

File: docs/adr/ADR-0009-ssl-certificate-management.md

Changes:

1. Replace “Layer 4: LLM_SSL_VERIFY Escape Hatch” with “Layer 4: centralized CERTS_PATH trust contract”.
2. Mark old env vars as removed/deprecated:
   • LLM_SSL_VERIFY removed
   • LLM_CA_CERT_URLS removed
3. Update key files table.
4. Update diagnostics/runbook.
5. State policy:
   • application code never disables TLS verification via env var
   • all trust anchors come from mounted CERTS_PATH

9. Tests

Backend unit tests:

• New tests for backend/src/core/ssl.py:
  • returns SSLContext
  • prefers capath=/etc/ssl/certs when present
  • does not read LLM_SSL_VERIFY
  • cannot return False

Shell/script tests, if existing harness supports:

• Cert normalization:
  • PEM .crt accepted
  • .pem accepted
  • DER .cer accepted/converted
  • server.key, server.p12 skipped
  • invalid file skipped with warning

Integration/smoke:

• Start backend with certs mounted.
• Run:

python3 /tmp/diag_container.py --target lite.ai.rusal.com:443

Expected after correct CA placed in ./certs:

• OpenSSL capath OK
• Python SSLContext OK
• httpx OK

10. Migration/operator steps

For production operators:

1. Remove from .env.enterprise-clean:

LLM_SSL_VERIFY=...
LLM_CA_CERT_URLS=...

2. Put all corporate CA files under ./certs:

./certs/RUSAL_ROOT.crt
./certs/RGM_Issuing.crt
./certs/lite_ai_issuing.crt

3. Restart all containers:

docker compose --env-file .env.enterprise-clean -f docker-compose.enterprise-clean.yml up -d --force-recreate

4. Run diagnostics:

docker cp scripts/diag_container.py ss_tools-backend-1:/tmp/
docker compose exec backend python3 /tmp/diag_container.py --target lite.ai.rusal.com:443

5. Verify expected:

openssl capath: OK
Python SSLContext(capath): OK
httpx(capath): OK

Acceptance criteria

• No runtime code reads LLM_SSL_VERIFY.
• No compose/env example exposes LLM_SSL_VERIFY or LLM_CA_CERT_URLS.
• Backend, frontend, and agent all mount CERTS_PATH and install certs into their system trust stores.
• Backend imports the same trust certs into NSS for Chromium/Playwright.
• Python LLM clients use one central SSL helper and never return verify=False from env config.
• Diagnostic script reports centralized CERTS_PATH trust state and no longer references LLM-specific env vars.
• ADR-0009 reflects the new centralized design.
• Existing auth/encryption/key recovery tests continue passing.

Risks and mitigations

• Risk: Removing LLM_SSL_VERIFY=false removes an easy emergency bypass.

  • Mitigation: keep only a code-local debug override unavailable in compose/examples? Recommended: no runtime bypass; rely on correct CA installation.

• Risk: Operators currently rely on LLM_CA_CERT_URLS for PKI downloads.

  • Mitigation: document how to download/copy CA files into ./certs; do not download at runtime.

• Risk: Agent previously did not install CAs.

  • Mitigation: add agent entrypoint and smoke-test HTTPS from inside agent.

• Risk: Frontend server.crt may accidentally be imported as CA.

  • Mitigation: explicit skip list for server/private files across all installers.

Open question

Should we completely remove LLM_SSL_VERIFY support from code, or keep a hidden ALLOW_INSECURE_SSL=false emergency variable that is not documented or present in compose/examples? Recommended: completely remove SSL bypass from application runtime.