Enhance the reliability and observability of the task execution engine
by introducing retry mechanisms, idempotency, and structured progress
tracking.
- Implement centralized retry logic with exponential backoff support
in `JobLifecycle`.
- Add `retry_task` API endpoint and `TaskManager` method for manual
task restarts.
- Introduce task idempotency using `_idempotency_key` to prevent
duplicate executions.
- Add `retry_count`, `max_retries`, `last_error`, and `progress` fields
to the `Task` model and ensure persistence via `TaskPersistenceService`.
- Upgrade `SchedulerService` to use differential synchronization with
the persistent `SQLAlchemyJobStore` for better job durability.
- Implement structured heartbeat logging to support real-time progress
updates.
- Update project documentation and ADRs to reflect the new plugin
runtime and task resilience patterns.
- Add comprehensive unit and integration tests for the new task
lifecycle features.
SSL fix (ADR-0009 Finding 7):
- Replace ssl.create_default_context() with system_ssl_context(capath)
in client_registry.py, async_network.py, notifications/providers.py,
git/_base.py. Previous fix (0.5.1) only covered LLM clients; the
Superset API client still used ssl.create_default_context() which
loads cafile (flat bundle) where OpenSSL 3.x ignores intermediate CA
certificates. system_ssl_context() uses capath only (hash symlinks).
Agent fix:
- Extract _check_llm_provider_health + _llm_status from agent/app.py
into agent/_llm_health.py. The /api/agent/llm-status endpoint was
importing from agent/app.py which triggers 'import gradio' at module
level. Backend container does not have gradio installed, causing
ModuleNotFoundError (500 error) every 30s on health check polling.
Config:
- Add ALLOWED_ORIGINS to docker-compose.yml + docker-compose.enterprise-clean.yml
ADR-0009 updated: Layer 2 table expanded with 4 missing clients,
Finding 5 reconciled (LLM_CA_CERT_URLS restored in 0.5.1), version 0.5.2.
Verified: 1110 unit tests passed, gradio import isolation confirmed.
Backend entrypoint sources certs.sh for CA certificate installation,
but the file was never copied into the Docker image — causing
container crash loop on startup (regression in 7f932610).
Changes:
- docker/backend.Dockerfile, docker/all-in-one.Dockerfile: COPY certs.sh
- docker/backend.entrypoint.sh: remove dead install_llm_ca_certs,
install_certificates (replaced by docker/certs.sh); update @INVARIANT
- backend/src/core/ssl.py: fix describe_context() to report actual
system cert count (SSLContext.capath attr does not exist in Python 3)
- __tests__: rewrite stale tests asserting LLM_SSL_VERIFY=false →
verify=False; behaviour is permanently removed — always CERT_REQUIRED
- backend/tests/integration/test_backend_container.py [new]: 5 tests
verifying certs.sh presence, sourceability, and full-stack smoke
(testcontainers PG → entrypoint → migrations → health)
- conftest.py: restore health-wait loop and fixtures in superset_container
- ADR-0009, README.md, scripts, .env: align docs with centralized SSL
- ADR-0009: comprehensive SSL certificate management strategy
- _get_ssl_verify() returns SSLContext with cafile= instead of bool True
(httpx deprecates verify=<str>, requests needs system CA, not certifi)
- _get_verify() in translate plugin returns system CA path
- All tests updated for SSLContext return type
- All QA findings C1-C3, H1-H4, M1 incorporated into ADR
