- DRAFT validation was removed from orchestrator_validation.py (moved to service layer)
- Integration test expected ValueError for DRAFT jobs — fixed to test actual behavior
- 194/194 integration tests pass, 0 failures
Enhance the reliability and observability of the task execution engine
by introducing retry mechanisms, idempotency, and structured progress
tracking.
- Implement centralized retry logic with exponential backoff support
in `JobLifecycle`.
- Add `retry_task` API endpoint and `TaskManager` method for manual
task restarts.
- Introduce task idempotency using `_idempotency_key` to prevent
duplicate executions.
- Add `retry_count`, `max_retries`, `last_error`, and `progress` fields
to the `Task` model and ensure persistence via `TaskPersistenceService`.
- Upgrade `SchedulerService` to use differential synchronization with
the persistent `SQLAlchemyJobStore` for better job durability.
- Implement structured heartbeat logging to support real-time progress
updates.
- Update project documentation and ADRs to reflect the new plugin
runtime and task resilience patterns.
- Add comprehensive unit and integration tests for the new task
lifecycle features.
Introduce a deployment recording system to track dashboard versions
across environments and improve the Git management user experience.
- Add `Deployment` model and Alembic migration to persist deployment
history.
- Implement `GitDeploymentRecorder` and `GitFingerprint` plugins to
automate deployment logging and content hashing.
- Add `get_deployment_status` API endpoint to retrieve real-time
environment states.
- Refactor `GitLifecycleHeader` to prioritize Call-to-Action (CTA)
buttons and improve visual hierarchy.
- Update `GitWorkspacePanel` to emphasize version saving and
streamline commit workflows.
- Enhance `GitEnvironmentTimeline` with deployment status integration,
collapsible UI, and improved theme consistency.
- Add auto-navigation logic in `GitManagerModel` to guide users to
relevant tabs based on recommended actions.
- Clean up obsolete documentation and update i18n strings for
git visualization features.
Implement a new Git environment timeline component to visualize dashboard
versions across different deployment stages (Development, Pre-production,
and Production). This includes new backend endpoints and frontend
services to support historical data retrieval and version comparison.
- Add `get_branch_commits` and `get_commit_diff` endpoints to backend
API and Git service.
- Implement `GitEnvironmentTimeline` Svelte component for visual
representation of deployment history.
- Update `GitManagerModel` to manage timeline state, including
environment histories and version selection for comparison.
- Add `getBranchCommits` and `getCommitDiff` methods to `gitService`.
- Improve UX by separating the branch selector from the version map.
- Add comprehensive i18n support for the new visualization features.
- Add pendingRemovals tracking and isEventRemoving() to maintenance store for better in-flight UX in EventsTable.
- Strongly type maintenance API client and store (replace unknown[]).
- Migrate selectedTask/taskLogs to dedicated lib/stores/selectedTask.svelte.ts (proper rune store with separate subscribers).
- Remove legacy central stores.svelte.ts and related dead code (src/pages/, old tests) to comply with ADR-0006 (global stores must be individual files in lib/stores/).
- Update mocks, components, models and tests accordingly.
- Fix related bugs (subscriber decoupling, test mocks).
- Add dedicated test_selectedTask.ts.
- Minor cleanups: semantic tokens, @RATIONALE for model-first, contracts update.
- All key tests (maintenance, selectedTask, store) pass.
Refs: ADR-0006, specs/031-maintenance-banner
Remove several legacy agent configuration files from the `.kilo/` directory and add new unit tests for task CRUD operations in the backend API routes.
- delete .kilo/agent/swarm-master.md
- delete .kilo/agents/closure-gate.md
- delete .kilo/agents/mcp-backend-coder.md
- delete .kilo/agents/swarm-master.md
- delete .kilo/agents/tester.md
- add backend/src/api/routes/__tests__/test_tasks_crud.py
- update backend/src/api/routes/__tests__/conftest.py to include test database URLs
Root cause: saveJob() used stale sourceDatasourceId (set once on load)
instead of live datasourceId (updated by ConfigTabForm via $bindable).
Since sourceDatasourceId was always truthy, the || fallback to datasourceId
never triggered — the old datasource ID was always sent to PUT.
Fixes:
- Removed dead sourceDatasourceId atom; saveJob() uses datasourceId directly
- Bound sourceTable via $bindable through ConfigTabForm; updated on select
- loadDatasourceColumns() syncs databaseDialect from Superset columns API
- saveJob() sends undefined for database_dialect="unknown" to force re-detect
- Backend: added direct_db+connection_id validation on update (mirroring create)
- Removed redundant "Язык ист." column from TranslationPreview table
- Removed unused getDetectedLang function
Tests:
- TranslationJobModel: datasourceId save mapping, dialect sync, unknown→undefined
- TranslateJobService: reject direct_db without connection_id, preserve existing
Verified: browser — translate_cross datasource persisted after save+reload,
dialect detected as "clickhouse", columns loaded (2), translation column preserved.
Previous lazy-import fix still required langchain at function call time.
Root cause: langchain_openai/langchain_core are only in requirements-agent.txt,
not requirements-backend.txt. The /api/agent/llm-status endpoint runs in
the backend container which has 'openai' but not 'langchain'.
Rewrite _check_llm_provider_health() to use:
- AsyncOpenAI from 'openai' package (in both containers)
- httpx to call /api/agent/llm-config on localhost (same as agent does)
- system_ssl_context() for SSL (same as LLM test endpoint)
- openai exceptions (APIConnectionError, APITimeoutError, etc.)
No langchain dependency at all — works in both backend and agent containers.
Verified: 977 tests passed, import without langchain succeeds.
_llm_health.py imported langchain_core, langchain_openai, and openai at
module level. These packages are only installed in the agent container
(requirements-agent.txt), not the backend container (requirements-backend.txt).
Moved all langchain/openai imports inside _check_llm_provider_health() with
ImportError handled gracefully — returns 'unavailable' status instead of
ModuleNotFoundError 500 error.
Root cause: the /api/agent/llm-status endpoint runs in the backend container,
which has httpx but not langchain. The agent container has all LLM deps.
Verified: import without langchain succeeds, health check returns 'unavailable'.
- Footer.svelte: replace hardcoded '2025' with new Date().getFullYear()
- build.sh: append git short hash to APP_VERSION for all bundle commands
(bundle, bundle:embeddings, bundle:light). Format: '0.5.2+6f029903'.
Previously only the tag was passed (e.g. '0.5.2'), and .git was not
available in Docker context so vite.config.js fell back to '0.0.0'.
Verified: frontend assets contain '0.5.2+6f029903', getFullYear() replaces
hardcoded 2025.
SSL fix (ADR-0009 Finding 7):
- Replace ssl.create_default_context() with system_ssl_context(capath)
in client_registry.py, async_network.py, notifications/providers.py,
git/_base.py. Previous fix (0.5.1) only covered LLM clients; the
Superset API client still used ssl.create_default_context() which
loads cafile (flat bundle) where OpenSSL 3.x ignores intermediate CA
certificates. system_ssl_context() uses capath only (hash symlinks).
Agent fix:
- Extract _check_llm_provider_health + _llm_status from agent/app.py
into agent/_llm_health.py. The /api/agent/llm-status endpoint was
importing from agent/app.py which triggers 'import gradio' at module
level. Backend container does not have gradio installed, causing
ModuleNotFoundError (500 error) every 30s on health check polling.
Config:
- Add ALLOWED_ORIGINS to docker-compose.yml + docker-compose.enterprise-clean.yml
ADR-0009 updated: Layer 2 table expanded with 4 missing clients,
Finding 5 reconciled (LLM_CA_CERT_URLS restored in 0.5.1), version 0.5.2.
Verified: 1110 unit tests passed, gradio import isolation confirmed.
Backend entrypoint sources certs.sh for CA certificate installation,
but the file was never copied into the Docker image — causing
container crash loop on startup (regression in 7f932610).
Changes:
- docker/backend.Dockerfile, docker/all-in-one.Dockerfile: COPY certs.sh
- docker/backend.entrypoint.sh: remove dead install_llm_ca_certs,
install_certificates (replaced by docker/certs.sh); update @INVARIANT
- backend/src/core/ssl.py: fix describe_context() to report actual
system cert count (SSLContext.capath attr does not exist in Python 3)
- __tests__: rewrite stale tests asserting LLM_SSL_VERIFY=false →
verify=False; behaviour is permanently removed — always CERT_REQUIRED
- backend/tests/integration/test_backend_container.py [new]: 5 tests
verifying certs.sh presence, sourceability, and full-stack smoke
(testcontainers PG → entrypoint → migrations → health)
- conftest.py: restore health-wait loop and fixtures in superset_container
- ADR-0009, README.md, scripts, .env: align docs with centralized SSL
Production SSL failure for lite.ai.rusal.com confirmed: LLM_CA_CERT_URLS
not set → LLM CA certs not downloaded. System CA store has corporate
certs from /opt/certs but lite.ai.rusal.com uses a different CA.
Diagnostic script scripts/diag_container.py covers all 6 check categories
per ADR-0009: env, system CA, openssl capath/cafile, Python SSLContext,
httpx connectivity, encryption health.
Fix: set LLM_CA_CERT_URLS in .env.enterprise-clean or place the CA .crt
in ./certs/ on the host. Run diag_container.py to verify.
Modal now mounts at document.body level via svelte mount/unmount.
Extracted WizardContent.svelte for portal rendering.
Eliminates 32px margin from page layout wrapper
(max-w-7xl/space-y-6/px-4 container).
Added KeyRecoveryWizard as portal wrapper, WizardContent as
the actual modal component.
Critical (C1): Added has_permission('security', 'READ') to /health and
/fingerprint endpoints; has_permission('security', 'WRITE') to /recover.
Previously any authenticated user could enumerate encrypted secrets and
overwrite stored values.
High (Bug #1): Fixed recover endpoint — updated/failed status now correctly
reflects whether a non-empty replacement value was submitted. Empty values
now report 'skipped' instead of falsely 'updated'.
High (Bug #2, #3): Replaced all hardcoded English strings in
KeyRecoveryWizard.svelte and SystemSettings.svelte with $t.settings.*
i18n lookups. Keys already existed in en/ru settings.json.
High (H1): Added _sanitize_error() helper to truncate + scrub exception
messages before logging, preventing potential secret leakage in log output.
Cleanup: Moved LLMProviderConfig import to module top. Instantiate
ConnectionService once before for-loop. Added @TEST_EDGE declarations
and @PRE/@SIDE_EFFECT to C4 contract.
Tests: 226 passed
Canonical variables:
- AUTH_SECRET_KEY — JWT signing key for backend + agent (was split across
AUTH_SECRET_KEY / JWT_SECRET)
- SERVICE_JWT — agent→backend service token
- No JWT_SECRET fallback: decoder fails with migration guidance if only
JWT_SECRET is set
Compose files unified:
- docker-compose.yml, docker-compose.enterprise-clean.yml,
docker-compose.e2e.yml — all use AUTH_SECRET_KEY
- build.sh generated compose now passes AUTH_SECRET_KEY + ENCRYPTION_KEY
to backend
Env examples unified and completed:
- .env.example — comprehensive template, all compose vars
- .env.enterprise-clean.example — production template
- backend/.env.example — backend-only run
- docker/.env.agent.example — agent-only run
- NEW: .env.current.example, .env.master.example,
.env.e2e.example, frontend/.env.example
Tests aligned:
- conftest sets AUTH_SECRET_KEY (canonical value matched across test files)
- test mocks use canonical name
- 1176 passed, 0 failed
Replaced src.core.auth.jwt.decode_token import with local _jwt_decoder.py:
- Tries JWT_SECTRET first, falls back to AUTH_SECTRET_KEY (avoids key mismatch
when both env vars are set in different test files)
- verify_aud disabled — backend tokens have audience; agent ignores
- No auth DB dependency — removed AUTH_DATABASE_URL requirement from agent
Cleanup:
- Removed src/core/auth/ from agent Dockerfile COPY
- Removed AUTH_SECTRET_KEY, AUTH_DATABASE_URL from agent compose env
- conftest sets AUTH_SECTRET_KEY for test consistency
- Updated test patches to new import path
Tests: 1176 passed, 0 failed
- Added python-jose[cryptography] to requirements-agent.txt
- Made sqlalchemy.orm.Session and TokenBlacklist imports lazy in jwt.py
so decode_token works without pulling ORM models into agent image
- Added src/core/auth/__init__.py, config.py, jwt.py to agent Dockerfile COPY
- Added AUTH_SECRET_KEY env var to agent compose (reads from JWT_SECRET)
