49a566359a
feat: translate module — runtime knobs, GRACE anchors, two-layer testing, QA fixes
...
Implementation:
- Performance knobs: llm_batch_max_rows, llm_concurrency, insert_concurrency,
multi_lang_mode, batch_aggressiveness, max_in_flight_batches
- Alembic migration f7a8b9c0d1e2 (idempotent, nullable, non-destructive)
- LLM provider capabilities: throughput_class, reasoning_control,
supports_json_object, default/max_llm_concurrency
- TargetSchemaValidationRequest with conditional validator (sqllab/direct_db)
- Scheduler: background dispatch, local imports for lazy bootstrap
GRACE-Poly compliance:
- Semantic anchors on orchestrator_aggregator, orchestrator_sql, llm_provider
- Shared module _llm_http.py: _apply_reasoning_control extraction (INV_4)
- Alembic migration anchors per C3/C1 template
- Four renamed Svelte components: RunOutcomeSummary, DetectionQualityCard,
LanguageStatList, SourceLanguageOverride
QA (this session):
- 11 backend test regressions fixed: spec'd MagicMock null fields for new
columns, _check_translation_cache_bulk retarget, scheduler mock wiring,
language_detection=auto assertions, token budget constant update
- 4 frontend eslint errors fixed: SvelteSet/SvelteDate imports,
unused lang parameter, dead isTransientError
- Production bugfix: job_to_response() mapped 6 missing fields
- Pre-existing auth test failure documented (dependencies.py untouched)
- Axiom: 6440 contracts, 0 warnings
2026-07-20 14:19:47 +03:00
31b9a19a0c
chore: commit remaining workspace updates
...
Agent:
- lifecycle: run tracking, middleware hardening, langgraph setup
- tests: agent lifecycle + langgraph setup coverage
Backend:
- async_job_runner: resilience hardening, tests
- agent_conversations: run lifecycle integration
- translate: scheduler + orchestrator SQL adjustments
- schemas/services: agent_lifecycle model extensions
Frontend:
- TaskDrawer: UX improvements
- TaskLogPanel/Viewer: safety hardening, i18n (en/ru)
- FilterBar: report filters contract + tests
- Reports page: layout adjustments
Specs:
- 036-agent-test-stabilization: runs contract, modules, events
- 037-superset-baseline-engine: catalog schema, testing API, modules
- 038-dashboard-scenario-model: scenario schema, capture profile, modules
- 039-dashboard-scenario-ui: screen models, release verification UX, modules
- dashboard-verification-usecases: new cross-cutting spec
2026-07-17 19:11:09 +03:00
8f0d123ff8
fix migration resume and release workflow
2026-07-16 12:31:59 +03:00
45ce585aba
chore: commit remaining workspace updates
2026-07-16 07:53:41 +03:00
20105f51c0
feat(translate): add run preflight and focus execution UX
2026-07-16 07:52:52 +03:00
20071b8c7a
security: fullstack hardening — task ownership, mapping validation, API-key scoping, test fixes
...
Backend:
- Add validate_mapping_database_ownership() to verify source/target UUIDs
belong to declared environments before persisting mappings (mappings.py)
- Add API-key environment scoping to get_mappings (filter) and
suggest_mappings_api (enforce) (mappings.py)
- Add user_id Column to TaskRecord model + Alembic migration (task.py)
- Persist task.user_id on save, restore on load (persistence.py)
- Wire current_user.id into migrate_dashboards + backup_dashboards
task creation (_action_routes.py)
- Fix test_migration_routes.py: module-level patch leak → autouse fixture,
SupersetClient→AsyncSupersetClient, AsyncMock for sync_environment/run
- Fix 7 Pydantic serializer warnings: 'PENDING'→TaskStatus.PENDING
in test_tasks.py + import TaskStatus
Frontend:
- Deepen isDryRunResult(): validate selection field, risk.items entries
(all 5 fields), and diff object uuids individually (ExecutorModel.svelte.ts)
Prior work included: task password redaction, resume ownership checks,
canonical dry-run DTO alignment, migration UI callback fixes, credential
exposure reduction, assistant dry-run await fix.
2026-07-15 23:02:23 +03:00
30c8acf7ae
fix(translate): handle invalid LLM JSON responses
2026-07-15 20:06:39 +03:00
9922d7c87a
fix(translate,scheduler): language detection, LLM error handling, and scheduler persistence
...
Translation pipeline fixes (from production error log analysis):
Lang detect:
- Add "ru" to _COMMON_SOURCE_CODES for Cyrillic source detection
- Version detector cache keys (v2:) and include detector version in
source hash — stale cache entries from old algorithm are invalidated
- Integrate _character_block_fallback into batch_detect() pipeline;
only assign Cyrillic fallback when exactly one Cyrillic target exists
(multiple ru/uk/be targets stay undetermined for LLM arbitration)
Batch processing:
- Case-insensitive cache language matching (cached_by_lang lookup)
- Propagate needs_review=True for undetected language rows in pre/cache path
LLM call (critical — fixes silent error hiding):
- Validate LLM row IDs against expected set; log unknown identifiers
- Retry only missing rows on incomplete response (bounded by recursion depth)
- Exhausted retries → FAILED (new _handle_incomplete_response method)
- Parse failures → FAILED instead of SKIPPED (_handle_parse_failure)
- NULL/Empty translations → FAILED via new _add_failed helper
- Source-language identity mapping preserved as TranslationLanguage entry
(carries source_language_detected metadata that _build_insert_rows
relies on for detected_src_lang derivation)
- finish_reason propagated to parser for truncation diagnostics
LLM parse:
- Structured incomplete-set logging with expected/received/missing counts
Target schema validation:
- Handle all non-success SQL Lab statuses (failed, timeout, error, stopped)
- Timeout gets explicit message prefix
- Route returns HTTP 502 with correct HTTPException passthrough
Scheduler persistence fix (production pickle error):
- Backup and validation APScheduler jobs now use module-level callbacks
(execute_scheduled_backup, execute_scheduled_validation) instead of
bound SchedulerService methods — prevents pickle failure from
serializing TaskManager + dynamically loaded plugin classes
- Callback func identity verified via pickle round-trip smoke test
Tests:
- Update assertions: FAILED replaces SKIPPED for LLM error paths
- Restore source-language contract tests with identity-mapped values
- Add scheduler callback identity and args verification tests
- Update detector cache key tests for versioned format
- Update target schema error route test: 200→502
Orthogonal code review: MEDIUM finding (empty rec.languages when all
targets match detected source → _build_insert_rows fallback to "und")
fixed by preserving source-language TranslationLanguage entries.
2026-07-15 13:07:41 +03:00
8f4ee25415
fix(alembic): merge three migration heads + add smoke test for chain integrity
...
- Created merge migration 7eaf84b7f6be joining heads:
- 6b8ca3b7405f (previous merge of c0d1e2f3a4b5 + f2b3c4d5e6f7)
- b4c5d6e7f8a9 (include_source_reference to translation_jobs)
- f4a5b6c7d8e9 (preproduction validation to deployment records)
- Added smoke test (test_smoke_migration_chain.py) that:
- Checks exactly 1 head (catches branch divergence)
- Walks full chain verifying all down_revision links exist
- Confirms all .py files are loaded as revisions
- Runs WITHOUT a database (real ScriptDirectory, no mocks)
- Catches what existing tests missed:
* test_alembic_migrations.py skips on non-PostgreSQL
* test_check_migration_chain.py uses mocks, not real files
2026-07-14 17:33:16 +03:00
9b3cc54646
feat: add backup integrity verification
2026-07-14 16:05:28 +03:00
c3ad0afc17
refactor: remove rejected dataset review feature
2026-07-14 15:56:31 +03:00
2a56ea5fc9
test: fix backend and frontend test contracts
2026-07-14 10:43:38 +03:00
66497da72b
feat(mapper): secure xlsx upload and dataset selection
2026-07-14 00:34:10 +03:00
d630573402
fix(backup): orthogonal code review fixes + UI/UX overhaul
...
## Critical fixes
- H1: Fix env_id/env/environment_id triple inconsistency in API route
(_action_routes.py now passes 'environment_id', matches scheduler)
- H2: Decompose BackupPlugin.execute() — CC 13 → 5 methods, CC ≤ 5 each
- H3: Fix unhandled int() ValueError on non-numeric dashboard_ids
- H4: Add concurrent guard test with API-style params (env key fallback)
- H5: Make RetentionPolicy configurable via StorageConfig
(retention_daily/weekly/monthly in config model + backend plugin)
- Fix: storage DELETE route missing 'await' on async delete_file (bug)
## UI/UX overhaul
- NEW: centralized cron utility (frontend/src/lib/utils/cron.ts)
- validateCron(), calcNextCronRun(), formatNextRun()
- Used by BackupManager, BackupDashboardModal
- BackupManager: replace custom BackupList with shared FileList
- Adds: download, delete, bulk actions, search, sort, pagination
- Adds: cron next-run preview below schedule input
- Adds: auto-open TaskDrawer on backup task creation
- BackupDashboardModal: dead 'Cron Help' button removed
- Adds: live cron validation + next-run preview
- StorageSettings: adds retention daily/weekly/monthly fields
- BackupCreateRequest type fixed to match actual API contract
- i18n: 7 new keys for en/ru (next_run, retention_*)
2026-07-13 20:58:02 +03:00
c2bd6cb441
feat(git): clarify dashboard release flow
2026-07-13 12:55:24 +03:00
00d2619c86
fix: align DRAFT integration test with removed DRAFT check in validate_job_preconditions
...
- DRAFT validation was removed from orchestrator_validation.py (moved to service layer)
- Integration test expected ValueError for DRAFT jobs — fixed to test actual behavior
- 194/194 integration tests pass, 0 failures
2026-07-12 19:44:11 +03:00
a39a76c87f
feat(agent-centric-logging): consolidate CoT infra in shared, close REASON→REFLECT chains
...
- shared/cot_logger.py is SSOT; backend/cot_logger.py deleted
- elapsed_ms timing in all REFLECT markers
- Frontend: REASON→REFLECT/EXPLORE in all fetch/post/delete/requestApi
- Dynamic src: route.GET.api.plugins instead of hardcoded api.request_handler
- trace_id generated immediately (no 'no-trace'), X-Trace-ID in both directions
- Global error handlers (window error + unhandledrejection + error.svelte)
- Fixed duplicate logging (shared/logger.py double StreamHandler)
- propagate=False in configure_logger (was in ConfigManager = duplicated startup logs)
- belief_scope: 'Coherence OK' → '{anchor}: completed' + elapsed_ms
- Fixed 28 pre-existing test failures (scheduler sig, DB columns, DRAFT validation, etc)
2026-07-12 19:30:57 +03:00
24d3b7d1f9
refactor(task-manager): implement task resilience and execution lifecycle improvements
...
Enhance the reliability and observability of the task execution engine
by introducing retry mechanisms, idempotency, and structured progress
tracking.
- Implement centralized retry logic with exponential backoff support
in `JobLifecycle`.
- Add `retry_task` API endpoint and `TaskManager` method for manual
task restarts.
- Introduce task idempotency using `_idempotency_key` to prevent
duplicate executions.
- Add `retry_count`, `max_retries`, `last_error`, and `progress` fields
to the `Task` model and ensure persistence via `TaskPersistenceService`.
- Upgrade `SchedulerService` to use differential synchronization with
the persistent `SQLAlchemyJobStore` for better job durability.
- Implement structured heartbeat logging to support real-time progress
updates.
- Update project documentation and ADRs to reflect the new plugin
runtime and task resilience patterns.
- Add comprehensive unit and integration tests for the new task
lifecycle features.
2026-07-12 15:31:56 +03:00
0cb1f80cd6
feat(git): implement deployment tracking and enhance lifecycle UX
...
Introduce a deployment recording system to track dashboard versions
across environments and improve the Git management user experience.
- Add `Deployment` model and Alembic migration to persist deployment
history.
- Implement `GitDeploymentRecorder` and `GitFingerprint` plugins to
automate deployment logging and content hashing.
- Add `get_deployment_status` API endpoint to retrieve real-time
environment states.
- Refactor `GitLifecycleHeader` to prioritize Call-to-Action (CTA)
buttons and improve visual hierarchy.
- Update `GitWorkspacePanel` to emphasize version saving and
streamline commit workflows.
- Enhance `GitEnvironmentTimeline` with deployment status integration,
collapsible UI, and improved theme consistency.
- Add auto-navigation logic in `GitManagerModel` to guide users to
relevant tabs based on recommended actions.
- Clean up obsolete documentation and update i18n strings for
git visualization features.
2026-07-12 14:57:03 +03:00
b39d9991b9
fix(logs): resolve old prod execution issues + full GRACE-Poly compliance
...
- translate: pure-skip scheduled runs now COMPLETE (not FAILED); extract _compute_final_status
- scheduler: cron validation at update; pre-validate + skip legacy bad expr (/15*) in load
- git: robust get_branch_commits (existence check for 'prod' to avoid fatal)
- health: use explore for network/auth failures (reduces spam)
- Full semantic protocol updates across modules:
- complete #region/#endregion + @BRIEF/@PRE/@POST/@INVARIANT/@RELATION/@RATIONALE
- belief_scope + CoT markers (reason/explore/reflect)
- semantics-testing compliance in tests
- Addresses exact symptoms from container_backend.log and backend/logs/app.*
Related to 039-dashboard-scenario-ui log analysis.
2026-07-10 12:35:07 +03:00
3b13c67c0d
feat(translate): language detection, async HTTP LLM, history model, agent improvements
...
- Add async HTTP-based LLM transport (_llm_async_http.py)
- Add orthogonal LLM call tests
- Improve language detection (_lang_detect.py) and batch insert
- Update translate schemas, service utils, preview constants/prompts
- Add TranslateHistoryModel with pagination and filtering
- Update agent confirmation, persistence, langgraph setup, run, tools
- Improve LLM health checking in shared module
- Update translate runs API, history route
2026-07-08 19:35:49 +03:00
d0facf4a3a
fix(translate): datasource change not persisted on save + preview column cleanup
...
Root cause: saveJob() used stale sourceDatasourceId (set once on load)
instead of live datasourceId (updated by ConfigTabForm via $bindable).
Since sourceDatasourceId was always truthy, the || fallback to datasourceId
never triggered — the old datasource ID was always sent to PUT.
Fixes:
- Removed dead sourceDatasourceId atom; saveJob() uses datasourceId directly
- Bound sourceTable via $bindable through ConfigTabForm; updated on select
- loadDatasourceColumns() syncs databaseDialect from Superset columns API
- saveJob() sends undefined for database_dialect="unknown" to force re-detect
- Backend: added direct_db+connection_id validation on update (mirroring create)
- Removed redundant "Язык ист." column from TranslationPreview table
- Removed unused getDetectedLang function
Tests:
- TranslationJobModel: datasourceId save mapping, dialect sync, unknown→undefined
- TranslateJobService: reject direct_db without connection_id, preserve existing
Verified: browser — translate_cross datasource persisted after save+reload,
dialect detected as "clickhouse", columns loaded (2), translation column preserved.
2026-07-07 23:50:08 +03:00
34aeeb92a2
fix(backend+frontend): migration deadlock, async pattern, timeout safety, fire-and-forget translations
...
Backend:
- MigrationPlugin.execute — remove AsyncJobRunner.run() deadlock on post-migration
ID sync (2 deadlocks total: plugins/migration.py + api/routes/migration.py
trigger_sync_now). Replace blocking runner.run with direct await.
- MigrationPlugin.execute — fix temp file leak (dry_run=True prevented cleanup).
- MigrationPlugin.execute — IdMappingService(SessionLocal()) now closed in finally.
- MigrationPlugin.execute — wire IdMappingService into MigrationEngine constructor
so cross-filter patching actually works instead of silently skipping.
- MigrationPlugin.execute — SupersetClient.aclose() in finally to prevent
httpx connection pool leak.
- TaskManager — _async_tasks dict leak (add_done_callback cleanup).
- JobLifecycle — CancelledError handler (tasks stuck in RUNNING after cancel).
- JobLifecycle — persist_task BEFORE _broadcast_task_status (crash consistency).
- JobLifecycle — wait_for_resolution/wait_for_input now have 3600s timeout.
- AsyncJobRunner.run — 300s timeout on future.result() (APScheduler thread safety).
Translate scheduler:
- execute_scheduled_translation — replace blocking runner.run(orch.execute_run(run))
with fire-and-forget TranslationOrchestrator.execute_background(). APScheduler
thread is freed in ms instead of blocking for the full translation duration.
Translation runs of 10k+ rows (200+ LLM batches, hours) no longer hit the
300s runner timeout.
- TranslationOrchestrator.execute_background — new static method: opens own DB
session, dispatches asyncio.create_task, handles errors + notification.
- Scheduler last_run_at updated at dispatch time (not after completion).
Frontend:
- MigrationModel.stepReady[3] now requires dryRunResult != null (was always
true, allowing UI to reach step 3 without dry-run).
- WizardModel.goToStep gate for step 3 uses stepReady[3].
- +page.svelte — dryRunResult no longer self-clears via reactive loop.
- Progress bar step 3 indicator gate fixed for new readiness logic.
Tests:
- 2 new regression tests for migration sync (deadlock-free, completes cleanly).
- test_migration_plugin.py — _make_mock_superset_client/_make_mock_mapping_service
helpers for proper async mock behavior (aclose, sync_environment AsyncMock).
- 10 scheduled-translation tests updated for fire-and-forget pattern.
- MigrationModel.test.ts — step 3 invariant + goToStep block test.
- All affected tests: 104 backend + 79 frontend = 183 passed.
2026-07-07 21:04:23 +03:00
3f6d7222c3
fix(agent): proxy gradio config and reduce translate blocking
2026-07-07 17:37:36 +03:00
cfb95fa3c8
refactor(agent): extract agent+shared into standalone packages with full GRACE semantic markup
...
- Move agent code from backend/src/agent/ to agent/src/ss_tools/agent/
- Extract shared stdlib-only utilities to shared/src/ss_tools/shared/
- Add #region/#endregion contracts to all ~140 functions (INV_1 compliance)
- Update docker files, entrypoint, build scripts for new package layout
- Backend now imports ss_tools.shared._llm_health (no gradio/langchain deps)
- Add specs for 036-039 feature plans
2026-07-07 15:18:24 +03:00
1182b29fe0
fix(certs): restore HTTP CA auto-download and test edge matrix
...
Restore LLM_CA_CERT_URLS support for corporate PKI HTTP cert delivery.
Downloaded certificates are installed into the llm/ CA category and share
the same system trust + NSS import path as CERTS_PATH-mounted certs.
Changes:
- certs.sh: add download_llm_ca_certs with HTTP-only curl policy,
PEM/DER handling, retries, llm/ storage, hash symlinks for llm+custom
- backend/agent entrypoints: run download before install_all_certs
- compose/env/docs: expose LLM_CA_CERT_URLS again and update ADR-0009
- integration tests: cover PEM/DER/CER, invalid certs, non-HTTP skips,
query-string filenames, private/server skips, empty inputs, combined
LLM_CA_CERT_URLS + CERTS_PATH channels, and NSS imports
Verified:
- 194 passed, 1 skipped integration tests
- full backend container smoke passed with testcontainers PostgreSQL
- docker bundle rebuilt for 0.5.0
2026-07-07 10:23:49 +03:00
702ee25ae4
fix(docker): add missing certs.sh COPY to backend/all-in-one Dockerfile
...
Backend entrypoint sources certs.sh for CA certificate installation,
but the file was never copied into the Docker image — causing
container crash loop on startup (regression in 7f932610
2026-07-07 00:58:41 +03:00
aafd89f2c7
test(integration): add encrypted key Superset container test with SSL_KEY_PASSPHRASE
...
conftest.py: superset_container fixture supports {encrypted_key: True}
- Writes encrypted private key to container
- Decrypts with openssl rsa -passin env:SSL_KEY_PASSPHRASE
- Starts Flask/Werkzeug TLS server with decrypted key
- Falls back to unencrypted key for existing tests
test_superset_tls_custom_ca.py:
- NEW: TestEncryptedPrivateKeySuperset class
- test_superset_health_over_tls_with_encrypted_key
- test_openssl_capath_encrypted_key
- Uses parametrize({tls: True, encrypted_key: True})
Validates full production flow: encrypted key → SSL_KEY_PASSPHRASE
→ openssl decrypt → TLS handshake → client trust via capath.
2026-07-06 21:21:42 +03:00
7f93261060
refactor(ssl): centralize SSL trust management, remove LLM_SSL_VERIFY
...
Centralized SSL via one contract: CERTS_PATH=/opt/certs mounted into all containers.
Backend:
- NEW: backend/src/core/ssl.py — system_ssl_context(), httpx_verify(),
cert_dir_inventory()
- LLMClient._get_ssl_verify() → delegates to core.ssl
- _llm_async_http._get_verify() → delegates to core.ssl
- Removed LLM_SSL_VERIFY env reading from all runtime code
Docker:
- NEW: docker/certs.sh — shared cert installer (PEM/DER/cer to .crt conversion,
update-ca-certificates, hash symlinks, NSS import)
- NEW: docker/agent.entrypoint.sh — agent entrypoint with cert installation
- backend.entrypoint.sh → uses certs.sh instead of install_llm_ca_certs
- Dockerfile.agent → adds ca-certificates, openssl, entrypoint
Compose:
- Removed LLM_CA_CERT_URLS and LLM_SSL_VERIFY from all compose files
- Added CERTS_PATH volume mount to agent (dev + enterprise)
- Added certs volume mount to backend/agent in dev compose
Env examples:
- Removed LLM_SSL_VERIFY, LLM_CA_CERT_URLS from .env.example,
.env.enterprise-clean.example, .env.current.example, .env.master.example,
backend/.env.example
- Enhanced CERTS_PATH comments with accepted formats
Diagnostics:
- diag_container.py: removed LLM_* checks, added CERTS_PATH inventory,
uses core.ssl for context creation
Tests:
- Updated test_llm_analysis_service, test_llm_async_http,
test_client_headers to verify centralized ssl context (no env disable)
- 4/4 SSL tests pass
2026-07-06 21:00:28 +03:00
2716f15804
refactor(env): unify Docker env vars — canonical AUTH_SECRET_KEY, remove JWT_SECRET fallback
...
Canonical variables:
- AUTH_SECRET_KEY — JWT signing key for backend + agent (was split across
AUTH_SECRET_KEY / JWT_SECRET)
- SERVICE_JWT — agent→backend service token
- No JWT_SECRET fallback: decoder fails with migration guidance if only
JWT_SECRET is set
Compose files unified:
- docker-compose.yml, docker-compose.enterprise-clean.yml,
docker-compose.e2e.yml — all use AUTH_SECRET_KEY
- build.sh generated compose now passes AUTH_SECRET_KEY + ENCRYPTION_KEY
to backend
Env examples unified and completed:
- .env.example — comprehensive template, all compose vars
- .env.enterprise-clean.example — production template
- backend/.env.example — backend-only run
- docker/.env.agent.example — agent-only run
- NEW: .env.current.example, .env.master.example,
.env.e2e.example, frontend/.env.example
Tests aligned:
- conftest sets AUTH_SECRET_KEY (canonical value matched across test files)
- test mocks use canonical name
- 1176 passed, 0 failed
2026-07-06 14:24:17 +03:00
ef3ce378bf
refactor(agent): lightweight JWT decoder — JWT_SECTRET first, AUTH_SECTRET_KEY fallback
...
Replaced src.core.auth.jwt.decode_token import with local _jwt_decoder.py:
- Tries JWT_SECTRET first, falls back to AUTH_SECTRET_KEY (avoids key mismatch
when both env vars are set in different test files)
- verify_aud disabled — backend tokens have audience; agent ignores
- No auth DB dependency — removed AUTH_DATABASE_URL requirement from agent
Cleanup:
- Removed src/core/auth/ from agent Dockerfile COPY
- Removed AUTH_SECTRET_KEY, AUTH_DATABASE_URL from agent compose env
- conftest sets AUTH_SECTRET_KEY for test consistency
- Updated test patches to new import path
Tests: 1176 passed, 0 failed
2026-07-06 13:45:22 +03:00
dcee86c260
refactor(docker): split requirements by role, slim enterprise bundle by default
...
Split requirements.txt into role-specific files:
- requirements-backend.txt — FastAPI API runtime
- requirements-agent.txt — Gradio/LangGraph agent (no embeddings)
- requirements-embeddings.txt — opt-in sentence-transformers
Default enterprise bundle (./build.sh bundle) builds slim agent without
sentence-transformers/torch (~500 MB saved). Embeddings opt-in via
./build.sh bundle:embeddings <tag>. Embedding router degrades gracefully
when package is absent (ImportError catch in _embedding_router.py).
Updated Dockerfiles:
- backend.Dockerfile → requirements-backend.txt
- Dockerfile.agent → requirements-agent.txt + WITH_EMBEDDINGS build arg
- all-in-one.Dockerfile → requirements-backend.txt
Added embeddings variant to build.sh with full manifest/digest output.
Hardened .dockerignore (cache, report, model, archive patterns).
Other branch work: align git route mocks with async services, remove
legacy agent test files superseded by contract tests.
2026-07-06 10:06:07 +03:00
6c123cbc73
test(git): align route mocks with async services
2026-07-06 01:23:20 +03:00
0256425c89
test(backend): update legacy agent and auth expectations
2026-07-06 01:23:09 +03:00
5d80f83869
feat(reports): add task status settings and tests
2026-07-06 01:22:39 +03:00
f607c920b8
test(agent-chat): audit guardrail and error handling
2026-07-06 01:20:12 +03:00
1bb6a5b5a7
feat(agent-chat): complete context guardrail event coverage
2026-07-05 14:14:42 +03:00
aa10be99d1
feat(agent-chat): 035-agent-chat-context — контекст, guardrails, tools, database discovery
...
== User stories ==
US1: Контекст с дашборда/датасета → /agent с URL params
US2: Guardrails card — env badge, 7 risk tones, countdown, permission_denied
US3: Tools optimization — retry, timeout, summarise, RBAC + context affinity
== Backend ==
- _context.py (NEW): UIContext validation (7 checks)
- _tool_filter.py (NEW): RBAC + context affinity pipeline
- _confirmation.py: build_confirmation_contract_v2, permission_denied_payload
- tools.py: superset_list_databases, retry/summarise/timeout wrappers
- app.py: _inject_uicontext, _inject_env_id_into_tools, database
prefetch в runtime context
- _persistence.py: prefetch_databases()
- agent_superset_explore.py: GET /databases endpoint
- _llm_async_http.py, _persistence.py: fix double /v1 в LLM URL
(LM Studio Unexpected endpoint)
== Frontend ==
- AgentChatModel.svelte.ts: 5 atoms, 3 actions, countdown, context
- AgentChat.svelte: production banner, process steps, debug panel
- ConfirmationCard.svelte: 7 risk tones, permission_denied, countdown
- ToolCallCard.svelte: retrying/timeout/cancelled states
- StreamProcessor.svelte.ts: tool_retry, timeout, permission_denied
- TopNavbar: sparkles icon + Ассистент
- sidebarNavigation: AI section
- DashboardHeader, datasets/+page: contextual AI buttons
- Icon: sparkles, brain, cpu icons
- tailwind: assistant category colors
- i18n: en/ru nav keys
== Tests ==
- 159 backend agent tests (+16 US3: retry, timeout, summarise, contracts)
- 2544 frontend tests (+11 model + component tests)
- 15 JSON fixtures (10 API + 5 model)
== Specs ==
- specs/035-agent-chat-context/: spec, UX, plan, tasks, research,
data-model, contracts, quickstart, traceability, fixtures, checklists
Closes #035
2026-07-04 22:47:17 +03:00
ece8d7d256
cleanup: убрать мёртвые env-переменные, консолидировать чтение в agent/_config.py
...
Удалены из кода:
- JWT_SECRET — мёртвая (decode_token использует AUTH_SECRET_KEY)
- SESSION_SECRET_KEY — заменён на прямой AUTH_SECRET_KEY
- POSTGRES_URL — deprecated fallback, удалён из database.py и reencrypt.py
Консолидировано чтение env-переменных agent-модуля:
- Создан agent/_config.py — единый модуль для FASTAPI_URL,
SERVICE_JWT, GRADIO_*, STORAGE_ROOT, AGENT_* (9 констант)
- Все agent/*.py импортируют из _config вместо разрозненных os.getenv
Удалены or-дефолты (безопасность):
- agent/langgraph_setup.py — удалён hardcoded DB URL postgres:postgres
- agent/langgraph_setup.py — удалены fallback API URL и model name
- scripts/reencrypt.py — удалён hardcoded DB URL postgres:postgres
- plugins/llm_analysis/service.py — удалены or-дефолты URL/app name
.env.example — минимализация:
- backend/.env.example: только 4 обязательные переменные
- root/.env.example: обязательные + docker + SSO/админ
Обновлены тесты (139 passed)
2026-07-04 14:58:43 +03:00
b1efc38306
Improve agent UX and spec sync
2026-07-03 16:47:10 +03:00
3fae6add87
test(tls): add encrypted (passphrase-protected) private key integration tests
...
- ca_chain fixture: add server_key_encrypted (BestAvailableEncryption) +
server_key_passphrase to returned dict; existing fields unchanged
- TestEncryptedPrivateKey: 6 new tests covering
- ssl.SSLContext.load_cert_chain() with correct/wrong/missing passphrase
- asyncio SSL server end-to-end with encrypted key + full chain trust
- openssl rsa -check with wrong/missing passphrase (CLI validation)
- All 6 tests pass; 14/14 in test_superset_tls_custom_ca.py
2026-07-03 16:46:12 +03:00
6209647700
feat(reports): Task Status Center — unified /reports dashboard
...
Страница /reports трансформирована в Центр статусов задач:
Backend:
- GET /api/reports/summary — агрегированные счётчики тип×статус (5 корзин)
- GET/PUT /api/settings/reports — глобальные настройки отчётов
- _filter_tasks_by_rbac() — row-level фильтрация по роли
- normalize_task_report: LLM-валидация с ошибками → FAILED/PARTIAL
- get_summary(): 5 корзин pending/running/awaiting_input/success/failed
Frontend:
- TaskCenterModel.svelte.ts (400 строк) — Screen Model
- SummaryPanel — сводная панель с цветовым кодированием и active filter
- ReportCard — humanized labels, duration, task_id, failed border
- FilterBar — search + sort + time range с label'ами
- Pagination — showingText, уникальные id для select
- Quick views: «Упавшие», «В работе», «Успешные»
- TaskDrawer: scroll-to-error, footer скрыт для terminal, «Н/Д» fix
Тесты: 48 backend + 38 frontend (2521 всего)
Build: ✅ Console errors: 0
2026-07-02 18:53:58 +03:00
db998ce085
feat(semantic): curator-driven protocol hardening — decision memory + relation repair
...
- Add @RATIONALE/@REJECTED to 103+ C4/C5 contracts across backend core, services, API routes, and frontend models
- Fix 109 unresolved @RELATION edges (Auth.*, SupersetClient.*, AgentChat.*, ADR cross-refs)
- Add 13 @ingroup tags for DSA/HCA attention grouping
- Repair 29 stale graph edges via index rebuild
- Update .kilo agent prompts and skills for GRACE-Poly v2.6 compliance
- Git integration: merge routes, branch lifecycle, remote providers, UX components
- 0 broken anchor pairs, index rebuilt with 0 parse warnings
2026-07-02 08:53:19 +03:00
87ac90bb8d
fix(git): fix 17 missing async/await bugs + UX overhaul
...
Backend:
- fix 17 missing 'await' in git route handlers causing silent no-ops
(branches, diff, history, commit, push, pull, merge, promote, sync)
- fix async coroutine passed to run_blocking in git_plugin.py
Frontend:
- add collapsible 'How it works' onboarding (GitHelpPanel)
- add status legend with color-coded repository statuses
- i18n: add 50+ missing keys, replace hardcoded strings
- add Refresh button in modal header
- add PROD deploy confirmation dialog (replaces browser prompt())
- add CommitHistory to workspace tab with timeline nodes
- add post-commit success banner with next-step guidance
- increase success toast duration to 8s
- group local/remote branches in selector (optgroup)
- format last_modified dates timezone-aware
- change PROD badge from red to neutral indigo
- extract shared resolveGitStatusToken to git-utils.ts
- fix 'slug' label regression
- remove dead init_repo_button key
UI/UX audit fixes:
- add descriptions to Create/Init buttons in init panel
- add actionable CTA to server mismatch warning
- improve checkbox text phrasing
2026-07-01 20:47:25 +03:00
78d2664e2e
fix(agent): minimal safety net, zero-config router, LLM provider status UI
...
- Remove deterministic intent matching (keyword lists, infer_tool,
fast_confirmation, negation guard, classification sets)
- Embedding descriptions auto-generated from tool docstrings
- LLM provider health endpoint GET /api/agent/llm-status
- 3 error codes: LLM_PROVIDER_UNAVAILABLE, LLM_TIMEOUT, LLM_AUTH_ERROR
- Frontend banner with auto-retry 30s + input disable
- i18n for LLM status messages (assistant.json ru/en)
- 138 passing backend tests
2026-07-01 16:47:21 +03:00
ec6d46ea70
tasks 033 updated
2026-06-30 19:05:17 +03:00
f3ff12a221
feat(agent-superset): extend SupersetClient with agent-critical methods + DDL/DML guard + tests (126 passed)
...
Ported from mcp-superset research module, integrated into existing async SupersetClient:
New mixins (4 files):
- safety.py: DDL/DML guard (13 keywords), comment/string stripping, viz_type validation
- _sql_lab.py: execute_sql, format_sql, results, estimate, CSV export, query history
- _saved_queries.py: saved queries CRUD (5 methods)
- _audit.py: permissions audit matrix (user × dashboard × dataset × RLS)
Extended mixins (4 files):
- _dashboards_write.py: +create, +update (general), +copy, +publish, +unpublish
- _dashboards_crud.py: +standalone get_dashboard_charts/datasets, +get_dashboard
- _datasets.py: +create, +delete, +duplicate, +refresh_schema, +get_or_create, +export/import
- _databases.py: +update, +test_connection, +schemas/tables/catalogs, +validate_sql, +select_star, +table_metadata
FastAPI proxy (2 files):
- agent_superset.py (284L): SQL Lab + Dashboards/Datasets write endpoints
- agent_superset_explore.py (240L): DB explore + Audit + Saved Queries endpoints
Agent tools (2 files):
- tools.py: +7 LangChain @tools (24 total), intent-routing keywords
- _tool_resolver.py: updated SAFE/GUARDED/FAST_CONFIRM classification sets
Tests (3 files, 126 tests):
- test_superset_safety.py (51): DDL/DML bypass/legitimate/safe scenarios
- test_superset_extended.py (42): all mixin methods with mock AsyncAPIClient
- test_superset_tools.py (33): agent tool registration, intent matching, @tool .ainvoke()
2026-06-30 17:26:51 +03:00
f8bfdeb1f6
chore: remaining pre-existing changes (storage, stream processor, tests, run.sh)
2026-06-30 15:21:15 +03:00
b61252667d
feat(agent+ui): fullstack agent module refactoring + UI/UX improvements
...
## Backend: agent module GRACE-Poly compliance
- Split app.py (749→~280 lines) into _tool_resolver, _confirmation, _persistence
- All 18 naked functions wrapped in #region/#endregion contracts
- Fixed @DEFGROUP→@defgroup typos; added @DATA_CONTRACT, @SIDE_EFFECT, CoT logs
- Conversation list API: added last_role, has_tool_calls, has_error, risk_level fields
- Message state detection: Russian/English error patterns (недоступен, unavailable)
- State field preserved in save_conversation messages
- HITL titles: descriptive tool names instead of generic "HITL resume"
## Backend: conversation title generation (two-layer)
- Layer 1: clean_title() — rule-based, strips file markers, pre-fetch blocks, JSON/CSV,
URLs, code; truncates at 80 chars word boundary (25 unit tests, all edge cases)
- Layer 2: generate_llm_title() — async best-effort LLM titling via /v1/chat/completions
with per-conversation lock, graceful degradation on failure
## Frontend: conversation list indicators (orthogonal system)
- Status dot (green/yellow/red/blue) per conversation state
- Icon column: tool activity, errors, waiting, completed
- Risk stripe (left border accent) + message count badge + relative time
- Fixed group labels: "Сегодня"/"Вчера" instead of "3 ч"/"5 ч"
- Hide "Окружение: —" when env is empty
## Frontend: guardrails card verification + fixes
- Confirmed all interaction modes: Enter/click confirm, Escape/click deny
- Auto-populate envId from environmentContextStore in DashboardDetailModel
- Better error message: missing_context_hint with recovery guidance
## Design system: semantic tokens
- Added category-* gradient tokens to tailwind.config.js
- Sidebar + Breadcrumbs use semantic tokens (10 categories)
- Raw Tailwind reduced from ~50 to 6 occurrences
- Added skip-to-content link in root layout (+layout.svelte)
- Added aria-label on DashboardDataGrid row checkboxes
## Protocol: INV_7 pragmatic exception
- Modules may exceed 400 lines when contract-dense (every function has #region)
- Recorded in semantics-core SKILL.md with rationale
Total: 5841+ contracts, 2993+ edges, backend 41/41, frontend 2501/2501
2026-06-30 15:21:05 +03:00
576fff8cc6
fix(agent-chat): streaming state leak, document parser magic bytes, HITL flow
...
### Bugfixes — Agent Chat 'Думаю' State Leak
- fix(agent-chat): loadHistory() now resets streamingState/idle + cancels stale
submission — prevents 'Думаю' state leak across conversation switches
- fix(agent-chat): onDisconnected/onDisconnectedPermanent cascade to
streamingState — prevents permanent hang on connection loss during stream
- fix(agent-chat): guard on isLoadingHistory — prevents false commit
of 'agent unavailable' fallback when switching conversations
- fix(agent-chat): remove race in _sendNow empty-response check vs Svelte
microtask (duplicate logic removed, handles correctly)
- fix(stream-processor): confirm_resolved now appends msg.text to partialText
instead of dropping it
### Bugfixes — Backend PDF Upload
- fix(document-parser): _detect_format_by_magic() — reads file header magic
bytes as fallback when Gradio loses filename
- fix(document-parser): improved name extraction — tries orig_name, path stem
- fix(document-parser): @RELATION AgentChatTypes -> AgentChat.Types
### HITL Flow & Agent Chat Improvements
- feat(agent): HITL resume confirm/deny with userId/userJwt/envId propagation
- feat(agent): confirm_required metadata fallback via aget_state() after
'Event loop is closed' error during interrupt
- feat(agent): interrupt_before re-enabled via AGENT_CONFIRM_TOOLS env var
- feat(frontend): debug panel with connection/stream state monitoring
- feat(frontend): AgentChatModel constructor options + onBeforeSend callback
- feat(frontend): crypto.randomUUID() for local conversation ID on first send
### Backend Agent Refactoring
- refactor(agent): langgraph_setup — monkey-patch for PydanticSerializationError
- refactor(agent): tools.py — dual identity headers, expanded tool set
- refactor(agent): run.py — _find_free_port, Gradio server port fallback
- refactor(agent): app.py — file size validation, message truncation, HITL path
### Frontend
- feat(dashboard-hub): DashboardHubModel with filters, pagination, git actions
- feat(ui): DateRangeFilter component
- feat(i18n): new dashboard keys; cache tooltips fix
- fix(i18n): full run tooltips — cache is NOT ignored
### Semantic Protocol
- chore(agents): update all agents with canonical format
- chore(skills): sync semantics-core, semantics-contracts, molecular-cot-logging
### Housekeeping
- chore: remove stale semantic reports (10 files, Jan 2026)
- chore: update 033-gradio-agent-chat specs, contracts, UX, tasks, tests
- chore: add .agents/ directory (mirrors .opencode/ agent layouts)
- chore: update run.sh with DEV_MODE, port management
2026-06-29 17:15:25 +03:00
