perf: fix translate deadlock, speed, trace_id, UI bugs — fullstack patch

## Backend (7 production files + 6 test files)

### P0-2: LLM output truncation cascade fix
- _token_budget.py: OUTPUT_PER_ROW_PER_LANG 120→200, OUTPUT_SAFETY_FACTOR 0.70→0.55
- Prevents finish_reason=length → split → retry cascade (3 calls → 1 call per batch)
- P2-8: added qwen-flash/qwen-plus/qwen-max/qwen-coder to PROVIDER_DEFAULTS

### P1-4/P1-5: EncryptionManager singleton
- encryption.py: get_encryption_manager() process-wide singleton
- llm_provider.py: use singleton instead of new EncryptionManager() per batch
- Eliminates ~90 redundant Fernet key validations per translation run

### P1-6: Cache-hit log aggregation
- _batch_proc.py: one log per batch (batch_rows + cache_hits) instead of per-row
- 1076 log lines → ~30 per run

### P1-7: Timezone-aware datetime fix
- scheduler.py: _ensure_aware() helper for naive DB datetime → UTC-aware
- Fixes TypeError in scheduled translation concurrency check

### P2-9: Connection test timeout
- connection_service.py: asyncio.wait_for(15s) on all dialect tests
- Prevents 2-minute UI hangs from DNS/TCP stalls

### Trace ID propagation
- middleware/trace.py: inject x-trace-id response header via ASGI send wrapper

### Test fixes & integration tests
- test_scheduler.py: AsyncMock for execute_run, mock get_async_job_runner
- test_sql_insert_service.py: AsyncMock for execute_sql
- test_token_budget.py: batch_size 50→45 for new OUTPUT_PER_ROW_PER_LANG=200
- test_encryption.py: +2 singleton tests
- test_scheduler_ensure_aware.py: +4 (naive→aware, passthrough, None, subtraction)
- test_batch_classify_persist.py: +2 cache-hit aggregation tests
- test_connection_service_edge.py: +2 timeout tests
- test_trace_middleware.py: +4 x-trace-id header tests
- test_token_budget.py: +4 qwen-flash/O200 tests

## Frontend (7 production files + 5 test files)

### Trace ID propagation
- api.ts: _captureTraceId() reads x-trace-id → setTraceId() in fetchApi/requestApi/postApi/deleteApi

### Duplicate datasource columns fetch
- ConfigTabForm.svelte: guard availableColumns.length === 0 before fetch

### Admin pages Svelte 5 runes fix
- admin/users/+page.svelte: plain let → () for all template-bound vars
- admin/roles/+page.svelte: same fix
- Both pages were stuck on «Загрузка...» due to mixed reactivity models

### Validation popover positioning
- +page.svelte: pass trigger HTMLElement instead of event
- DashboardHubModel.svelte.ts: toggleValidationPopover(HTMLElement), closeValidationPopover()
- Added X close button + click-outside overlay + i18n

### Test fixes & integration tests
- api.test.ts: mock setTraceId/getTraceId, +3 _captureTraceId tests
- provider_config.integration.test.ts: handleDelete→promptDeleteProvider
- DatasetPreview.test.ts: dashboards/ → ROUTES.dashboards
- test_config_tab_form.svelte.js: +2 columns fetch guard tests (NEW)
- admin-users.test.ts: +3 loading→table tests (NEW)
- admin-roles.test.ts: +2 loading→table tests (NEW)

## Semantic curation
- Removed @COMPLEXITY N from 6 route files + metrics.py (duplicate of [C:N])
- Added [C:N] to 2 orphan child contracts in metrics.py
- Added [C:N] + @BRIEF to 4 frontend anchors
- Fixed #region → # #region consistency in validation_tasks.py

## Verification
- Backend: 608 pytest passed (0 failures)
- Frontend: 2472 vitest passed (128 files, 0 failures)
- Frontend build: ✓ built in 18s
- Browser: dashboards, admin/users, admin/roles, validation popover — all green
This commit is contained in:
2026-06-18 23:54:57 +03:00
parent 4a6fe8db58
commit 3133e50645
59 changed files with 1774 additions and 13928 deletions

View File

@@ -35,6 +35,7 @@
# @SIDE_EFFECT Opens/closes native DB connection. Network I/O.
# #endregion
import asyncio
import time as time_module
from typing import Any
@@ -78,6 +79,10 @@ def _validate_name_uniqueness(
# @ingroup Core
# @BRIEF Service class wrapping all DatabaseConnection CRUD + test operations.
class ConnectionService:
# Hard cap on connection test duration — prevents UI hangs when
# DNS/TCP handshake stalls (observed 2+ min in production).
TEST_CONNECTION_TIMEOUT_SEC = 15.0
def __init__(self, config_manager: ConfigManager):
self.config_manager = config_manager
self._encryption: EncryptionManager | None = None
@@ -253,13 +258,22 @@ class ConnectionService:
start = time_module.time()
try:
# Use dialect-appropriate test
# Use dialect-appropriate test with hard timeout
if conn.dialect in ("postgresql",):
version = await self._test_postgresql(conn)
version = await asyncio.wait_for(
self._test_postgresql(conn),
timeout=self.TEST_CONNECTION_TIMEOUT_SEC,
)
elif conn.dialect == "clickhouse":
version = self._test_clickhouse(conn)
version = await asyncio.wait_for(
asyncio.to_thread(self._test_clickhouse, conn),
timeout=self.TEST_CONNECTION_TIMEOUT_SEC,
)
elif conn.dialect == "mysql":
version = self._test_mysql(conn)
version = await asyncio.wait_for(
asyncio.to_thread(self._test_mysql, conn),
timeout=self.TEST_CONNECTION_TIMEOUT_SEC,
)
else:
return {"success": False, "error": f"Unsupported dialect: {conn.dialect}"}
@@ -269,6 +283,13 @@ class ConnectionService:
"latency_ms": elapsed_ms,
"db_version": version,
}
except asyncio.TimeoutError:
elapsed_ms = int((time_module.time() - start) * 1000)
return {
"success": False,
"latency_ms": elapsed_ms,
"error": f"Connection test timed out after {self.TEST_CONNECTION_TIMEOUT_SEC:.0f}s",
}
except Exception as e:
elapsed_ms = int((time_module.time() - start) * 1000)
return {

View File

@@ -64,6 +64,9 @@ def _require_fernet_key() -> bytes:
# @SIDE_EFFECT Initializes Fernet cryptography state from process environment.
# @DATA_CONTRACT Input[str] -> Output[str]
# @INVARIANT Uses only a validated secret key from environment.
# @RATIONALE Fernet key validation (Fernet(key)) is O(1) but repeated ~90× per translation
# run — cached at module level via get_encryption_manager() singleton to eliminate
# redundant env-var reads and key validation.
#
# @TEST_CONTRACT EncryptionManagerModel ->
# {
@@ -109,4 +112,31 @@ class EncryptionManager:
# #endregion EncryptionManager
# Module-level singleton — avoids redundant env reads and Fernet construction
# across ~90 LLMProviderService instantiations per translation run.
_encryption_manager: EncryptionManager | None = None
# #region get_encryption_manager [C:4] [TYPE Function] [SEMANTICS encryption,singleton,cache]
# @BRIEF Return the process-wide EncryptionManager singleton.
# @PRE ENCRYPTION_KEY env var is set to a valid Fernet key.
# @POST Returns the same EncryptionManager instance on every call (idempotent).
# @SIDE_EFFECT Creates EncryptionManager on first call (reads env, validates Fernet key).
# @RATIONALE Eliminates redundant ENCRYPTION_KEY reads and Fernet key validation —
# LLMProviderService was creating ~90 EncryptionManager instances per
# translation run (2 per batch). The singleton reduces this to 1.
def get_encryption_manager() -> EncryptionManager:
"""Return the process-wide EncryptionManager singleton.
Creates on first call; returns cached instance thereafter.
Eliminates redundant ENCRYPTION_KEY reads and Fernet key validation.
"""
global _encryption_manager
if _encryption_manager is None:
_encryption_manager = EncryptionManager()
return _encryption_manager
# #endregion get_encryption_manager
# #endregion EncryptionCore

View File

@@ -38,7 +38,7 @@ import uuid
from starlette.types import ASGIApp, Receive, Scope, Send
from ..cot_logger import seed_trace_id, set_trace_id
from ..cot_logger import get_trace_id, seed_trace_id, set_trace_id
# #region TraceContextMiddleware [C:2] [TYPE Class] [SEMANTICS middleware, trace, context, asgi]
@@ -57,7 +57,7 @@ class TraceContextMiddleware:
# #region TraceContextMiddleware.__call__ [C:2] [TYPE Function] [SEMANTICS asgi, call, trace, seed, header]
# @ingroup Core
# @BRIEF ASGI __call__ that seeds trace_id in the root task context before forwarding.
# @BRIEF ASGI __call__ that seeds trace_id and injects X-Trace-Id response header.
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
if scope["type"] != "http":
await self.app(scope, receive, send)
@@ -82,7 +82,19 @@ class TraceContextMiddleware:
else:
seed_trace_id()
await self.app(scope, receive, send)
# Capture trace_id for response header injection
response_trace_id = get_trace_id()
async def send_with_trace(message: dict) -> None:
if message["type"] == "http.response.start":
headers = list(message.get("headers", []))
headers.append(
(b"x-trace-id", response_trace_id.encode("ascii"))
)
message["headers"] = headers
await send(message)
await self.app(scope, receive, send_with_trace)
# #endregion TraceContextMiddleware.__call__