3133e50645
perf: fix translate deadlock, speed, trace_id, UI bugs — fullstack patch
...
## Backend (7 production files + 6 test files)
### P0-2: LLM output truncation cascade fix
- _token_budget.py: OUTPUT_PER_ROW_PER_LANG 120→200, OUTPUT_SAFETY_FACTOR 0.70→0.55
- Prevents finish_reason=length → split → retry cascade (3 calls → 1 call per batch)
- P2-8: added qwen-flash/qwen-plus/qwen-max/qwen-coder to PROVIDER_DEFAULTS
### P1-4/P1-5: EncryptionManager singleton
- encryption.py: get_encryption_manager() process-wide singleton
- llm_provider.py: use singleton instead of new EncryptionManager() per batch
- Eliminates ~90 redundant Fernet key validations per translation run
### P1-6: Cache-hit log aggregation
- _batch_proc.py: one log per batch (batch_rows + cache_hits) instead of per-row
- 1076 log lines → ~30 per run
### P1-7: Timezone-aware datetime fix
- scheduler.py: _ensure_aware() helper for naive DB datetime → UTC-aware
- Fixes TypeError in scheduled translation concurrency check
### P2-9: Connection test timeout
- connection_service.py: asyncio.wait_for(15s) on all dialect tests
- Prevents 2-minute UI hangs from DNS/TCP stalls
### Trace ID propagation
- middleware/trace.py: inject x-trace-id response header via ASGI send wrapper
### Test fixes & integration tests
- test_scheduler.py: AsyncMock for execute_run, mock get_async_job_runner
- test_sql_insert_service.py: AsyncMock for execute_sql
- test_token_budget.py: batch_size 50→45 for new OUTPUT_PER_ROW_PER_LANG=200
- test_encryption.py: +2 singleton tests
- test_scheduler_ensure_aware.py: +4 (naive→aware, passthrough, None, subtraction)
- test_batch_classify_persist.py: +2 cache-hit aggregation tests
- test_connection_service_edge.py: +2 timeout tests
- test_trace_middleware.py: +4 x-trace-id header tests
- test_token_budget.py: +4 qwen-flash/O200 tests
## Frontend (7 production files + 5 test files)
### Trace ID propagation
- api.ts: _captureTraceId() reads x-trace-id → setTraceId() in fetchApi/requestApi/postApi/deleteApi
### Duplicate datasource columns fetch
- ConfigTabForm.svelte: guard availableColumns.length === 0 before fetch
### Admin pages Svelte 5 runes fix
- admin/users/+page.svelte: plain let → () for all template-bound vars
- admin/roles/+page.svelte: same fix
- Both pages were stuck on «Загрузка...» due to mixed reactivity models
### Validation popover positioning
- +page.svelte: pass trigger HTMLElement instead of event
- DashboardHubModel.svelte.ts: toggleValidationPopover(HTMLElement), closeValidationPopover()
- Added X close button + click-outside overlay + i18n
### Test fixes & integration tests
- api.test.ts: mock setTraceId/getTraceId, +3 _captureTraceId tests
- provider_config.integration.test.ts: handleDelete→promptDeleteProvider
- DatasetPreview.test.ts: dashboards/ → ROUTES.dashboards
- test_config_tab_form.svelte.js: +2 columns fetch guard tests (NEW)
- admin-users.test.ts: +3 loading→table tests (NEW)
- admin-roles.test.ts: +2 loading→table tests (NEW)
## Semantic curation
- Removed @COMPLEXITY N from 6 route files + metrics.py (duplicate of [C:N])
- Added [C:N] to 2 orphan child contracts in metrics.py
- Added [C:N] + @BRIEF to 4 frontend anchors
- Fixed #region → # #region consistency in validation_tasks.py
## Verification
- Backend: 608 pytest passed (0 failures)
- Frontend: 2472 vitest passed (128 files, 0 failures)
- Frontend build: ✓ built in 18s
- Browser: dashboards, admin/users, admin/roles, validation popover — all green
2026-06-18 23:54:57 +03:00
8e8a3c3235
feat: attention-optimized semantic protocol v2.7
...
Core changes:
- Add @defgroup/@ingroup to 1791 C2+ contracts (555 files) for HCA 128× pre-training DSA grouping
- Add §0.1 Pre-Training Frequency matrix to semantics-core
- Add §VIII Attention Architecture rules (ATTN_1-4) with MLA/CSA/HCA/DSA mechanics
- Add @defgroup/@ingroup to canonical syntax (§II) and all contract examples
Agent prompts (5 files):
- Add ZERO-STATE RATIONALE with MLA/CSA/HCA/DSA compression mechanics
- Add pre-training note: @RATIONALE/@REJECTED are in-context learned tags
- svelte-coder: add missing #region contract, fix Svelte rule violations
- python-coder/fullstack-coder: honor function contracts from speckit plan
- qa-tester: add attention compliance audit (P3 ATTN_1-4 checks)
Skills (6 files):
- Translate all axiom_config descriptions to English
- Fix doc_dirs to index .opencode/ and .specify/
- Deduplicate 5× complexity_rules → single global_tags catalog
- Reduce semantics-svelte 591→485 lines (remove duplicate code blocks)
- Fix semantics-testing: 'Short IDs' → 'Short hierarchical IDs'
- Fix all examples: flat IDs → hierarchical Domain.Name format
- Fix Svelte examples: replace raw Tailwind + <button> with semantic tokens + /ui
Speckit workflow (commands + templates):
- speckit.plan: add Function-Level Contracts for C3+ with @PRE/@POST/@TEST_EDGE
- speckit.plan: add Attention Compliance Gate (ATTN_1-4 before contract generation)
- speckit.tasks: add function contract inlining format (constraints in task description)
- speckit.specify: load semantics-core for spec density rules
- spec-template: add #region contract, @SEMANTICS grouping, hierarchical IDs
- ux-reference-template: add #region wrapper
- plan-template: add attention gate, @defgroup/@ingroup guidance
- tasks-template: add attention audit + rebuild + orphan check tasks
- constitution.md: translate to English, add Principle VIII (attention-optimized contracts)
Reference modules rewritten (hierarchical IDs + full contracts):
- Auth.Jwt: 6 child contracts with @RATIONALE/@REJECTED/@TEST_EDGE
- Api.Auth: 5 endpoints with @TEST_EDGE + molecular CoT markers
- Migration.Model: @defgroup Migration with 18 @ACTION + 6 @INVARIANT
Scripts:
- add_defgroup_ingroup.py: zero-risk additive @ingroup migration (1791 insertions)
- migrate_hierarchical.py: flat→hierarchical ID dry-run analysis (792 contracts)
- merge_prompts.py: merge all prompts/skills/commands into one review file
Config:
- axiom_config.yaml: 749→395 lines (-47%), English, doc_dirs include prompts
- Fix test_datasets.py import collision (rename → test_datasets_routes.py)
- Fix test_preview.py: SupersetClient→get_superset_client, AsyncMock, logger f-string
2026-06-08 16:30:59 +03:00
814f2da139
perf(translate): fix slow translation startup — CJK estimation, output budget, provider token config
...
Root cause: batch sizing underestimated CJK token density (1.5→1.0 chars/token)
and ignored output budget as primary constraint, causing cascading finish_reason=length.
Changes:
- _token_budget.py: CJK_RATIO 1.5→1.0, OTHER_RATIO 2.2→1.8, safety factors 0.75/0.70
- _token_budget.py: new _compute_max_rows_by_output() — output budget is PRIMARY constraint
- _batch_sizer.py: resolve_provider_config() with DB-level context_window/max_output_tokens
- _batch_sizer.py: INPUT_SAFETY_FACTOR applied, max_rows_by_output used as row cap
- _llm_http.py: log actual usage.prompt_tokens/.completion_tokens from provider
- _llm_call.py: retry only missing rows after finish_reason=length (save partial result)
- models/llm.py + schema: provider-level context_window / max_output_tokens (nullable)
- services/llm_provider.py: get_provider_token_config() helper
- Alembic migration: add columns to llm_providers
- Svelte ProviderConfig: collapsible Advanced: Token Limits section
- 12 new tests (token budget, batch sizer, provider config)
- All 492 tests pass
2026-06-03 23:25:08 +03:00
5e5f958eaa
feat(llm): auto-detect max images per request for LLM providers
...
Add binary-search probe endpoint POST /providers/{id}/probe-max-images
that discovers the per-request image limit by sending incrementally
more 1x1 JPEGs via the provider's own API. Result is cached in the
new max_images column on the provider config.
- LLMProviderConfig: add max_images: int | None
- LLMProvider (SQLAlchemy): add max_images column
- Migration ed28d34edde7: clean ADD COLUMN
- LLMProviderService: create/update/set_max_images
- POST /providers/{id}/probe-max-images: binary search + error parsing
- ProviderConfig.svelte: 'Detect' button in edit modal + HelpTooltip
- i18n (en/ru): 11 new keys for probe UI
2026-05-31 22:31:36 +03:00
a9a453109c
security: critical auth fixes, test migration to SQLite+FK, 44 test fixes
...
CRITICAL (CVE-class):
- C-4: Remove DEV_MODE fallback with hardcoded postgres:postgres credentials
- C-3: WebSocket endpoints now require JWT or API key token (?token=) auth
- C-1: Superset environment passwords encrypted via Fernet at rest in DB
- H-4: SESSION_SECRET_KEY separated from JWT AUTH_SECRET_KEY
- H-1: Log injection via %s-formatting instead of f-strings
- H-5: X-Trace-ID header validated as UUID4 to prevent trace poisoning
INFRASTRUCTURE:
- New src/core/encryption.py — EncryptionManager extracted from llm_provider
- Test DB: per-module sqlite:///:memory: with PRAGMA foreign_keys=ON
- testcontainers PostgreSQL via TEST_DB=postgres env var
- conftest temp-file global engine (no 10GB shared-cache leak)
TEST FIXES (44 pre-existing → 0):
- test_smoke_plugins: module-level sys.modules mock isolated to per-test fixture
- test_migration_engine: EXT:Python:uuid → uuid syntax fix
- test_dashboards_api: mock env attributes + correct patch target
- test_constants_audit_fixes: sync expected constant names with actual
- test_defensive_guards: patch.object instead of module-level Repo mock
- test_clean_release_cli: removed empty config.json directory
- FK violations: TaskRecord parents in log_persistence, Environment in
mapping_service, ReleaseCandidate in candidate_manifest_services
ORTHOGONAL TESTS (18 new):
- test_security_orthogonal.py: bcrypt 72-byte limit, unicode, API key
format invariants, Fernet robustness, encryption key lifecycle,
log injection protection, JWT edge cases
MODEL FIXES:
- MetricSnapshot.job_id: nullable with SET NULL (was broken FK for
aggregate prune snapshots, hidden by SQLite)
CLEANUP:
- Removed stale :memory:test_main/test_auth/test_tasks file databases
- Removed duplicate #endregion in encryption_key.py
2026-05-26 14:58:49 +03:00
9ffa8af1dc
semantics
2026-05-26 09:30:41 +03:00
68740cd9ed
semantic
2026-05-20 23:54:53 +03:00
084f782065
lang detect
2026-05-20 17:15:31 +03:00
58ac89c21e
fix(translate): token budget overhaul + truncation retry + smart batch sizing + LLM provider fixes
...
Token Budget (_token_budget.py):
- DEFAULT_MAX_OUTPUT_TOKENS 8192→16384 (adequate for 50 rows×4 langs)
- Add PROVIDER_DEFAULTS with per-model context_window/max_output_tokens
- OUTPUT_PER_ROW_PER_LANG 60→120, add JSON_OVERHEAD_PER_ROW=50
- PROMPT_BASE_TOKENS 300→600, MAX_OUTPUT_HEADROOM 1000→3000
- CJK-aware token estimator (_estimate_tokens_for_text: 1.5 chars/tok for CJK)
- Output-aware batch sizing (_apply_output_aware_batch_sizing)
- Warn on dictionary cap mismatch
Executor (executor.py):
- Return finish_reason from _call_openai_compatible→_call_llm
- Truncation detection + batch splitting on finish_reason=length
- Smart batch sizing (_auto_size_batches): greedily split by row token budget
- Fix disable_reasoning hardcoded 8192→use calculated max_tokens
- TypeError guard on choices[0] access, base_url validation
- Broadened response_format fallback (matches response_format|structured|json_object)
- Remove invalid extra_body, fix partial-recovery regex for integer row_id
- 18 new tests for estimate_row_tokens and _auto_size_batches
Preview (preview.py):
- Align token constants with _token_budget.py (CHARS_PER_TOKEN=2.2, OUTPUT=120)
- Return finish_reason, broadened response_format fallback
- Fix hardcoded 8192, base_url validation
LLM Services:
- render_prompt: detect unfilled {placeholders}, log WARNING
- llm_provider: distinguish crypto exceptions (InvalidTag vs ValueError vs generic)
- Respect Retry-After header on HTTP 429
2026-05-16 09:58:31 +03:00
30ba70933d
feat(llm): mask API keys in UI responses and prevent masked-key leakage in fetch/test/save payloads
...
- Add mask_api_key() and is_masked_or_placeholder() to llm_provider service
- Return masked keys in all provider CRUD endpoints
- Reject masked/placeholder keys in fetch_models and test_provider_config
- Show masked key with Change button in ProviderConfig.svelte edit form
- Exclude masked keys from fetch-models, test, and submit payloads on frontend
- Update semantics-core skill with clarified complexity tier rules
- Switch agent modes from subagent to all
2026-05-15 23:34:09 +03:00
c6189876b3
chore(lint): apply ruff --fix (4443 auto-fixes)
...
Auto-fixed categories:
- F401: unused imports removed
- I001: import blocks sorted
- W293: trailing whitespace stripped
- UP035: deprecated typing imports replaced
- SIM: simplify suggestions applied
- ARG: unused args prefixed with underscore
- T201: print statements removed
- F841: unused variables removed
- RUF059: unpacked variables prefixed
Remaining ~1500 unfixable errors (C901, B904, N806, E402) require manual work.
Backend smoke tests: 13/13 passed.
2026-05-14 11:20:17 +03:00
a3c7c402b7
fix(llm): add fetch-models endpoint, fix SQL Lab INSERT (client_id truncation, sync mode, target_column, timestamp normalization)
...
- Add POST /api/llm/providers/fetch-models route with LLMClient.fetch_models()
- Add target_column to TranslationJob model/schema/service/orchestrator
- Fix SQL Lab execute: truncate client_id to 11 chars (varchar(11))
- Switch SQL Lab to sync mode (runAsync: false) — no Celery workers
- Fix polling: unwrap nested result from Superset query API
- Fix ClickHouse timestamp: normalize float timestamps to YYYY-MM-DD
2026-05-13 20:06:15 +03:00
39ab647851
semantics
2026-05-13 14:15:33 +03:00
306c5ae742
semantics: complete DEF-to-region migration, fix regressions
...
- Convert legacy [DEF🆔 Type] anchors to #region/#endregion across 329 files
- Reinstate _normalize_timestamp_value in sql_generator.py
- Fix MarkerLogger→logger migration in events.py (molecular CoT markers)
- Fix dataset_review orchestrator dependencies (_build_execution_snapshot)
- Fix config_manager stale-record deletion (moved to save path only)
- Add 77 missing [/DEF:] closers in 5 unbalanced test files
- Update assistant_chat.integration.test.js for #region format
- Apply molecular-cot-logging markers (REASON/REFLECT/EXPLORE) via logger.* methods
2026-05-12 23:54:55 +03:00
fe8978f716
semantics
2026-05-12 20:06:16 +03:00
7901ce0f39
semantics
2026-03-27 21:27:31 +03:00
81406bc2e2
fix: commit semantic repair changes
2026-03-21 11:22:25 +03:00
f9fc2811a2
fix(027): stabilize shared acceptance gates and compatibility collateral
2026-03-17 11:07:49 +03:00
67867f8220
refactor(semantics): migrate legacy @TIER to @COMPLEXITY annotations
...
- Replaced @TIER: TRIVIAL with @COMPLEXITY: 1
- Replaced @TIER: STANDARD with @COMPLEXITY: 3
- Replaced @TIER: CRITICAL with @COMPLEXITY: 5
- Manually elevated specific critical/complex components to levels 2 and 4
- Ignored legacy, specs, and node_modules directories
- Updated generated semantic map
2026-03-16 10:06:44 +03:00
4efce79df5
Fix LLM validation and dashboard health hot paths
2026-03-15 13:18:51 +03:00
5e6fe2fe61
security: rotate bootstrap and clean workspace
2026-03-13 12:14:37 +03:00
1bf7b53bae
test contracts
2026-02-26 19:40:00 +03:00
cbf9135db6
semantic update
2026-02-24 21:08:12 +03:00
5ab1549cdd
linter + новые таски
2026-02-10 12:53:01 +03:00
0743641181
semantic update
2026-02-08 22:53:54 +03:00
2d09ff4dc8
Похоже работает
2026-02-07 11:26:06 +03:00
161d70648c
Вроде работает
2026-01-30 11:10:16 +03:00
