Commit Graph

355 Commits

Author SHA1 Message Date
e754efc7bd feat(llm): add LiteLLM provider — enum, client, tests, QA follow-ups
- Add LITELLM = "litellm" to LLMProviderType enum (already in HEAD from prev commit)
- Add comment in LLMClient.__init__ explaining standard Bearer auth for LiteLLM
- Add regression test test_provider_type_enum_values — guard against value drift
- Add test_litellm_client_uses_default_bearer_auth — verify no extra headers
- Clean up duplicate @RELATION lines in models.py
- All 20 backend + 4 frontend tests pass
2026-05-15 22:11:55 +03:00
27168664b8 fix(translate): normalize Unix timestamps to YYYY-MM-DD for ClickHouse Date columns
- Add _normalize_timestamp_value to key column values in orchestrator.py and executor.py before SQL generation
- Fix test mocks for .options(joinedload()) chain, explicit None attrs for mock job
- Add global translation run progress indicator (TranslationRunGlobalIndicator + store)
- Fix translate page import missing translationRunStore
- All 208 translate tests pass
2026-05-15 22:06:27 +03:00
fdf48491a1 fix(translate): Kilo API response_format, reasoning_effort skip, structured_outputs fallback, refusal handling
- Enable response_format (json_object) for all providers including Kilo/OpenRouter
- Skip reasoning_effort for Kilo/OpenRouter (returns 400 — mandatory reasoning)
- Add structured_outputs fallback: retry once without response_format if upstream
  provider (e.g. StepFun) rejects it with 'structured_outputs is not supported'
- Handle model refusal field (refusal) instead of treating as empty content
- Fix None-handling guards for message/finish_reason/content fields
- Apply same fixes to both preview.py and executor.py _call_openai_compatible
- Connect orphaned TermCorrectionPopup, BulkReplaceModal, BulkCorrectionSidebar
2026-05-15 18:12:29 +03:00
0fbf8f65bf fix(ui): responsive parent layout - remove max-w-4xl constraint
The job config page was capped at 896px (max-w-4xl), making the
10×4 translation preview table extremely squished. Changed to
max-w-full xl:max-w-7xl — full width on all screens, 1280px max
on ultrawide.
2026-05-15 10:45:33 +03:00
e55f679262 fix(ui): dynamically resizable TranslationPreview table for 100-10000 char texts
- Removed max-w-[200px] compression on source text column → now
  gets proportional space
- Scrollable containers (max-h-32 overflow-y-auto) for long text cells
- Language columns: min-w-[200px]→[250px] with whitespace-pre-wrap
- Source language badge and status columns remain narrow (w-28, w-24)
2026-05-15 10:40:57 +03:00
70aec51e0b fix(translate): partial JSON recovery for truncated LLM responses when DeepSeek runs out of tokens
DeepSeek v4 Flash returns finish_reason=length when reasoning uses
all output tokens. Partial row extraction via regex recovers complete
rows from truncated JSON instead of failing with 400 error.

Also: extra_body reasoning_effort for Kilo/OpenRouter proxied providers
2026-05-15 10:31:51 +03:00
ff4804164a feat(translate): universal reasoning suppression for DeepSeek/Qwen/MiniMax/Kimi
Added disable_reasoning toggle that works via:
1. API parameter: reasoning_effort: 'none' (DeepSeek, Qwen)
2. System prompt instruction: 'Respond directly with ONLY the JSON.
   Do NOT include any reasoning, thinking, or chain-of-thought'
   (works for ALL models universally)

Backend: model column, schema, preview+executor payload
Frontend: checkbox in LLM settings, i18n en+ru
2026-05-15 10:13:37 +03:00
4641c82397 feat(translate): auto batch_size estimator for LLM token budget
New _token_budget.py calculates safe batch_size and max_tokens based on source text length, target languages, dictionary size, and model context window (64K DeepSeek v4 Flash). preview.py: auto-reduces sample_size when texts are long. executor.py: per-batch dynamic max_tokens. 12 new tests.
2026-05-15 09:53:54 +03:00
365d710806 fix(translate): increase max_tokens 4096→8192 for DeepSeek multi-language output
finish_reason=length from DeepSeek v4 Flash — model runs out of
output tokens due to reasoning_content (CoT) + multi-language (4 langs).
8192 tokens ensures complete JSON response for preview and execution.
2026-05-15 09:46:37 +03:00
5d4649d24e fix(translate): add detailed LLM response logging for empty content debug
Preview + executor now log finish_reason, message_keys, and raw
response structure when LLM returns empty content, enabling
diagnosis of DeepSeek API issues (empty content with finish_reason=length)
2026-05-15 09:28:21 +03:00
7947700188 fix(translate): add EXPLORE logging for LLM JSON parse failures
Preview + executor now log the raw LLM response (first 1000 chars)
when JSON parsing fails, enabling debugging of malformed LLM output.

Previously ValueErrors from JSON parse failures were silently returned
as 400 without any diagnostic data in logs.
2026-05-15 09:15:36 +03:00
a74c50206a feat(translate): enhanced target table column mapping + fix multi-language save
Multi-language save fix:
- MultiSelect.svelte: selected prop changed to $bindable([])
  (without $bindable, bind:selected never propagated to parent component,
  so target_languages always stayed as default ['en'])

Target table column mapping (INSERT enhancement):
- target_language_column — stores language code (e.g. 'ru', 'en')
- target_source_column — stores original source text
- target_source_language_column — stores detected BCP-47 source language
- SQL generator includes these columns in INSERT when configured
- Frontend: 4 input fields under 'Target Column Mapping' section
  with clear labels, placeholders, and i18n hints

Backend: model, schema, service, orchestrator
Frontend: job config page, i18n en+ru
DB: 3 new columns added to production
2026-05-15 09:09:39 +03:00
30d7f3f725 feat(i18n): localize all translate components (spec 028)
P0 — Wired i18n to 4 previously-unlocalized surfaces:
- BulkReplaceModal (~25 strings → translate.bulk_replace.*)
- CorrectionCell (~20 strings → translate.corrections.cell_*)
- TermCorrectionPopup (~30 strings → translate.term_correction.*)
- history/+page.svelte (~50 strings → translate.history.*)

P1 — Added 85+ keys to en.json + ru.json (identical structures):
- 11 previously-missing keys (common.close, config.status, etc.)
- bulk_replace.*, corrections.*, term_correction.*, history.*, dictionaries.*

P2 — Dictionary detail page: import form, filter-by-language, context display
P3 — Russian hardcoded strings in job config replaced with i18n

Tests: 280 pass, build succeeds
2026-05-15 09:00:33 +03:00
77d55c8e69 refactor(translate): remove all deprecated columns from model + DB
Removed 6 legacy columns replaced by Phase 11 multi-language models:
- TranslationJob.target_language → target_languages (JSON)
- TranslationRecord.{llm_translation, user_edit, final_value} → TranslationLanguage
- TerminologyDictionary.{source_language, target_language} → DictionaryEntry per-entry

Affected files: model, schemas, 5 plugins, 4 test files
Alembic migration: aa1b2c3d4e5f (DROP COLUMN IF EXISTS)
Production DB: columns dropped via ALTER TABLE

Tests: 234 passed, 0 failed
2026-05-15 00:06:41 +03:00
6717e2551d fix(translate): add missing target_languages column to production DB
Migration 2a7b8c9d0e1f adds target_languages (JSON) to translation_jobs.
Uses ADD COLUMN IF NOT EXISTS for idempotent PostgreSQL deployment.

Fixes 500 error on GET /api/translate/jobs:
  column translation_jobs.target_languages does not exist

Migration chain is now clean: 5 linear revisions, no orphans.
2026-05-14 23:51:25 +03:00
cc1e57088d chore: remove legacy SQLite migration script + fix stale comments
- Delete migrate_sqlite_to_postgres.py (one-time script, no longer needed)
- Fix @INVARIANT in mappings.py: SQLite → PostgreSQL
- Remove deprecated source_language column from TranslationJob model
  + Alembic migration 8dd0a93af539 to drop column if exists
- Remove source_language from TranslateJobResponse schema

Fixes recurring 'source_language does not exist' scheduler error
2026-05-14 23:48:13 +03:00
fdc452a945 feat(docker): add all-in-one Dockerfile without Playwright (<200MB target)
Multi-stage build:
- Stage 1: node:20-alpine builds frontend (discarded)
- Stage 2: python:3.11-slim runs backend + serves SPA static files
- Playwright removed from deps (saves ~350MB)
- Uvicorn serves both API and frontend on :8000
- Healthcheck via curl
- Entrypoint with admin bootstrap support

Estimated image size: ~190MB (vs ~550MB with playwright)
2026-05-14 21:37:08 +03:00
8bea44f640 chore: update semantic index, agent configs, and app startup 2026-05-14 21:33:26 +03:00
9f3f6611a1 feat(translate): complete Phase 10-11 — full spec 028 closure (T075-T134)
Phase 10 closure:
- T075: NotificationService wiring for scheduled-run failures
- T077: Semantic audit via axiom MCP (0 warnings)
- T078: Quickstart validation — all 165+280 tests pass

Phase 11 completion:
- T083-T086: Multi-language models/schemas/Alembic migration
- T087-T090: Auto-detect source language (BCP-47) + tests
- T091-T095: Multilingual dictionaries with language-pair filtering
- T096-T108: Multi-language job config → preview → execution pipeline
- T109-T118: Inline correction + BulkReplaceModal + CorrectionCell + tests
- T119-T122: Per-language history and MetricSnapshot
- T123-T134: Context-aware corrections (Jaccard similarity, priority flagging, context_data)

New files: Alembic migration, test_scheduler, test_inline_correction,
BulkReplaceModal, vitest for CorrectionCell + BulkReplaceModal

Total: 134/134 tasks complete. Backend 165p, Frontend 280p.
2026-05-14 21:14:04 +03:00
bb21fd3165 fix(translate): M1-M3 medium + L1-L2 low bugs from QA review
M1: ReDoS guard — 500 char max + try/except on re.compile
M2: Performance note — O(n×m) regex documented for future optimization
M3: Verified test key names correct (metrics.py transforms cumulative_* → short)
L1: Removed @staticmethod no-op from module-level function
L2: Added aria-label to emoji indicators in CorrectionCell
2026-05-14 17:44:50 +03:00
1853d008e9 fix(translate): CR1-CR2 migration + H1-H4 code review bugs
CR1: Add needs_review, language_overridden to translation_languages migration
CR2: Add per_language_metrics to translation_metric_snapshots migration
H1: BulkFindReplaceService.preview() now uses replacement_text (not find pattern)
H2: get_run_records now includes per-language data via selectinload
H3: Preview per-language approve/reject uses correct (jobId, rowId, lang) args
H4: _compute_config_hash now includes target_languages
2026-05-14 17:38:52 +03:00
bb0fbfdafd feat(translate): multi-language optimization (Phase 11)
- Auto-detection of source language per row via LLM (US6)
- Multi-target translation — one LLM call for N languages (US1-US3)
- Language-aware storage: TranslationLanguage, per-language stats
- Multilingual dictionaries with language-pair-aware filtering (US7)
- Inline correction on any run result + submit-to-dictionary (US8)
- Context-aware dictionary: auto-capture row context, usage notes,
  Jaccard similarity, priority flagging in LLM prompts (US8b)
- Configurable preview sample size 1-100, cost warning at >30
- Per-language history & metrics with MetricSnapshot preservation
- 36 files, +5022/-373, all specs GRACE-Poly v2.6 compliant
2026-05-14 17:12:41 +03:00
5e741a4332 specks updated 2026-05-14 15:27:41 +03:00
c6189876b3 chore(lint): apply ruff --fix (4443 auto-fixes)
Auto-fixed categories:
- F401: unused imports removed
- I001: import blocks sorted
- W293: trailing whitespace stripped
- UP035: deprecated typing imports replaced
- SIM: simplify suggestions applied
- ARG: unused args prefixed with underscore
- T201: print statements removed
- F841: unused variables removed
- RUF059: unpacked variables prefixed

Remaining ~1500 unfixable errors (C901, B904, N806, E402) require manual work.

Backend smoke tests: 13/13 passed.
2026-05-14 11:20:17 +03:00
a9a5eff518 fix(lint): apply QA-found gaps in agent instruction files
- python-coder.md: ruff check src/ tests/ → ruff check .
- speckit.test.md: add frontend npm run lint
- speckit.plan.md: ruff check . + frontend lint
- speckit.tasks.md: ruff check . + frontend lint
- fullstack-coder.md: move npm run lint to frontend block
2026-05-14 11:19:22 +03:00
9b8c485562 chore: configure ruff + eslint linters for agentic workflow
- Replace pylint with ruff (backend/ruff.toml)
  - line-length=300, max-complexity=10, per-file-ignores for tests
  - Exclude D (docstrings replaced by contracts) and ANN
- Add eslint flat config for frontend (eslint-plugin-svelte)
  - Allow console.info/warn/error (belief-state protocol)
- Remove .pylintrc (references non-existent plugin)
- Update agent docs: ruff check . + npm run lint
- Add ruff>=0.11.0 to backend/requirements.txt
- Add eslint + plugins to frontend devDependencies
2026-05-14 10:28:30 +03:00
4074506114 chore: gitignore runtime artifacts, commit pre-existing dictionary+specs changes 2026-05-14 10:17:09 +03:00
8d0bc6fb93 feat(translate,scheduler,trace): new-key-only mode, stale PENDING protection, trace_id propagation
Translations:
- FR-045: new-key-only execution mode — translate only rows with unseen keys
- Compare source row keys against TranslationRecord.source_data from last succeeded run
- Baseline expired (>90 days) fallback to full mode with baseline_expired event
- run_noop early return when zero new keys (skip LLM + SQL)

Scheduler reliability:
- Stale PENDING protection: runs older than 1h auto-marked FAILED, no longer block schedule
- load_schedules() reloads active translation schedules from DB on restart
- add_translation_job/remove_translation_job register/unregister with APScheduler
- execution_mode column on translation_schedules with additive DB migration

Automation view:
- GET /settings/automation/translation-schedules endpoint
- Translation schedules displayed on Automation page below validation policies

Trace propagation:
- seed_trace_id() in all background entry points:
  TaskManager._run_task/_flusher_loop, SchedulerService._trigger_backup,
  websocket_endpoint, IdMappingService, all standalone scripts

Migrations:
- dictionary_entries: origin_run_id, origin_row_key, origin_user_id
- translation_schedules: execution_mode

Tests:
- 3 new test modules (22 tests): core_scheduler, executor_filter, scheduler_execution+guard
- Fix pre-existing translate test isolation (conftest.py)
- Fix test_list_runs_filter_status parameter
2026-05-14 10:13:56 +03:00
d1695fe536 fix(translate,automation): fix schedule import/param errors, add translation schedules to Automation view
- Fix ModuleNotFoundError: 4 lazy imports in _schedule_routes.py changed
  from 3-dot to 4-dot relative imports (src.api.dependencies -> src.dependencies)
- Fix TypeError: update_schedule() param name mismatch (timezone -> timezone_str)
- Add add_translation_job/remove_translation_job to SchedulerService
  to register translation schedules with APScheduler via execute_scheduled_translation
- Add GET /settings/automation/translation-schedules endpoint
  returning all translation schedules with joined job names
- Update Automation settings page to display translation schedules
  (cron, timezone, active badge, last run) below validation policies
2026-05-13 21:08:10 +03:00
1f8b8c9f47 fix(qa): address qa-tester findings — mock target_column, FetchModelsRequest schema, error handling
- Fix MagicMock poisoning: add target_column=None to _make_mock_job/mock_job fixtures (3 files)
- Add FetchModelsRequest Pydantic model for type-safe fetch-models endpoint
- logger.warning→logger.error for fetch_models 502 path
- Early validation: require api_key or provider_id in fetch-models
- 50 tests pass, 0 failures (test_clickhouse_insert_integration, test_orchestrator, test_preview)
2026-05-13 20:17:14 +03:00
a3c7c402b7 fix(llm): add fetch-models endpoint, fix SQL Lab INSERT (client_id truncation, sync mode, target_column, timestamp normalization)
- Add POST /api/llm/providers/fetch-models route with LLMClient.fetch_models()
- Add target_column to TranslationJob model/schema/service/orchestrator
- Fix SQL Lab execute: truncate client_id to 11 chars (varchar(11))
- Switch SQL Lab to sync mode (runAsync: false) — no Celery workers
- Fix polling: unwrap nested result from Superset query API
- Fix ClickHouse timestamp: normalize float timestamps to YYYY-MM-DD
2026-05-13 20:06:15 +03:00
39ab647851 semantics 2026-05-13 14:15:33 +03:00
83b8f4d999 logs: migrate to molecular CoT JSON protocol, fix regressions
- Replace BeliefFormatter with CotJsonFormatter (single-line JSON with
  ts, level, trace_id, src, marker, intent, payload, error, span_id)
- Migrate belief_scope to use cot_log() for structured JSON output
- Update monkey-patched reason/reflect/explore to pass extra= dict
- Strip 200+ inline [REASON]/[REFLECT]/[EXPLORE] markers from 50+ files
- Remove belief_scope from hot-path utilities (_parse_datetime,
  _json_load_if_needed) to eliminate startup log noise
- Register TraceContextMiddleware in app.py (was implemented but unused)
- Seed trace_id in startup_event/shutdown_event for background tasks
- Fix broken relative imports in translate routes (3 dots → 4 dots)
- Migrate 43 translate_routes log calls to logger.reason/explore
- Fix SyntaxError (positional arg after extra=) in _repo_operations_routes
- Fix IndentationError in rbac_permission_catalog except-block
- Add smoke test tests/test_smoke_app.py (catches import errors before run)
2026-05-13 09:44:50 +03:00
b6f46fe9f5 logger: migrate belief_scope from legacy markers to molecular CoT protocol
Replace [Entry]/[Exit]/[Action]/[COHERENCE:OK] with [REASON]/[REFLECT]/[EXPLORE]
in belief_scope context manager, BeliefFormatter, and 34 source files.
Per molecular-cot-logging skill invariants.
2026-05-13 08:46:24 +03:00
306c5ae742 semantics: complete DEF-to-region migration, fix regressions
- Convert legacy [DEF🆔Type] anchors to #region/#endregion across 329 files
- Reinstate _normalize_timestamp_value in sql_generator.py
- Fix MarkerLogger→logger migration in events.py (molecular CoT markers)
- Fix dataset_review orchestrator dependencies (_build_execution_snapshot)
- Fix config_manager stale-record deletion (moved to save path only)
- Add 77 missing [/DEF:] closers in 5 unbalanced test files
- Update assistant_chat.integration.test.js for #region format
- Apply molecular-cot-logging markers (REASON/REFLECT/EXPLORE) via logger.* methods
2026-05-12 23:54:55 +03:00
fe8978f716 semantics 2026-05-12 20:06:16 +03:00
9ea177558b agents 2026-05-12 19:37:22 +03:00
b17b5333c7 log refactor 2026-05-12 19:30:15 +03:00
1d59df2233 refactor 2026-05-12 14:52:27 +03:00
9f995f22ae refactor: migrate translate engine to GRACE-Poly v2.6 semantic protocol
- Convert all 84 contracts from legacy [DEF:] to #region/#endregion syntax
- Fix complexity tiers: 14 modules re-tiered (6 C4 route modules, 7 C4→C5 plugin services)
- Remove forbidden tags: @RATIONALE/@REJECTED stripped from C1–C4 contracts
- Add required tags: @PRE/@POST/@SIDE_EFFECT on C4, @RELATION on C3, @DATA_CONTRACT/@INVARIANT on C5
- Add belief runtime markers (reason/reflect/explore) to 7 service.py functions
- Fix @LAYER: route files → UI, plugins → Domain, superset_executor → Infra
- Fix pre-existing test mock_service fixture in test_orchestrator.py
- 196/196 translation tests pass, zero regressions
2026-05-12 14:32:28 +03:00
fefdee98d0 agents 2026-05-11 22:58:01 +03:00
2abba06e52 feat: add job status display and 'Mark as READY' button to Run tab
- Add status badge (DRAFT/READY/RUNNING/COMPLETED/FAILED) in Run tab
- Add 'Mark as READY' button to transition from DRAFT to READY
- Include status field in save payload to prevent reverting on save
- Disable Run button when job is in DRAFT status
2026-05-09 23:19:46 +03:00
dbb8bd6c4e fix: persist environment selection, support Kilo AI provider, resolve Superset virtual columns for translation preview
- Add environment_id to TranslationJob model/schema/API + DB migration
- Pass environmentId from page to TranslationPreview and fetchPreview
- Fix _fetch_sample_rows: use result_type='samples' to include virtual cols
- Add 'kilo' and 'openrouter' to supported LLM provider types
- Make response_format conditional (skip for non-OpenAI upstream providers)
- Remove source_table field from form (redundant with datasource)
- Restore datasourceSearch display from saved job on page load
- Add request/response logging for LLM API calls
2026-05-09 23:10:15 +03:00
67ba04d4ff feat: implement LLM table translation service with searchable datasource dropdown, i18n, sidebar, and RBAC
- Add backend plugin modules: preview, executor, orchestrator, events, sql_generator, superset_executor, dictionary, scheduler, metrics
- Add API routes for translate jobs, runs, dictionaries, preview, schedule, metrics, corrections
- Add ORM models (12 tables) and Pydantic schemas (15 DTOs)
- Register translate router in app.py with RBAC permission guards
- Add frontend pages: job list, job config, dictionary list/editor, history
- Add 7 reusable Svelte 5 components: Preview, RunProgress, RunResult, TermCorrection, BulkCorrection, ScheduleConfig, MetricsDashboard
- Add searchable datasource dropdown with Superset API integration
- Add i18n (en/ru) with ~300 keys across common, config, jobs, dictionaries, preview namespaces
- Add Translation sidebar category with Jobs/Dictionaries/History sub-items
- Hide health monitor error toast via suppressToast API option
- Add 69 backend tests and 44 frontend test files
- Fix: SupersetClient env resolution (string -> Environment object)
- Fix: Dataset detail API returns proper database dict
- Fix: Database dialect extraction fallback when metadata incomplete
2026-05-09 19:34:25 +03:00
bf82e17418 chore: add comprehensive log patterns to gitignore, purge logs from history
- Add *.log.* and logs/ patterns to .gitignore
- Remove duplicate backend/logs and logs/app.log.1 entries
- Purge all log files, backend/logs/, and .axiom/runtime/ from
  entire git history via git-filter-repo
2026-05-09 10:17:26 +03:00
201886eeb0 refactor: decompose 6 monolithic modules (>1000 LOC) into domain packages
Decompose 6 large modules into packages with <400-line modules to satisfy INV_7:

- superset_client.py (2145->0) -> superset_client/ package (12 modules)
- assistant.py (2683->0) -> assistant/ package (12 modules)
- git_service.py (2101->11 shim) -> services/git/ package (9 modules)
- git.py (1757->0) -> routes/git/ package (11 modules)
- dashboards.py (1619->0) -> routes/dashboards/ package (8 modules)
- translate.py (1587->0) -> routes/translate/ package (11 modules)
- superset_context_extractor.py (1397->0) -> utils/superset_context_extractor/ (7 modules)

All modules <400 lines (INV_7). All 80 tests pass.
All external imports preserved via __init__.py re-exports or shim.
Semantic [DEF] contracts with @LAYER/@SEMANTICS added to all Module types.
2026-05-09 10:13:07 +03:00
344b47ca23 tasks ready 2026-05-08 18:01:49 +03:00
1b15fd3fa7 semantic cleanup 2026-05-08 10:07:05 +03:00
54ce6a8126 fix(git): replace git diff --cached with git status --porcelain in get_status
repo.is_dirty() and repo.index.diff("HEAD") internally call git diff --cached,
which fails with exit 129 (unknown option) in some environments.

Introduce _parse_status_porcelain() using git status --porcelain,
a self-contained command that avoids --cached entirely.

All 25 git route tests continue to pass.
2026-05-08 09:55:10 +03:00
0b227a307c refactor 2026-04-24 17:10:02 +03:00