Commit Graph

343 Commits

Author SHA1 Message Date
30d7f3f725 feat(i18n): localize all translate components (spec 028)
P0 — Wired i18n to 4 previously-unlocalized surfaces:
- BulkReplaceModal (~25 strings → translate.bulk_replace.*)
- CorrectionCell (~20 strings → translate.corrections.cell_*)
- TermCorrectionPopup (~30 strings → translate.term_correction.*)
- history/+page.svelte (~50 strings → translate.history.*)

P1 — Added 85+ keys to en.json + ru.json (identical structures):
- 11 previously-missing keys (common.close, config.status, etc.)
- bulk_replace.*, corrections.*, term_correction.*, history.*, dictionaries.*

P2 — Dictionary detail page: import form, filter-by-language, context display
P3 — Russian hardcoded strings in job config replaced with i18n

Tests: 280 pass, build succeeds
2026-05-15 09:00:33 +03:00
77d55c8e69 refactor(translate): remove all deprecated columns from model + DB
Removed 6 legacy columns replaced by Phase 11 multi-language models:
- TranslationJob.target_language → target_languages (JSON)
- TranslationRecord.{llm_translation, user_edit, final_value} → TranslationLanguage
- TerminologyDictionary.{source_language, target_language} → DictionaryEntry per-entry

Affected files: model, schemas, 5 plugins, 4 test files
Alembic migration: aa1b2c3d4e5f (DROP COLUMN IF EXISTS)
Production DB: columns dropped via ALTER TABLE

Tests: 234 passed, 0 failed
2026-05-15 00:06:41 +03:00
6717e2551d fix(translate): add missing target_languages column to production DB
Migration 2a7b8c9d0e1f adds target_languages (JSON) to translation_jobs.
Uses ADD COLUMN IF NOT EXISTS for idempotent PostgreSQL deployment.

Fixes 500 error on GET /api/translate/jobs:
  column translation_jobs.target_languages does not exist

Migration chain is now clean: 5 linear revisions, no orphans.
2026-05-14 23:51:25 +03:00
cc1e57088d chore: remove legacy SQLite migration script + fix stale comments
- Delete migrate_sqlite_to_postgres.py (one-time script, no longer needed)
- Fix @INVARIANT in mappings.py: SQLite → PostgreSQL
- Remove deprecated source_language column from TranslationJob model
  + Alembic migration 8dd0a93af539 to drop column if exists
- Remove source_language from TranslateJobResponse schema

Fixes recurring 'source_language does not exist' scheduler error
2026-05-14 23:48:13 +03:00
fdc452a945 feat(docker): add all-in-one Dockerfile without Playwright (<200MB target)
Multi-stage build:
- Stage 1: node:20-alpine builds frontend (discarded)
- Stage 2: python:3.11-slim runs backend + serves SPA static files
- Playwright removed from deps (saves ~350MB)
- Uvicorn serves both API and frontend on :8000
- Healthcheck via curl
- Entrypoint with admin bootstrap support

Estimated image size: ~190MB (vs ~550MB with playwright)
2026-05-14 21:37:08 +03:00
8bea44f640 chore: update semantic index, agent configs, and app startup 2026-05-14 21:33:26 +03:00
9f3f6611a1 feat(translate): complete Phase 10-11 — full spec 028 closure (T075-T134)
Phase 10 closure:
- T075: NotificationService wiring for scheduled-run failures
- T077: Semantic audit via axiom MCP (0 warnings)
- T078: Quickstart validation — all 165+280 tests pass

Phase 11 completion:
- T083-T086: Multi-language models/schemas/Alembic migration
- T087-T090: Auto-detect source language (BCP-47) + tests
- T091-T095: Multilingual dictionaries with language-pair filtering
- T096-T108: Multi-language job config → preview → execution pipeline
- T109-T118: Inline correction + BulkReplaceModal + CorrectionCell + tests
- T119-T122: Per-language history and MetricSnapshot
- T123-T134: Context-aware corrections (Jaccard similarity, priority flagging, context_data)

New files: Alembic migration, test_scheduler, test_inline_correction,
BulkReplaceModal, vitest for CorrectionCell + BulkReplaceModal

Total: 134/134 tasks complete. Backend 165p, Frontend 280p.
2026-05-14 21:14:04 +03:00
bb21fd3165 fix(translate): M1-M3 medium + L1-L2 low bugs from QA review
M1: ReDoS guard — 500 char max + try/except on re.compile
M2: Performance note — O(n×m) regex documented for future optimization
M3: Verified test key names correct (metrics.py transforms cumulative_* → short)
L1: Removed @staticmethod no-op from module-level function
L2: Added aria-label to emoji indicators in CorrectionCell
2026-05-14 17:44:50 +03:00
1853d008e9 fix(translate): CR1-CR2 migration + H1-H4 code review bugs
CR1: Add needs_review, language_overridden to translation_languages migration
CR2: Add per_language_metrics to translation_metric_snapshots migration
H1: BulkFindReplaceService.preview() now uses replacement_text (not find pattern)
H2: get_run_records now includes per-language data via selectinload
H3: Preview per-language approve/reject uses correct (jobId, rowId, lang) args
H4: _compute_config_hash now includes target_languages
2026-05-14 17:38:52 +03:00
bb0fbfdafd feat(translate): multi-language optimization (Phase 11)
- Auto-detection of source language per row via LLM (US6)
- Multi-target translation — one LLM call for N languages (US1-US3)
- Language-aware storage: TranslationLanguage, per-language stats
- Multilingual dictionaries with language-pair-aware filtering (US7)
- Inline correction on any run result + submit-to-dictionary (US8)
- Context-aware dictionary: auto-capture row context, usage notes,
  Jaccard similarity, priority flagging in LLM prompts (US8b)
- Configurable preview sample size 1-100, cost warning at >30
- Per-language history & metrics with MetricSnapshot preservation
- 36 files, +5022/-373, all specs GRACE-Poly v2.6 compliant
2026-05-14 17:12:41 +03:00
5e741a4332 specks updated 2026-05-14 15:27:41 +03:00
c6189876b3 chore(lint): apply ruff --fix (4443 auto-fixes)
Auto-fixed categories:
- F401: unused imports removed
- I001: import blocks sorted
- W293: trailing whitespace stripped
- UP035: deprecated typing imports replaced
- SIM: simplify suggestions applied
- ARG: unused args prefixed with underscore
- T201: print statements removed
- F841: unused variables removed
- RUF059: unpacked variables prefixed

Remaining ~1500 unfixable errors (C901, B904, N806, E402) require manual work.

Backend smoke tests: 13/13 passed.
2026-05-14 11:20:17 +03:00
a9a5eff518 fix(lint): apply QA-found gaps in agent instruction files
- python-coder.md: ruff check src/ tests/ → ruff check .
- speckit.test.md: add frontend npm run lint
- speckit.plan.md: ruff check . + frontend lint
- speckit.tasks.md: ruff check . + frontend lint
- fullstack-coder.md: move npm run lint to frontend block
2026-05-14 11:19:22 +03:00
9b8c485562 chore: configure ruff + eslint linters for agentic workflow
- Replace pylint with ruff (backend/ruff.toml)
  - line-length=300, max-complexity=10, per-file-ignores for tests
  - Exclude D (docstrings replaced by contracts) and ANN
- Add eslint flat config for frontend (eslint-plugin-svelte)
  - Allow console.info/warn/error (belief-state protocol)
- Remove .pylintrc (references non-existent plugin)
- Update agent docs: ruff check . + npm run lint
- Add ruff>=0.11.0 to backend/requirements.txt
- Add eslint + plugins to frontend devDependencies
2026-05-14 10:28:30 +03:00
4074506114 chore: gitignore runtime artifacts, commit pre-existing dictionary+specs changes 2026-05-14 10:17:09 +03:00
8d0bc6fb93 feat(translate,scheduler,trace): new-key-only mode, stale PENDING protection, trace_id propagation
Translations:
- FR-045: new-key-only execution mode — translate only rows with unseen keys
- Compare source row keys against TranslationRecord.source_data from last succeeded run
- Baseline expired (>90 days) fallback to full mode with baseline_expired event
- run_noop early return when zero new keys (skip LLM + SQL)

Scheduler reliability:
- Stale PENDING protection: runs older than 1h auto-marked FAILED, no longer block schedule
- load_schedules() reloads active translation schedules from DB on restart
- add_translation_job/remove_translation_job register/unregister with APScheduler
- execution_mode column on translation_schedules with additive DB migration

Automation view:
- GET /settings/automation/translation-schedules endpoint
- Translation schedules displayed on Automation page below validation policies

Trace propagation:
- seed_trace_id() in all background entry points:
  TaskManager._run_task/_flusher_loop, SchedulerService._trigger_backup,
  websocket_endpoint, IdMappingService, all standalone scripts

Migrations:
- dictionary_entries: origin_run_id, origin_row_key, origin_user_id
- translation_schedules: execution_mode

Tests:
- 3 new test modules (22 tests): core_scheduler, executor_filter, scheduler_execution+guard
- Fix pre-existing translate test isolation (conftest.py)
- Fix test_list_runs_filter_status parameter
2026-05-14 10:13:56 +03:00
d1695fe536 fix(translate,automation): fix schedule import/param errors, add translation schedules to Automation view
- Fix ModuleNotFoundError: 4 lazy imports in _schedule_routes.py changed
  from 3-dot to 4-dot relative imports (src.api.dependencies -> src.dependencies)
- Fix TypeError: update_schedule() param name mismatch (timezone -> timezone_str)
- Add add_translation_job/remove_translation_job to SchedulerService
  to register translation schedules with APScheduler via execute_scheduled_translation
- Add GET /settings/automation/translation-schedules endpoint
  returning all translation schedules with joined job names
- Update Automation settings page to display translation schedules
  (cron, timezone, active badge, last run) below validation policies
2026-05-13 21:08:10 +03:00
1f8b8c9f47 fix(qa): address qa-tester findings — mock target_column, FetchModelsRequest schema, error handling
- Fix MagicMock poisoning: add target_column=None to _make_mock_job/mock_job fixtures (3 files)
- Add FetchModelsRequest Pydantic model for type-safe fetch-models endpoint
- logger.warning→logger.error for fetch_models 502 path
- Early validation: require api_key or provider_id in fetch-models
- 50 tests pass, 0 failures (test_clickhouse_insert_integration, test_orchestrator, test_preview)
2026-05-13 20:17:14 +03:00
a3c7c402b7 fix(llm): add fetch-models endpoint, fix SQL Lab INSERT (client_id truncation, sync mode, target_column, timestamp normalization)
- Add POST /api/llm/providers/fetch-models route with LLMClient.fetch_models()
- Add target_column to TranslationJob model/schema/service/orchestrator
- Fix SQL Lab execute: truncate client_id to 11 chars (varchar(11))
- Switch SQL Lab to sync mode (runAsync: false) — no Celery workers
- Fix polling: unwrap nested result from Superset query API
- Fix ClickHouse timestamp: normalize float timestamps to YYYY-MM-DD
2026-05-13 20:06:15 +03:00
39ab647851 semantics 2026-05-13 14:15:33 +03:00
83b8f4d999 logs: migrate to molecular CoT JSON protocol, fix regressions
- Replace BeliefFormatter with CotJsonFormatter (single-line JSON with
  ts, level, trace_id, src, marker, intent, payload, error, span_id)
- Migrate belief_scope to use cot_log() for structured JSON output
- Update monkey-patched reason/reflect/explore to pass extra= dict
- Strip 200+ inline [REASON]/[REFLECT]/[EXPLORE] markers from 50+ files
- Remove belief_scope from hot-path utilities (_parse_datetime,
  _json_load_if_needed) to eliminate startup log noise
- Register TraceContextMiddleware in app.py (was implemented but unused)
- Seed trace_id in startup_event/shutdown_event for background tasks
- Fix broken relative imports in translate routes (3 dots → 4 dots)
- Migrate 43 translate_routes log calls to logger.reason/explore
- Fix SyntaxError (positional arg after extra=) in _repo_operations_routes
- Fix IndentationError in rbac_permission_catalog except-block
- Add smoke test tests/test_smoke_app.py (catches import errors before run)
2026-05-13 09:44:50 +03:00
b6f46fe9f5 logger: migrate belief_scope from legacy markers to molecular CoT protocol
Replace [Entry]/[Exit]/[Action]/[COHERENCE:OK] with [REASON]/[REFLECT]/[EXPLORE]
in belief_scope context manager, BeliefFormatter, and 34 source files.
Per molecular-cot-logging skill invariants.
2026-05-13 08:46:24 +03:00
306c5ae742 semantics: complete DEF-to-region migration, fix regressions
- Convert legacy [DEF🆔Type] anchors to #region/#endregion across 329 files
- Reinstate _normalize_timestamp_value in sql_generator.py
- Fix MarkerLogger→logger migration in events.py (molecular CoT markers)
- Fix dataset_review orchestrator dependencies (_build_execution_snapshot)
- Fix config_manager stale-record deletion (moved to save path only)
- Add 77 missing [/DEF:] closers in 5 unbalanced test files
- Update assistant_chat.integration.test.js for #region format
- Apply molecular-cot-logging markers (REASON/REFLECT/EXPLORE) via logger.* methods
2026-05-12 23:54:55 +03:00
fe8978f716 semantics 2026-05-12 20:06:16 +03:00
9ea177558b agents 2026-05-12 19:37:22 +03:00
b17b5333c7 log refactor 2026-05-12 19:30:15 +03:00
1d59df2233 refactor 2026-05-12 14:52:27 +03:00
9f995f22ae refactor: migrate translate engine to GRACE-Poly v2.6 semantic protocol
- Convert all 84 contracts from legacy [DEF:] to #region/#endregion syntax
- Fix complexity tiers: 14 modules re-tiered (6 C4 route modules, 7 C4→C5 plugin services)
- Remove forbidden tags: @RATIONALE/@REJECTED stripped from C1–C4 contracts
- Add required tags: @PRE/@POST/@SIDE_EFFECT on C4, @RELATION on C3, @DATA_CONTRACT/@INVARIANT on C5
- Add belief runtime markers (reason/reflect/explore) to 7 service.py functions
- Fix @LAYER: route files → UI, plugins → Domain, superset_executor → Infra
- Fix pre-existing test mock_service fixture in test_orchestrator.py
- 196/196 translation tests pass, zero regressions
2026-05-12 14:32:28 +03:00
fefdee98d0 agents 2026-05-11 22:58:01 +03:00
2abba06e52 feat: add job status display and 'Mark as READY' button to Run tab
- Add status badge (DRAFT/READY/RUNNING/COMPLETED/FAILED) in Run tab
- Add 'Mark as READY' button to transition from DRAFT to READY
- Include status field in save payload to prevent reverting on save
- Disable Run button when job is in DRAFT status
2026-05-09 23:19:46 +03:00
dbb8bd6c4e fix: persist environment selection, support Kilo AI provider, resolve Superset virtual columns for translation preview
- Add environment_id to TranslationJob model/schema/API + DB migration
- Pass environmentId from page to TranslationPreview and fetchPreview
- Fix _fetch_sample_rows: use result_type='samples' to include virtual cols
- Add 'kilo' and 'openrouter' to supported LLM provider types
- Make response_format conditional (skip for non-OpenAI upstream providers)
- Remove source_table field from form (redundant with datasource)
- Restore datasourceSearch display from saved job on page load
- Add request/response logging for LLM API calls
2026-05-09 23:10:15 +03:00
67ba04d4ff feat: implement LLM table translation service with searchable datasource dropdown, i18n, sidebar, and RBAC
- Add backend plugin modules: preview, executor, orchestrator, events, sql_generator, superset_executor, dictionary, scheduler, metrics
- Add API routes for translate jobs, runs, dictionaries, preview, schedule, metrics, corrections
- Add ORM models (12 tables) and Pydantic schemas (15 DTOs)
- Register translate router in app.py with RBAC permission guards
- Add frontend pages: job list, job config, dictionary list/editor, history
- Add 7 reusable Svelte 5 components: Preview, RunProgress, RunResult, TermCorrection, BulkCorrection, ScheduleConfig, MetricsDashboard
- Add searchable datasource dropdown with Superset API integration
- Add i18n (en/ru) with ~300 keys across common, config, jobs, dictionaries, preview namespaces
- Add Translation sidebar category with Jobs/Dictionaries/History sub-items
- Hide health monitor error toast via suppressToast API option
- Add 69 backend tests and 44 frontend test files
- Fix: SupersetClient env resolution (string -> Environment object)
- Fix: Dataset detail API returns proper database dict
- Fix: Database dialect extraction fallback when metadata incomplete
2026-05-09 19:34:25 +03:00
bf82e17418 chore: add comprehensive log patterns to gitignore, purge logs from history
- Add *.log.* and logs/ patterns to .gitignore
- Remove duplicate backend/logs and logs/app.log.1 entries
- Purge all log files, backend/logs/, and .axiom/runtime/ from
  entire git history via git-filter-repo
2026-05-09 10:17:26 +03:00
201886eeb0 refactor: decompose 6 monolithic modules (>1000 LOC) into domain packages
Decompose 6 large modules into packages with <400-line modules to satisfy INV_7:

- superset_client.py (2145->0) -> superset_client/ package (12 modules)
- assistant.py (2683->0) -> assistant/ package (12 modules)
- git_service.py (2101->11 shim) -> services/git/ package (9 modules)
- git.py (1757->0) -> routes/git/ package (11 modules)
- dashboards.py (1619->0) -> routes/dashboards/ package (8 modules)
- translate.py (1587->0) -> routes/translate/ package (11 modules)
- superset_context_extractor.py (1397->0) -> utils/superset_context_extractor/ (7 modules)

All modules <400 lines (INV_7). All 80 tests pass.
All external imports preserved via __init__.py re-exports or shim.
Semantic [DEF] contracts with @LAYER/@SEMANTICS added to all Module types.
2026-05-09 10:13:07 +03:00
344b47ca23 tasks ready 2026-05-08 18:01:49 +03:00
1b15fd3fa7 semantic cleanup 2026-05-08 10:07:05 +03:00
54ce6a8126 fix(git): replace git diff --cached with git status --porcelain in get_status
repo.is_dirty() and repo.index.diff("HEAD") internally call git diff --cached,
which fails with exit 129 (unknown option) in some environments.

Introduce _parse_status_porcelain() using git status --porcelain,
a self-contained command that avoids --cached entirely.

All 25 git route tests continue to pass.
2026-05-08 09:55:10 +03:00
0b227a307c refactor 2026-04-24 17:10:02 +03:00
b7a9ef71b2 agents + semantic 2026-04-21 11:30:41 +03:00
e04bddcc4a refactor(dashboards): extract dashboard detail components
Split the large dashboard detail page into smaller, focused components: DashboardHeader, DashboardGitManager, DashboardLinkedResources, and DashboardTaskHistory to improve maintainability.
2026-04-21 08:36:48 +03:00
7a84716322 refactor: extract EnvironmentsTab to smaller semantic boundary 2026-04-20 19:18:19 +03:00
277ce7b3b3 skills + agents 2026-04-19 20:05:45 +03:00
2228a00035 semantics 2026-04-02 12:12:23 +03:00
80ec0c5eb1 prune contracts 2026-04-01 22:31:10 +03:00
89924afa8e + 2026-04-01 22:04:19 +03:00
f5c90af287 semantics 2026-04-01 21:57:51 +03:00
ace77ed64c semantics update 2026-04-01 15:30:13 +03:00
03d46cb32a semantic 2026-04-01 14:51:45 +03:00
af61096ddb root update 2026-04-01 14:25:52 +03:00
68b5b8d505 configs update 2026-04-01 14:07:02 +03:00