Commit Graph

335 Commits

Author SHA1 Message Date
ddd7769f0e fix(translate): CR1-CR2 migration + H1-H4 code review bugs
CR1: Add needs_review, language_overridden to translation_languages migration
CR2: Add per_language_metrics to translation_metric_snapshots migration
H1: BulkFindReplaceService.preview() now uses replacement_text (not find pattern)
H2: get_run_records now includes per-language data via selectinload
H3: Preview per-language approve/reject uses correct (jobId, rowId, lang) args
H4: _compute_config_hash now includes target_languages
2026-05-14 17:38:52 +03:00
fa3e74b220 feat(translate): multi-language optimization (Phase 11)
- Auto-detection of source language per row via LLM (US6)
- Multi-target translation — one LLM call for N languages (US1-US3)
- Language-aware storage: TranslationLanguage, per-language stats
- Multilingual dictionaries with language-pair-aware filtering (US7)
- Inline correction on any run result + submit-to-dictionary (US8)
- Context-aware dictionary: auto-capture row context, usage notes,
  Jaccard similarity, priority flagging in LLM prompts (US8b)
- Configurable preview sample size 1-100, cost warning at >30
- Per-language history & metrics with MetricSnapshot preservation
- 36 files, +5022/-373, all specs GRACE-Poly v2.6 compliant
2026-05-14 17:12:41 +03:00
99e88c8ba9 specks updated 2026-05-14 15:27:41 +03:00
16c7b947dd chore(lint): apply ruff --fix (4443 auto-fixes)
Auto-fixed categories:
- F401: unused imports removed
- I001: import blocks sorted
- W293: trailing whitespace stripped
- UP035: deprecated typing imports replaced
- SIM: simplify suggestions applied
- ARG: unused args prefixed with underscore
- T201: print statements removed
- F841: unused variables removed
- RUF059: unpacked variables prefixed

Remaining ~1500 unfixable errors (C901, B904, N806, E402) require manual work.

Backend smoke tests: 13/13 passed.
2026-05-14 11:20:17 +03:00
4f311fe594 fix(lint): apply QA-found gaps in agent instruction files
- python-coder.md: ruff check src/ tests/ → ruff check .
- speckit.test.md: add frontend npm run lint
- speckit.plan.md: ruff check . + frontend lint
- speckit.tasks.md: ruff check . + frontend lint
- fullstack-coder.md: move npm run lint to frontend block
2026-05-14 11:19:22 +03:00
1992c6e077 chore: configure ruff + eslint linters for agentic workflow
- Replace pylint with ruff (backend/ruff.toml)
  - line-length=300, max-complexity=10, per-file-ignores for tests
  - Exclude D (docstrings replaced by contracts) and ANN
- Add eslint flat config for frontend (eslint-plugin-svelte)
  - Allow console.info/warn/error (belief-state protocol)
- Remove .pylintrc (references non-existent plugin)
- Update agent docs: ruff check . + npm run lint
- Add ruff>=0.11.0 to backend/requirements.txt
- Add eslint + plugins to frontend devDependencies
2026-05-14 10:28:30 +03:00
e1e0d72f09 chore: gitignore runtime artifacts, commit pre-existing dictionary+specs changes 2026-05-14 10:17:09 +03:00
d081277ae9 feat(translate,scheduler,trace): new-key-only mode, stale PENDING protection, trace_id propagation
Translations:
- FR-045: new-key-only execution mode — translate only rows with unseen keys
- Compare source row keys against TranslationRecord.source_data from last succeeded run
- Baseline expired (>90 days) fallback to full mode with baseline_expired event
- run_noop early return when zero new keys (skip LLM + SQL)

Scheduler reliability:
- Stale PENDING protection: runs older than 1h auto-marked FAILED, no longer block schedule
- load_schedules() reloads active translation schedules from DB on restart
- add_translation_job/remove_translation_job register/unregister with APScheduler
- execution_mode column on translation_schedules with additive DB migration

Automation view:
- GET /settings/automation/translation-schedules endpoint
- Translation schedules displayed on Automation page below validation policies

Trace propagation:
- seed_trace_id() in all background entry points:
  TaskManager._run_task/_flusher_loop, SchedulerService._trigger_backup,
  websocket_endpoint, IdMappingService, all standalone scripts

Migrations:
- dictionary_entries: origin_run_id, origin_row_key, origin_user_id
- translation_schedules: execution_mode

Tests:
- 3 new test modules (22 tests): core_scheduler, executor_filter, scheduler_execution+guard
- Fix pre-existing translate test isolation (conftest.py)
- Fix test_list_runs_filter_status parameter
2026-05-14 10:13:56 +03:00
78674834a5 fix(translate,automation): fix schedule import/param errors, add translation schedules to Automation view
- Fix ModuleNotFoundError: 4 lazy imports in _schedule_routes.py changed
  from 3-dot to 4-dot relative imports (src.api.dependencies -> src.dependencies)
- Fix TypeError: update_schedule() param name mismatch (timezone -> timezone_str)
- Add add_translation_job/remove_translation_job to SchedulerService
  to register translation schedules with APScheduler via execute_scheduled_translation
- Add GET /settings/automation/translation-schedules endpoint
  returning all translation schedules with joined job names
- Update Automation settings page to display translation schedules
  (cron, timezone, active badge, last run) below validation policies
2026-05-13 21:08:10 +03:00
be11a06de4 fix(qa): address qa-tester findings — mock target_column, FetchModelsRequest schema, error handling
- Fix MagicMock poisoning: add target_column=None to _make_mock_job/mock_job fixtures (3 files)
- Add FetchModelsRequest Pydantic model for type-safe fetch-models endpoint
- logger.warning→logger.error for fetch_models 502 path
- Early validation: require api_key or provider_id in fetch-models
- 50 tests pass, 0 failures (test_clickhouse_insert_integration, test_orchestrator, test_preview)
2026-05-13 20:17:14 +03:00
10803a5019 fix(llm): add fetch-models endpoint, fix SQL Lab INSERT (client_id truncation, sync mode, target_column, timestamp normalization)
- Add POST /api/llm/providers/fetch-models route with LLMClient.fetch_models()
- Add target_column to TranslationJob model/schema/service/orchestrator
- Fix SQL Lab execute: truncate client_id to 11 chars (varchar(11))
- Switch SQL Lab to sync mode (runAsync: false) — no Celery workers
- Fix polling: unwrap nested result from Superset query API
- Fix ClickHouse timestamp: normalize float timestamps to YYYY-MM-DD
2026-05-13 20:06:15 +03:00
ffcfec9acf semantics 2026-05-13 14:15:33 +03:00
6e9bd522e9 logs: migrate to molecular CoT JSON protocol, fix regressions
- Replace BeliefFormatter with CotJsonFormatter (single-line JSON with
  ts, level, trace_id, src, marker, intent, payload, error, span_id)
- Migrate belief_scope to use cot_log() for structured JSON output
- Update monkey-patched reason/reflect/explore to pass extra= dict
- Strip 200+ inline [REASON]/[REFLECT]/[EXPLORE] markers from 50+ files
- Remove belief_scope from hot-path utilities (_parse_datetime,
  _json_load_if_needed) to eliminate startup log noise
- Register TraceContextMiddleware in app.py (was implemented but unused)
- Seed trace_id in startup_event/shutdown_event for background tasks
- Fix broken relative imports in translate routes (3 dots → 4 dots)
- Migrate 43 translate_routes log calls to logger.reason/explore
- Fix SyntaxError (positional arg after extra=) in _repo_operations_routes
- Fix IndentationError in rbac_permission_catalog except-block
- Add smoke test tests/test_smoke_app.py (catches import errors before run)
2026-05-13 09:44:50 +03:00
614f185fbc logger: migrate belief_scope from legacy markers to molecular CoT protocol
Replace [Entry]/[Exit]/[Action]/[COHERENCE:OK] with [REASON]/[REFLECT]/[EXPLORE]
in belief_scope context manager, BeliefFormatter, and 34 source files.
Per molecular-cot-logging skill invariants.
2026-05-13 08:46:24 +03:00
f2a24447ed semantics: complete DEF-to-region migration, fix regressions
- Convert legacy [DEF🆔Type] anchors to #region/#endregion across 329 files
- Reinstate _normalize_timestamp_value in sql_generator.py
- Fix MarkerLogger→logger migration in events.py (molecular CoT markers)
- Fix dataset_review orchestrator dependencies (_build_execution_snapshot)
- Fix config_manager stale-record deletion (moved to save path only)
- Add 77 missing [/DEF:] closers in 5 unbalanced test files
- Update assistant_chat.integration.test.js for #region format
- Apply molecular-cot-logging markers (REASON/REFLECT/EXPLORE) via logger.* methods
2026-05-12 23:54:55 +03:00
13f40048c1 semantics 2026-05-12 20:06:16 +03:00
0d56842ac6 agents 2026-05-12 19:37:22 +03:00
623cad065d log refactor 2026-05-12 19:30:15 +03:00
3a6a0be95a refactor 2026-05-12 14:52:27 +03:00
316d97ae25 refactor: migrate translate engine to GRACE-Poly v2.6 semantic protocol
- Convert all 84 contracts from legacy [DEF:] to #region/#endregion syntax
- Fix complexity tiers: 14 modules re-tiered (6 C4 route modules, 7 C4→C5 plugin services)
- Remove forbidden tags: @RATIONALE/@REJECTED stripped from C1–C4 contracts
- Add required tags: @PRE/@POST/@SIDE_EFFECT on C4, @RELATION on C3, @DATA_CONTRACT/@INVARIANT on C5
- Add belief runtime markers (reason/reflect/explore) to 7 service.py functions
- Fix @LAYER: route files → UI, plugins → Domain, superset_executor → Infra
- Fix pre-existing test mock_service fixture in test_orchestrator.py
- 196/196 translation tests pass, zero regressions
2026-05-12 14:32:28 +03:00
72d034746d agents 2026-05-11 22:58:01 +03:00
262f9ca39e feat: add job status display and 'Mark as READY' button to Run tab
- Add status badge (DRAFT/READY/RUNNING/COMPLETED/FAILED) in Run tab
- Add 'Mark as READY' button to transition from DRAFT to READY
- Include status field in save payload to prevent reverting on save
- Disable Run button when job is in DRAFT status
2026-05-09 23:19:46 +03:00
c2bff20937 fix: persist environment selection, support Kilo AI provider, resolve Superset virtual columns for translation preview
- Add environment_id to TranslationJob model/schema/API + DB migration
- Pass environmentId from page to TranslationPreview and fetchPreview
- Fix _fetch_sample_rows: use result_type='samples' to include virtual cols
- Add 'kilo' and 'openrouter' to supported LLM provider types
- Make response_format conditional (skip for non-OpenAI upstream providers)
- Remove source_table field from form (redundant with datasource)
- Restore datasourceSearch display from saved job on page load
- Add request/response logging for LLM API calls
2026-05-09 23:10:15 +03:00
85121191d4 feat: implement LLM table translation service with searchable datasource dropdown, i18n, sidebar, and RBAC
- Add backend plugin modules: preview, executor, orchestrator, events, sql_generator, superset_executor, dictionary, scheduler, metrics
- Add API routes for translate jobs, runs, dictionaries, preview, schedule, metrics, corrections
- Add ORM models (12 tables) and Pydantic schemas (15 DTOs)
- Register translate router in app.py with RBAC permission guards
- Add frontend pages: job list, job config, dictionary list/editor, history
- Add 7 reusable Svelte 5 components: Preview, RunProgress, RunResult, TermCorrection, BulkCorrection, ScheduleConfig, MetricsDashboard
- Add searchable datasource dropdown with Superset API integration
- Add i18n (en/ru) with ~300 keys across common, config, jobs, dictionaries, preview namespaces
- Add Translation sidebar category with Jobs/Dictionaries/History sub-items
- Hide health monitor error toast via suppressToast API option
- Add 69 backend tests and 44 frontend test files
- Fix: SupersetClient env resolution (string -> Environment object)
- Fix: Dataset detail API returns proper database dict
- Fix: Database dialect extraction fallback when metadata incomplete
2026-05-09 19:34:25 +03:00
c9b86f2bdf chore: add comprehensive log patterns to gitignore, purge logs from history
- Add *.log.* and logs/ patterns to .gitignore
- Remove duplicate backend/logs and logs/app.log.1 entries
- Purge all log files, backend/logs/, and .axiom/runtime/ from
  entire git history via git-filter-repo
2026-05-09 10:17:26 +03:00
c899a369f3 refactor: decompose 6 monolithic modules (>1000 LOC) into domain packages
Decompose 6 large modules into packages with <400-line modules to satisfy INV_7:

- superset_client.py (2145->0) -> superset_client/ package (12 modules)
- assistant.py (2683->0) -> assistant/ package (12 modules)
- git_service.py (2101->11 shim) -> services/git/ package (9 modules)
- git.py (1757->0) -> routes/git/ package (11 modules)
- dashboards.py (1619->0) -> routes/dashboards/ package (8 modules)
- translate.py (1587->0) -> routes/translate/ package (11 modules)
- superset_context_extractor.py (1397->0) -> utils/superset_context_extractor/ (7 modules)

All modules <400 lines (INV_7). All 80 tests pass.
All external imports preserved via __init__.py re-exports or shim.
Semantic [DEF] contracts with @LAYER/@SEMANTICS added to all Module types.
2026-05-09 10:13:07 +03:00
867fb8dce1 tasks ready 2026-05-08 18:01:49 +03:00
13fcd544d2 semantic cleanup 2026-05-08 10:07:05 +03:00
48366d237a fix(git): replace git diff --cached with git status --porcelain in get_status
repo.is_dirty() and repo.index.diff("HEAD") internally call git diff --cached,
which fails with exit 129 (unknown option) in some environments.

Introduce _parse_status_porcelain() using git status --porcelain,
a self-contained command that avoids --cached entirely.

All 25 git route tests continue to pass.
2026-05-08 09:55:10 +03:00
a69446473b refactor 2026-04-24 17:10:02 +03:00
216c5ca4be agents + semantic 2026-04-21 11:30:41 +03:00
d7d0028d5d refactor(dashboards): extract dashboard detail components
Split the large dashboard detail page into smaller, focused components: DashboardHeader, DashboardGitManager, DashboardLinkedResources, and DashboardTaskHistory to improve maintainability.
2026-04-21 08:36:48 +03:00
24cb045f76 refactor: extract EnvironmentsTab to smaller semantic boundary 2026-04-20 19:18:19 +03:00
6a2741182b skills + agents 2026-04-19 20:05:45 +03:00
721caee611 semantics 2026-04-02 12:12:23 +03:00
e9fe80784f prune contracts 2026-04-01 22:31:10 +03:00
f1a809e431 + 2026-04-01 22:04:19 +03:00
78d6995e13 semantics 2026-04-01 21:57:51 +03:00
69e86a97e7 semantics update 2026-04-01 15:30:13 +03:00
3dbac3ce69 semantic 2026-04-01 14:51:45 +03:00
b04cf0f782 root update 2026-04-01 14:25:52 +03:00
bc2e43a14b configs update 2026-04-01 14:07:02 +03:00
f8e08757cf mcp tuning 2026-04-01 13:29:41 +03:00
8fbb6d9da6 agents ADR promts 2026-03-27 22:00:37 +03:00
e83879cf06 semantics 2026-03-27 21:27:31 +03:00
d17f75a65d feat(ui): add chat-driven dataset review flow
Move dataset review clarification into the assistant workspace and
rework the review page into a chat-centric layout with execution rails.

Add session-scoped assistant actions for mappings, semantic fields,
and SQL preview generation. Introduce optimistic locking for dataset
review mutations, propagate session versions through API responses,
and mask imported filter values before assistant exposure.

Refresh tests, i18n, and spec artifacts to match the new workflow.

BREAKING CHANGE: dataset review mutation endpoints now require the
X-Session-Version header, and clarification is no longer handled
through ClarificationDialog-based flows
2026-03-26 13:33:12 +03:00
1ae9e03ff7 + clean 2026-03-22 13:00:17 +03:00
6a49e630fc some cleaning in promts 2026-03-22 13:00:06 +03:00
3b3946a72d standards clean 2026-03-22 12:40:02 +03:00
0a8159b8b7 swarm update 2026-03-21 18:34:24 +03:00