Commit Graph

332 Commits

Author SHA1 Message Date
c6189876b3 chore(lint): apply ruff --fix (4443 auto-fixes)
Auto-fixed categories:
- F401: unused imports removed
- I001: import blocks sorted
- W293: trailing whitespace stripped
- UP035: deprecated typing imports replaced
- SIM: simplify suggestions applied
- ARG: unused args prefixed with underscore
- T201: print statements removed
- F841: unused variables removed
- RUF059: unpacked variables prefixed

Remaining ~1500 unfixable errors (C901, B904, N806, E402) require manual work.

Backend smoke tests: 13/13 passed.
2026-05-14 11:20:17 +03:00
a9a5eff518 fix(lint): apply QA-found gaps in agent instruction files
- python-coder.md: ruff check src/ tests/ → ruff check .
- speckit.test.md: add frontend npm run lint
- speckit.plan.md: ruff check . + frontend lint
- speckit.tasks.md: ruff check . + frontend lint
- fullstack-coder.md: move npm run lint to frontend block
2026-05-14 11:19:22 +03:00
9b8c485562 chore: configure ruff + eslint linters for agentic workflow
- Replace pylint with ruff (backend/ruff.toml)
  - line-length=300, max-complexity=10, per-file-ignores for tests
  - Exclude D (docstrings replaced by contracts) and ANN
- Add eslint flat config for frontend (eslint-plugin-svelte)
  - Allow console.info/warn/error (belief-state protocol)
- Remove .pylintrc (references non-existent plugin)
- Update agent docs: ruff check . + npm run lint
- Add ruff>=0.11.0 to backend/requirements.txt
- Add eslint + plugins to frontend devDependencies
2026-05-14 10:28:30 +03:00
4074506114 chore: gitignore runtime artifacts, commit pre-existing dictionary+specs changes 2026-05-14 10:17:09 +03:00
8d0bc6fb93 feat(translate,scheduler,trace): new-key-only mode, stale PENDING protection, trace_id propagation
Translations:
- FR-045: new-key-only execution mode — translate only rows with unseen keys
- Compare source row keys against TranslationRecord.source_data from last succeeded run
- Baseline expired (>90 days) fallback to full mode with baseline_expired event
- run_noop early return when zero new keys (skip LLM + SQL)

Scheduler reliability:
- Stale PENDING protection: runs older than 1h auto-marked FAILED, no longer block schedule
- load_schedules() reloads active translation schedules from DB on restart
- add_translation_job/remove_translation_job register/unregister with APScheduler
- execution_mode column on translation_schedules with additive DB migration

Automation view:
- GET /settings/automation/translation-schedules endpoint
- Translation schedules displayed on Automation page below validation policies

Trace propagation:
- seed_trace_id() in all background entry points:
  TaskManager._run_task/_flusher_loop, SchedulerService._trigger_backup,
  websocket_endpoint, IdMappingService, all standalone scripts

Migrations:
- dictionary_entries: origin_run_id, origin_row_key, origin_user_id
- translation_schedules: execution_mode

Tests:
- 3 new test modules (22 tests): core_scheduler, executor_filter, scheduler_execution+guard
- Fix pre-existing translate test isolation (conftest.py)
- Fix test_list_runs_filter_status parameter
2026-05-14 10:13:56 +03:00
d1695fe536 fix(translate,automation): fix schedule import/param errors, add translation schedules to Automation view
- Fix ModuleNotFoundError: 4 lazy imports in _schedule_routes.py changed
  from 3-dot to 4-dot relative imports (src.api.dependencies -> src.dependencies)
- Fix TypeError: update_schedule() param name mismatch (timezone -> timezone_str)
- Add add_translation_job/remove_translation_job to SchedulerService
  to register translation schedules with APScheduler via execute_scheduled_translation
- Add GET /settings/automation/translation-schedules endpoint
  returning all translation schedules with joined job names
- Update Automation settings page to display translation schedules
  (cron, timezone, active badge, last run) below validation policies
2026-05-13 21:08:10 +03:00
1f8b8c9f47 fix(qa): address qa-tester findings — mock target_column, FetchModelsRequest schema, error handling
- Fix MagicMock poisoning: add target_column=None to _make_mock_job/mock_job fixtures (3 files)
- Add FetchModelsRequest Pydantic model for type-safe fetch-models endpoint
- logger.warning→logger.error for fetch_models 502 path
- Early validation: require api_key or provider_id in fetch-models
- 50 tests pass, 0 failures (test_clickhouse_insert_integration, test_orchestrator, test_preview)
2026-05-13 20:17:14 +03:00
a3c7c402b7 fix(llm): add fetch-models endpoint, fix SQL Lab INSERT (client_id truncation, sync mode, target_column, timestamp normalization)
- Add POST /api/llm/providers/fetch-models route with LLMClient.fetch_models()
- Add target_column to TranslationJob model/schema/service/orchestrator
- Fix SQL Lab execute: truncate client_id to 11 chars (varchar(11))
- Switch SQL Lab to sync mode (runAsync: false) — no Celery workers
- Fix polling: unwrap nested result from Superset query API
- Fix ClickHouse timestamp: normalize float timestamps to YYYY-MM-DD
2026-05-13 20:06:15 +03:00
39ab647851 semantics 2026-05-13 14:15:33 +03:00
83b8f4d999 logs: migrate to molecular CoT JSON protocol, fix regressions
- Replace BeliefFormatter with CotJsonFormatter (single-line JSON with
  ts, level, trace_id, src, marker, intent, payload, error, span_id)
- Migrate belief_scope to use cot_log() for structured JSON output
- Update monkey-patched reason/reflect/explore to pass extra= dict
- Strip 200+ inline [REASON]/[REFLECT]/[EXPLORE] markers from 50+ files
- Remove belief_scope from hot-path utilities (_parse_datetime,
  _json_load_if_needed) to eliminate startup log noise
- Register TraceContextMiddleware in app.py (was implemented but unused)
- Seed trace_id in startup_event/shutdown_event for background tasks
- Fix broken relative imports in translate routes (3 dots → 4 dots)
- Migrate 43 translate_routes log calls to logger.reason/explore
- Fix SyntaxError (positional arg after extra=) in _repo_operations_routes
- Fix IndentationError in rbac_permission_catalog except-block
- Add smoke test tests/test_smoke_app.py (catches import errors before run)
2026-05-13 09:44:50 +03:00
b6f46fe9f5 logger: migrate belief_scope from legacy markers to molecular CoT protocol
Replace [Entry]/[Exit]/[Action]/[COHERENCE:OK] with [REASON]/[REFLECT]/[EXPLORE]
in belief_scope context manager, BeliefFormatter, and 34 source files.
Per molecular-cot-logging skill invariants.
2026-05-13 08:46:24 +03:00
306c5ae742 semantics: complete DEF-to-region migration, fix regressions
- Convert legacy [DEF🆔Type] anchors to #region/#endregion across 329 files
- Reinstate _normalize_timestamp_value in sql_generator.py
- Fix MarkerLogger→logger migration in events.py (molecular CoT markers)
- Fix dataset_review orchestrator dependencies (_build_execution_snapshot)
- Fix config_manager stale-record deletion (moved to save path only)
- Add 77 missing [/DEF:] closers in 5 unbalanced test files
- Update assistant_chat.integration.test.js for #region format
- Apply molecular-cot-logging markers (REASON/REFLECT/EXPLORE) via logger.* methods
2026-05-12 23:54:55 +03:00
fe8978f716 semantics 2026-05-12 20:06:16 +03:00
9ea177558b agents 2026-05-12 19:37:22 +03:00
b17b5333c7 log refactor 2026-05-12 19:30:15 +03:00
1d59df2233 refactor 2026-05-12 14:52:27 +03:00
9f995f22ae refactor: migrate translate engine to GRACE-Poly v2.6 semantic protocol
- Convert all 84 contracts from legacy [DEF:] to #region/#endregion syntax
- Fix complexity tiers: 14 modules re-tiered (6 C4 route modules, 7 C4→C5 plugin services)
- Remove forbidden tags: @RATIONALE/@REJECTED stripped from C1–C4 contracts
- Add required tags: @PRE/@POST/@SIDE_EFFECT on C4, @RELATION on C3, @DATA_CONTRACT/@INVARIANT on C5
- Add belief runtime markers (reason/reflect/explore) to 7 service.py functions
- Fix @LAYER: route files → UI, plugins → Domain, superset_executor → Infra
- Fix pre-existing test mock_service fixture in test_orchestrator.py
- 196/196 translation tests pass, zero regressions
2026-05-12 14:32:28 +03:00
fefdee98d0 agents 2026-05-11 22:58:01 +03:00
2abba06e52 feat: add job status display and 'Mark as READY' button to Run tab
- Add status badge (DRAFT/READY/RUNNING/COMPLETED/FAILED) in Run tab
- Add 'Mark as READY' button to transition from DRAFT to READY
- Include status field in save payload to prevent reverting on save
- Disable Run button when job is in DRAFT status
2026-05-09 23:19:46 +03:00
dbb8bd6c4e fix: persist environment selection, support Kilo AI provider, resolve Superset virtual columns for translation preview
- Add environment_id to TranslationJob model/schema/API + DB migration
- Pass environmentId from page to TranslationPreview and fetchPreview
- Fix _fetch_sample_rows: use result_type='samples' to include virtual cols
- Add 'kilo' and 'openrouter' to supported LLM provider types
- Make response_format conditional (skip for non-OpenAI upstream providers)
- Remove source_table field from form (redundant with datasource)
- Restore datasourceSearch display from saved job on page load
- Add request/response logging for LLM API calls
2026-05-09 23:10:15 +03:00
67ba04d4ff feat: implement LLM table translation service with searchable datasource dropdown, i18n, sidebar, and RBAC
- Add backend plugin modules: preview, executor, orchestrator, events, sql_generator, superset_executor, dictionary, scheduler, metrics
- Add API routes for translate jobs, runs, dictionaries, preview, schedule, metrics, corrections
- Add ORM models (12 tables) and Pydantic schemas (15 DTOs)
- Register translate router in app.py with RBAC permission guards
- Add frontend pages: job list, job config, dictionary list/editor, history
- Add 7 reusable Svelte 5 components: Preview, RunProgress, RunResult, TermCorrection, BulkCorrection, ScheduleConfig, MetricsDashboard
- Add searchable datasource dropdown with Superset API integration
- Add i18n (en/ru) with ~300 keys across common, config, jobs, dictionaries, preview namespaces
- Add Translation sidebar category with Jobs/Dictionaries/History sub-items
- Hide health monitor error toast via suppressToast API option
- Add 69 backend tests and 44 frontend test files
- Fix: SupersetClient env resolution (string -> Environment object)
- Fix: Dataset detail API returns proper database dict
- Fix: Database dialect extraction fallback when metadata incomplete
2026-05-09 19:34:25 +03:00
bf82e17418 chore: add comprehensive log patterns to gitignore, purge logs from history
- Add *.log.* and logs/ patterns to .gitignore
- Remove duplicate backend/logs and logs/app.log.1 entries
- Purge all log files, backend/logs/, and .axiom/runtime/ from
  entire git history via git-filter-repo
2026-05-09 10:17:26 +03:00
201886eeb0 refactor: decompose 6 monolithic modules (>1000 LOC) into domain packages
Decompose 6 large modules into packages with <400-line modules to satisfy INV_7:

- superset_client.py (2145->0) -> superset_client/ package (12 modules)
- assistant.py (2683->0) -> assistant/ package (12 modules)
- git_service.py (2101->11 shim) -> services/git/ package (9 modules)
- git.py (1757->0) -> routes/git/ package (11 modules)
- dashboards.py (1619->0) -> routes/dashboards/ package (8 modules)
- translate.py (1587->0) -> routes/translate/ package (11 modules)
- superset_context_extractor.py (1397->0) -> utils/superset_context_extractor/ (7 modules)

All modules <400 lines (INV_7). All 80 tests pass.
All external imports preserved via __init__.py re-exports or shim.
Semantic [DEF] contracts with @LAYER/@SEMANTICS added to all Module types.
2026-05-09 10:13:07 +03:00
344b47ca23 tasks ready 2026-05-08 18:01:49 +03:00
1b15fd3fa7 semantic cleanup 2026-05-08 10:07:05 +03:00
54ce6a8126 fix(git): replace git diff --cached with git status --porcelain in get_status
repo.is_dirty() and repo.index.diff("HEAD") internally call git diff --cached,
which fails with exit 129 (unknown option) in some environments.

Introduce _parse_status_porcelain() using git status --porcelain,
a self-contained command that avoids --cached entirely.

All 25 git route tests continue to pass.
2026-05-08 09:55:10 +03:00
0b227a307c refactor 2026-04-24 17:10:02 +03:00
b7a9ef71b2 agents + semantic 2026-04-21 11:30:41 +03:00
e04bddcc4a refactor(dashboards): extract dashboard detail components
Split the large dashboard detail page into smaller, focused components: DashboardHeader, DashboardGitManager, DashboardLinkedResources, and DashboardTaskHistory to improve maintainability.
2026-04-21 08:36:48 +03:00
7a84716322 refactor: extract EnvironmentsTab to smaller semantic boundary 2026-04-20 19:18:19 +03:00
277ce7b3b3 skills + agents 2026-04-19 20:05:45 +03:00
2228a00035 semantics 2026-04-02 12:12:23 +03:00
80ec0c5eb1 prune contracts 2026-04-01 22:31:10 +03:00
89924afa8e + 2026-04-01 22:04:19 +03:00
f5c90af287 semantics 2026-04-01 21:57:51 +03:00
ace77ed64c semantics update 2026-04-01 15:30:13 +03:00
03d46cb32a semantic 2026-04-01 14:51:45 +03:00
af61096ddb root update 2026-04-01 14:25:52 +03:00
68b5b8d505 configs update 2026-04-01 14:07:02 +03:00
d4d212fd35 mcp tuning 2026-04-01 13:29:41 +03:00
d1d18df898 agents ADR promts 2026-03-27 22:00:37 +03:00
7901ce0f39 semantics 2026-03-27 21:27:31 +03:00
202823aba5 feat(ui): add chat-driven dataset review flow
Move dataset review clarification into the assistant workspace and
rework the review page into a chat-centric layout with execution rails.

Add session-scoped assistant actions for mappings, semantic fields,
and SQL preview generation. Introduce optimistic locking for dataset
review mutations, propagate session versions through API responses,
and mask imported filter values before assistant exposure.

Refresh tests, i18n, and spec artifacts to match the new workflow.

BREAKING CHANGE: dataset review mutation endpoints now require the
X-Session-Version header, and clarification is no longer handled
through ClarificationDialog-based flows
2026-03-26 13:33:12 +03:00
c56b8dad92 + clean 2026-03-22 13:00:17 +03:00
b8447e9beb some cleaning in promts 2026-03-22 13:00:06 +03:00
8847a99209 standards clean 2026-03-22 12:40:02 +03:00
a6b4c4ab59 swarm update 2026-03-21 18:34:24 +03:00
1f904a6321 promts 2026-03-21 16:09:13 +03:00
10c15d0e93 feat(ui): surface review readiness and blockers
Add readiness hints, blocker summaries, and progress cues across
the dataset review workspace to clarify partial recovery and
launch gating states.

Highlight read-only preview snapshots, pending semantic review,
clarification queue status, and unresolved validation blockers to
guide the next recommended action.

Document known limitations in backend test doubles so permission,
task dispatch, and uncovered persistence branches are explicit.
2026-03-21 15:48:28 +03:00
c76f75785e subagents promts 2026-03-21 15:48:07 +03:00