Commit Graph

599 Commits

Author SHA1 Message Date
d29c70f8a4 032: add async regression tests (21 tests covering all fixed bug patterns) 2026-06-05 10:40:51 +03:00
2c9698256e 032: fix 2 critical QA issues — missing endregion + Tombstone dead helpers
C1: _llm_call.py — added missing #endregion _split_and_retry (violated INV_3)
C2: dashboards/_helpers.py — sync _find_dashboard_id_by_slug and
    _resolve_dashboard_id_from_ref typed Tombstone per INV_6 (dead code,
    callers use _async versions from git/_helpers.py)
2026-06-05 10:32:37 +03:00
b7f9d524d7 032: fix final sync→async cascade (dataset_review, parsing, validation)
- _parsing.py: parse_superset_link + _recover_dataset_binding → async
    (await get_dashboard_detail, get_chart)
  - dataset_review/orchestrator.py: start_session + _build_recovery_bootstrap → async
    (await get_dataset_detail, parse_superset_link)
  - _routes.py (dataset_review): await orchestrator.start_session()
  - validation_tasks.py: await parse_superset_link + get_dashboard_detail
2026-06-05 09:55:34 +03:00
958e5056cf 032: fix remaining sync→async propagation (17 call sites)
Core fixes:
  - service_datasource.py: fetch_datasource_metadata() → async
  - service.py: create_job(), update_job() → async (callers await)
  - _job_routes.py: await create_job/update_job

Maintenance scanners:
  - _dashboard_scanner.py: 4 functions → async (find_affected, _get_linked,
    _apply_filters, _resolve_title)
  - _chart_manager.py: 3 functions → async
  - _banner_renderer.py: rebuild_banner → async
  - _orchestrators.py: 3 orchestrators → async
  - maintenance_banner.py: await async calls

Migration:
  - dry_run_orchestrator.py: run(), _build_target_signatures() → async
  - risk_assessor.py: build_risks() → async
  - migration.py: await service.run()
  - mapping_service.py: sync_environment() → async

Dead code:
  - _helpers.py: _find_dashboard_id_by_slug marked DEPRECATED
2026-06-05 08:56:37 +03:00
531f1d5994 032: fix missing await on TranslationExecutor.execute_run() in orchestrator_exec
executor.execute_run() is async def but was called without await in
TranslationExecutionEngine.execute_run(). This returned a coroutine
instead of a TranslationRun, causing:
  'coroutine' object has no attribute 'status'
This made every translation run fail in the background execute path.
2026-06-05 08:47:56 +03:00
a8a4f8b83d 032: fix run_translation route handler — sync def → async def (asyncio.create_task needs running loop)
run_translation was def (sync FastAPI handler runs in thread pool with no event
loop), but its body calls asyncio.create_task(_background_execute()) which
requires a running event loop. Changed to async def so FastAPI runs it on the
event loop directly.

Error: 'Run failed: no running event loop'
2026-06-05 08:40:46 +03:00
410b6427e3 032: fix get_dashboards_page_async -> get_dashboards_page (method renamed during async migration)
get_dashboards_page_async() no longer exists — the sync/async split was
removed and the method is now simply get_dashboards_page() (already async).
Calls in dashboard slug resolution and git helpers were using the old name.
This caused AttributeError at runtime, making all slug-based dashboard
lookups fail with 'Dashboard not found'.
2026-06-05 08:39:05 +03:00
bdabff02af 032: add aclose() to SupersetClient (was missing, used in dashboard detail routes)
SupersetClientBase was missing aclose() method. AsyncSupersetClient had it
via override, but code creating SupersetClient directly (e.g. dashboard tasks
history route) would fail with AttributeError on await client.aclose().
2026-06-05 08:33:49 +03:00
cf7f69b4c1 032: dead code cleanup — remove sync APIClient, _llm_http, preview_llm_client, fix retry chains 2026-06-05 08:31:18 +03:00
90b191529d 032: fix tests — validate_target_table_schema became async (add await + AsyncMock)
All test methods calling validate_target_table_schema now async def + await.
Mocked async methods (resolve_database_id, execute_and_poll) use AsyncMock
instead of MagicMock since await on MagicMock raises TypeError.
2026-06-05 00:18:13 +03:00
ec4669561f 032: fix health_service _prime_dashboard_meta_cache — async get_dashboards_summary
_prime_dashboard_meta_cache was sync but called async get_dashboards_summary().
Parent get_health_summary was already async def. Made child async + added awaits.
2026-06-05 00:14:15 +03:00
3853f5505d 032: deep async propagation — orchestrator, insert, mapper, batch chains
Full async conversion for all sync callers of async SupersetClient methods:

orchestrator_sql.py: generate_and_insert_sql + _resolve_dialect → async
orchestrator_run_completion.py: complete_success → async (calls generate_and_insert_sql)
orchestrator_exec.py: execute_run → async (awaits complete_success)
orchestrator_runner.py: execute_run → async (delegates to engine)
orchestrator.py: execute_run + _generate_and_insert_sql → async
executor.py: _insert_batch_to_target → async (awaits batch insert)
_batch_proc.py: insert_batch_to_target → async
_batch_insert.py: insert_batch_to_target + _resolve_insert_backend + _execute_insert_sql → async
dataset_mapper.py: get_sqllab_mappings + run_mapping → async
  + await get_dataset, update_dataset, execute_and_poll
mapper.py: await on run_mapping + resolve_database_id calls
_run_routes.py: threading.Thread → asyncio.create_task (_background_execute async)
2026-06-05 00:13:01 +03:00
b47e0c8c73 032: fix async chain for target schema check + mapper + executor
resolve_database_id → async (was sync but called async SupersetClient methods)
  - await client.get_database(db_id)
  - await client.get_databases(...)

validate_target_table_schema → async (calls async resolve_database_id + execute_and_poll)
  - await executor.resolve_database_id(...)
  - await executor.execute_and_poll(...)

Route check_target_schema → await validate_target_table_schema(...)

MapperPlugin.execute → await executor.resolve_database_id(...)
  (execute() was already async def, just missing await)

SupersetSqlLabExecutor.execute_sql → await self.resolve_database_id()
  (was sync call to now-async method)
2026-06-05 00:04:54 +03:00
bd257607ea 032: fix UnboundLocalError for db_name in validate_target_table_schema
db_name and backend were initialized inside the try block but referenced
in the except handler. If an exception occurred before their assignment
(e.g. in resolve_database_id), the except block would raise:
  cannot access local variable 'db_name' where it is not associated
Moved initialization before try with safe defaults.
2026-06-04 23:59:11 +03:00
9fcd9d96a9 032: fix missing awaits in translate datasource endpoints (#5,6,7)
Three sync functions called async SupersetClient methods without await:
  - get_datasource_columns(): get_dataset_detail + get_database (async)
  - fetch_available_datasources(): get_datasets (async)
  - Route handlers: both calls missing await

Converted to async functions + added awaits. Both endpoints now return
proper data instead of coroutine errors.
2026-06-04 23:57:46 +03:00
80c8b2eabd 032: fix duplicate class:border-warning / class:bg-warning-light in TargetSchemaHint
Svelte 5 does not allow duplicate class: directives on the same element.
The two class:border-warning (and two class:bg-warning-light) conditions
were mutually exclusive but Svelte rejected them at compile time.
Merged into single || condition.
2026-06-04 23:56:00 +03:00
6423c7fc83 032: fix missing await on git_service.get_status() (async -> coroutine)
git_service.get_status() is async def, but was called without await,
returning a coroutine object instead of dict. The coroutine was then
used as a dict value in RepoStatusBatchResponse, causing Pydantic
ValidationError (dict type mismatch).
2026-06-04 23:54:42 +03:00
fd037206b1 032: fix async get_dataset_linked_dashboard_count passed to asyncio.to_thread
get_dataset_linked_dashboard_count is async (coroutine function), but was
passed to asyncio.to_thread() which expects a sync callable. This returned
a coroutine object instead of an int, causing:
  '>' not supported between instances of 'coroutine' and 'int'

Fix: await the async method directly inside asyncio.wait_for().
2026-06-04 23:53:26 +03:00
30a082b43d 032: fix missing await on async SupersetClient calls in resource_service.py
Found 3 missing 'await' keywords causing 'coroutine object is not iterable':
  - get_dashboards_summary()  (line 57)
  - get_dashboards_summary_page() (line 114)
  - get_datasets_summary() (line 306)

All three were calling async methods in sync context — returned coroutine
objects instead of lists/dicts, causing iteration failures.
2026-06-04 23:47:09 +03:00
25427515f1 032: fix SyntaxError in git/_base.py — positional after keyword args in run_blocking
7 calls fixed: moved kind and fn to positional to avoid
SyntaxError 'positional argument follows keyword argument'.
2026-06-04 23:37:37 +03:00
3c98c0e375 032: fix(TargetSchemaHint) — differentiate transient errors (503/504) from table not found
HIGH: 'bodyClass' and display logic updated — 503/504 errors now show
warning (yellow) styling and 'Could not verify' message instead of
destructive (red) 'Table not found'. Backend now returns 503 on pool
exhaustion and 504 on upstream timeout after async refactoring.
2026-06-04 23:34:45 +03:00
f0c526c179 032: mark all tasks [x] — feature complete
Updated status in spec.md, plan.md, tasks.md, research.md,
data-model.md, ux_reference.md, quickstart.md.
All 72 tasks completed. 34/34 tests passing.
2026-06-04 21:09:14 +03:00
3130fae68a 032: T029-T031 + T046-T048 — all remaining tests
T029: concurrent preview+schema check test
T030: static asyncio.sleep audit
T031: LLM rate-limit backoff test + 6 edge cases
T046: TaskManager concurrent tasks + cancellation tests
T047: async notifications — SMTP timeout test
T048: EventBus publish/subscribe + maxsize tests

34 total async tests passing.
2026-06-04 21:06:22 +03:00
94469ba449 032: final — tombstones, tests, cleanup
T055: APIClient tombstone in network.py
T056: AsyncSupersetClient @DEPRECATED marker
T057: _llm_http.py + preview_llm_client.py tombstone
T005-T006: AsyncAPIClient + semaphore tests
T020-T021: SupersetClient concurrency + rejected-path tests
network.py cleaned from 584 to 220 lines (orphan code removed)

All 20 async tests pass.
2026-06-04 20:57:20 +03:00
9edef064c7 032: fix tests — AsyncMock for async SupersetClient methods
All 10 preview pipeline tests pass.
2026-06-04 20:42:31 +03:00
794073a7ff 032: final — async_superset_client collapse, profile/superset_lookup async 2026-06-04 20:37:55 +03:00
22dc6827f1 032: T059 — ADR-0011 async-backend decision record 2026-06-04 20:36:53 +03:00
c1867c767f 032: T045 — git services async (run_blocking for all blocking ops)
_merge.py partially done. Tests still pending.
2026-06-04 20:36:33 +03:00
ea85bbbf95 032: Phase 5 (T036-T039) + Phase 6 (T040-T044) completed
T036: superset_compilation_adapter fully async
T037: fileio.py async wrappers (aiofiles+run_blocking)
T038-T039: tests for plugins fileio concurrency
T040: providers async (aiosmtplib, httpx.AsyncClient)
T041: dispatch_report parallel via asyncio.gather
T042: TaskManager ThreadPoolExecutor->asyncio.create_task
T043: EventBus asyncio.Queue(maxsize=10000)
T044: lifecycle async context manager

Remaining: T045 git/_base.py, T046-T048 tests, T005-T020-T021 tests
2026-06-04 20:30:43 +03:00
777e2b53ac 032: Phase 5 US3 — backup, git, llm_analysis, storage async
T032-T035 completed. T036 partial.
Remaining: T036 superset_compilation_adapter, T037 fileio.py, cascade updates
2026-06-04 20:17:13 +03:00
2a86ab6fe1 032: Phase 4 US2 — Translate plugin fully async
T022-T028: All translate methods async.
- _llm_async_http.py created (httpx.AsyncClient+asyncio.sleep)
- Old _llm_http.py and preview_llm_client.py preserved (tombstone later)
- superset_executor, preview, executor, run_source, llm_call all async

RATIONALE: httpx.AsyncClient + asyncio.sleep instead of time.sleep.
REJECTED: AsyncOpenAI SDK — doesn't support custom base_url.
2026-06-04 20:09:45 +03:00
3b7778b1d1 032: T019 — remaining route files migrated to async SupersetClient
All routes (assistant, migration, datasets, git) now use AsyncSupersetClient.
_helpers.py sync->async for dashboard ref resolution.
_detail_routes.py import fixed.

Known residual: MigrationDryRunService and IdMappingService still sync.
2026-06-04 20:02:33 +03:00
f313fd11ef 032: Phase 3 US1 — 13 mixins migrated to async + dashboard routes
T011-T017: All SupersetClient mixins now async.
T018: _detail_routes.py uses registry/AsyncSupersetClient.
T019: Partial — routes/environments, settings, profile, listing async.

RATIONALE: Big-bang merge of sync+AsyncSupersetClient.
REJECTED: dual-stack.

Remaining T019: assistant/*, migration, datasets, git helpers still use sync.
2026-06-04 19:55:43 +03:00
5a0a2c56f1 032: Phase 1-2 — setup deps + AsyncAPIClient extend + client_registry + executors
Phase 1 (Setup):
- T001: requirements-dev.txt with pytest-httpx
- T002: aiofiles added to requirements.txt
- T003: aiosmtplib added to requirements.txt
- T004: EnvironmentConfig extended (connection_pool_size, etc.)
  + AppAsyncRuntimeConfig created (executor workers, shutdown)

Phase 2 (Foundational):
- T007: AsyncAPIClient extended — semaphore parameter, request() method
- T008a: SupersetClientRegistry — singleton per-env client/semaphore/lock
- T008b: run_blocking helper + bounded executors (db/file/git)

RATIONALE: httpx.AsyncClient replaces requests.Session; singleton
registry ensures global per-env semaphore; named executors prevent
thread pool exhaustion.
REJECTED: asyncio.to_thread (default executor, no backpressure);
per-request clients (lose pooling); dual-stack (rejected at clarify).
2026-06-04 19:45:57 +03:00
db03b970ed tasks ready 2026-06-04 19:41:08 +03:00
af3bef625d test(frontend): add model unit tests for Screen Models
Add L1 invariant tests for all Screen Models:
- DashboardDetailModel: test load, delete, pagination, column filter
- DashboardHubModel: test load, environment switching, selection, git actions
- DatasetDetailModel: test load, edit, delete
- DatasetsHubModel: test load, filter, pagination
- DictionaryDetailModel: test load entries, add, edit, delete, search
- LLMReportModel: test load, filter, report generation
- TranslateHistoryModel: test load runs, filter, pagination
- ValidationRunDetailModel: test load details, records
- ValidationTasksListModel: test load tasks, status transitions

Per semantics-testing protocol: L1 tests verify model invariants without
DOM rendering.
2026-06-04 16:17:59 +03:00
60f2987f1c feat(frontend): add admin/tools pages, i18n, UI improvements, route annotations
New pages:
- /admin: admin overview page with links to user/role/settings/LLM management
- /tools: tools overview page with links to mapper/debug/storage/backup tools

i18n:
- nav.json (en/ru): add description keys for admin and tools sub-items
- migration.json (en/ru): add help tooltips and step-by-step instructions
  for the database mapping workflow

UI components:
- EnvSelector: add optional helpText with HelpTooltip
- MappingTable: add HelpTooltip for status column
- MultiSelect: add id for accessibility, fix label element structure
- Input: fix reactive id assignment with ()
- Select: fix reactive id assignment with ()

Routes:
- routes.ts: add admin.overview() and tools.overview() routes
- dashboards/+page.svelte: add @RELATION BINDS_TO annotation
- migration/mappings/+page.svelte: add HelpTooltip, Card imports, help texts
- translate pages: minor annotation updates

Other:
- .gitignore: add backend/:memory (SQLite test artifact)
2026-06-04 16:17:52 +03:00
38495e6f82 feat(backend): add is_regex to dictionary API routes
Add is_regex field to list, add, and edit dictionary entry API responses,
and pass is_regex through to DictionaryEntryCRUD methods.
2026-06-04 16:17:36 +03:00
a49b537b72 feat(frontend): update translate components + BackupManager
Translate components:
- BulkReplaceModal: add dictionary selection dropdown to save replacements
  directly to a dictionary after applying bulk find-replace
- CorrectionCell: improve inline edit UX with better state handling
- TermCorrectionPopup: enhanced popup for term corrections
- ConfigTabForm: update form field bindings
- RunTabContent: minor layout adjustments
- ScheduleConfig: improved schedule configuration UI
- TranslationPreview, TranslationRunGlobalIndicator, TranslationRunProgress:
  UX polish and state management improvements

BackupManager:
- Add AbortSignal.timeout(30s) to prevent infinite loading state
- Add onDestroy AbortController cleanup to prevent stale state
- Add error toasts for failure states (was missing — state hung forever)
- Import API_REQUEST_TIMEOUT from api.ts
2026-06-04 16:17:14 +03:00
90a24d2032 refactor(frontend): migrate health center page to HealthCenterModel
Extract state management from inline health page into HealthCenterModel:

- HealthCenterModel.svelte.ts (new): hosts all state atoms (),
  derived values (), and core actions (load, filter, delete)
- health page reduced from ~120 to ~27 lines of script — thin shell
  delegating to model; only DOM/template concerns remain
- Integration test updated for model-based architecture
- HealthCenterModel.test.ts (new): model invariant tests
2026-06-04 16:17:03 +03:00
bc72504892 refactor(frontend): migrate dataset review to DatasetReviewModel
Extract all state management from the inline page into a dedicated
DatasetReviewModel class following the Screen Model pattern:

- DatasetReviewModel.svelte.ts (new): hosts all state atoms (),
  derived values (), and core actions (load, submit, export)
- review-workspace-helpers.ts moved from routes/ to /helpers/
- useReviewSession.ts moved from routes/ to /api/dataset-review/
- DatasetReviewModel.test.ts (new): model invariant tests
- [id]/+page.svelte: reduced from ~380 to ~190 lines — thin shell
  delegating to model; only navigation/DOM concerns remain inline
- Old files deleted: routes/datasets/review/{review-workspace-helpers,useReviewSession}.ts
- Updated ux test for new model-based architecture
2026-06-04 16:16:51 +03:00
3be05d7b88 feat(frontend): add AbortSignal/timeout support to API client
- Add FetchOptions.signal for request cancellation (timeout or unmount)
- Propagate signal to native fetch() in fetchApi, fetchApiBlob, postApi,
  requestApi, patchApi, putApi, and deleteApi
- Export API_REQUEST_TIMEOUT constant (30s default)
- Add @INVARIANT for signal propagation contract
- Add @RATIONALE documenting the anti-loop protocol motivation
2026-06-04 16:16:25 +03:00
2d4caefeff fix(backend): resolve test regressions
- Remove invalid sqlite=True parameter from composite index migration
  (sqlite=True is not a valid op.create_index parameter)
- Fix test_assistant_api assertions for updated response format
- Fix test_git_status_route edge case assertions
- Fix test_audit_service expected value after metric changes
- Fix test_session_repository assertion after store refactor
2026-06-04 16:16:18 +03:00
e0577e8caa test(backend): add is_regex dictionary enforcement and metrics tests
test_enforce_dictionary.py (new):
- Verify regex patterns are matched correctly in translation enforcement
- Verify invalid regex patterns are gracefully skipped

test_metrics_cumulative.py (new):
- Verify metrics calculations produce correct cumulative statistics

test_dictionary_crud.py (extended):
- test_add_entry_regex: verify creating entry with is_regex=True
- test_add_entry_regex_validation: verify invalid regex raises ValueError

test_dictionary_filter.py (extended):
- Add test coverage for regex-based dictionary entry filtering
2026-06-04 16:16:10 +03:00
000c2171b6 feat(backend+frontend): add is_regex support to dictionary entries
Add support for regex-based dictionary entries across the full stack:

Backend:
- DictionaryEntry model: add is_regex column (Boolean, default False)
- DictionaryEntryCRUD: validate regex on add_entry(), compile on creation
- _enforce_dictionary: match by regex pattern when is_regex=True
- dictionary_filter: support is_regex in filter/query
- metrics: include is_regex entries in metrics calculations
- Alembic migration: 20260604_add_is_regex_to_dictionary_entries
- Merge migration: 351afb8f961a (merge is_regex + composite index heads)

Frontend:
- DictionaryDetailModel: add is_regex field to DictionaryEntry interface,
  EditForm, and addForm; sync edit/add form state with backend schema
2026-06-04 16:16:02 +03:00
a95c15caf1 refactor(backend): split translate run routes into edit/history modules
Extract inline edit, bulk find-replace, and override language endpoints from
_run_routes.py into dedicated _run_edit_routes.py and _run_history_routes.py
modules to reduce module complexity below INV_7 limits.

Changes:
- _run_routes.py now only handles execution, retry, and cancel endpoints
- _run_edit_routes.py (new): inline edit, bulk find-replace, override language
- __init__.py registers the new route modules
- schemas/translate.py: add imports for extracted endpoints
2026-06-04 16:15:48 +03:00
78fa5ee0e0 fix(backend): migrate trace middleware to raw ASGI for contextvar isolation
BaseHTTPMiddleware (Starlette 0.50.0) uses anyio.create_task_group() internally,
creating separate asyncio tasks for dispatch vs call_next. ContextVars set in
dispatch() were not visible to outer middleware like log_requests.

Converting to raw ASGI middleware ensures trace_id is seeded in the root task
context, visible to ALL middleware layers.

Key changes:
- Replace BaseHTTPMiddleware with raw ASGI __call__(self, scope, receive, send)
- UUID v4 validation: check parsed.version == 4 explicitly instead of relying
  on uuid.UUID(hex=..., version=4) which silently mutates non-v4 UUIDs
- Add @RATIONALE and @REJECTED tags per semantics-core protocol
- Update app.py comment to document the architectural decision
2026-06-04 16:15:40 +03:00
d883dc2cdb chore: update agent model configs 2026-06-04 16:15:32 +03:00
c297db9f6b fix: QA issues — composite index, anchors, fallbacks
- Add composite index ix_translation_records_run_source_hash
  for NOT EXISTS dedup subquery performance + Alembic migration
- Remove duplicate #endregion in orchestrator.py (INV_3)
- Replace hardcoded RU fallback 'Статус' with 'Status'
- Add early return guard to loadMoreRecords()
- Show records summary always when recordsTotal > 0
0.2.3
2026-06-04 13:33:17 +03:00
0a02b19dfb feat: add deduplicate + metrics to Run tab
Backend:
- deduplicate param in GET /runs/{id}/records — NOT EXISTS subquery
  with (created_at, id) tiebreaker for one row per source_hash
- source_data and source_hash added to records JSON response
- fix missing status import in _run_history_routes.py (NameError)
- fix tab/space mixup in orchestrator_aggregator.py
- remove duplicated @RELATION edges in metrics.py
- fix #region/#endregion style and @PURPOSE→@BRIEF in metrics.py

Frontend:
- RunTabContent: summary metrics bar (fetchJobMetrics) with HelpTooltips
- RunTabContent: trigger_type badge, duration, cache rate in run rows
- TranslationRunResult: deduplicate=true by default, paginated Load more
- TranslationRunResult: per-language token_count and estimated_cost
- TranslationRunResult: collapsible batch breakdown with timing
- TranslationRunResult: Bulk Replace button in header and records table
- TranslationRunResult: source_data key values shown under source text

i18n: all new keys in EN + RU (load_more_records, showing_records,
sum_*, help_sum_*, trigger_*, batch_*, duration_label, cost_label,
cache_rate, load_more_records)
2026-06-04 13:23:10 +03:00