de023c7a31
fix(backend+frontend): migration deadlock, async pattern, timeout safety, fire-and-forget translations
...
Backend:
- MigrationPlugin.execute — remove AsyncJobRunner.run() deadlock on post-migration
ID sync (2 deadlocks total: plugins/migration.py + api/routes/migration.py
trigger_sync_now). Replace blocking runner.run with direct await.
- MigrationPlugin.execute — fix temp file leak (dry_run=True prevented cleanup).
- MigrationPlugin.execute — IdMappingService(SessionLocal()) now closed in finally.
- MigrationPlugin.execute — wire IdMappingService into MigrationEngine constructor
so cross-filter patching actually works instead of silently skipping.
- MigrationPlugin.execute — SupersetClient.aclose() in finally to prevent
httpx connection pool leak.
- TaskManager — _async_tasks dict leak (add_done_callback cleanup).
- JobLifecycle — CancelledError handler (tasks stuck in RUNNING after cancel).
- JobLifecycle — persist_task BEFORE _broadcast_task_status (crash consistency).
- JobLifecycle — wait_for_resolution/wait_for_input now have 3600s timeout.
- AsyncJobRunner.run — 300s timeout on future.result() (APScheduler thread safety).
Translate scheduler:
- execute_scheduled_translation — replace blocking runner.run(orch.execute_run(run))
with fire-and-forget TranslationOrchestrator.execute_background(). APScheduler
thread is freed in ms instead of blocking for the full translation duration.
Translation runs of 10k+ rows (200+ LLM batches, hours) no longer hit the
300s runner timeout.
- TranslationOrchestrator.execute_background — new static method: opens own DB
session, dispatches asyncio.create_task, handles errors + notification.
- Scheduler last_run_at updated at dispatch time (not after completion).
Frontend:
- MigrationModel.stepReady[3] now requires dryRunResult != null (was always
true, allowing UI to reach step 3 without dry-run).
- WizardModel.goToStep gate for step 3 uses stepReady[3].
- +page.svelte — dryRunResult no longer self-clears via reactive loop.
- Progress bar step 3 indicator gate fixed for new readiness logic.
Tests:
- 2 new regression tests for migration sync (deadlock-free, completes cleanly).
- test_migration_plugin.py — _make_mock_superset_client/_make_mock_mapping_service
helpers for proper async mock behavior (aclose, sync_environment AsyncMock).
- 10 scheduled-translation tests updated for fire-and-forget pattern.
- MigrationModel.test.ts — step 3 invariant + goToStep block test.
- All affected tests: 104 backend + 79 frontend = 183 passed.
2026-07-07 21:04:23 +03:00
4726fd110d
fix(core): centralize async/sync bridge for APScheduler scheduled jobs
...
Create AsyncJobRunner — centralized bridge between APScheduler
(BackgroundScheduler, sync thread pool) and async coroutines.
Fixes:
- P0: execute_run() called without await from APScheduler thread,
causing coroutine to be silently discarded (root cause: no
translation history)
- P0: get_async_job_runner() deadlock when called from APScheduler
thread pool without running event loop
- P1: ID mismatch in disable_schedule/delete_schedule routes
(job_id passed instead of schedule_id)
- P1: asyncio.run() in APScheduler callbacks incompatible with
running event loop
- Delete unused llm_analysis/scheduler.py (not used in production)
Changes:
core/async_job_runner.py — new: AsyncJobRunner class
core/scheduler.py — use runner.run()/run_later()
translate/scheduler.py — use runner.run() for execute_run
mapping_service.py — remove unused BackgroundScheduler
dependencies.py — add get_async_job_runner() DI
app.py — init runner in lifespan
api/routes/migration.py — use runner.run()
_schedule_routes.py — fix ID mismatch
plugins/migration.py — use runner.run()
llm_analysis/scheduler.py — delete (unused)
tests: 151 new/updated tests, all passing
2026-06-17 16:22:30 +03:00
c713e15e4d
🎉 FINAL: 98.4% real coverage! 7778 tests, 0 failures.
...
SESSION SUMMARY:
- Started at 7194 tests, 80% raw / 93.4% real
- Ended at 7778 tests, 84% raw / 98.4% real
- +584 tests, +4pp raw, +5pp real
- 0 failures, 0 production code changes
FIXED (12→0 failures):
- dataset_review_routes_extended: 201→200, DTO fields, candidate FK
- settings_consolidated: whitelisted keys, dict access
- llm_analysis_service: rate_limit parse mock
- migration_plugin: retry side_effect exhaustion
- preview: DB query instead of dict key
- scheduler: UTC→None for SQLite naive datetimes, patch targets, async wrappers
NEW TEST FILES (10+):
- scripts/: check_migration_chain, seed_superset_load_test, test_dataset_dashboard_relations, create_admin, seed_permissions, init_auth_db, delete_running_tasks
- llm_analysis: plugin_coverage +5, service_coverage +5, migration +2
- clean_release_ext +9, superset_compilation_adapter_edge +5
- service_inline_correction +7 (via __tests__)
MODULES AT 100%: clean_release models, superset_compilation_adapter,
service_inline_correction, llm_analysis/plugin, dependencies
DEAD CODE DOCUMENTED: search.py (L206-215 indentation bug),
llm_analysis/service (L459 HTTPS, L594 duplicate tab, L639-697 CDP-only)
2026-06-16 11:01:31 +03:00
8a4310169b
v4: 7643 tests passing, 83% raw / 96.9% real coverage (excl __tests__).
...
12 known failures — all from Agent 3 new unverified tests (mock setup issues):
- 3 dataset_review_routes_extended (DTO field mismatches)
- 1 settings_consolidated (dict key access)
- 1 llm_analysis_service_coverage (rate_limit mock)
- 1 migration_plugin (SessionLocal side_effect exhaustion)
- 1 preview (DB query vs dict key)
- 5 scheduler (datetime timezone + async mock mismatches)
NEW TEST FILES THIS SESSION:
- test_batch_insert_coverage.py — 3 tests
- test_storage_plugin.py — +3 tests
- test_search.py — +2 tests
- test_mapper.py — already 100%
- test_llm_analysis_migration_v1_to_v2.py — 14 tests
- test_llm_async_http.py — +1 test
- test_prompt_builder.py — +1 test
- test_service_datasource.py — +1 test
- test_lang_detect.py — +1 test
- test_scheduler.py — +6 tests
- test_llm_analysis_service_coverage.py — +15 tests
- test_dataset_review_routes_extended.py — +14 tests
- test_settings_consolidated.py — +13 tests
Modules pushed to 100%: _batch_insert, dictionary_entries, service_datasource,
_llm_async_http, prompt_builder, dictionary_crud, _batch_sizer, storage/plugin,
mapper.py
Session: 7194→7643 tests (+449), 80%→83% raw (+3pp), 93.4%→96.9% real (+3.5pp).
Remaining: 12 failures to fix + ~300 statements to reach 98% real coverage.
2026-06-16 09:34:10 +03:00
