Commit Graph

17 Commits

Author SHA1 Message Date
4205618ee6 chore: eliminate all deprecation warnings from tests and linter
Warnings fixed:
- datetime.utcnow() → datetime.now(UTC) across 48+ files (src/ + tests/)
- datetime.utcnow (callback ref) → lambda: datetime.now(UTC) in model fields (18 files)
- Pydantic class Config → model_config = ConfigDict(...) (16 files)
- Pydantic .dict() → .model_dump() (8 files)
- ConfigDict(allow_population_by_field_name=True) → validate_by_name=True
- SQLAlchemy declarative_base() import path updated
- FastAPI on_event → lifespan context manager (app.py)
- Import sorting (ruff I001) auto-fixed across all files
- Fixed broken re-export chains that ruff F401 cleanup broke:
  _validate_bcp47: service.py now imports from dictionary_validation directly
  job_to_response: _job_routes.py and test imports from service_utils directly
  fetch_datasource_metadata: restored re-export in service.py
- Added missing TranslateJobService import in _job_routes.py (was deleted by F401)
- Added ConfigDict(protected_namespaces=()) for DashboardDatasetItem schema field
- pytest.ini: replaced deprecated importmode with asyncio_mode

All 440 tests pass with zero deprecation warnings.
2026-05-26 19:18:28 +03:00
9337a4cecd fix: add missing schema columns and Alembic model imports
- Added 4 missing columns to _ensure_translation_jobs_columns() inline
  migration: target_language_column, target_source_column,
  target_source_language_column, disable_reasoning — these were in the
  SQLAlchemy model but had no Alembic migration or _ensure_*() coverage
- Added missing api_key, assistant, clean_release model imports to
  alembic/env.py so autogenerate can see all tables
2026-05-26 19:00:04 +03:00
2fa8edfa03 fix: is_admin fallback for legacy Admin roles + migration data fix
Migration now sets is_admin=true for existing Admin roles.
has_permission falls back to role.name == 'Admin' check if
is_admin flag is missing (backward compat for pre-migration roles).
2026-05-26 15:29:14 +03:00
6b960b0eb3 fix: add is_admin column migration + additive safety net
Alembic migration 86c7b1d6a710 adds is_admin BOOLEAN DEFAULT FALSE
to the roles table. Fixes login crash after M-3 change:

  psycopg2.errors.UndefinedColumn: column roles.is_admin does not exist

Also adds _ensure_roles_is_admin_column() as an additive migration
safety net for environments that don't run alembic automatically.

To apply: cd backend && source .venv/bin/activate && alembic upgrade head
2026-05-26 15:28:03 +03:00
31680a1bc9 fix(maintenance): auto-expiry + adaptive markdown height + tests
- Auto-expiry: expired events auto-end on GET /events via task manager
- Height: _estimate_markdown_height adapted to unit=8px scale with padding detection
- CRITICAL-1: removed dead import remove_chart_from_position (QA finding)
- CRITICAL-2: added chart alive check in ensure_banner_chart (QA finding)
- CRITICAL-3: fixed test assertions update_markdown_chart → update_banner_on_dashboard
- @POST contract: fixed return range [2,12] → [19,200]
- Tests: 8 new tests (auto-expiry + layout height)
2026-05-25 08:36:33 +03:00
0653a75ee7 fix(db): add ondelete cascade to all FK constraints, fix multimodal flag persistence
- Add ondelete=CASCADE/SET NULL to all environment and translate FK constraints
- Add is_multimodal to GET /api/settings/consolidated response
- Fix toggleActive to not overwrite is_multimodal with false
- New SearchableMultiSelect component for dashboard search with multi-select
- Fix validation task page: task data unwrapping, date formatting, dashboard multi-select
- Fix infinite effect loop on dashboards page via queueMicrotask guard
- 3 new Alembic migrations (merge f0e9d8c7b6a5, translate b0c1d2e3f4a5)
2026-05-22 09:21:24 +03:00
48cc02ea0b feat: LLM Validation section — 9 API endpoints + 3 Svelte pages + Playwright optimization
Backend:
- /api/validation/ CRUD for validation tasks (policies)
- Run history with 6 filters + pagination
- Alembic migrations (provider_id, policy_id)
- 28 pytest tests

Frontend:
- /validation — task list page with status filters + CRUD
- /validation/[id] — task config (Config + History tabs)
- /validation/history — run history with metrics
- Sidebar nav entry under Operations
- i18n en/ru

Playwright (ScreenshotService):
- Removed 25s hardcoded sleeps → conditional waits
- CDP fallback to full_page screenshot
- Total timeout enforcement (120s)
- Debug screenshots → temp dir with cleanup

QA fixes:
- Debug screenshot accumulation in production (tempdir + rmtree)
- Frontend API payload field name mismatch
- History issues field reference fix
- delete_runs scoped by policy_id

Belief protocol:
- @RATIONALE/@REJECTED on 27 C4+ contracts

Closes: VALIDATION-REWORK-2026
2026-05-21 20:29:36 +03:00
68740cd9ed semantic 2026-05-20 23:54:53 +03:00
084f782065 lang detect 2026-05-20 17:15:31 +03:00
6988e63967 chore: commit remaining pre-existing changes
- Agent configs (.opencode/agents/)
- Backend: alembic, routes, app, utils, scripts
- Frontend: package.json, vite, components, e2e infra
- Specs: 028-llm-datasource-supeset updates
- Docker e2e config and Playwright setup
2026-05-17 19:23:07 +03:00
b466ac6211 feat(translate): add translation cache with source_hash dedup + enforce dictionary post-processing
- Add source_hash column (SHA256 of source_text+source_data+dict_hash+config_hash) to TranslationRecord for cache dedup
- Add Alembic migration b1c2d3e4f5a6 for the new column and composite index (source_hash, status)
- Implement _compute_source_hash / _check_translation_cache in executor.py — before LLM call, check if same source+dict+config combo was already translated successfully
- Implement cache-aware routing in _process_batch: full cache hit → skip LLM, partial hit → route to LLM
- Store source_hash on all new TranslationRecord rows for future cache hits
- Strengthen dictionary prompt instructions from 'use when applicable' to 'MUST use — mandatory'
- Add _enforce_dictionary() post-processing: force-replace dictionary terms in LLM output if LLM ignored them
- Fix CRITICAL: use run.dict_snapshot_hash/config_hash (exist on TranslationRun, not TranslationJob)
- Fix MAJOR: verify cached languages cover ALL target_languages before accepting cache hit
- Fix MAJOR: prevent regex back-reference injection in dictionary enforcement (use lambda)
- Fix: add joinedload to cache lookup to avoid N+1 queries
- Fix: remove redundant single-column index (composite index is sufficient)
- Fix(frontend): parse paginated /translate/dictionaries response correctly (result.items)
2026-05-16 00:42:07 +03:00
77d55c8e69 refactor(translate): remove all deprecated columns from model + DB
Removed 6 legacy columns replaced by Phase 11 multi-language models:
- TranslationJob.target_language → target_languages (JSON)
- TranslationRecord.{llm_translation, user_edit, final_value} → TranslationLanguage
- TerminologyDictionary.{source_language, target_language} → DictionaryEntry per-entry

Affected files: model, schemas, 5 plugins, 4 test files
Alembic migration: aa1b2c3d4e5f (DROP COLUMN IF EXISTS)
Production DB: columns dropped via ALTER TABLE

Tests: 234 passed, 0 failed
2026-05-15 00:06:41 +03:00
6717e2551d fix(translate): add missing target_languages column to production DB
Migration 2a7b8c9d0e1f adds target_languages (JSON) to translation_jobs.
Uses ADD COLUMN IF NOT EXISTS for idempotent PostgreSQL deployment.

Fixes 500 error on GET /api/translate/jobs:
  column translation_jobs.target_languages does not exist

Migration chain is now clean: 5 linear revisions, no orphans.
2026-05-14 23:51:25 +03:00
cc1e57088d chore: remove legacy SQLite migration script + fix stale comments
- Delete migrate_sqlite_to_postgres.py (one-time script, no longer needed)
- Fix @INVARIANT in mappings.py: SQLite → PostgreSQL
- Remove deprecated source_language column from TranslationJob model
  + Alembic migration 8dd0a93af539 to drop column if exists
- Remove source_language from TranslateJobResponse schema

Fixes recurring 'source_language does not exist' scheduler error
2026-05-14 23:48:13 +03:00
9f3f6611a1 feat(translate): complete Phase 10-11 — full spec 028 closure (T075-T134)
Phase 10 closure:
- T075: NotificationService wiring for scheduled-run failures
- T077: Semantic audit via axiom MCP (0 warnings)
- T078: Quickstart validation — all 165+280 tests pass

Phase 11 completion:
- T083-T086: Multi-language models/schemas/Alembic migration
- T087-T090: Auto-detect source language (BCP-47) + tests
- T091-T095: Multilingual dictionaries with language-pair filtering
- T096-T108: Multi-language job config → preview → execution pipeline
- T109-T118: Inline correction + BulkReplaceModal + CorrectionCell + tests
- T119-T122: Per-language history and MetricSnapshot
- T123-T134: Context-aware corrections (Jaccard similarity, priority flagging, context_data)

New files: Alembic migration, test_scheduler, test_inline_correction,
BulkReplaceModal, vitest for CorrectionCell + BulkReplaceModal

Total: 134/134 tasks complete. Backend 165p, Frontend 280p.
2026-05-14 21:14:04 +03:00
1853d008e9 fix(translate): CR1-CR2 migration + H1-H4 code review bugs
CR1: Add needs_review, language_overridden to translation_languages migration
CR2: Add per_language_metrics to translation_metric_snapshots migration
H1: BulkFindReplaceService.preview() now uses replacement_text (not find pattern)
H2: get_run_records now includes per-language data via selectinload
H3: Preview per-language approve/reject uses correct (jobId, rowId, lang) args
H4: _compute_config_hash now includes target_languages
2026-05-14 17:38:52 +03:00
bb0fbfdafd feat(translate): multi-language optimization (Phase 11)
- Auto-detection of source language per row via LLM (US6)
- Multi-target translation — one LLM call for N languages (US1-US3)
- Language-aware storage: TranslationLanguage, per-language stats
- Multilingual dictionaries with language-pair-aware filtering (US7)
- Inline correction on any run result + submit-to-dictionary (US8)
- Context-aware dictionary: auto-capture row context, usage notes,
  Jaccard similarity, priority flagging in LLM prompts (US8b)
- Configurable preview sample size 1-100, cost warning at >30
- Per-language history & metrics with MetricSnapshot preservation
- 36 files, +5022/-373, all specs GRACE-Poly v2.6 compliant
2026-05-14 17:12:41 +03:00