fc7e6c3c15
test: add 7 more test modules — assistant cmd parser, history, dispatch, admin routes, dataset review, maintenance + semantic resolver
2026-06-15 16:06:18 +03:00
f75c15dbc6
test: massive coverage expansion — 15 new test modules + assistant tool fixes + orthogonal testing
...
- 10 translate plugin test files (100% coverage on 12 modules)
- assistant/handler tools: 85+ tests covering dispatch, registry, resolvers, routes, llm_planner, 13 tool handlers
- clean release: artifact_catalog_loader, mappers, approval, publication tests
- API routes: translate_helpers, validation_service extensions, datasets to 100%
- notifications: providers/service tests
- services: profile_preference_service
- docs/orthogonal-test-report.md — full speckit.tests audit
- Fixes: 3 git_base async mock failures, 4 assistant handler permission-check patches
- .gitignore: coverage artifacts
2026-06-15 15:38:59 +03:00
654f92e7ad
fix(test): resolve 3 remaining git_base test interaction failures
...
Root cause: unittest.mock.patch with new_callable=AsyncMock fails in
full-suite context due to asyncio event loop state from earlier tests.
Fix: use direct module dict patching (gb_mod.run_blocking = AsyncMock())
instead of patch() context manager. This bypasses the mock machinery
interaction with inherited event loop state.
Also add conftest.py in tests/services/git/ with event_loop fixture
for per-function isolation.
Result: 2602 passed, 0 failed, 3 skipped, 1 xpassed
2026-06-15 15:05:26 +03:00
ce0369ae5c
test(backend): add 55+ test files to push coverage to 98%
...
Subagents delivered tests across all uncovered backend modules:
Schemas (100%): agent, auth, health, profile, settings, validation
Services (98-100%): auth, profile, health, llm, mapping, resource,
security, git, superset_lookup, sql_table_extractor, rbac
API routes (new): auth, admin, health, environments, plugins,
dashboards (helpers, projection, actions, listing),
git (config, deps, env, helpers)
Clean Release (100%): DTO, facade, policy_engine, stages,
repos, preparation, source_isolation, compliance
Git services: base, remote_providers
Agent module: app, run, middleware, langgraph_setup
Core: trace, cleanup, ws_log_handler, timezone, auth (config/oauth/security), matching
Reports: normalizer, report_service, type_profiles
Notifications: service, providers
Also:
- .gitignore: add .coverage, *.cover, coverage-* dirs
- src/schemas/auth.py: fix AD group DN regex (comma in CN=...)
- Remove co-located src/services/__tests__/ (caused pytest module collision)
2026-06-15 13:55:57 +03:00
27a20cbbeb
fix(ssl): replace verify=True with ssl.create_default_context() for corporate CA support
...
Core fix: all httpx.AsyncClient instances now use ssl.create_default_context()
instead of bool verify=True, which uses certifi and ignores system CA store.
This makes corporate CA certificates installed via update-ca-certificates
visible to Python HTTP clients.
Files:
- async_network.py: AsyncAPIClient.__init__ converts True→SSLContext
- client_registry.py: get_client converts bool→SSLContext before passing
- notifications/providers.py: _get_http_client uses ssl.create_default_context()
- services/git/_base.py: GitServiceBase uses ssl.create_default_context()
- translate/_llm_async_http.py: _get_verify returns SSLContext (not string)
Test: new integration test with 3-tier PKI (Root→Intermediate→Server),
TLS-protected Superset container, and custom CA installation via
update-ca-certificates or SSL_CERT_DIR fallback.
- conftest.py: added ca_chain, install_custom_ca, superset_tls_env fixtures;
parameterized superset_container for TLS mode
- test_superset_tls_custom_ca.py: 8 tests (openssl -CApath, -CAfile certifi,
httpx capath, httpx certifi, AsyncAPIClient, SupersetClient, fingerprint, verify=False)
2026-06-15 10:32:17 +03:00
997329e2a5
fix(agent): resolve ModuleNotFoundError for backend, add E2E test infra
...
- Dockerfile.agent: fix CMD (python -m src.agent.run), use backend/requirements.txt,
minimal COPY (only src.agent + src.core.cot_logger), add GRACE contract
- docker-compose.yml: SERVICE_TOKEN_SECRET -> SERVICE_JWT (match code)
- docker-compose.enterprise-clean.yml: same env var fix
- docker/.env.agent.example: same env var fix
- build.sh: same env var fix
- chore: semantics-testing SKILL.md, backend tests, pyproject.toml
2026-06-14 15:41:46 +03:00
50d6b9226e
test(translate): add tests for ConnectionService, DbExecutor, orchestrator direct DB dispatch
...
- 9 new enhancement test files: test_connection_service.py,
test_db_executor.py, test_orchestrator_direct_db.py, test_batch_insert.py,
test_lang_stats.py, test_response_field_coverage.py, test_retry.py,
test_run_service.py, test_sql_insert_service.py
- 5 new integration tests: test_superset_sqllab_e2e.py,
test_translate_clickhouse.py, test_translate_corrections.py,
test_translate_schedules.py, test_translate_status_fk.py
- Updated existing tests for insert_method/connection_id fields
2026-06-11 19:12:45 +03:00
d24745a01e
fix(superset): resolve JWT auth — install psycopg2 + superset_config.py
...
Root cause: Superset 4.1.2 ignores SQLALCHEMY_DATABASE_URI env var —
always falls back to SQLite. With separate init+web containers, the
init container's SQLite DB is lost → web container has no admin user
→ JWT login returns 401.
Fix:
- Install psycopg2-binary at container start (python -m pip install)
- Create /tmp/superset_config.py via heredoc that reads SUPERSET_DB_URI
- Set SUPERSET_CONFIG_PATH=/tmp/superset_config.py
- Use Docker bridge IP (not localhost) for Postgres from inside container
Now all 10 integration tests pass, including:
- test_jwt_login_success (access_token + refresh_token)
- test_jwt_authenticated_api_call (Bearer token → /api/v1/dashboard/)
- 4 health checks + 2 form-auth + 2 client construct
2026-06-11 15:23:12 +03:00
1c9c2c298e
test(superset): add SQL Lab API and executor integration tests
...
- test_superset_sqllab_api_integration.py (8 tests): raw REST API with form-based
auth — login via /login/, CSRF token, /api/v1/database/ listing,
/api/v1/sqllab/execute/ (returns structured error without configured DB),
400 on missing database_id, CSRF protection, 404 for non-existent DB,
JWT failure documented per ADR-0012
- test_superset_sqllab_executor_integration.py (9 tests): httpx.AsyncClient
with form-based auth — DB listing, SQL Lab, database by ID,
SupersetSqlLabExecutor constructor + resolve_database_id,
SupersetClient with container URL, batch_insert module import,
raw API column listing
Total integration test suite: 25 tests across 3 files
(conftest: 6 fixtures — superset_container, superset_url, superset_admin_headers)
2026-06-11 14:48:27 +03:00
79f67e1952
test(superset): add Testcontainers setup for Apache Superset integration tests
...
- Add 6 fixtures to integration conftest: superset_db_url, superset_secret_key,
superset_admin_password, superset_container (init+web), superset_url,
superset_admin_headers
- Two-container architecture: init (db upgrade → create-admin → init) +
web (superset run -p 8088)
- Superset 4.1.2 pinned (6.x incompatible: no psycopg2, SUPERSET__ prefix required)
- 8 integration tests cover health, form-login auth, SupersetClient construction
- Document decision in ADR-0012 with architecture rationale, rejected alternatives,
and migration path to Superset 6.x
- Update docs/architecture.md with testing infrastructure overview
2026-06-11 14:45:31 +03:00
7760214170
test(agent): extend coverage — agent handler, confirmations, conversation API, tools
...
- test_agent_handler: additional edge cases for streaming, HITL, file upload
- test_confirmations: HITL confirm/deny lifecycle coverage
- test_conversation_api: conversation save/load persistence tests
- test_langchain_tools: tool registration, dual-auth header propagation
- ConversationList.test.ts (frontend): conversation list component tests
- conftest: shared fixtures for agent tests
- task_manager/manager: minor fixes from test coverage
- tasks.md/test-documentation.md: spec and test documentation updates
- speckit.test.md: speckit workflow documentation update
2026-06-10 16:38:06 +03:00
143f14d516
chore: remainder — backend test infra, agent config, docker, i18n, frontend ui
...
- Backend: alembic env, config manager/models, dependencies, translate plugin
- Backend tests: async_sync_regression, integration tests, git services, test_agent
- Docker: docker-compose.yml updates
- Agent: qa-tester.md update, semantics-testing SKILL.md update
- Frontend: TopNavbar, sidebarNavigation, FeaturesSettings, FeatureGate
- i18n: assistant.json en/ru locale updates
- New: frontend/src/lib/components/agent/ directory
2026-06-10 15:06:36 +03:00
f87c873027
test(app): add split app.py tests and extend migration_engine coverage
...
- app.py split from 1966-line test_app.py into 7 files (68 tests, 92% coverage):
- test_app_lifespan: lifespan, ensure_initial_admin_user
- test_app_handlers: exception handlers, HSTS
- test_app_middleware: log_requests, middleware chain
- test_app_ws_auth: _authenticate_websocket
- test_app_ws_endpoint: WS main loop, 5 endpoint handlers
- test_app_ws_events: task/maintenance/dataset/translate WS streams
- test_app_spa: SPA serving, TestClient integration
- migration_engine: extended coverage with init, edge cases, error paths
2026-06-10 14:57:18 +03:00
442596a3dc
test(services): add unit tests for profile_preference, resource, validation_service
...
- profile_preference_service: 19 tests, 100% coverage (CRUD, validation, encryption,
DTO conversion with mocked AuthRepository and EncryptionManager)
- resource_service: 50 tests, 100% coverage (dashboard/dataset enrichment with
git/task status, pagination, activity summary, datetime normalization)
- validation_service: 63 tests across 3 files, 97% coverage (provider validation,
environment validation, source resolution, run/record conversion, trigger_run,
create/update/delete tasks, list/filter runs, get_run_detail)
2026-06-10 14:57:11 +03:00
c33792c8ee
test(services): add unit tests for 7 service-layer modules
...
- profile_utils: 40 tests, 100% coverage (sanitize, normalize, mask, validate payload)
- security_badge_service: 17 tests, 100% coverage (role/permission extraction, security summary)
- services_mapping: 4 tests, 100% coverage (client resolution, get_suggestions)
- superset_lookup_service: 10 tests, 100% coverage (resolve environment, lookup success/degraded)
- notification_providers: 16 tests, 95% coverage (SMTP, Telegram, Slack providers)
- llm_provider: 25 tests, 91% coverage (mask_api_key, CRUD with encrypted API keys)
- rbac_permission_catalog: 12 tests, 79% coverage (route scanning, sync to DB)
2026-06-10 14:57:04 +03:00
54a0317e98
test(core): add unit tests for 7 core utility modules
...
- executors: 13 tests, 100% coverage (init/shutdown/run_blocking/run_cpu_blocking)
- fileio_utils: 33 tests, 45% coverage (sanitize_filename, get_filename_from_headers,
calculate_crc32, create_temp_file, remove_empty_directories, create_dashboard_export,
consolidate_archive_folders)
- client_registry: 14 tests, 92% coverage (get_client, get_superset_client,
get_semaphore, get_auth_lock, shutdown)
- cot_logger: 24 tests, 100% coverage (seed/set/get trace_id, push/pop span,
structured log with markers, MarkerLogger proxy)
- encryption: 12 tests, 100% coverage (key validation, encrypt/decrypt cycle)
- rate_limiter: 9 tests, 100% coverage (ban logic, window pruning, per-IP isolation)
- auth/logger: 10 tests, 100% coverage (_mask_details, log_security_event)
2026-06-10 14:56:48 +03:00
2222261157
tasks read
2026-06-09 11:44:20 +03:00
fbe0ba122c
037: fix 99 failing tests — missing await after async migration
...
Fixed async/sync boundary bugs across 14 test files. Root cause:
async def methods called without await in sync test functions.
Fixed files:
- test_translate_jobs.py (10): create_job/get_job/update_job/delete_job
- test_translate_scheduler.py (5): create_schedule/update/delete
- test_datasets.py (14): AsyncMock + corrected patch target
- test_mapping_service.py (11): sync_environment + MockSupersetClient
- test_defensive_guards.py (6): GitService/SupersetClient guards
- test_maintenance_service.py (29): all 6 maintenance services
- test_dry_run_orchestrator.py (1): run() without await
- test_dashboards_api.py (23): registry client via AsyncMock
- test_validation_tasks.py (4): trailing slash in POST URL
- test_superset_matrix.py (3): AsyncMock for compile_preview
- test_payload_reduction.py (6): LLMClient._optimize_image wrapper
- test_compliance_task_integration.py (2): event_bus ref
- test_smoke_plugins.py (1): flusher_stop_event fallback
- test_task_manager.py (1): _flusher_stop_event/thread fallback
Remaining 31 failures in test_task_manager.py (29) and
test_smoke_plugins.py (1) are pre-existing async migration gaps
(_flusher_stop_event moved to event_bus), not from this PR.
2026-06-05 15:43:35 +03:00
909b568784
032: add async regression tests (21 tests covering all fixed bug patterns)
2026-06-05 10:40:51 +03:00
5eb116509b
032: dead code cleanup — remove sync APIClient, _llm_http, preview_llm_client, fix retry chains
2026-06-05 08:31:18 +03:00
9f9ad91345
032: T029-T031 + T046-T048 — all remaining tests
...
T029: concurrent preview+schema check test
T030: static asyncio.sleep audit
T031: LLM rate-limit backoff test + 6 edge cases
T046: TaskManager concurrent tasks + cancellation tests
T047: async notifications — SMTP timeout test
T048: EventBus publish/subscribe + maxsize tests
34 total async tests passing.
2026-06-04 21:06:22 +03:00
b190414747
032: final — tombstones, tests, cleanup
...
T055: APIClient tombstone in network.py
T056: AsyncSupersetClient @DEPRECATED marker
T057: _llm_http.py + preview_llm_client.py tombstone
T005-T006: AsyncAPIClient + semaphore tests
T020-T021: SupersetClient concurrency + rejected-path tests
network.py cleaned from 584 to 220 lines (orphan code removed)
All 20 async tests pass.
2026-06-04 20:57:20 +03:00
5ca1131ba3
032: Phase 5 (T036-T039) + Phase 6 (T040-T044) completed
...
T036: superset_compilation_adapter fully async
T037: fileio.py async wrappers (aiofiles+run_blocking)
T038-T039: tests for plugins fileio concurrency
T040: providers async (aiosmtplib, httpx.AsyncClient)
T041: dispatch_report parallel via asyncio.gather
T042: TaskManager ThreadPoolExecutor->asyncio.create_task
T043: EventBus asyncio.Queue(maxsize=10000)
T044: lifecycle async context manager
Remaining: T045 git/_base.py, T046-T048 tests, T005-T020-T021 tests
2026-06-04 20:30:43 +03:00
371834cf43
chore: commit remaining maintenance and model changes
2026-06-03 23:26:20 +03:00
1e6ec34c2b
feat(backend): drop dictionary dialect columns, add allowed_languages
...
- Removes source_dialect and target_dialect from TerminologyDictionary model,
schemas, routes, helper serialization, and all tests
- Adds allowed_languages config field (BCP-47 language codes) to GlobalSettings
with validation, consolidated settings response, and API endpoint
- Adds alembic migration f1a2b3c4d5e6 to drop the two columns
2026-06-03 11:48:03 +03:00
4c5da7e4d9
alembic fix
2026-06-01 16:34:07 +03:00
b4b0deb856
js - ts + fix
2026-06-01 14:40:17 +03:00
d2b53c2a79
feat(validation): v2 LLM dashboard validation — plugin, routes, services
...
Core implementation of the v2 LLM dashboard validation pipeline:
- LLM plugin with Path A (screenshots) and Path B (logs-only) execution
- Validation task management (CRUD, schedule, run)
- WebSocket task progress with Python 3.13 asyncio fix
- Cross-task runs listing (GET /validation-tasks/runs/all)
- RecordResponse schema for validation records
- JSON prompt helper, per-dashboard status aggregation
- Prompt templates with docs/git-commit/validation presets
- Migration: v2 validation models + description column
- Tests: plugin persistence, prompt templates, batch, payload, url
2026-05-31 22:32:20 +03:00
21e32ac4cc
fix: legacy DB support + QA audit fixes
...
- Entrypoint: for legacy DBs, add is_admin via raw SQL then stamp head
(instead of running full migration chain which fails on existing tables)
- Fixed admin_api_keys.py Pydantic config (dict → ConfigDict)
- Added tests/test_alembic_migrations.py (PostgreSQL-only integration tests)
- Fixed infinite recursion in _create_table_if_not_exists (QA catch)
2026-05-26 20:35:36 +03:00
4205618ee6
chore: eliminate all deprecation warnings from tests and linter
...
Warnings fixed:
- datetime.utcnow() → datetime.now(UTC) across 48+ files (src/ + tests/)
- datetime.utcnow (callback ref) → lambda: datetime.now(UTC) in model fields (18 files)
- Pydantic class Config → model_config = ConfigDict(...) (16 files)
- Pydantic .dict() → .model_dump() (8 files)
- ConfigDict(allow_population_by_field_name=True) → validate_by_name=True
- SQLAlchemy declarative_base() import path updated
- FastAPI on_event → lifespan context manager (app.py)
- Import sorting (ruff I001) auto-fixed across all files
- Fixed broken re-export chains that ruff F401 cleanup broke:
_validate_bcp47: service.py now imports from dictionary_validation directly
job_to_response: _job_routes.py and test imports from service_utils directly
fetch_datasource_metadata: restored re-export in service.py
- Added missing TranslateJobService import in _job_routes.py (was deleted by F401)
- Added ConfigDict(protected_namespaces=()) for DashboardDatasetItem schema field
- pytest.ini: replaced deprecated importmode with asyncio_mode
All 440 tests pass with zero deprecation warnings.
2026-05-26 19:18:28 +03:00
2bc8ad87e1
security: rate limiter, is_admin flag, password policy, log masking
...
M-3: Role.name == 'Admin' string comparison → Role.is_admin boolean flag.
Admin role created with is_admin=True. has_permission checks getattr(role,
'is_admin', False) instead of comparing role names.
M-2: In-memory rate limiter on /api/auth/login. Tracks failed attempts
per IP (10 in 5 min window, 15 min ban). Thread-safe via threading.Lock.
M-5: Password policy — UserCreate.password: min_length=8, must include
uppercase, lowercase, and digit.
L-1: log_security_event details dict now masks sensitive fields:
password, secret, token, api_key, authorization. Recursive masking
for nested dicts. Sensitive field names matched via regex.
OTHER:
- 4 rate limiter tests (under limit, ban, success clears, IP isolation)
- 4 password policy tests (too short, no upper, no digit, valid)
- 2 log masking tests (flat keys, nested)
2026-05-26 15:17:19 +03:00
23719ecfbc
security: JWT aud/iss validation, token blacklist, encryption key crash-early
...
C-2 (CRITICAL): JWT decode_token now validates aud, iss, iat, jti claims.
Tokens without these claims are rejected. Audience 'ss-tools-api' and
issuer 'ss-tools' prevent cross-service token reuse.
H-2 (HIGH): Server-side JWT revocation via token_blacklist table.
- New TokenBlacklist model (SHA-256 hash only, never raw token)
- blacklist_token() on logout — revokes current session
- is_token_blacklisted() check in get_current_user
- Expired entries auto-pruned via _prune_blacklist()
H-3 (HIGH): Encryption key auto-generation removed — crash-early instead.
Previously, ensure_encryption_key() would auto-generate a Fernet key
on first run and write it to .env. This made encrypted data unrecoverable
when the key changed between deployments. Now raises RuntimeError with
a clear command to generate the key explicitly.
Also: update orthogonal security test for crash-early behavior.
2026-05-26 15:08:55 +03:00
a9a453109c
security: critical auth fixes, test migration to SQLite+FK, 44 test fixes
...
CRITICAL (CVE-class):
- C-4: Remove DEV_MODE fallback with hardcoded postgres:postgres credentials
- C-3: WebSocket endpoints now require JWT or API key token (?token=) auth
- C-1: Superset environment passwords encrypted via Fernet at rest in DB
- H-4: SESSION_SECRET_KEY separated from JWT AUTH_SECRET_KEY
- H-1: Log injection via %s-formatting instead of f-strings
- H-5: X-Trace-ID header validated as UUID4 to prevent trace poisoning
INFRASTRUCTURE:
- New src/core/encryption.py — EncryptionManager extracted from llm_provider
- Test DB: per-module sqlite:///:memory: with PRAGMA foreign_keys=ON
- testcontainers PostgreSQL via TEST_DB=postgres env var
- conftest temp-file global engine (no 10GB shared-cache leak)
TEST FIXES (44 pre-existing → 0):
- test_smoke_plugins: module-level sys.modules mock isolated to per-test fixture
- test_migration_engine: EXT:Python:uuid → uuid syntax fix
- test_dashboards_api: mock env attributes + correct patch target
- test_constants_audit_fixes: sync expected constant names with actual
- test_defensive_guards: patch.object instead of module-level Repo mock
- test_clean_release_cli: removed empty config.json directory
- FK violations: TaskRecord parents in log_persistence, Environment in
mapping_service, ReleaseCandidate in candidate_manifest_services
ORTHOGONAL TESTS (18 new):
- test_security_orthogonal.py: bcrypt 72-byte limit, unicode, API key
format invariants, Fernet robustness, encryption key lifecycle,
log injection protection, JWT edge cases
MODEL FIXES:
- MetricSnapshot.job_id: nullable with SET NULL (was broken FK for
aggregate prune snapshots, hidden by SQLite)
CLEANUP:
- Removed stale :memory:test_main/test_auth/test_tasks file databases
- Removed duplicate #endregion in encryption_key.py
2026-05-26 14:58:49 +03:00
f49b7d909e
feat: GRACE-Poly protocol optimization — ~3200→10 warnings (↓99.7%)
...
Full optimization cycle:
Protocol (15 files):
- 4-layer SSOT architecture for agent prompts & skills
- Anti-Corruption Protocol consolidated from 5 duplicates
- Tag-to-tier permissiveness matrix (all @tags allowed at all tiers)
Axiom config:
- complexity_rules: all 22+ tags available on C1-C5
- contract_type_overrides: removed (was narrowing per-type)
- 18 new tags added, LAYER enum expanded (Infra, Frontend, Atom, etc.)
- RELATION predicates expanded (USES, CONTAINS, BELONGS_TO, etc.)
Code fixes:
- 2216 @TAG: normalized to @TAG (colon→space)
- 518 [DEF] blocks migrated to #region/#endregion (37 files)
- VERIFIES→BINDS_TO, :Class/:Function suffixes removed, paths→IDs
- 1173-line _external_stubs.py deleted (EXT: handled natively)
- Batch EXT: reference audit (240 targets: 132 external, 99 internal, 9 fix)
- QA regression check: 0 regressions across all checks
Infrastructure:
- DuckDB rebuild stabilized (appender API, INSERT OR IGNORE)
- Anchor regex fix (parent-child BINDS_TO now resolves)
- EXT:*/DTO:/NEED_CONTEXT: regex fixed in validator
- 34MB Doxygen API portal (3194 contract pages)
2026-05-26 11:14:25 +03:00
9ffa8af1dc
semantics
2026-05-26 09:30:41 +03:00
320f82ab95
feat(auth): implement API key authentication and management
...
Introduce a new API key authentication mechanism to support service-to-service
communication (e.g., Airflow, CI/CD) without browser-based JWT login.
- Add `APIKey` model and `APIKeyPrincipal` dependency for RBAC.
- Implement `require_api_key_or_jwt` dependency with environment scoping.
- Add admin CRUD endpoints for API key lifecycle management.
- Add API key management UI in System Settings.
- Update maintenance API to support explicit `environment_id` and API key auth.
- Add i18n support for API keys and connection settings.
- Include Python and Bash usage examples in the `examples/` directory.
- Update technical specifications and documentation.
2026-05-25 11:35:27 +03:00
31680a1bc9
fix(maintenance): auto-expiry + adaptive markdown height + tests
...
- Auto-expiry: expired events auto-end on GET /events via task manager
- Height: _estimate_markdown_height adapted to unit=8px scale with padding detection
- CRITICAL-1: removed dead import remove_chart_from_position (QA finding)
- CRITICAL-2: added chart alive check in ensure_banner_chart (QA finding)
- CRITICAL-3: fixed test assertions update_markdown_chart → update_banner_on_dashboard
- @POST contract: fixed return range [2,12] → [19,200]
- Tests: 8 new tests (auto-expiry + layout height)
2026-05-25 08:36:33 +03:00
b529e8082a
feat(030): Dataset Lifecycle Workspace — Stats Bar, split-view, inline-edit, bulk actions
...
Backend:
- Add MetricItem, StatsCounts, ColumnDescriptionUpdate, MetricDescriptionUpdate DTOs
- Add metric_count to DatasetItem and DatasetDetailResponse
- Add server-side filtering via ?filter=unmapped|mapped|linked|all
- Add PUT endpoints for column/metric description inline-edit
- Add HTML stripping validation (max 2000 chars, plain text)
- Add WebSocket dataset.updated event on task completion
- RBAC: plugin:migration:WRITE for mutations, READ for reads
- 14 pytest tests (stats, filter, metrics, inline-edit, validation, 502/503)
Frontend:
- Rewrite +page.svelte as split-view orchestrator (387 lines)
- StatsBar.svelte — 4 metric tiles with aria-pressed, server-side filter dispatch
- DatasetList.svelte — card grid with progress bars, checkboxes, search
- DatasetPreview.svelte — presentational detail panel (container fetches)
- ColumnsTable.svelte — inline-edit with type chips, bold/italic fallback
- MetricsTable.svelte — inline-edit with expression hints
- 52 vitest tests across 7 test files
- i18n: 11 EN + 11 RU keys (with_mapping/с_маппингом)
Spec:
- 15 issues resolved (semantic label, filtering, endpoints, conflict, metric_count, RBAC, ID types, ports, i18n, validation, a11y, propagation, resize, perf, ownership)
Status: 14/14 backend tests pass, 52/52 frontend tests pass, build succeeds
Risk: Low — T051 browser validation pending (non-blocking)
2026-05-21 13:58:47 +03:00
68740cd9ed
semantic
2026-05-20 23:54:53 +03:00
cd868df261
fix(health): suppress 404 when health monitor disabled + fix audit test assertion
...
- Sidebar.svelte: defer healthStore.refresh() until feature flags confirm
health_monitor is enabled; skip entirely when disabled
- health.js: remove throw error from catch block — log and return null
instead, preventing 'Uncaught (in promise)' on expected 404
- test_report_audit_immutability.py: fix mock assertions —
audit_service uses logger.reason/reflect/explore, not logger.info
- HealthStore and Sidebar now produce zero network noise when
FEATURES__HEALTH_MONITOR=false
2026-05-17 14:18:02 +03:00
77d55c8e69
refactor(translate): remove all deprecated columns from model + DB
...
Removed 6 legacy columns replaced by Phase 11 multi-language models:
- TranslationJob.target_language → target_languages (JSON)
- TranslationRecord.{llm_translation, user_edit, final_value} → TranslationLanguage
- TerminologyDictionary.{source_language, target_language} → DictionaryEntry per-entry
Affected files: model, schemas, 5 plugins, 4 test files
Alembic migration: aa1b2c3d4e5f (DROP COLUMN IF EXISTS)
Production DB: columns dropped via ALTER TABLE
Tests: 234 passed, 0 failed
2026-05-15 00:06:41 +03:00
bb0fbfdafd
feat(translate): multi-language optimization (Phase 11)
...
- Auto-detection of source language per row via LLM (US6)
- Multi-target translation — one LLM call for N languages (US1-US3)
- Language-aware storage: TranslationLanguage, per-language stats
- Multilingual dictionaries with language-pair-aware filtering (US7)
- Inline correction on any run result + submit-to-dictionary (US8)
- Context-aware dictionary: auto-capture row context, usage notes,
Jaccard similarity, priority flagging in LLM prompts (US8b)
- Configurable preview sample size 1-100, cost warning at >30
- Per-language history & metrics with MetricSnapshot preservation
- 36 files, +5022/-373, all specs GRACE-Poly v2.6 compliant
2026-05-14 17:12:41 +03:00
c6189876b3
chore(lint): apply ruff --fix (4443 auto-fixes)
...
Auto-fixed categories:
- F401: unused imports removed
- I001: import blocks sorted
- W293: trailing whitespace stripped
- UP035: deprecated typing imports replaced
- SIM: simplify suggestions applied
- ARG: unused args prefixed with underscore
- T201: print statements removed
- F841: unused variables removed
- RUF059: unpacked variables prefixed
Remaining ~1500 unfixable errors (C901, B904, N806, E402) require manual work.
Backend smoke tests: 13/13 passed.
2026-05-14 11:20:17 +03:00
8d0bc6fb93
feat(translate,scheduler,trace): new-key-only mode, stale PENDING protection, trace_id propagation
...
Translations:
- FR-045: new-key-only execution mode — translate only rows with unseen keys
- Compare source row keys against TranslationRecord.source_data from last succeeded run
- Baseline expired (>90 days) fallback to full mode with baseline_expired event
- run_noop early return when zero new keys (skip LLM + SQL)
Scheduler reliability:
- Stale PENDING protection: runs older than 1h auto-marked FAILED, no longer block schedule
- load_schedules() reloads active translation schedules from DB on restart
- add_translation_job/remove_translation_job register/unregister with APScheduler
- execution_mode column on translation_schedules with additive DB migration
Automation view:
- GET /settings/automation/translation-schedules endpoint
- Translation schedules displayed on Automation page below validation policies
Trace propagation:
- seed_trace_id() in all background entry points:
TaskManager._run_task/_flusher_loop, SchedulerService._trigger_backup,
websocket_endpoint, IdMappingService, all standalone scripts
Migrations:
- dictionary_entries: origin_run_id, origin_row_key, origin_user_id
- translation_schedules: execution_mode
Tests:
- 3 new test modules (22 tests): core_scheduler, executor_filter, scheduler_execution+guard
- Fix pre-existing translate test isolation (conftest.py)
- Fix test_list_runs_filter_status parameter
2026-05-14 10:13:56 +03:00
a3c7c402b7
fix(llm): add fetch-models endpoint, fix SQL Lab INSERT (client_id truncation, sync mode, target_column, timestamp normalization)
...
- Add POST /api/llm/providers/fetch-models route with LLMClient.fetch_models()
- Add target_column to TranslationJob model/schema/service/orchestrator
- Fix SQL Lab execute: truncate client_id to 11 chars (varchar(11))
- Switch SQL Lab to sync mode (runAsync: false) — no Celery workers
- Fix polling: unwrap nested result from Superset query API
- Fix ClickHouse timestamp: normalize float timestamps to YYYY-MM-DD
2026-05-13 20:06:15 +03:00
83b8f4d999
logs: migrate to molecular CoT JSON protocol, fix regressions
...
- Replace BeliefFormatter with CotJsonFormatter (single-line JSON with
ts, level, trace_id, src, marker, intent, payload, error, span_id)
- Migrate belief_scope to use cot_log() for structured JSON output
- Update monkey-patched reason/reflect/explore to pass extra= dict
- Strip 200+ inline [REASON]/[REFLECT]/[EXPLORE] markers from 50+ files
- Remove belief_scope from hot-path utilities (_parse_datetime,
_json_load_if_needed) to eliminate startup log noise
- Register TraceContextMiddleware in app.py (was implemented but unused)
- Seed trace_id in startup_event/shutdown_event for background tasks
- Fix broken relative imports in translate routes (3 dots → 4 dots)
- Migrate 43 translate_routes log calls to logger.reason/explore
- Fix SyntaxError (positional arg after extra=) in _repo_operations_routes
- Fix IndentationError in rbac_permission_catalog except-block
- Add smoke test tests/test_smoke_app.py (catches import errors before run)
2026-05-13 09:44:50 +03:00
b17b5333c7
log refactor
2026-05-12 19:30:15 +03:00
67ba04d4ff
feat: implement LLM table translation service with searchable datasource dropdown, i18n, sidebar, and RBAC
...
- Add backend plugin modules: preview, executor, orchestrator, events, sql_generator, superset_executor, dictionary, scheduler, metrics
- Add API routes for translate jobs, runs, dictionaries, preview, schedule, metrics, corrections
- Add ORM models (12 tables) and Pydantic schemas (15 DTOs)
- Register translate router in app.py with RBAC permission guards
- Add frontend pages: job list, job config, dictionary list/editor, history
- Add 7 reusable Svelte 5 components: Preview, RunProgress, RunResult, TermCorrection, BulkCorrection, ScheduleConfig, MetricsDashboard
- Add searchable datasource dropdown with Superset API integration
- Add i18n (en/ru) with ~300 keys across common, config, jobs, dictionaries, preview namespaces
- Add Translation sidebar category with Jobs/Dictionaries/History sub-items
- Hide health monitor error toast via suppressToast API option
- Add 69 backend tests and 44 frontend test files
- Fix: SupersetClient env resolution (string -> Environment object)
- Fix: Dataset detail API returns proper database dict
- Fix: Database dialect extraction fallback when metadata incomplete
2026-05-09 19:34:25 +03:00
201886eeb0
refactor: decompose 6 monolithic modules (>1000 LOC) into domain packages
...
Decompose 6 large modules into packages with <400-line modules to satisfy INV_7:
- superset_client.py (2145->0) -> superset_client/ package (12 modules)
- assistant.py (2683->0) -> assistant/ package (12 modules)
- git_service.py (2101->11 shim) -> services/git/ package (9 modules)
- git.py (1757->0) -> routes/git/ package (11 modules)
- dashboards.py (1619->0) -> routes/dashboards/ package (8 modules)
- translate.py (1587->0) -> routes/translate/ package (11 modules)
- superset_context_extractor.py (1397->0) -> utils/superset_context_extractor/ (7 modules)
All modules <400 lines (INV_7). All 80 tests pass.
All external imports preserved via __init__.py re-exports or shim.
Semantic [DEF] contracts with @LAYER/@SEMANTICS added to all Module types.
2026-05-09 10:13:07 +03:00
7901ce0f39
semantics
2026-03-27 21:27:31 +03:00
