Commit Graph

67 Commits

Author SHA1 Message Date
eda346979b refactor: remove _ensure_*() safety net, keep only Alembic
- Simplified init_db() to only call Base.metadata.create_all() on all
  three engines — no more _ensure_*() inline additive migrations
- run_alembic_migrations() now raises on failure (no safety net to
  fall back to)
- All schema changes (add/drop/rename columns, data migrations) must
  go through Alembic. create_all() handles new tables only.

@REJECTED _ensure_*() inline migrations removed because they:
  - Duplicated Alembic logic without versioning
  - Created hidden schema drift (columns existed in DB but had no
    corresponding Alembic migration)
  - Made audits and fresh DB provisioning unreliable
2026-05-26 19:02:59 +03:00
81fc7e0902 feat: run Alembic migrations automatically on startup
- Added alembic==1.18.4 to requirements.txt (was missing, transitive only)
- Added run_alembic_migrations() to startup_event in app.py — runs
   before init_db() to apply all pending migrations
- Combined approach: Alembic for complex migrations first, then inline
  _ensure_*() additive safety net in init_db()
- This ensures DB schema is always up to date on container startup
  without manual intervention
2026-05-26 18:56:29 +03:00
8dcc0f86f8 fix: conditional HSTS via FORCE_HTTPS, safe for enterprise without certs
HSTS (Strict-Transport-Security) is now opt-in via FORCE_HTTPS=true.
Without it enabled, no HSTS header is sent — safe for dev environments
and enterprise deployments behind HTTP-only proxies without HTTPS certs.

When FORCE_HTTPS=true, sends 'max-age=31536000; includeSubDomains'.
Enterprise recommendation: set HSTS at nginx/ingress level instead.

.env.example documents the risk: enabling without certs breaks site access.
2026-05-26 15:25:55 +03:00
10cdb8a81a fix: replace TrustedHostMiddleware with proper HSTS middleware
TrustedHostMiddleware blocked Vite proxied requests (Host: localhost:5173
vs allowed_hosts list). Replaced with simple HSTSMiddleware that adds
Strict-Transport-Security header without blocking any requests.
2026-05-26 15:24:40 +03:00
ee4ca3cdf6 security: HSTS, AD group validation, INITIAL_ADMIN_PASSWORD warning
L-2: HSTS middleware via TrustedHostMiddleware — restricts allowed hosts
  to ALLOWED_ORIGINS list, prevents Host header injection.

L-4: AD group name validation — ADGroupMappingCreate.ad_group validated
  with regex: DOMAIN\groupname or CN=...,DC=... format. Empty or
  invalid characters rejected.

L-5: INITIAL_ADMIN_PASSWORD warning — log warning that env-var password
  is visible via /proc to other processes on the same host.
2026-05-26 15:20:29 +03:00
2bc8ad87e1 security: rate limiter, is_admin flag, password policy, log masking
M-3: Role.name == 'Admin' string comparison → Role.is_admin boolean flag.
  Admin role created with is_admin=True. has_permission checks getattr(role,
  'is_admin', False) instead of comparing role names.

M-2: In-memory rate limiter on /api/auth/login. Tracks failed attempts
  per IP (10 in 5 min window, 15 min ban). Thread-safe via threading.Lock.

M-5: Password policy — UserCreate.password: min_length=8, must include
  uppercase, lowercase, and digit.

L-1: log_security_event details dict now masks sensitive fields:
  password, secret, token, api_key, authorization. Recursive masking
  for nested dicts. Sensitive field names matched via regex.

OTHER:
- 4 rate limiter tests (under limit, ban, success clears, IP isolation)
- 4 password policy tests (too short, no upper, no digit, valid)
- 2 log masking tests (flat keys, nested)
2026-05-26 15:17:19 +03:00
a9a453109c security: critical auth fixes, test migration to SQLite+FK, 44 test fixes
CRITICAL (CVE-class):
- C-4: Remove DEV_MODE fallback with hardcoded postgres:postgres credentials
- C-3: WebSocket endpoints now require JWT or API key token (?token=) auth
- C-1: Superset environment passwords encrypted via Fernet at rest in DB
- H-4: SESSION_SECRET_KEY separated from JWT AUTH_SECRET_KEY
- H-1: Log injection via %s-formatting instead of f-strings
- H-5: X-Trace-ID header validated as UUID4 to prevent trace poisoning

INFRASTRUCTURE:
- New src/core/encryption.py — EncryptionManager extracted from llm_provider
- Test DB: per-module sqlite:///:memory: with PRAGMA foreign_keys=ON
- testcontainers PostgreSQL via TEST_DB=postgres env var
- conftest temp-file global engine (no 10GB shared-cache leak)

TEST FIXES (44 pre-existing → 0):
- test_smoke_plugins: module-level sys.modules mock isolated to per-test fixture
- test_migration_engine: EXT:Python:uuid → uuid syntax fix
- test_dashboards_api: mock env attributes + correct patch target
- test_constants_audit_fixes: sync expected constant names with actual
- test_defensive_guards: patch.object instead of module-level Repo mock
- test_clean_release_cli: removed empty config.json directory
- FK violations: TaskRecord parents in log_persistence, Environment in
  mapping_service, ReleaseCandidate in candidate_manifest_services

ORTHOGONAL TESTS (18 new):
- test_security_orthogonal.py: bcrypt 72-byte limit, unicode, API key
  format invariants, Fernet robustness, encryption key lifecycle,
  log injection protection, JWT edge cases

MODEL FIXES:
- MetricSnapshot.job_id: nullable with SET NULL (was broken FK for
  aggregate prune snapshots, hidden by SQLite)

CLEANUP:
- Removed stale :memory:test_main/test_auth/test_tasks file databases
- Removed duplicate #endregion in encryption_key.py
2026-05-26 14:58:49 +03:00
9ffa8af1dc semantics 2026-05-26 09:30:41 +03:00
320f82ab95 feat(auth): implement API key authentication and management
Introduce a new API key authentication mechanism to support service-to-service
communication (e.g., Airflow, CI/CD) without browser-based JWT login.

- Add `APIKey` model and `APIKeyPrincipal` dependency for RBAC.
- Implement `require_api_key_or_jwt` dependency with environment scoping.
- Add admin CRUD endpoints for API key lifecycle management.
- Add API key management UI in System Settings.
- Update maintenance API to support explicit `environment_id` and API key auth.
- Add i18n support for API keys and connection settings.
- Include Python and Bash usage examples in the `examples/` directory.
- Update technical specifications and documentation.
2026-05-25 11:35:27 +03:00
31680a1bc9 fix(maintenance): auto-expiry + adaptive markdown height + tests
- Auto-expiry: expired events auto-end on GET /events via task manager
- Height: _estimate_markdown_height adapted to unit=8px scale with padding detection
- CRITICAL-1: removed dead import remove_chart_from_position (QA finding)
- CRITICAL-2: added chart alive check in ensure_banner_chart (QA finding)
- CRITICAL-3: fixed test assertions update_markdown_chart → update_banner_on_dashboard
- @POST contract: fixed return range [2,12] → [19,200]
- Tests: 8 new tests (auto-expiry + layout height)
2026-05-25 08:36:33 +03:00
48cc02ea0b feat: LLM Validation section — 9 API endpoints + 3 Svelte pages + Playwright optimization
Backend:
- /api/validation/ CRUD for validation tasks (policies)
- Run history with 6 filters + pagination
- Alembic migrations (provider_id, policy_id)
- 28 pytest tests

Frontend:
- /validation — task list page with status filters + CRUD
- /validation/[id] — task config (Config + History tabs)
- /validation/history — run history with metrics
- Sidebar nav entry under Operations
- i18n en/ru

Playwright (ScreenshotService):
- Removed 25s hardcoded sleeps → conditional waits
- CDP fallback to full_page screenshot
- Total timeout enforcement (120s)
- Debug screenshots → temp dir with cleanup

QA fixes:
- Debug screenshot accumulation in production (tempdir + rmtree)
- Frontend API payload field name mismatch
- History issues field reference fix
- delete_runs scoped by policy_id

Belief protocol:
- @RATIONALE/@REJECTED on 27 C4+ contracts

Closes: VALIDATION-REWORK-2026
2026-05-21 20:29:36 +03:00
36643024cf refactor(connections): remove ConnectionConfig, migrate mapper to SQL Lab
Backend:
- Remove ConnectionConfig model, CRUD routes (connections.py), and tests
- Remove ensure_connection_configs_table from database.py
- Migrate MapperPlugin from psycopg2 direct PostgreSQL to SupersetSqlLabExecutor
- Add DatasetMapper.get_sqllab_mappings() with default information_schema query
- Update MapColumnsRequest: connection_id → database_id + sql_query
- Fix pydantic_settings v2 deprecation (Field(env=...) → validation_alias)
- Clean up app.py, routes/__init__.py, database.py imports

Frontend:
- Remove DatabaseConnectionsTab, ConnectionForm, ConnectionList components
- Remove /settings/connections route page and Navbar link
- Remove connectionService.js, connections i18n files
- Update MapperTool.svelte: postgres → sqllab source
- Update Map Columns modal: database selector + SQL query instead of connection_id
- Clean up settings page tabs, nav links, test mocks
- Clean up i18n keys (settings, nav, mapper)

Tests: 25/25 pass (14 030-feature + 11 route tests)
Build: succeeds
2026-05-21 18:47:41 +03:00
b529e8082a feat(030): Dataset Lifecycle Workspace — Stats Bar, split-view, inline-edit, bulk actions
Backend:
- Add MetricItem, StatsCounts, ColumnDescriptionUpdate, MetricDescriptionUpdate DTOs
- Add metric_count to DatasetItem and DatasetDetailResponse
- Add server-side filtering via ?filter=unmapped|mapped|linked|all
- Add PUT endpoints for column/metric description inline-edit
- Add HTML stripping validation (max 2000 chars, plain text)
- Add WebSocket dataset.updated event on task completion
- RBAC: plugin:migration:WRITE for mutations, READ for reads
- 14 pytest tests (stats, filter, metrics, inline-edit, validation, 502/503)

Frontend:
- Rewrite +page.svelte as split-view orchestrator (387 lines)
- StatsBar.svelte — 4 metric tiles with aria-pressed, server-side filter dispatch
- DatasetList.svelte — card grid with progress bars, checkboxes, search
- DatasetPreview.svelte — presentational detail panel (container fetches)
- ColumnsTable.svelte — inline-edit with type chips, bold/italic fallback
- MetricsTable.svelte — inline-edit with expression hints
- 52 vitest tests across 7 test files
- i18n: 11 EN + 11 RU keys (with_mapping/с_маппингом)

Spec:
- 15 issues resolved (semantic label, filtering, endpoints, conflict, metric_count, RBAC, ID types, ports, i18n, validation, a11y, propagation, resize, perf, ownership)

Status: 14/14 backend tests pass, 52/52 frontend tests pass, build succeeds
Risk: Low — T051 browser validation pending (non-blocking)
2026-05-21 13:58:47 +03:00
68740cd9ed semantic 2026-05-20 23:54:53 +03:00
b916ef94d5 semantics 2026-05-19 18:36:15 +03:00
6988e63967 chore: commit remaining pre-existing changes
- Agent configs (.opencode/agents/)
- Backend: alembic, routes, app, utils, scripts
- Frontend: package.json, vite, components, e2e infra
- Specs: 028-llm-datasource-supeset updates
- Docker e2e config and Playwright setup
2026-05-17 19:23:07 +03:00
8bea44f640 chore: update semantic index, agent configs, and app startup 2026-05-14 21:33:26 +03:00
c6189876b3 chore(lint): apply ruff --fix (4443 auto-fixes)
Auto-fixed categories:
- F401: unused imports removed
- I001: import blocks sorted
- W293: trailing whitespace stripped
- UP035: deprecated typing imports replaced
- SIM: simplify suggestions applied
- ARG: unused args prefixed with underscore
- T201: print statements removed
- F841: unused variables removed
- RUF059: unpacked variables prefixed

Remaining ~1500 unfixable errors (C901, B904, N806, E402) require manual work.

Backend smoke tests: 13/13 passed.
2026-05-14 11:20:17 +03:00
8d0bc6fb93 feat(translate,scheduler,trace): new-key-only mode, stale PENDING protection, trace_id propagation
Translations:
- FR-045: new-key-only execution mode — translate only rows with unseen keys
- Compare source row keys against TranslationRecord.source_data from last succeeded run
- Baseline expired (>90 days) fallback to full mode with baseline_expired event
- run_noop early return when zero new keys (skip LLM + SQL)

Scheduler reliability:
- Stale PENDING protection: runs older than 1h auto-marked FAILED, no longer block schedule
- load_schedules() reloads active translation schedules from DB on restart
- add_translation_job/remove_translation_job register/unregister with APScheduler
- execution_mode column on translation_schedules with additive DB migration

Automation view:
- GET /settings/automation/translation-schedules endpoint
- Translation schedules displayed on Automation page below validation policies

Trace propagation:
- seed_trace_id() in all background entry points:
  TaskManager._run_task/_flusher_loop, SchedulerService._trigger_backup,
  websocket_endpoint, IdMappingService, all standalone scripts

Migrations:
- dictionary_entries: origin_run_id, origin_row_key, origin_user_id
- translation_schedules: execution_mode

Tests:
- 3 new test modules (22 tests): core_scheduler, executor_filter, scheduler_execution+guard
- Fix pre-existing translate test isolation (conftest.py)
- Fix test_list_runs_filter_status parameter
2026-05-14 10:13:56 +03:00
39ab647851 semantics 2026-05-13 14:15:33 +03:00
83b8f4d999 logs: migrate to molecular CoT JSON protocol, fix regressions
- Replace BeliefFormatter with CotJsonFormatter (single-line JSON with
  ts, level, trace_id, src, marker, intent, payload, error, span_id)
- Migrate belief_scope to use cot_log() for structured JSON output
- Update monkey-patched reason/reflect/explore to pass extra= dict
- Strip 200+ inline [REASON]/[REFLECT]/[EXPLORE] markers from 50+ files
- Remove belief_scope from hot-path utilities (_parse_datetime,
  _json_load_if_needed) to eliminate startup log noise
- Register TraceContextMiddleware in app.py (was implemented but unused)
- Seed trace_id in startup_event/shutdown_event for background tasks
- Fix broken relative imports in translate routes (3 dots → 4 dots)
- Migrate 43 translate_routes log calls to logger.reason/explore
- Fix SyntaxError (positional arg after extra=) in _repo_operations_routes
- Fix IndentationError in rbac_permission_catalog except-block
- Add smoke test tests/test_smoke_app.py (catches import errors before run)
2026-05-13 09:44:50 +03:00
306c5ae742 semantics: complete DEF-to-region migration, fix regressions
- Convert legacy [DEF🆔Type] anchors to #region/#endregion across 329 files
- Reinstate _normalize_timestamp_value in sql_generator.py
- Fix MarkerLogger→logger migration in events.py (molecular CoT markers)
- Fix dataset_review orchestrator dependencies (_build_execution_snapshot)
- Fix config_manager stale-record deletion (moved to save path only)
- Add 77 missing [/DEF:] closers in 5 unbalanced test files
- Update assistant_chat.integration.test.js for #region format
- Apply molecular-cot-logging markers (REASON/REFLECT/EXPLORE) via logger.* methods
2026-05-12 23:54:55 +03:00
fe8978f716 semantics 2026-05-12 20:06:16 +03:00
b17b5333c7 log refactor 2026-05-12 19:30:15 +03:00
67ba04d4ff feat: implement LLM table translation service with searchable datasource dropdown, i18n, sidebar, and RBAC
- Add backend plugin modules: preview, executor, orchestrator, events, sql_generator, superset_executor, dictionary, scheduler, metrics
- Add API routes for translate jobs, runs, dictionaries, preview, schedule, metrics, corrections
- Add ORM models (12 tables) and Pydantic schemas (15 DTOs)
- Register translate router in app.py with RBAC permission guards
- Add frontend pages: job list, job config, dictionary list/editor, history
- Add 7 reusable Svelte 5 components: Preview, RunProgress, RunResult, TermCorrection, BulkCorrection, ScheduleConfig, MetricsDashboard
- Add searchable datasource dropdown with Superset API integration
- Add i18n (en/ru) with ~300 keys across common, config, jobs, dictionaries, preview namespaces
- Add Translation sidebar category with Jobs/Dictionaries/History sub-items
- Hide health monitor error toast via suppressToast API option
- Add 69 backend tests and 44 frontend test files
- Fix: SupersetClient env resolution (string -> Environment object)
- Fix: Dataset detail API returns proper database dict
- Fix: Database dialect extraction fallback when metadata incomplete
2026-05-09 19:34:25 +03:00
f5c90af287 semantics 2026-04-01 21:57:51 +03:00
7901ce0f39 semantics 2026-03-27 21:27:31 +03:00
5dd10f748a Bootstrap initial admin via env and add compose profiles 2026-03-17 19:16:25 +03:00
268ac66a38 feat(us1): add dataset review orchestration automatic review slice 2026-03-17 10:57:49 +03:00
8cfa7b7c3c chore: update semantic contracts and git merge handling 2026-03-16 20:34:28 +03:00
67867f8220 refactor(semantics): migrate legacy @TIER to @COMPLEXITY annotations
- Replaced @TIER: TRIVIAL with @COMPLEXITY: 1
- Replaced @TIER: STANDARD with @COMPLEXITY: 3
- Replaced @TIER: CRITICAL with @COMPLEXITY: 5
- Manually elevated specific critical/complex components to levels 2 and 4
- Ignored legacy, specs, and node_modules directories
- Updated generated semantic map
2026-03-16 10:06:44 +03:00
63940020db chore(semantic): checkpoint remediation progress 2026-03-15 21:08:00 +03:00
bcce55fbf9 feat add connections management and health summary improvements 2026-03-15 16:40:43 +03:00
b4b17d89b4 код написан 2026-03-10 12:00:18 +03:00
6775c7e3db feat(clean-release): complete compliance redesign phases and polish tasks T047-T052 2026-03-10 09:11:26 +03:00
b8eafa28a2 feat(rbac): hide unauthorized menu sections and enforce route guards 2026-03-06 10:50:28 +03:00
ede337b977 test: remediate audit findings for task log viewer, report card and logger tests 2026-03-03 21:01:24 +03:00
1bf7b53bae test contracts 2026-02-26 19:40:00 +03:00
cbf9135db6 semantic update 2026-02-24 21:08:12 +03:00
80f0d30386 feat(assistant): implement spec 021 chat assistant flow with semantic contracts 2026-02-23 19:37:56 +03:00
2bcae76d29 таски готовы 2026-02-23 10:18:56 +03:00
1fcb7ca67c semantic update 2026-02-20 10:41:15 +03:00
00d4960da0 refactor 2026-02-18 17:29:46 +03:00
e242b61c5b fix 2026-02-15 11:11:30 +03:00
5ab1549cdd linter + новые таски 2026-02-10 12:53:01 +03:00
0743641181 semantic update 2026-02-08 22:53:54 +03:00
2d09ff4dc8 Похоже работает 2026-02-07 11:26:06 +03:00
161d70648c Вроде работает 2026-01-30 11:10:16 +03:00
0fef9cb59a Передаем на тест 2026-01-27 16:32:08 +03:00
aabbaa44c3 Передаем на тест 2026-01-25 18:33:00 +03:00