c7ae60b76a
security: rate limiter, is_admin flag, password policy, log masking
...
M-3: Role.name == 'Admin' string comparison → Role.is_admin boolean flag.
Admin role created with is_admin=True. has_permission checks getattr(role,
'is_admin', False) instead of comparing role names.
M-2: In-memory rate limiter on /api/auth/login. Tracks failed attempts
per IP (10 in 5 min window, 15 min ban). Thread-safe via threading.Lock.
M-5: Password policy — UserCreate.password: min_length=8, must include
uppercase, lowercase, and digit.
L-1: log_security_event details dict now masks sensitive fields:
password, secret, token, api_key, authorization. Recursive masking
for nested dicts. Sensitive field names matched via regex.
OTHER:
- 4 rate limiter tests (under limit, ban, success clears, IP isolation)
- 4 password policy tests (too short, no upper, no digit, valid)
- 2 log masking tests (flat keys, nested)
2026-05-26 15:17:19 +03:00
bb03929ca1
security: critical auth fixes, test migration to SQLite+FK, 44 test fixes
...
CRITICAL (CVE-class):
- C-4: Remove DEV_MODE fallback with hardcoded postgres:postgres credentials
- C-3: WebSocket endpoints now require JWT or API key token (?token=) auth
- C-1: Superset environment passwords encrypted via Fernet at rest in DB
- H-4: SESSION_SECRET_KEY separated from JWT AUTH_SECRET_KEY
- H-1: Log injection via %s-formatting instead of f-strings
- H-5: X-Trace-ID header validated as UUID4 to prevent trace poisoning
INFRASTRUCTURE:
- New src/core/encryption.py — EncryptionManager extracted from llm_provider
- Test DB: per-module sqlite:///:memory: with PRAGMA foreign_keys=ON
- testcontainers PostgreSQL via TEST_DB=postgres env var
- conftest temp-file global engine (no 10GB shared-cache leak)
TEST FIXES (44 pre-existing → 0):
- test_smoke_plugins: module-level sys.modules mock isolated to per-test fixture
- test_migration_engine: EXT:Python:uuid → uuid syntax fix
- test_dashboards_api: mock env attributes + correct patch target
- test_constants_audit_fixes: sync expected constant names with actual
- test_defensive_guards: patch.object instead of module-level Repo mock
- test_clean_release_cli: removed empty config.json directory
- FK violations: TaskRecord parents in log_persistence, Environment in
mapping_service, ReleaseCandidate in candidate_manifest_services
ORTHOGONAL TESTS (18 new):
- test_security_orthogonal.py: bcrypt 72-byte limit, unicode, API key
format invariants, Fernet robustness, encryption key lifecycle,
log injection protection, JWT edge cases
MODEL FIXES:
- MetricSnapshot.job_id: nullable with SET NULL (was broken FK for
aggregate prune snapshots, hidden by SQLite)
CLEANUP:
- Removed stale :memory:test_main/test_auth/test_tasks file databases
- Removed duplicate #endregion in encryption_key.py
2026-05-26 14:58:49 +03:00
d5cf5f0cf7
semantics
2026-05-26 09:30:41 +03:00
18b5953b69
feat(auth): implement API key authentication and management
...
Introduce a new API key authentication mechanism to support service-to-service
communication (e.g., Airflow, CI/CD) without browser-based JWT login.
- Add `APIKey` model and `APIKeyPrincipal` dependency for RBAC.
- Implement `require_api_key_or_jwt` dependency with environment scoping.
- Add admin CRUD endpoints for API key lifecycle management.
- Add API key management UI in System Settings.
- Update maintenance API to support explicit `environment_id` and API key auth.
- Add i18n support for API keys and connection settings.
- Include Python and Bash usage examples in the `examples/` directory.
- Update technical specifications and documentation.
2026-05-25 11:35:27 +03:00
dfc74b5c65
fix(maintenance): auto-expiry + adaptive markdown height + tests
...
- Auto-expiry: expired events auto-end on GET /events via task manager
- Height: _estimate_markdown_height adapted to unit=8px scale with padding detection
- CRITICAL-1: removed dead import remove_chart_from_position (QA finding)
- CRITICAL-2: added chart alive check in ensure_banner_chart (QA finding)
- CRITICAL-3: fixed test assertions update_markdown_chart → update_banner_on_dashboard
- @POST contract: fixed return range [2,12] → [19,200]
- Tests: 8 new tests (auto-expiry + layout height)
2026-05-25 08:36:33 +03:00
2ce1557d7c
feat: LLM Validation section — 9 API endpoints + 3 Svelte pages + Playwright optimization
...
Backend:
- /api/validation/ CRUD for validation tasks (policies)
- Run history with 6 filters + pagination
- Alembic migrations (provider_id, policy_id)
- 28 pytest tests
Frontend:
- /validation — task list page with status filters + CRUD
- /validation/[id] — task config (Config + History tabs)
- /validation/history — run history with metrics
- Sidebar nav entry under Operations
- i18n en/ru
Playwright (ScreenshotService):
- Removed 25s hardcoded sleeps → conditional waits
- CDP fallback to full_page screenshot
- Total timeout enforcement (120s)
- Debug screenshots → temp dir with cleanup
QA fixes:
- Debug screenshot accumulation in production (tempdir + rmtree)
- Frontend API payload field name mismatch
- History issues field reference fix
- delete_runs scoped by policy_id
Belief protocol:
- @RATIONALE/@REJECTED on 27 C4+ contracts
Closes: VALIDATION-REWORK-2026
2026-05-21 20:29:36 +03:00
74e646225f
refactor(connections): remove ConnectionConfig, migrate mapper to SQL Lab
...
Backend:
- Remove ConnectionConfig model, CRUD routes (connections.py), and tests
- Remove ensure_connection_configs_table from database.py
- Migrate MapperPlugin from psycopg2 direct PostgreSQL to SupersetSqlLabExecutor
- Add DatasetMapper.get_sqllab_mappings() with default information_schema query
- Update MapColumnsRequest: connection_id → database_id + sql_query
- Fix pydantic_settings v2 deprecation (Field(env=...) → validation_alias)
- Clean up app.py, routes/__init__.py, database.py imports
Frontend:
- Remove DatabaseConnectionsTab, ConnectionForm, ConnectionList components
- Remove /settings/connections route page and Navbar link
- Remove connectionService.js, connections i18n files
- Update MapperTool.svelte: postgres → sqllab source
- Update Map Columns modal: database selector + SQL query instead of connection_id
- Clean up settings page tabs, nav links, test mocks
- Clean up i18n keys (settings, nav, mapper)
Tests: 25/25 pass (14 030-feature + 11 route tests)
Build: succeeds
2026-05-21 18:47:41 +03:00
bb11b6f9e5
feat(030): Dataset Lifecycle Workspace — Stats Bar, split-view, inline-edit, bulk actions
...
Backend:
- Add MetricItem, StatsCounts, ColumnDescriptionUpdate, MetricDescriptionUpdate DTOs
- Add metric_count to DatasetItem and DatasetDetailResponse
- Add server-side filtering via ?filter=unmapped|mapped|linked|all
- Add PUT endpoints for column/metric description inline-edit
- Add HTML stripping validation (max 2000 chars, plain text)
- Add WebSocket dataset.updated event on task completion
- RBAC: plugin:migration:WRITE for mutations, READ for reads
- 14 pytest tests (stats, filter, metrics, inline-edit, validation, 502/503)
Frontend:
- Rewrite +page.svelte as split-view orchestrator (387 lines)
- StatsBar.svelte — 4 metric tiles with aria-pressed, server-side filter dispatch
- DatasetList.svelte — card grid with progress bars, checkboxes, search
- DatasetPreview.svelte — presentational detail panel (container fetches)
- ColumnsTable.svelte — inline-edit with type chips, bold/italic fallback
- MetricsTable.svelte — inline-edit with expression hints
- 52 vitest tests across 7 test files
- i18n: 11 EN + 11 RU keys (with_mapping/с_маппингом)
Spec:
- 15 issues resolved (semantic label, filtering, endpoints, conflict, metric_count, RBAC, ID types, ports, i18n, validation, a11y, propagation, resize, perf, ownership)
Status: 14/14 backend tests pass, 52/52 frontend tests pass, build succeeds
Risk: Low — T051 browser validation pending (non-blocking)
2026-05-21 13:58:47 +03:00
8805c5d329
semantic
2026-05-20 23:54:53 +03:00
82c1b92a51
semantics
2026-05-19 18:36:15 +03:00
2b93692070
chore: commit remaining pre-existing changes
...
- Agent configs (.opencode/agents/)
- Backend: alembic, routes, app, utils, scripts
- Frontend: package.json, vite, components, e2e infra
- Specs: 028-llm-datasource-supeset updates
- Docker e2e config and Playwright setup
2026-05-17 19:23:07 +03:00
db18348ab1
chore: update semantic index, agent configs, and app startup
2026-05-14 21:33:26 +03:00
16c7b947dd
chore(lint): apply ruff --fix (4443 auto-fixes)
...
Auto-fixed categories:
- F401: unused imports removed
- I001: import blocks sorted
- W293: trailing whitespace stripped
- UP035: deprecated typing imports replaced
- SIM: simplify suggestions applied
- ARG: unused args prefixed with underscore
- T201: print statements removed
- F841: unused variables removed
- RUF059: unpacked variables prefixed
Remaining ~1500 unfixable errors (C901, B904, N806, E402) require manual work.
Backend smoke tests: 13/13 passed.
2026-05-14 11:20:17 +03:00
d081277ae9
feat(translate,scheduler,trace): new-key-only mode, stale PENDING protection, trace_id propagation
...
Translations:
- FR-045: new-key-only execution mode — translate only rows with unseen keys
- Compare source row keys against TranslationRecord.source_data from last succeeded run
- Baseline expired (>90 days) fallback to full mode with baseline_expired event
- run_noop early return when zero new keys (skip LLM + SQL)
Scheduler reliability:
- Stale PENDING protection: runs older than 1h auto-marked FAILED, no longer block schedule
- load_schedules() reloads active translation schedules from DB on restart
- add_translation_job/remove_translation_job register/unregister with APScheduler
- execution_mode column on translation_schedules with additive DB migration
Automation view:
- GET /settings/automation/translation-schedules endpoint
- Translation schedules displayed on Automation page below validation policies
Trace propagation:
- seed_trace_id() in all background entry points:
TaskManager._run_task/_flusher_loop, SchedulerService._trigger_backup,
websocket_endpoint, IdMappingService, all standalone scripts
Migrations:
- dictionary_entries: origin_run_id, origin_row_key, origin_user_id
- translation_schedules: execution_mode
Tests:
- 3 new test modules (22 tests): core_scheduler, executor_filter, scheduler_execution+guard
- Fix pre-existing translate test isolation (conftest.py)
- Fix test_list_runs_filter_status parameter
2026-05-14 10:13:56 +03:00
ffcfec9acf
semantics
2026-05-13 14:15:33 +03:00
6e9bd522e9
logs: migrate to molecular CoT JSON protocol, fix regressions
...
- Replace BeliefFormatter with CotJsonFormatter (single-line JSON with
ts, level, trace_id, src, marker, intent, payload, error, span_id)
- Migrate belief_scope to use cot_log() for structured JSON output
- Update monkey-patched reason/reflect/explore to pass extra= dict
- Strip 200+ inline [REASON]/[REFLECT]/[EXPLORE] markers from 50+ files
- Remove belief_scope from hot-path utilities (_parse_datetime,
_json_load_if_needed) to eliminate startup log noise
- Register TraceContextMiddleware in app.py (was implemented but unused)
- Seed trace_id in startup_event/shutdown_event for background tasks
- Fix broken relative imports in translate routes (3 dots → 4 dots)
- Migrate 43 translate_routes log calls to logger.reason/explore
- Fix SyntaxError (positional arg after extra=) in _repo_operations_routes
- Fix IndentationError in rbac_permission_catalog except-block
- Add smoke test tests/test_smoke_app.py (catches import errors before run)
2026-05-13 09:44:50 +03:00
f2a24447ed
semantics: complete DEF-to-region migration, fix regressions
...
- Convert legacy [DEF🆔 Type] anchors to #region/#endregion across 329 files
- Reinstate _normalize_timestamp_value in sql_generator.py
- Fix MarkerLogger→logger migration in events.py (molecular CoT markers)
- Fix dataset_review orchestrator dependencies (_build_execution_snapshot)
- Fix config_manager stale-record deletion (moved to save path only)
- Add 77 missing [/DEF:] closers in 5 unbalanced test files
- Update assistant_chat.integration.test.js for #region format
- Apply molecular-cot-logging markers (REASON/REFLECT/EXPLORE) via logger.* methods
2026-05-12 23:54:55 +03:00
13f40048c1
semantics
2026-05-12 20:06:16 +03:00
623cad065d
log refactor
2026-05-12 19:30:15 +03:00
85121191d4
feat: implement LLM table translation service with searchable datasource dropdown, i18n, sidebar, and RBAC
...
- Add backend plugin modules: preview, executor, orchestrator, events, sql_generator, superset_executor, dictionary, scheduler, metrics
- Add API routes for translate jobs, runs, dictionaries, preview, schedule, metrics, corrections
- Add ORM models (12 tables) and Pydantic schemas (15 DTOs)
- Register translate router in app.py with RBAC permission guards
- Add frontend pages: job list, job config, dictionary list/editor, history
- Add 7 reusable Svelte 5 components: Preview, RunProgress, RunResult, TermCorrection, BulkCorrection, ScheduleConfig, MetricsDashboard
- Add searchable datasource dropdown with Superset API integration
- Add i18n (en/ru) with ~300 keys across common, config, jobs, dictionaries, preview namespaces
- Add Translation sidebar category with Jobs/Dictionaries/History sub-items
- Hide health monitor error toast via suppressToast API option
- Add 69 backend tests and 44 frontend test files
- Fix: SupersetClient env resolution (string -> Environment object)
- Fix: Dataset detail API returns proper database dict
- Fix: Database dialect extraction fallback when metadata incomplete
2026-05-09 19:34:25 +03:00
78d6995e13
semantics
2026-04-01 21:57:51 +03:00
e83879cf06
semantics
2026-03-27 21:27:31 +03:00
0bc6f59a7a
Bootstrap initial admin via env and add compose profiles
2026-03-17 19:16:25 +03:00
d56084e5c8
feat(us1): add dataset review orchestration automatic review slice
2026-03-17 10:57:49 +03:00
30323d8273
chore: update semantic contracts and git merge handling
2026-03-16 20:34:28 +03:00
96d8712b6f
refactor(semantics): migrate legacy @TIER to @COMPLEXITY annotations
...
- Replaced @TIER: TRIVIAL with @COMPLEXITY: 1
- Replaced @TIER: STANDARD with @COMPLEXITY: 3
- Replaced @TIER: CRITICAL with @COMPLEXITY: 5
- Manually elevated specific critical/complex components to levels 2 and 4
- Ignored legacy, specs, and node_modules directories
- Updated generated semantic map
2026-03-16 10:06:44 +03:00
613a923c36
chore(semantic): checkpoint remediation progress
2026-03-15 21:08:00 +03:00
12b522f852
feat add connections management and health summary improvements
2026-03-15 16:40:43 +03:00
b6bb732a40
код написан
2026-03-10 12:00:18 +03:00
b4e0e21446
feat(clean-release): complete compliance redesign phases and polish tasks T047-T052
2026-03-10 09:11:26 +03:00
ac74a424e1
feat(rbac): hide unauthorized menu sections and enforce route guards
2026-03-06 10:50:28 +03:00
b539d4e8c3
test: remediate audit findings for task log viewer, report card and logger tests
2026-03-03 21:01:24 +03:00
b756948bac
test contracts
2026-02-26 19:40:00 +03:00
8936445e4a
semantic update
2026-02-24 21:08:12 +03:00
b98c7a8e4c
feat(assistant): implement spec 021 chat assistant flow with semantic contracts
2026-02-23 19:37:56 +03:00
f7ac6a9156
таски готовы
2026-02-23 10:18:56 +03:00
2280c24be5
semantic update
2026-02-20 10:41:15 +03:00
f686c0d85a
refactor
2026-02-18 17:29:46 +03:00
cae752e112
fix
2026-02-15 11:11:30 +03:00
b503e7cad1
linter + новые таски
2026-02-10 12:53:01 +03:00
f6a3e2cfbb
semantic update
2026-02-08 22:53:54 +03:00
718e29de7a
Похоже работает
2026-02-07 11:26:06 +03:00
9ced48dd55
Вроде работает
2026-01-30 11:10:16 +03:00
7ea90e200d
Передаем на тест
2026-01-27 16:32:08 +03:00
b336cab132
Передаем на тест
2026-01-25 18:33:00 +03:00
76610ce412
Работает создание коммитов и перенос в новый enviroment
2026-01-23 13:57:44 +03:00
8f8ffa5408
refactor complete
2026-01-22 17:37:17 +03:00
ced5b7b1ac
ашч
2026-01-21 14:00:48 +03:00
ea3705fe4a
semantic markup update
2026-01-18 21:29:54 +03:00
6d20309bfc
semantics update
2026-01-13 09:11:27 +03:00
