Commit Graph

124 Commits

Author SHA1 Message Date
320f82ab95 feat(auth): implement API key authentication and management
Introduce a new API key authentication mechanism to support service-to-service
communication (e.g., Airflow, CI/CD) without browser-based JWT login.

- Add `APIKey` model and `APIKeyPrincipal` dependency for RBAC.
- Implement `require_api_key_or_jwt` dependency with environment scoping.
- Add admin CRUD endpoints for API key lifecycle management.
- Add API key management UI in System Settings.
- Update maintenance API to support explicit `environment_id` and API key auth.
- Add i18n support for API keys and connection settings.
- Include Python and Bash usage examples in the `examples/` directory.
- Update technical specifications and documentation.
2026-05-25 11:35:27 +03:00
31680a1bc9 fix(maintenance): auto-expiry + adaptive markdown height + tests
- Auto-expiry: expired events auto-end on GET /events via task manager
- Height: _estimate_markdown_height adapted to unit=8px scale with padding detection
- CRITICAL-1: removed dead import remove_chart_from_position (QA finding)
- CRITICAL-2: added chart alive check in ensure_banner_chart (QA finding)
- CRITICAL-3: fixed test assertions update_markdown_chart → update_banner_on_dashboard
- @POST contract: fixed return range [2,12] → [19,200]
- Tests: 8 new tests (auto-expiry + layout height)
2026-05-25 08:36:33 +03:00
7219c47357 fix(datasets): fix oversized buttons UI + populate linked_dashboard_count in list endpoint
StatsBar — переделаны огромные карточки-кнопки в компактные пилюли
(rounded-full px-3 py-1 text-sm вместо p-3 grid grid-cols-4).

DatasetList — экшн-кнопки из bg-primary в outline-стиль, карточки
плотнее (p-2.5, py-0.5), ESLint-фиксы (#each key).

Backend — linked_dashboard_count теперь заполняется в списке датасетов
через /dataset/{id}/related_objects (конкурентно, Semaphore=3, timeout=10s).
Ранее поле было только в get_dataset_detail и StatsBar показывал 0.
2026-05-24 09:38:21 +03:00
36643024cf refactor(connections): remove ConnectionConfig, migrate mapper to SQL Lab
Backend:
- Remove ConnectionConfig model, CRUD routes (connections.py), and tests
- Remove ensure_connection_configs_table from database.py
- Migrate MapperPlugin from psycopg2 direct PostgreSQL to SupersetSqlLabExecutor
- Add DatasetMapper.get_sqllab_mappings() with default information_schema query
- Update MapColumnsRequest: connection_id → database_id + sql_query
- Fix pydantic_settings v2 deprecation (Field(env=...) → validation_alias)
- Clean up app.py, routes/__init__.py, database.py imports

Frontend:
- Remove DatabaseConnectionsTab, ConnectionForm, ConnectionList components
- Remove /settings/connections route page and Navbar link
- Remove connectionService.js, connections i18n files
- Update MapperTool.svelte: postgres → sqllab source
- Update Map Columns modal: database selector + SQL query instead of connection_id
- Clean up settings page tabs, nav links, test mocks
- Clean up i18n keys (settings, nav, mapper)

Tests: 25/25 pass (14 030-feature + 11 route tests)
Build: succeeds
2026-05-21 18:47:41 +03:00
b529e8082a feat(030): Dataset Lifecycle Workspace — Stats Bar, split-view, inline-edit, bulk actions
Backend:
- Add MetricItem, StatsCounts, ColumnDescriptionUpdate, MetricDescriptionUpdate DTOs
- Add metric_count to DatasetItem and DatasetDetailResponse
- Add server-side filtering via ?filter=unmapped|mapped|linked|all
- Add PUT endpoints for column/metric description inline-edit
- Add HTML stripping validation (max 2000 chars, plain text)
- Add WebSocket dataset.updated event on task completion
- RBAC: plugin:migration:WRITE for mutations, READ for reads
- 14 pytest tests (stats, filter, metrics, inline-edit, validation, 502/503)

Frontend:
- Rewrite +page.svelte as split-view orchestrator (387 lines)
- StatsBar.svelte — 4 metric tiles with aria-pressed, server-side filter dispatch
- DatasetList.svelte — card grid with progress bars, checkboxes, search
- DatasetPreview.svelte — presentational detail panel (container fetches)
- ColumnsTable.svelte — inline-edit with type chips, bold/italic fallback
- MetricsTable.svelte — inline-edit with expression hints
- 52 vitest tests across 7 test files
- i18n: 11 EN + 11 RU keys (with_mapping/с_маппингом)

Spec:
- 15 issues resolved (semantic label, filtering, endpoints, conflict, metric_count, RBAC, ID types, ports, i18n, validation, a11y, propagation, resize, perf, ownership)

Status: 14/14 backend tests pass, 52/52 frontend tests pass, build succeeds
Risk: Low — T051 browser validation pending (non-blocking)
2026-05-21 13:58:47 +03:00
68740cd9ed semantic 2026-05-20 23:54:53 +03:00
b916ef94d5 semantics 2026-05-19 18:36:15 +03:00
6988e63967 chore: commit remaining pre-existing changes
- Agent configs (.opencode/agents/)
- Backend: alembic, routes, app, utils, scripts
- Frontend: package.json, vite, components, e2e infra
- Specs: 028-llm-datasource-supeset updates
- Docker e2e config and Playwright setup
2026-05-17 19:23:07 +03:00
f872e610a9 refactor: decompose oversized contracts to satisfy INV_7 fractal limit
Break monolithic modules >400 lines into focused sub-modules while
preserving backward-compatible imports and all test coverage:

Backend (Python):
- TranslationExecutor: 1974→241 lines, split into 9 sub-modules
- Translate plugin: orchestrator (1137→148), preview (1303→244),
  service (1052→275), dictionary (1007→68)
- ProfileService: 857→172 with 4 extracted sub-modules
- TaskManager: 708→322 with graph/event_bus/lifecycle extracted
- Test dictionary: 1199→split into 6 focused test files

Frontend (Svelte):
- SettingsPage: 1451→291 with 6 extracted tab components
- GitManager: 1220→228 with 5 extracted panels
- DatasetReviewWorkspace: 1202→314
- translate.js API: 664→28 barrel with 6 domain modules

Protocol:
- Remove single-contract 150-line limit from INV_7 (keep CC≤10)
- Fix unclosed #endregion tags across 11 files
- Fix 19 test regressions from stale mock paths
- All 294 tests passing
2026-05-17 19:18:32 +03:00
c6189876b3 chore(lint): apply ruff --fix (4443 auto-fixes)
Auto-fixed categories:
- F401: unused imports removed
- I001: import blocks sorted
- W293: trailing whitespace stripped
- UP035: deprecated typing imports replaced
- SIM: simplify suggestions applied
- ARG: unused args prefixed with underscore
- T201: print statements removed
- F841: unused variables removed
- RUF059: unpacked variables prefixed

Remaining ~1500 unfixable errors (C901, B904, N806, E402) require manual work.

Backend smoke tests: 13/13 passed.
2026-05-14 11:20:17 +03:00
8d0bc6fb93 feat(translate,scheduler,trace): new-key-only mode, stale PENDING protection, trace_id propagation
Translations:
- FR-045: new-key-only execution mode — translate only rows with unseen keys
- Compare source row keys against TranslationRecord.source_data from last succeeded run
- Baseline expired (>90 days) fallback to full mode with baseline_expired event
- run_noop early return when zero new keys (skip LLM + SQL)

Scheduler reliability:
- Stale PENDING protection: runs older than 1h auto-marked FAILED, no longer block schedule
- load_schedules() reloads active translation schedules from DB on restart
- add_translation_job/remove_translation_job register/unregister with APScheduler
- execution_mode column on translation_schedules with additive DB migration

Automation view:
- GET /settings/automation/translation-schedules endpoint
- Translation schedules displayed on Automation page below validation policies

Trace propagation:
- seed_trace_id() in all background entry points:
  TaskManager._run_task/_flusher_loop, SchedulerService._trigger_backup,
  websocket_endpoint, IdMappingService, all standalone scripts

Migrations:
- dictionary_entries: origin_run_id, origin_row_key, origin_user_id
- translation_schedules: execution_mode

Tests:
- 3 new test modules (22 tests): core_scheduler, executor_filter, scheduler_execution+guard
- Fix pre-existing translate test isolation (conftest.py)
- Fix test_list_runs_filter_status parameter
2026-05-14 10:13:56 +03:00
d1695fe536 fix(translate,automation): fix schedule import/param errors, add translation schedules to Automation view
- Fix ModuleNotFoundError: 4 lazy imports in _schedule_routes.py changed
  from 3-dot to 4-dot relative imports (src.api.dependencies -> src.dependencies)
- Fix TypeError: update_schedule() param name mismatch (timezone -> timezone_str)
- Add add_translation_job/remove_translation_job to SchedulerService
  to register translation schedules with APScheduler via execute_scheduled_translation
- Add GET /settings/automation/translation-schedules endpoint
  returning all translation schedules with joined job names
- Update Automation settings page to display translation schedules
  (cron, timezone, active badge, last run) below validation policies
2026-05-13 21:08:10 +03:00
a3c7c402b7 fix(llm): add fetch-models endpoint, fix SQL Lab INSERT (client_id truncation, sync mode, target_column, timestamp normalization)
- Add POST /api/llm/providers/fetch-models route with LLMClient.fetch_models()
- Add target_column to TranslationJob model/schema/service/orchestrator
- Fix SQL Lab execute: truncate client_id to 11 chars (varchar(11))
- Switch SQL Lab to sync mode (runAsync: false) — no Celery workers
- Fix polling: unwrap nested result from Superset query API
- Fix ClickHouse timestamp: normalize float timestamps to YYYY-MM-DD
2026-05-13 20:06:15 +03:00
39ab647851 semantics 2026-05-13 14:15:33 +03:00
83b8f4d999 logs: migrate to molecular CoT JSON protocol, fix regressions
- Replace BeliefFormatter with CotJsonFormatter (single-line JSON with
  ts, level, trace_id, src, marker, intent, payload, error, span_id)
- Migrate belief_scope to use cot_log() for structured JSON output
- Update monkey-patched reason/reflect/explore to pass extra= dict
- Strip 200+ inline [REASON]/[REFLECT]/[EXPLORE] markers from 50+ files
- Remove belief_scope from hot-path utilities (_parse_datetime,
  _json_load_if_needed) to eliminate startup log noise
- Register TraceContextMiddleware in app.py (was implemented but unused)
- Seed trace_id in startup_event/shutdown_event for background tasks
- Fix broken relative imports in translate routes (3 dots → 4 dots)
- Migrate 43 translate_routes log calls to logger.reason/explore
- Fix SyntaxError (positional arg after extra=) in _repo_operations_routes
- Fix IndentationError in rbac_permission_catalog except-block
- Add smoke test tests/test_smoke_app.py (catches import errors before run)
2026-05-13 09:44:50 +03:00
b6f46fe9f5 logger: migrate belief_scope from legacy markers to molecular CoT protocol
Replace [Entry]/[Exit]/[Action]/[COHERENCE:OK] with [REASON]/[REFLECT]/[EXPLORE]
in belief_scope context manager, BeliefFormatter, and 34 source files.
Per molecular-cot-logging skill invariants.
2026-05-13 08:46:24 +03:00
306c5ae742 semantics: complete DEF-to-region migration, fix regressions
- Convert legacy [DEF🆔Type] anchors to #region/#endregion across 329 files
- Reinstate _normalize_timestamp_value in sql_generator.py
- Fix MarkerLogger→logger migration in events.py (molecular CoT markers)
- Fix dataset_review orchestrator dependencies (_build_execution_snapshot)
- Fix config_manager stale-record deletion (moved to save path only)
- Add 77 missing [/DEF:] closers in 5 unbalanced test files
- Update assistant_chat.integration.test.js for #region format
- Apply molecular-cot-logging markers (REASON/REFLECT/EXPLORE) via logger.* methods
2026-05-12 23:54:55 +03:00
fe8978f716 semantics 2026-05-12 20:06:16 +03:00
b17b5333c7 log refactor 2026-05-12 19:30:15 +03:00
1d59df2233 refactor 2026-05-12 14:52:27 +03:00
dbb8bd6c4e fix: persist environment selection, support Kilo AI provider, resolve Superset virtual columns for translation preview
- Add environment_id to TranslationJob model/schema/API + DB migration
- Pass environmentId from page to TranslationPreview and fetchPreview
- Fix _fetch_sample_rows: use result_type='samples' to include virtual cols
- Add 'kilo' and 'openrouter' to supported LLM provider types
- Make response_format conditional (skip for non-OpenAI upstream providers)
- Remove source_table field from form (redundant with datasource)
- Restore datasourceSearch display from saved job on page load
- Add request/response logging for LLM API calls
2026-05-09 23:10:15 +03:00
201886eeb0 refactor: decompose 6 monolithic modules (>1000 LOC) into domain packages
Decompose 6 large modules into packages with <400-line modules to satisfy INV_7:

- superset_client.py (2145->0) -> superset_client/ package (12 modules)
- assistant.py (2683->0) -> assistant/ package (12 modules)
- git_service.py (2101->11 shim) -> services/git/ package (9 modules)
- git.py (1757->0) -> routes/git/ package (11 modules)
- dashboards.py (1619->0) -> routes/dashboards/ package (8 modules)
- translate.py (1587->0) -> routes/translate/ package (11 modules)
- superset_context_extractor.py (1397->0) -> utils/superset_context_extractor/ (7 modules)

All modules <400 lines (INV_7). All 80 tests pass.
All external imports preserved via __init__.py re-exports or shim.
Semantic [DEF] contracts with @LAYER/@SEMANTICS added to all Module types.
2026-05-09 10:13:07 +03:00
1b15fd3fa7 semantic cleanup 2026-05-08 10:07:05 +03:00
0b227a307c refactor 2026-04-24 17:10:02 +03:00
277ce7b3b3 skills + agents 2026-04-19 20:05:45 +03:00
2228a00035 semantics 2026-04-02 12:12:23 +03:00
80ec0c5eb1 prune contracts 2026-04-01 22:31:10 +03:00
f5c90af287 semantics 2026-04-01 21:57:51 +03:00
ace77ed64c semantics update 2026-04-01 15:30:13 +03:00
7901ce0f39 semantics 2026-03-27 21:27:31 +03:00
202823aba5 feat(ui): add chat-driven dataset review flow
Move dataset review clarification into the assistant workspace and
rework the review page into a chat-centric layout with execution rails.

Add session-scoped assistant actions for mappings, semantic fields,
and SQL preview generation. Introduce optimistic locking for dataset
review mutations, propagate session versions through API responses,
and mask imported filter values before assistant exposure.

Refresh tests, i18n, and spec artifacts to match the new workflow.

BREAKING CHANGE: dataset review mutation endpoints now require the
X-Session-Version header, and clarification is no longer handled
through ClarificationDialog-based flows
2026-03-26 13:33:12 +03:00
2da548fd71 fix: finalize semantic repair and test updates 2026-03-21 15:07:06 +03:00
81406bc2e2 fix: commit semantic repair changes 2026-03-21 11:22:25 +03:00
4d4f204f1c semantics 2026-03-20 20:01:58 +03:00
62d7c660ef subagents 2026-03-20 17:20:24 +03:00
7cff6bdcf8 semantic 2026-03-18 08:45:15 +03:00
b51a68168c feat(027): Final Phase T038-T043 implementation
- T038: SessionEvent logger and persistence logic
  - Added SessionEventLogger service with explicit audit event persistence
  - Added SessionEvent model with events relationship on DatasetReviewSession
  - Integrated event logging into orchestrator flows and API mutation endpoints

- T039: Semantic source version propagation
  - Added source_version column to SemanticFieldEntry
  - Added propagate_source_version_update() to SemanticResolver
  - Preserves locked/manual field invariants during propagation

- T040: Batch approval API and UI actions
  - Added batch semantic approval endpoint (/fields/semantic/approve-batch)
  - Added batch mapping approval endpoint (/mappings/approve-batch)
  - Added batch approval actions to SemanticLayerReview and ExecutionMappingReview components
  - Aligned batch semantics with single-item approval contracts

- T041: Superset compatibility matrix tests
  - Added test_superset_matrix.py with preview and SQL Lab fallback coverage
  - Tests verify client method preference and matrix fallback behavior

- T042: RBAC audit sweep on session-mutation endpoints
  - Added _require_owner_mutation_scope() helper
  - Applied owner guards to update_session, delete_session, and all mutation endpoints
  - Ensured no bypass of existing permission checks

- T043: i18n coverage for dataset-review UI
  - Added workspace state labels (empty/importing/review) to en.json and ru.json
  - Added batch action labels for semantics and mappings
  - Fixed workspace state comparison to lowercase strings
  - Removed hardcoded workspace state display strings

Signed-off-by: Implementation Specialist <impl@ss-tools>
2026-03-17 14:29:33 +03:00
f9fc2811a2 fix(027): stabilize shared acceptance gates and compatibility collateral 2026-03-17 11:07:49 +03:00
268ac66a38 feat(us1): add dataset review orchestration automatic review slice 2026-03-17 10:57:49 +03:00
a6a4288443 feat: initial dataset review orchestration flow implementation 2026-03-16 23:43:03 +03:00
188620a224 fix 2026-03-16 21:27:33 +03:00
8cfa7b7c3c chore: update semantic contracts and git merge handling 2026-03-16 20:34:28 +03:00
54a7bc3242 semantic 2026-03-16 16:45:08 +03:00
67867f8220 refactor(semantics): migrate legacy @TIER to @COMPLEXITY annotations
- Replaced @TIER: TRIVIAL with @COMPLEXITY: 1
- Replaced @TIER: STANDARD with @COMPLEXITY: 3
- Replaced @TIER: CRITICAL with @COMPLEXITY: 5
- Manually elevated specific critical/complex components to levels 2 and 4
- Ignored legacy, specs, and node_modules directories
- Updated generated semantic map
2026-03-16 10:06:44 +03:00
2f22bdd2ff chore(semantics): checkpoint orphan-reduction hub normalization batch 2026-03-15 22:14:05 +03:00
1ff034a73f chore(semantic): remediate backend core contracts 2026-03-15 21:23:44 +03:00
63940020db chore(semantic): checkpoint remediation progress 2026-03-15 21:08:00 +03:00
bcce55fbf9 feat add connections management and health summary improvements 2026-03-15 16:40:43 +03:00
4efce79df5 Fix LLM validation and dashboard health hot paths 2026-03-15 13:18:51 +03:00
d867f52ae5 feat: Implement LLM provider deletion and refactor ConfigManager to preserve unknown payload sections. 2026-03-14 09:19:08 +03:00