busya/ss-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
336 Commits 3 Branches 0 Tags
bb21fd316569ea40a5fa056c4cfd8963943034ad
Commit Graph

50 Commits

Author SHA1 Message Date
busya
bb0fbfdafd feat(translate): multi-language optimization (Phase 11) 
- Auto-detection of source language per row via LLM (US6)
- Multi-target translation — one LLM call for N languages (US1-US3)
- Language-aware storage: TranslationLanguage, per-language stats
- Multilingual dictionaries with language-pair-aware filtering (US7)
- Inline correction on any run result + submit-to-dictionary (US8)
- Context-aware dictionary: auto-capture row context, usage notes,
  Jaccard similarity, priority flagging in LLM prompts (US8b)
- Configurable preview sample size 1-100, cost warning at >30
- Per-language history & metrics with MetricSnapshot preservation
- 36 files, +5022/-373, all specs GRACE-Poly v2.6 compliant
 2026-05-14 17:12:41 +03:00
busya
c6189876b3 chore(lint): apply ruff --fix (4443 auto-fixes) 
Auto-fixed categories:
- F401: unused imports removed
- I001: import blocks sorted
- W293: trailing whitespace stripped
- UP035: deprecated typing imports replaced
- SIM: simplify suggestions applied
- ARG: unused args prefixed with underscore
- T201: print statements removed
- F841: unused variables removed
- RUF059: unpacked variables prefixed

Remaining ~1500 unfixable errors (C901, B904, N806, E402) require manual work.

Backend smoke tests: 13/13 passed.
 2026-05-14 11:20:17 +03:00
busya
8d0bc6fb93 feat(translate,scheduler,trace): new-key-only mode, stale PENDING protection, trace_id propagation 
Translations:
- FR-045: new-key-only execution mode — translate only rows with unseen keys
- Compare source row keys against TranslationRecord.source_data from last succeeded run
- Baseline expired (>90 days) fallback to full mode with baseline_expired event
- run_noop early return when zero new keys (skip LLM + SQL)

Scheduler reliability:
- Stale PENDING protection: runs older than 1h auto-marked FAILED, no longer block schedule
- load_schedules() reloads active translation schedules from DB on restart
- add_translation_job/remove_translation_job register/unregister with APScheduler
- execution_mode column on translation_schedules with additive DB migration

Automation view:
- GET /settings/automation/translation-schedules endpoint
- Translation schedules displayed on Automation page below validation policies

Trace propagation:
- seed_trace_id() in all background entry points:
  TaskManager._run_task/_flusher_loop, SchedulerService._trigger_backup,
  websocket_endpoint, IdMappingService, all standalone scripts

Migrations:
- dictionary_entries: origin_run_id, origin_row_key, origin_user_id
- translation_schedules: execution_mode

Tests:
- 3 new test modules (22 tests): core_scheduler, executor_filter, scheduler_execution+guard
- Fix pre-existing translate test isolation (conftest.py)
- Fix test_list_runs_filter_status parameter
 2026-05-14 10:13:56 +03:00
busya
a3c7c402b7 fix(llm): add fetch-models endpoint, fix SQL Lab INSERT (client_id truncation, sync mode, target_column, timestamp normalization) 
- Add POST /api/llm/providers/fetch-models route with LLMClient.fetch_models()
- Add target_column to TranslationJob model/schema/service/orchestrator
- Fix SQL Lab execute: truncate client_id to 11 chars (varchar(11))
- Switch SQL Lab to sync mode (runAsync: false) — no Celery workers
- Fix polling: unwrap nested result from Superset query API
- Fix ClickHouse timestamp: normalize float timestamps to YYYY-MM-DD
 2026-05-13 20:06:15 +03:00
busya
83b8f4d999 logs: migrate to molecular CoT JSON protocol, fix regressions 
- Replace BeliefFormatter with CotJsonFormatter (single-line JSON with
  ts, level, trace_id, src, marker, intent, payload, error, span_id)
- Migrate belief_scope to use cot_log() for structured JSON output
- Update monkey-patched reason/reflect/explore to pass extra= dict
- Strip 200+ inline [REASON]/[REFLECT]/[EXPLORE] markers from 50+ files
- Remove belief_scope from hot-path utilities (_parse_datetime,
  _json_load_if_needed) to eliminate startup log noise
- Register TraceContextMiddleware in app.py (was implemented but unused)
- Seed trace_id in startup_event/shutdown_event for background tasks
- Fix broken relative imports in translate routes (3 dots → 4 dots)
- Migrate 43 translate_routes log calls to logger.reason/explore
- Fix SyntaxError (positional arg after extra=) in _repo_operations_routes
- Fix IndentationError in rbac_permission_catalog except-block
- Add smoke test tests/test_smoke_app.py (catches import errors before run)
 2026-05-13 09:44:50 +03:00
busya
b17b5333c7 log refactor 2026-05-12 19:30:15 +03:00
busya
67ba04d4ff feat: implement LLM table translation service with searchable datasource dropdown, i18n, sidebar, and RBAC 
- Add backend plugin modules: preview, executor, orchestrator, events, sql_generator, superset_executor, dictionary, scheduler, metrics
- Add API routes for translate jobs, runs, dictionaries, preview, schedule, metrics, corrections
- Add ORM models (12 tables) and Pydantic schemas (15 DTOs)
- Register translate router in app.py with RBAC permission guards
- Add frontend pages: job list, job config, dictionary list/editor, history
- Add 7 reusable Svelte 5 components: Preview, RunProgress, RunResult, TermCorrection, BulkCorrection, ScheduleConfig, MetricsDashboard
- Add searchable datasource dropdown with Superset API integration
- Add i18n (en/ru) with ~300 keys across common, config, jobs, dictionaries, preview namespaces
- Add Translation sidebar category with Jobs/Dictionaries/History sub-items
- Hide health monitor error toast via suppressToast API option
- Add 69 backend tests and 44 frontend test files
- Fix: SupersetClient env resolution (string -> Environment object)
- Fix: Dataset detail API returns proper database dict
- Fix: Database dialect extraction fallback when metadata incomplete
 2026-05-09 19:34:25 +03:00
busya
201886eeb0 refactor: decompose 6 monolithic modules (>1000 LOC) into domain packages 
Decompose 6 large modules into packages with <400-line modules to satisfy INV_7:

- superset_client.py (2145->0) -> superset_client/ package (12 modules)
- assistant.py (2683->0) -> assistant/ package (12 modules)
- git_service.py (2101->11 shim) -> services/git/ package (9 modules)
- git.py (1757->0) -> routes/git/ package (11 modules)
- dashboards.py (1619->0) -> routes/dashboards/ package (8 modules)
- translate.py (1587->0) -> routes/translate/ package (11 modules)
- superset_context_extractor.py (1397->0) -> utils/superset_context_extractor/ (7 modules)

All modules <400 lines (INV_7). All 80 tests pass.
All external imports preserved via __init__.py re-exports or shim.
Semantic [DEF] contracts with @LAYER/@SEMANTICS added to all Module types.
 2026-05-09 10:13:07 +03:00
busya
7901ce0f39 semantics 2026-03-27 21:27:31 +03:00
busya
e86204df6c test semantics 2026-03-21 15:23:06 +03:00
busya
2da548fd71 fix: finalize semantic repair and test updates 2026-03-21 15:07:06 +03:00
busya
81406bc2e2 fix: commit semantic repair changes 2026-03-21 11:22:25 +03:00
busya
cf601a717e fix: commit verified semantic repair changes 2026-03-21 11:05:20 +03:00
busya
10a5a7f741 fix(final-phase): finalize dataset review audit blockers 2026-03-17 18:23:02 +03:00
busya
800916d40b fix(tests): Add model imports to fix SQLAlchemy registration in matrix tests 2026-03-17 14:33:15 +03:00
busya
b51a68168c feat(027): Final Phase T038-T043 implementation 
- T038: SessionEvent logger and persistence logic
  - Added SessionEventLogger service with explicit audit event persistence
  - Added SessionEvent model with events relationship on DatasetReviewSession
  - Integrated event logging into orchestrator flows and API mutation endpoints

- T039: Semantic source version propagation
  - Added source_version column to SemanticFieldEntry
  - Added propagate_source_version_update() to SemanticResolver
  - Preserves locked/manual field invariants during propagation

- T040: Batch approval API and UI actions
  - Added batch semantic approval endpoint (/fields/semantic/approve-batch)
  - Added batch mapping approval endpoint (/mappings/approve-batch)
  - Added batch approval actions to SemanticLayerReview and ExecutionMappingReview components
  - Aligned batch semantics with single-item approval contracts

- T041: Superset compatibility matrix tests
  - Added test_superset_matrix.py with preview and SQL Lab fallback coverage
  - Tests verify client method preference and matrix fallback behavior

- T042: RBAC audit sweep on session-mutation endpoints
  - Added _require_owner_mutation_scope() helper
  - Applied owner guards to update_session, delete_session, and all mutation endpoints
  - Ensured no bypass of existing permission checks

- T043: i18n coverage for dataset-review UI
  - Added workspace state labels (empty/importing/review) to en.json and ru.json
  - Added batch action labels for semantics and mappings
  - Fixed workspace state comparison to lowercase strings
  - Removed hardcoded workspace state display strings

Signed-off-by: Implementation Specialist <impl@ss-tools>
 2026-03-17 14:29:33 +03:00
busya
f9fc2811a2 fix(027): stabilize shared acceptance gates and compatibility collateral 2026-03-17 11:07:49 +03:00
busya
67867f8220 refactor(semantics): migrate legacy @TIER to @COMPLEXITY annotations 
- Replaced @TIER: TRIVIAL with @COMPLEXITY: 1
- Replaced @TIER: STANDARD with @COMPLEXITY: 3
- Replaced @TIER: CRITICAL with @COMPLEXITY: 5
- Manually elevated specific critical/complex components to levels 2 and 4
- Ignored legacy, specs, and node_modules directories
- Updated generated semantic map
 2026-03-16 10:06:44 +03:00
busya
5e6fe2fe61 security: rotate bootstrap and clean workspace 2026-03-13 12:14:37 +03:00
busya
a544a90cc9 Commit remaining workspace changes 2026-03-13 11:45:06 +03:00
busya
0e94268980 Add docker admin bootstrap for clean release 2026-03-13 11:41:44 +03:00
busya
a95109da4a feat: add slug-only dashboard profile filter and unify backend imports 2026-03-11 12:20:34 +03:00
busya
6775c7e3db feat(clean-release): complete compliance redesign phases and polish tasks T047-T052 2026-03-10 09:11:26 +03:00
busya
1b2f76ee23 feat: Implement user profile preferences for start page, Git identity, and task drawer auto-open, alongside Git server default branch configuration. 2026-03-08 10:19:38 +03:00
busya
b8eafa28a2 feat(rbac): hide unauthorized menu sections and enforce route guards 2026-03-06 10:50:28 +03:00
busya
787f4e55f2 clean ui 2026-03-04 19:33:47 +03:00
busya
0c68fbcef5 test: remediate and stabilize auxiliary backend and frontend tests 
- Standardized task log, LLM provider, and report profile tests.
- Relocated auxiliary tests into __tests__ directories for consistency.
- Updated git_service and defensive guards with minor stability fixes discovered during testing.
- Added UX integration tests for the reports list component.
 2026-03-04 13:54:06 +03:00
busya
664e47d540 feat(clean-release): complete and verify backend test suite (33 passing tests) 
- Relocated and standardized tests for clean_release subsystem into __tests__ sub-packages.
- Implemented missing unit tests for preparation_service, audit_service, and stages.
- Enhanced API contract tests for candidate preparation and compliance reporting.
- Updated 023-clean-repo-enterprise coverage matrix with final verification results.
- Fixed relative import issues and model validation mismatches during test migration.
 2026-03-04 13:53:43 +03:00
busya
1bb654e14c move test 2026-03-04 09:18:42 +03:00
busya
4351cdfe39 [ 
{
        "file": "frontend/src/components/__tests__/task_log_viewer.test.js",
        "verdict": "APPROVED",
        "rejection_reason": "NONE",
        "audit_details": {
            "target_invoked": true,
            "pre_conditions_tested": true,
            "post_conditions_tested": true,
            "test_fixture_used": true,
            "edges_covered": true,
            "invariants_verified": true,
            "ux_states_tested": true,
            "semantic_anchors_present": true
        },
        "coverage_summary": {
            "total_edges": 2,
            "edges_tested": 2,
            "total_invariants": 1,
            "invariants_tested": 1,
            "total_ux_states": 3,
            "ux_states_tested": 3
        },
        "tier_compliance": {
            "source_tier": "CRITICAL",
            "meets_tier_requirements": true
        },
        "feedback": "Remediation successful: test tier matches CRITICAL, missing missing @TEST_EDGE no_task_id coverage added, test for @UX_FEEDBACK (autoScroll) added properly, missing inline=false (show=true) tested properly. Semantic RELATION tag fixed to VERIFIES."
    },
    {
        "file": "frontend/src/lib/components/reports/__tests__/report_card.ux.test.js",
        "verdict": "APPROVED",
        "rejection_reason": "NONE",
        "audit_details": {
            "target_invoked": true,
            "pre_conditions_tested": true,
            "post_conditions_tested": true,
            "test_fixture_used": true,
            "edges_covered": true,
            "invariants_verified": true,
            "ux_states_tested": true,
            "semantic_anchors_present": true
        },
        "coverage_summary": {
            "total_edges": 2,
            "edges_tested": 2,
            "total_invariants": 1,
            "invariants_tested": 1,
            "total_ux_states": 2,
            "ux_states_tested": 2
        },
        "tier_compliance": {
            "source_tier": "CRITICAL",
            "meets_tier_requirements": true
        },
        "feedback": "Remediation successful: @TEST_EDGE random_status and @TEST_EDGE empty_report_object tests explicitly assert on outcomes, @TEST_FIXTURE tested completely, Test tier switched to CRITICAL."
    },
    {
        "file": "backend/tests/test_logger.py",
        "verdict": "APPROVED",
        "rejection_reason": "NONE",
        "audit_details": {
            "target_invoked": true,
            "pre_conditions_tested": true,
            "post_conditions_tested": true,
            "test_fixture_used": true,
            "edges_covered": true,
            "invariants_verified": true,
            "ux_states_tested": false,
            "semantic_anchors_present": true
        },
        "coverage_summary": {
            "total_edges": 0,
            "edges_tested": 0,
            "total_invariants": 0,
            "invariants_tested": 0,
            "total_ux_states": 0,
            "ux_states_tested": 0
        },
        "tier_compliance": {
            "source_tier": "STANDARD",
            "meets_tier_requirements": true
        },
        "feedback": "Remediation successful: Test module semantic anchors added [DEF] and [/DEF] explicitly. Added missing @TIER tag and @RELATION: VERIFIES -> src/core/logger.py at the top of the file."
    }
]
 2026-03-03 21:05:29 +03:00
busya
ede337b977 test: remediate audit findings for task log viewer, report card and logger tests 2026-03-03 21:01:24 +03:00
busya
4f273f33ea chore: commit remaining workspace changes 2026-03-03 19:51:17 +03:00
busya
efce41b0c5 dry run migration 2026-02-27 20:48:18 +03:00
busya
7f96a983a9 [ 
{
    "file": "backend/src/api/routes/__tests__/test_dashboards.py",
    "verdict": "APPROVED",
    "rejection_reason": "NONE",
    "audit_details": {
      "target_invoked": true,
      "pre_conditions_tested": true,
      "post_conditions_tested": true,
      "test_data_used": true
    },
    "feedback": "All 9 previous findings remediated. @TEST_FIXTURE data aligned, all @TEST_EDGE scenarios covered, all @PRE negative tests present, all @SIDE_EFFECT assertions added. Full contract compliance."
  },
  {
    "file": "backend/src/api/routes/__tests__/test_datasets.py",
    "verdict": "APPROVED",
    "rejection_reason": "NONE",
    "audit_details": {
      "target_invoked": true,
      "pre_conditions_tested": true,
      "post_conditions_tested": true,
      "test_data_used": true
    },
    "feedback": "All 6 previous findings remediated. Full @PRE boundary coverage including page_size>100, empty IDs, missing env. @SIDE_EFFECT assertions added. 503 error path tested."
  },
  {
    "file": "backend/src/core/auth/__tests__/test_auth.py",
    "verdict": "APPROVED",
    "rejection_reason": "NONE",
    "audit_details": {
      "target_invoked": true,
      "pre_conditions_tested": true,
      "post_conditions_tested": true,
      "test_data_used": true
    },
    "feedback": "All 4 previous findings remediated. @SIDE_EFFECT last_login verified. Inactive user @PRE negative test added. Empty hash edge case covered. provision_adfs_user tested for both new and existing user paths."
  },
  {
    "file": "backend/src/services/__tests__/test_resource_service.py",
    "verdict": "APPROVED",
    "rejection_reason": "NONE",
    "audit_details": {
      "target_invoked": true,
      "pre_conditions_tested": true,
      "post_conditions_tested": true,
      "test_data_used": true
    },
    "feedback": "Both prior recommendations implemented. Full edge case coverage for _get_last_task_for_resource. No anti-patterns detected."
  },
  {
    "file": "backend/tests/test_resource_hubs.py",
    "verdict": "APPROVED",
    "rejection_reason": "NONE",
    "audit_details": {
      "target_invoked": true,
      "pre_conditions_tested": true,
      "post_conditions_tested": true,
      "test_data_used": true
    },
    "feedback": "Pagination boundary tests added. All @TEST_EDGE scenarios now covered. No anti-patterns detected."
  },
  {
    "file": "frontend/src/lib/components/assistant/__tests__/assistant_chat.integration.test.js",
    "verdict": "APPROVED",
    "rejection_reason": "NONE",
    "audit_details": {
      "target_invoked": true,
      "pre_conditions_tested": true,
      "post_conditions_tested": true,
      "test_data_used": true
    },
    "feedback": "No changes since previous audit. Contract scanning remains sound."
  },
  {
    "file": "frontend/src/lib/components/assistant/__tests__/assistant_confirmation.integration.test.js",
    "verdict": "APPROVED",
    "rejection_reason": "NONE",
    "audit_details": {
      "target_invoked": true,
      "pre_conditions_tested": true,
      "post_conditions_tested": true,
      "test_data_used": true
    },
    "feedback": "No changes since previous audit. Confirmation flow testing remains sound."
  }
]
 2026-02-27 09:59:57 +03:00
busya
31cdc7958d { "verdict": "APPROVED", "rejection_reason": "NONE", "audit_details": { "target_invoked": true, "pre_conditions_tested": true, "post_conditions_tested": true, "test_data_used": true }, "feedback": "The test suite robustly verifies the 
MigrationEngine
 contracts. It avoids Tautologies by cleanly substituting IdMappingService without mocking the engine itself. Cross-filter parsing asserts against hard-coded, predefined validation dictionaries (no Logic Mirroring). It successfully addresses @PRE negative cases (e.g. invalid zip paths, missing YAMLs) and rigorously validates @POST file transformations (e.g. in-place UUID substitutions and archive reconstruction)." }
 2026-02-25 17:47:55 +03:00
busya
cd259a8292 feat: Enhance ID mapping service robustness, add defensive guards, and expand migration engine and API testing. 2026-02-25 14:44:21 +03:00
busya
94691ed1a5 ready for test 2026-02-25 13:35:09 +03:00
busya
f850d5d28d { "verdict": "APPROVED", "rejection_reason": "NONE", "audit_details": { "target_invoked": true, "pre_conditions_tested": true, "post_conditions_tested": true, "test_data_used": true }, "feedback": "Both test files have successfully passed the audit. The 'task_log_viewer.test.js' suite now correctly imports and mounts the real Svelte component using Test Library, fully eliminating the logic mirror/tautology issue. The 'test_logger.py' suite now properly implements negative tests for the @PRE constraint in 'belief_scope' and fully verifies all @POST effects triggered by 'configure_logger'." } 2026-02-24 21:55:13 +03:00
busya
f1f46f4e11 semantic update 2026-02-23 13:15:48 +03:00
busya
2bcae76d29 таски готовы 2026-02-23 10:18:56 +03:00
busya
d89c92ad35 tests ready 2026-02-19 13:33:20 +03:00
busya
00d4960da0 refactor 2026-02-18 17:29:46 +03:00
busya
5ab1549cdd linter + новые таски 2026-02-10 12:53:01 +03:00
busya
0743641181 semantic update 2026-02-08 22:53:54 +03:00
busya
c874946ed4 tested 2026-01-27 23:49:19 +03:00
busya
2288abfc5a semantic cleanup 2026-01-23 21:58:32 +03:00
busya
943dc06dfd refactor complete 2026-01-22 17:37:17 +03:00
busya
d8c0694538 semantic markup update 2026-01-18 21:29:54 +03:00
busya
651abc84cc feat(logging): implement configurable belief state logging 
- Add LoggingConfig model and logging field to GlobalSettings
- Implement belief_scope context manager for structured logging
- Add configure_logger for dynamic level and file rotation settings
- Add logging configuration UI to Settings page
- Update ConfigManager to apply logging settings on initialization and updates
 2025-12-27 05:39:33 +03:00
busya
82fd409b6a fixed css 2025-12-20 23:33:47 +03:00