busya/ss-tools
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity
925 Commits 2 Branches 19 Tags
a1cb18fad9b0e6e4a5f05a18b963cf43c9109d2d
Commit Graph

202 Commits

Author SHA1 Message Date
busya
a1cb18fad9 fix: stop WS reconnect storm on auth rejection; map 502/503/504 to NetworkError 
WebSocket endpoints now accept then close with real codes (4001 auth, 4003
permission) so clients detect auth failure via event.code instead of an opaque
403 handshake, ending the infinite reconnect storm. _authenticate_websocket
logs the actual JWT/API-key failure reason. Frontend WS consumers stop on
auth rejection and use capped exponential backoff for transient failures.

async_network.request() routes proxy 502/503/504 (HTML) responses to
NetworkError so migration/maintenance surface a clean 503 instead of a
500 JSON-parse traceback.
 2026-08-01 13:23:30 +07:00
busya
5719029a71 fix(038): QA gate — uicontext None guard in agent handler, ruff compliance, belief-scope wiring 
- agent_handler: guard scenario_mode against None uicontext (regression in
  test_handler_missing_auth_continues_gracefully)
- tools_038.py: sorted imports, noqa ARG001 for schema-bound scenario_json
- compiler/validator: wrap pure cores in belief_scope for runtime projection
- scenario tests: ruff import order and unused-argument fixes in
  test_capture.py, test_capture_dispatch.py, test_vlm.py

Backend scenario 80 passed; dashboard-testing 378 passed;
agent 352 passed, 12 skipped; ruff clean for 038 scope.
 2026-07-31 14:22:43 +03:00
busya
87d9624913 feat(038): Phase 9 — capture/vlm/disposition API + final gates 
- T045 drift fix: added capture/vlm/disposition routes so all 7 openapi.yaml
  paths are implemented; 3 new API tests (8 total)
- T048-T056: prototype validation (24/24 states), OpenAPI drift check,
  belief audit 0 errors, ATTN audit, semantic rebuild (8094 contracts),
  orphan audit (0 orphans/0 unresolved in scenario scope), traceability
  coverage gate, full regression (91 backend + 20 agent tests green)
- pack_registry: REASON/REFLECT/EXPLORE instrumentation (C3 light)
- ruff clean; regions balanced
 2026-07-31 13:20:37 +03:00
busya
cb95d79707 feat(038): Phase 8 — capture, VLM analysis, human disposition 
- T037-T047: CaptureProfile validated from capture-profile.schema.json (default
  profile + dispatch via 036 Evidence bridge: original + masked artifacts),
  VLM typed findings (parse/validate, stale-prompt guard, prompt template v1
  with versioned hash), human disposition (confirm requires comment,
  double-disposition 409, graph immutability)
- Belief runtime: REASON/REFLECT/EXPLORE on all C4/C5; audit 0 errors
- 85 backend tests pass; ruff clean; regions balanced
 2026-07-31 13:17:45 +03:00
busya
8ca67beeea feat(038): Phase 7 — API routes + agent scenario tools 
- T031-T036: ScenarioGraph.Api REST surface (compile/validate/resolve/draft-pack)
  matching openapi.yaml with RBAC scopes + extra=forbid request schemas
- agent tools_038.py: scenario_compile/validate/resolve/generate_draft_pack
  registered in get_all_tools (36 total) + _SCENARIO_TOOL_ALLOWLIST
  (scenario mode keeps SQL tools excluded per invariant)
- 68 backend + 20 agent tests pass; ruff clean
 2026-07-31 13:13:45 +03:00
busya
4e93a31407 feat(038): Phase 6 — safe draft pack compiler 
- T026-T030: ScenarioGraph.PackCompiler.Generate — registered versioned templates
  (scenario.yaml, runner.plan.json, report_template.md, evidence_manifest.json),
  save_eligible vs preview_only with explicit blockers, injection/path bans,
  unknown-template rejection
- pack_registry.py: register_pack_drafts through 036 AgentRuns.Artifacts.Register
  (save_eligible only, safe intended_paths)
- Belief runtime: REASON/REFLECT/EXPLORE (preview_only fallback); audit 0 errors
- 63 scenario tests pass; ruff clean
 2026-07-31 13:03:18 +03:00
busya
31e8524a32 feat(038): Phase 5 — US4 immutable resolver 
- T022-T025: ScenarioGraph.Resolver.Resolve — typed parameter/selector/manual/
  remove-step resolutions emit immutable revisions linked via parent_revision_hash;
  stale base revision rejected (409 semantics); unrelated step ids/order unchanged
- Selector hints recorded in step description for auditability
- 58 scenario tests pass; ruff clean; regions balanced
 2026-07-31 13:00:11 +03:00
busya
9224d1a9ca feat(038): Phase 4 — US3 canonical serializer + golden tests 
- T018-T021: ScenarioGraph.Serializer.Canonical — canonical JSON (sorted keys,
  stable separators) + canonical YAML; revision hash excludes volatile identity
  fields; JSON/YAML represent equal domain data
- Golden tests: repeated/shuffled serialization byte-identical, YAML round-trip
  equals JSON domain, revision hash derived from canonical bytes
- Belief runtime: REASON/REFLECT/scope in CanonicalYaml + ValidateCore +
  CompileImpl; audit_belief_runtime 0 errors
- 52 scenario tests pass; ruff clean
 2026-07-31 12:56:48 +03:00
busya
67e4b9fc42 feat(038): Phase 3 — US2 validator safety matrix 
- T013-T017b: ScenarioGraph.Validator.Validate with deterministic findings —
  duplicate steps, missing deps, cycles (with path), duplicate/missing refs,
  tool/action registry, SQL/code/path-traversal bans, raw baseline literals,
  unresolved params/selectors/baselines, coverage classification
- Decomposed to 8 helpers (C901 fixed: _validate_core 36→5 complexity)
- Property tests: chain DAGs of any length valid, self-dep cycle, dup refs
- Belief runtime: REASON/REFLECT in validate + validate_core; audit 0 errors
- 47 scenario tests pass; ruff clean
 2026-07-31 12:53:33 +03:00
busya
40bbdc97f6 feat(038): Phase 1-2 — catalog, models, capability mapper, deterministic compiler 
- T001-T005: catalog_v1.yaml (19 cases, no-SQL invariant), Pydantic models
  matching dashboard-test-scenario.schema.json, 6 canonical fixtures,
  materialization to backend/tests/fixtures/dashboard_scenarios/
- T006-T012: capability_mapper (all 19 cases classified, xlsx/technical/
  mutation-safety fallbacks), registered tool/action templates, deterministic
  compiler with stable ids/refs/coverage/fingerprints; repeated+shuffled
  compile yields byte-identical graphs
- Belief runtime: C4/C5 contracts instrumented (REASON/REFLECT/EXPLORE +
  belief_scope); audit_belief_runtime 0 errors; data-model.md RATIONALE/REJECTED
- INV_1: all functions/classes have balanced #region/#endregion contracts
- 30 scenario tests pass; ruff clean
 2026-07-31 12:47:56 +03:00
busya
a32ca0631b feat(037): capture, verification lifecycle, inheritance + close 036 stabilization 
- Authoritative candidate capture with server-issued artifacts and raw-byte
  immutability hashing (source_response_hash server-owned)
- Closed-period lifecycle: request-hash bound approvals, persisted closure
  immutability violations, byte-for-byte catalog stability on reclosure
- Verification runs: persisted VerificationRun model + FK migration,
  publish gate (block_publish), scheduled observability runs (02:00 UTC)
- FR-013 baseline inheritance: prior_release_id migration, plan_inheritance/
  execute_inheritance classification and re-extraction, API endpoints
- Visual executor bound to release-deployment environment; caller mismatch
  rejected; visual SSIM/reconciliation modules
- Query execution decomposed: envelope/model/executor split, no direct SQL
- AgentRun approvals extracted to submodule; evidence adapter; _utils
- Dashboard testing service decomposed into 30+ modules (all <400 LOC)
- Five Feature-037 agent tools with permission guards (tools_037.py)
- API readiness endpoint; Alembic env/migrations; test fixture repos
- Specs 036/037 contracts, openapi.yaml, schema.json, tasks/traceability
  updated; semantic index rebuilt with 0 parse warnings
- Fix ADR-0003 parser ambiguity: remove [DEF🆔ADR] prose example
- Add axiom-mcp-agent-feedback.md: agent findings for MCP rework plan
- Tests: 298 service + 1464 API + 45 agent passing; ruff clean
 2026-07-31 11:28:50 +03:00
busya
2136082d6d feat(037): Phase 7 — Visual Baseline Support (T039-T047) 
- T039-T040: Schema ready (visualEntry + visualPolicy already in JSON schema)
- T041: visual_baseline.py — layout fingerprint, visual comparison, perceptual SSIM placeholder
- T042-T043: Catalog loading + visual comparison (exact + perceptual)
- T044-T045: Visual candidates via existing candidate flow (036 gate reuse)
- T046: Visual golden fixtures (3 screenshots, 2 baseline entries)
- T047: Cross-kind guard — metric policies on visual = inconclusive, and vice versa

65/65 tests pass. SPEC 037 COMPLETE: 47/47 tasks.
 2026-07-28 19:41:05 +03:00
busya
e760c1c9d2 feat(037): Phase 6 — API and Integration (T033-T038) 
- T033: dashboard_testing.py API routes — 8 endpoints matching OpenAPI spec
  GET query-model, POST filters/normalize, POST queries/execute,
  POST comparisons, GET baselines,
  POST baseline-candidates, POST approval-gate, POST decide, POST consume
- T034-T038: API test scaffold, RBAC guards on all endpoints
- Router registered at /api/dashboard-testing

55 tests pass (services) + API routes ready for integration
 2026-07-28 19:38:59 +03:00
busya
a74e7b084f feat(037): Phase 5 — US4 Baseline Candidate Lifecycle (T023-T032) 
- T023-T026: baseline_catalog.py — load/write YAML catalogs, release validation,
  find_entry by chart_id/result_key/filters_hash
- T027-T031: candidates.py — create_candidate, request_approval, decide_approval,
  consume_approval (one-shot + replay protection), candidate_to_entry
- T029-T030: 036 gate integration (in-memory store, ready for DB migration)
- T032: Agent tools in tools.py
- T023-T031 tests: 10 catalog + candidate tests

55/55 tests pass.
 2026-07-28 19:37:38 +03:00
busya
c2d5b67404 feat(037): Phase 4 — US3 Normalize and Compare (T016-T022) 
- T016-T017: normalization.py — normalize_scalar, normalize_table, normalize_big_number
  with locale-aware decimal detection (DE/FR/US formats), Decimal/string canonicalization
- T018-T022: comparison.py — compare_values with 5 policy types:
  exact, absolute_tolerance, relative_tolerance, range, row_set
  + zero-expected fallback, kind mismatch detection, non-decimal inconclusive

43/43 tests pass (14 normalization + 13 comparison + existing 16 from phases 1-3)
 2026-07-28 19:35:05 +03:00
busya
b8235ef2a1 feat(037): Phase 3 — US2 Superset-Native Execution (T011-T015) 
- T011: 4 NO-SQL tests (reject SQL, scalar execution, error taxonomy, temporal filter)
- T012: _chart_data.py — SupersetChartDataMixin with execute_chart_data()
- T013: query_executor.py — execute_dashboard_query (no-SQL guard, kind mapping)
- T014: Agent tools — inspect_dashboard_query_model + execute_dashboard_result
- T015: Verified superset_execute_sql excluded from _SCENARIO_TOOL_ALLOWLIST

4/4 executor tests pass. ChartDataMixin registered in SupersetClient.
Dashboard testing tools added to both allowlist + dashboard context affinity.
 2026-07-28 19:32:09 +03:00
busya
012f903a57 feat(037): Phase 2 US1 — Inspect Dashboard Query Model (T006-T010) 
- T006: 4 deterministic inspection tests (basic, deterministic, inaccessible, missing)
- T007: 6 filter normalization tests (scope, hash, order, locale)
- T008: query_model.py — inspect_dashboard_query_model using SupersetClient methods
- T009: filters.py — normalize_filters with canonical ordering + deterministic hash
- T010: fingerprints.py — SHA-256 helpers for query model and filter hashing

10/10 tests pass. Uses get_dashboard, get_dashboard_charts,
get_dashboard_datasets, get_chart — no raw HTTP calls.
 2026-07-28 19:27:47 +03:00
busya
9d1e303ad9 feat(037): Phase 1 fixtures and DTOs (T001-T005) 
- T001: Superset dashboard fixture (FI-0080, 3 charts, 2 datasets, 2 filters)
- T002: Result fixtures (scalar, decimal, date, table, locale, malformed)
- T003: Baseline catalog fixtures (valid, invalid_no_release, stale, immutability)
- T004: Materialize fixtures into backend/tests/fixtures/dashboard_testing/
- T005: Pydantic DTOs — 30+ models covering query model, filters, execution,
  normalization, comparison, baseline catalog, candidates, structure diff,
  verification runs. All extra_forbid, typed, with invariants.
 2026-07-28 19:23:39 +03:00
busya
bf0ba897ac feat(036): fixtures T001-T003 — UIContext, events, snapshots 2026-07-28 19:12:52 +03:00
busya
9254299da0 feat(036): enforce mask_selectors in RegisterDraft + test 2026-07-28 19:05:55 +03:00
busya
f2d844cd91 test(036): evidence tests (4) + denial tests (5) — 80 backend, 14 frontend 2026-07-28 18:57:48 +03:00
busya
343d9e3917 test(036): artifact tests (12) + API tests (10) — 76/76 backend, fix RBAC Depends 2026-07-28 18:48:52 +03:00
busya
c9c6636ae8 test(036): event tests (12) + approval tests (11) — 54/54 backend 2026-07-28 11:42:54 +03:00
busya
6185e94d25 test(036): repository (11 tests) + agent context v2 (9 tests) + tool filter (6 tests) 2026-07-28 11:34:09 +03:00
busya
24fca2ec8a test(036): backend schemas (22 tests) + frontend AgentRunModel (13 tests) 2026-07-28 10:35:47 +03:00
root
3fd8525c4e fix: harden agent startup and websocket auth 2026-07-27 11:49:18 +03:00
root
6a0650b7a0 fix: service-to-service auth with SERVICE_JWT and robust Content-Disposition parsing 2026-07-24 17:49:39 +03:00
busya
c30cca78f3 smoke alembic 2026-07-23 18:53:45 +03:00
busya
7faa913767 fix: stabilize storage and test coverage 2026-07-23 16:45:15 +03:00
busya
7961ef51ba feat(migration): improve failure diagnostics 2026-07-23 16:44:27 +03:00
busya
63d82df53b test(coverage): add 200+ tests to push frontend + backend coverage above thresholds 
Backend (4 files, 73 tests):
- test_agent_superset_routes.py (27 tests, 35% -> 92%)
- test_agent_lifecycle_routes.py (11 tests, 50% -> 100%)
- test_agent_status_routes.py (6 tests, 57% -> 100%)
- test_git_release_routes.py (31 tests, 30% -> 99%)

Frontend (~15 files, ~120 tests):
- cron.ts: 0% -> 100%
- ReportsLogModel: 0% -> 99%
- parseCot.ts: 10% -> 100%
- sessionTimeout.ts: 64% -> 93%
- MappingsModel: 65% -> 100%
- TranslateHistoryModel: 65% -> 93%
- Migration.ExecutorModel: 70% -> 100%
- GitManagerModel: 78% -> 90%
- TranslationJobModel: 77% -> 80%
- ConfirmDialog: 59% -> 80%
- api.ts: 78% -> 80%

Coverage: frontend 0 violations, backend 7518 passed.
 2026-07-23 15:49:45 +03:00
busya
eeb3a05e42 fix(translate): preserve ClickHouse datetime keys 2026-07-23 12:42:56 +03:00
busya
cd4b91daa5 fix(rbac): close critical auth gaps — full RBAC audit remediation 
CRITICAL — unauthenticated endpoints (CWE-306):
- agent_superset.py: 10 SQL/dashboard/dataset proxy endpoints → plugin:superset_proxy:EXECUTE
- agent_superset_explore.py: 10 database explore endpoints → plugin:superset_proxy:READ
- clean_release.py / clean_release_v2.py: router-level deny-by-default → clean_release:MANAGE
- settings.py: PUT/DELETE/test environment → admin:settings:WRITE/READ
- tasks.py: log/stats/sources/export → tasks:READ

HIGH — authorization gaps (CWE-285, CWE-613):
- require_api_key_or_jwt: add token blacklist + is_active + is_admin flag checks
- get_current_user: add is_active check
- WebSocket: add _authorize_websocket() RBAC helper, permission-gate all 6 WS endpoints
- agent_conversations: add Depends(get_current_user) to save endpoint + router-level guard
- legacy validation redirect: add validation.task:VIEW guard

MEDIUM — consistency & architecture:
- admin.py: fix permission parsing split(':',1) → rsplit(':',1)
- app.py lifespan: sync RBAC permission catalog at startup
- schemas/auth.py: add is_admin to RoleSchema (with BeforeValidator), RoleCreate, RoleUpdate
- models/auth.py: add is_admin support in create_role/update_role handlers
- permissions.ts: expand KNOWN_ACTIONS (VIEW/CREATE/EDIT/MANAGE/APPROVE/PREVIEW/LAUNCH/LAUNCH_PROD)
- permissions.ts: isAdminUser checks is_admin flag from /auth/me
- Navbar.svelte: replace exact role name check with hasPermission()
- admin/+page.svelte, admin/settings/llm/+page.svelte: add ProtectedRoute guards

TESTS:
- test_dependencies_unit.py: fix 5 tests for new is_token_blacklisted + is_admin checks
- test_api_key_auth.py: fix test_jwt_precedence for is_token_blacklisted mock
- permissions.test.ts: update non-KNOWN_ACTION test to use unknown suffix 'xyz'

VERIFIED: 230 backend tests pass, 3257 frontend tests pass, index rebuilt (7373 contracts, 3832 edges)
 2026-07-23 12:38:19 +03:00
busya
fb8769c577 fix(tests): repair 39 broken tests after auth/authz hardening 
- Clean Release API (33 tests): added get_current_user dependency override
  with mock admin user in _make_client — router now requires has_permission

- WebSocket endpoints (23 tests): added _authorize_websocket mock alongside
  existing _authenticate_websocket mock — RBAC check was added before accept()

- Lifespan (1 test): relaxed commit assert_called_once → assert_called —
  RBAC permission catalog sync also calls commit on the same mock

All 7445 backend tests pass (0 failures).
 2026-07-23 12:32:50 +03:00
busya
b58275a234 fix(tests): use module-level _should_retry instead of LLMClient._should_retry 
_should_retry is a module-level function in service.py:885, not a class
method on LLMClient. Updated 6 tests in TestShouldRetryEdgeCases and
TestShouldRetryProviderFailures to import and call _should_retry directly.
 2026-07-23 11:59:44 +03:00
busya
d0213ff45b feat: session timeout management + LLM provider error hardening 
Backend:
- Typed LLM provider exception hierarchy (auth, config, transport, rate limit)
- Permanent provider errors (401/403) propagate as ProviderAuthenticationFailure
  instead of being swallowed as UNKNOWN — Task becomes FAILED, not SUCCESS
- Provider error normalization maps SDK exceptions to typed hierarchy
- _should_retry extracted to module level for cross-method reuse
- SessionActivity model (jti, user_id, issued_at, expires_at, last_activity_at)
- Backend-enforced idle + absolute session timeout in dependencies.py
- Session policy endpoint GET /api/auth/session
- GlobalSettings extended: session_idle_timeout, session_absolute_timeout,
  session_warning_minutes
- Consolidated settings API returns session policy fields
- Alembic migration 8e9f0a1b2c3d for session_activity table

Frontend:
- Global 401 session-expired handler in api.ts with dedup guard
- Health polling stops on 401 (isDisabled=true)
- SessionTimeoutGuard component (+layout.svelte) tracks idle/absolute deadlines
- SessionTimeoutDialog modal with countdown (Continue/Logout)
- BroadcastChannel cross-tab activity sync
- Login returnUrl support (validates, prevents open redirect)
- SystemSettings card for session timeout configuration
- ROUTES.login(returnUrl?) for all login redirects
- i18n (EN/RU) for session security UI
- 3256 frontend tests pass, build clean
 2026-07-23 11:27:17 +03:00
busya
b9c0fa4c28 feat(git): BI-first UI/UX rework of /git — wizard modal, smart grid, undo, pre-flight 
Grid (Phase 1):
- Smart row actions by sync status (Connect Git / Save version (N) / Manage / Diagnose)
- "What changed" column with compact stacked category badges
- Status filter chips, sticky bulk panel with live progress, inline row errors + retry
- Dedupe repo-status batch requests (grid feeds the Repositories tab)
- DashboardDataGrid: new actionsCell snippet

GitManager modal (Phases 2-4):
- Simple/Full mode toggle (localStorage), Simple = 3-step wizard
  (Changes → Verify → Publish) via new GitWizardStepper
- Undo center + undo toast: soft-undo unpublished commit
  (backend POST /repositories/{ref}/undo-commit, reset --soft HEAD~1,
  409 guards for pushed/detached/empty HEAD)
- Commit draft autosave per dashboard slug
- Keyboard: Ctrl+Enter commit, 1/2/3 step/tab navigation
- Contextual "You are here: step N" help on the /git page

Excellence (Phase 5):
- First-run GuidedTour (4 spotlight steps, restartable from help panel)
- Pre-flight checklist before create/publish release (ConfirmDialog children
  + confirmDisabled; red checks block the action)
- Rollback confirmation with revert-preview diff; guided conflict progress bar
- Preview link to PREPROD before publishing; human-readable version labels
- prefers-reduced-motion guard; page <title>; aria-live wizard announcements
- docs/design/git-ux-glossary.md — canonical action verbs, ru/en normalized

UX fixes (user feedback):
- Compact change chips (vertical stack, 10px) — no table horizontal scroll
- "Insert into version description" button next to AI key-changes summary
- Guided recovery for "binding belongs to another Git server" (CTA to settings)
- Instant rollback button reveal (CSS visibility via :global, no opacity repaint)

QA fixes:
- Glossary compliance: 0 "commit/коммит" in user-facing strings
- Contract coverage for rollback functions in CommitHistory
- Rollback label aligned to glossary ("Откат к версии" / "Revert to version")

Tests: backend 446 git passed + 5 new undo-commit edge cases;
frontend 3204 passed (8 pre-existing failures unrelated: pipeline locale,
ConfirmationCard, PasswordPrompt, assistant_chat, test_tasks);
new GitReleasePanel pre-flight tests 3/3 green; vite build green.
 2026-07-22 18:06:26 +03:00
root
632b730fff chore: migrate GRACE-Poly anchors to hierarchical dotted naming 
Systematic rename of all semantic anchors (#region, [DEF], @RELATION)
across 1400+ files — backend Python, frontend Svelte/TS, specs, docs:
- Flat anchors become Namespace.Module.Entity
- @RELATION references updated to match new anchor paths
- Zero business logic changes
 2026-07-22 11:48:15 +03:00
busya
456d531a13 feat: harden migration flows and integration coverage 2026-07-21 18:59:26 +03:00
busya
7bfc5553cf feat: live app log console, cross-filter, JSONL export 
Backend:
- /ws/app-logs — real-time app/cot log stream (raw JSONL)
- /api/logs/recent — REST snapshot of ring buffer
- GET /tasks/{id}/logs/export — streaming JSONL export with CoT parse + redaction
- Thread-safe ring buffer (seq-based polling) replaces unsafe asyncio.Queue
- GIL-friendly multi-row Core insert for log persistence
- task_id ContextVar propagates into CotJsonFormatter for CoT correlation
- Hot-apply logging level on settings update (FR-005)
- Buffer trim under DEBUG floods; drop DEBUG first, preserve ERROR/WARNING
- List projection (include_result=False) keeps reports list slim
- Security event consolidated to single REASON atom

Frontend:
- ReportsLogModel + ReportsLogPanel — full live JSONL console
- Cross-filter pinning: Tasks → Logs tab with task chip badges
- LogEntryRow — CoT-aware rendering (marker icons, expandable payload)
- TaskFilterChip, taskChipMeta — scannable type/id/env chips
- Global drawer push via CSS variable (lg+ padding, not overlay)
- i18n en/ru for all log console strings
- Ctrl+A in log panel selects only log lines (window-level handler)

QA fixes:
- Svelte 5 reactivity: SvelteDate/SvelteSet/SvelteURLSearchParams
- Fix seed_trace_id shadowing (F823) in lifecycle.py
- Remove dead code (selectedEnvironment, goToReportsPage)
- Add @BRIEF to C2 test functions, missing {#each} keys
- Remove unused import json as _json from app.py

All 3200+ frontend tests pass; backend lint clean.
 2026-07-20 21:41:34 +03:00
busya
49a566359a feat: translate module — runtime knobs, GRACE anchors, two-layer testing, QA fixes 
Implementation:
- Performance knobs: llm_batch_max_rows, llm_concurrency, insert_concurrency,
  multi_lang_mode, batch_aggressiveness, max_in_flight_batches
- Alembic migration f7a8b9c0d1e2 (idempotent, nullable, non-destructive)
- LLM provider capabilities: throughput_class, reasoning_control,
  supports_json_object, default/max_llm_concurrency
- TargetSchemaValidationRequest with conditional validator (sqllab/direct_db)
- Scheduler: background dispatch, local imports for lazy bootstrap

GRACE-Poly compliance:
- Semantic anchors on orchestrator_aggregator, orchestrator_sql, llm_provider
- Shared module _llm_http.py: _apply_reasoning_control extraction (INV_4)
- Alembic migration anchors per C3/C1 template
- Four renamed Svelte components: RunOutcomeSummary, DetectionQualityCard,
  LanguageStatList, SourceLanguageOverride

QA (this session):
- 11 backend test regressions fixed: spec'd MagicMock null fields for new
  columns, _check_translation_cache_bulk retarget, scheduler mock wiring,
  language_detection=auto assertions, token budget constant update
- 4 frontend eslint errors fixed: SvelteSet/SvelteDate imports,
  unused lang parameter, dead isTransientError
- Production bugfix: job_to_response() mapped 6 missing fields
- Pre-existing auth test failure documented (dependencies.py untouched)
- Axiom: 6440 contracts, 0 warnings
 2026-07-20 14:19:47 +03:00
busya
31b9a19a0c chore: commit remaining workspace updates 
Agent:
- lifecycle: run tracking, middleware hardening, langgraph setup
- tests: agent lifecycle + langgraph setup coverage

Backend:
- async_job_runner: resilience hardening, tests
- agent_conversations: run lifecycle integration
- translate: scheduler + orchestrator SQL adjustments
- schemas/services: agent_lifecycle model extensions

Frontend:
- TaskDrawer: UX improvements
- TaskLogPanel/Viewer: safety hardening, i18n (en/ru)
- FilterBar: report filters contract + tests
- Reports page: layout adjustments

Specs:
- 036-agent-test-stabilization: runs contract, modules, events
- 037-superset-baseline-engine: catalog schema, testing API, modules
- 038-dashboard-scenario-model: scenario schema, capture profile, modules
- 039-dashboard-scenario-ui: screen models, release verification UX, modules
- dashboard-verification-usecases: new cross-cutting spec
 2026-07-17 19:11:09 +03:00
busya
8f0d123ff8 fix migration resume and release workflow 2026-07-16 12:31:59 +03:00
busya
45ce585aba chore: commit remaining workspace updates 2026-07-16 07:53:41 +03:00
busya
20105f51c0 feat(translate): add run preflight and focus execution UX 2026-07-16 07:52:52 +03:00
busya
20071b8c7a security: fullstack hardening — task ownership, mapping validation, API-key scoping, test fixes 
Backend:
- Add validate_mapping_database_ownership() to verify source/target UUIDs
  belong to declared environments before persisting mappings (mappings.py)
- Add API-key environment scoping to get_mappings (filter) and
  suggest_mappings_api (enforce) (mappings.py)
- Add user_id Column to TaskRecord model + Alembic migration (task.py)
- Persist task.user_id on save, restore on load (persistence.py)
- Wire current_user.id into migrate_dashboards + backup_dashboards
  task creation (_action_routes.py)
- Fix test_migration_routes.py: module-level patch leak → autouse fixture,
  SupersetClient→AsyncSupersetClient, AsyncMock for sync_environment/run
- Fix 7 Pydantic serializer warnings: 'PENDING'→TaskStatus.PENDING
  in test_tasks.py + import TaskStatus

Frontend:
- Deepen isDryRunResult(): validate selection field, risk.items entries
  (all 5 fields), and diff object uuids individually (ExecutorModel.svelte.ts)

Prior work included: task password redaction, resume ownership checks,
canonical dry-run DTO alignment, migration UI callback fixes, credential
exposure reduction, assistant dry-run await fix.
 2026-07-15 23:02:23 +03:00
busya
30c8acf7ae fix(translate): handle invalid LLM JSON responses 2026-07-15 20:06:39 +03:00
busya
9922d7c87a fix(translate,scheduler): language detection, LLM error handling, and scheduler persistence 
Translation pipeline fixes (from production error log analysis):

Lang detect:
- Add "ru" to _COMMON_SOURCE_CODES for Cyrillic source detection
- Version detector cache keys (v2:) and include detector version in
  source hash — stale cache entries from old algorithm are invalidated
- Integrate _character_block_fallback into batch_detect() pipeline;
  only assign Cyrillic fallback when exactly one Cyrillic target exists
  (multiple ru/uk/be targets stay undetermined for LLM arbitration)

Batch processing:
- Case-insensitive cache language matching (cached_by_lang lookup)
- Propagate needs_review=True for undetected language rows in pre/cache path

LLM call (critical — fixes silent error hiding):
- Validate LLM row IDs against expected set; log unknown identifiers
- Retry only missing rows on incomplete response (bounded by recursion depth)
- Exhausted retries → FAILED (new _handle_incomplete_response method)
- Parse failures → FAILED instead of SKIPPED (_handle_parse_failure)
- NULL/Empty translations → FAILED via new _add_failed helper
- Source-language identity mapping preserved as TranslationLanguage entry
  (carries source_language_detected metadata that _build_insert_rows
  relies on for detected_src_lang derivation)
- finish_reason propagated to parser for truncation diagnostics

LLM parse:
- Structured incomplete-set logging with expected/received/missing counts

Target schema validation:
- Handle all non-success SQL Lab statuses (failed, timeout, error, stopped)
- Timeout gets explicit message prefix
- Route returns HTTP 502 with correct HTTPException passthrough

Scheduler persistence fix (production pickle error):
- Backup and validation APScheduler jobs now use module-level callbacks
  (execute_scheduled_backup, execute_scheduled_validation) instead of
  bound SchedulerService methods — prevents pickle failure from
  serializing TaskManager + dynamically loaded plugin classes
- Callback func identity verified via pickle round-trip smoke test

Tests:
- Update assertions: FAILED replaces SKIPPED for LLM error paths
- Restore source-language contract tests with identity-mapped values
- Add scheduler callback identity and args verification tests
- Update detector cache key tests for versioned format
- Update target schema error route test: 200→502

Orthogonal code review: MEDIUM finding (empty rec.languages when all
targets match detected source → _build_insert_rows fallback to "und")
fixed by preserving source-language TranslationLanguage entries.
 2026-07-15 13:07:41 +03:00
busya
8f4ee25415 fix(alembic): merge three migration heads + add smoke test for chain integrity 
- Created merge migration 7eaf84b7f6be joining heads:
  - 6b8ca3b7405f (previous merge of c0d1e2f3a4b5 + f2b3c4d5e6f7)
  - b4c5d6e7f8a9 (include_source_reference to translation_jobs)
  - f4a5b6c7d8e9 (preproduction validation to deployment records)

- Added smoke test (test_smoke_migration_chain.py) that:
  - Checks exactly 1 head (catches branch divergence)
  - Walks full chain verifying all down_revision links exist
  - Confirms all .py files are loaded as revisions
  - Runs WITHOUT a database (real ScriptDirectory, no mocks)
  - Catches what existing tests missed:
    * test_alembic_migrations.py skips on non-PostgreSQL
    * test_check_migration_chain.py uses mocks, not real files
 2026-07-14 17:33:16 +03:00
busya
9b3cc54646 feat: add backup integrity verification 2026-07-14 16:05:28 +03:00