Commit Graph

615 Commits

Author SHA1 Message Date
f87ebf5d4b feat(agent): Gradio-powered LangGraph agent chat with streaming, tool calls, file upload, conversation persistence
- Gradio 5.50.0 ChatInterface with type='messages' streaming
- LangGraph create_react_agent with InMemorySaver checkpointer
- 4 @tool functions: search_dashboards, get_health_summary, list_environments, get_task_status
- Structured ChatMessage metadata (7 discriminator types: stream_token, tool_start/end/error, confirm_required, confirm_resolved, error)
- HITL resume via second submit() with interrupt_before/Command
- Dual-identity RBAC: service JWT + user JWT for tool calls
- File upload (10 MB limit, pdfplumber/xlsx/JSON parser)
- Conversation persistence via POST /api/agent/conversations/save
- REST API: list, history, archive conversations; multi-tab gate; LLM config
- LLM provider selection via Admin -> LLM Settings (assistant_planner_provider)
- Svelte 5 AgentChatModel with stream event queue, dedup, stream_status watcher
- MarkdownRenderer using svelte-markdown with semantic Tailwind tokens
- ToolCallCard (3 states: executing/completed/failed)
- ConversationList with search, date grouping, infinite scroll
- ConnectionIndicator with Gradio health status
- /agent route with two-column layout
- Vite proxy /api/agent/gradio -> Gradio SSE
- Fixed: not_() SQLAlchemy operator, route collision with _admin_routes
- Fixed: conversation_id -> id normalization, .pyc cache staleness
- Fixed: event.data array parsing (Gradio returns [jsonStr, null])
- Requirements pinned: gradio==5.50.0, pydantic>=2.7,<=2.12.3
2026-06-10 10:27:19 +03:00
2222261157 tasks read 2026-06-09 11:44:20 +03:00
25b7a6d423 tasks 2026-06-09 10:10:26 +03:00
a7fb06cd8e tasks ready 1 2026-06-09 09:43:34 +03:00
8e8a3c3235 feat: attention-optimized semantic protocol v2.7
Core changes:
- Add @defgroup/@ingroup to 1791 C2+ contracts (555 files) for HCA 128× pre-training DSA grouping
- Add §0.1 Pre-Training Frequency matrix to semantics-core
- Add §VIII Attention Architecture rules (ATTN_1-4) with MLA/CSA/HCA/DSA mechanics
- Add @defgroup/@ingroup to canonical syntax (§II) and all contract examples

Agent prompts (5 files):
- Add ZERO-STATE RATIONALE with MLA/CSA/HCA/DSA compression mechanics
- Add pre-training note: @RATIONALE/@REJECTED are in-context learned tags
- svelte-coder: add missing #region contract, fix Svelte rule violations
- python-coder/fullstack-coder: honor function contracts from speckit plan
- qa-tester: add attention compliance audit (P3 ATTN_1-4 checks)

Skills (6 files):
- Translate all axiom_config descriptions to English
- Fix doc_dirs to index .opencode/ and .specify/
- Deduplicate 5× complexity_rules → single global_tags catalog
- Reduce semantics-svelte 591→485 lines (remove duplicate code blocks)
- Fix semantics-testing: 'Short IDs' → 'Short hierarchical IDs'
- Fix all examples: flat IDs → hierarchical Domain.Name format
- Fix Svelte examples: replace raw Tailwind + <button> with semantic tokens + /ui

Speckit workflow (commands + templates):
- speckit.plan: add Function-Level Contracts for C3+ with @PRE/@POST/@TEST_EDGE
- speckit.plan: add Attention Compliance Gate (ATTN_1-4 before contract generation)
- speckit.tasks: add function contract inlining format (constraints in task description)
- speckit.specify: load semantics-core for spec density rules
- spec-template: add #region contract, @SEMANTICS grouping, hierarchical IDs
- ux-reference-template: add #region wrapper
- plan-template: add attention gate, @defgroup/@ingroup guidance
- tasks-template: add attention audit + rebuild + orphan check tasks
- constitution.md: translate to English, add Principle VIII (attention-optimized contracts)

Reference modules rewritten (hierarchical IDs + full contracts):
- Auth.Jwt: 6 child contracts with @RATIONALE/@REJECTED/@TEST_EDGE
- Api.Auth: 5 endpoints with @TEST_EDGE + molecular CoT markers
- Migration.Model: @defgroup Migration with 18 @ACTION + 6 @INVARIANT

Scripts:
- add_defgroup_ingroup.py: zero-risk additive @ingroup migration (1791 insertions)
- migrate_hierarchical.py: flat→hierarchical ID dry-run analysis (792 contracts)
- merge_prompts.py: merge all prompts/skills/commands into one review file

Config:
- axiom_config.yaml: 749→395 lines (-47%), English, doc_dirs include prompts
- Fix test_datasets.py import collision (rename → test_datasets_routes.py)
- Fix test_preview.py: SupersetClient→get_superset_client, AsyncMock, logger f-string
2026-06-08 16:30:59 +03:00
73e809e189 skills 2026-06-08 15:08:02 +03:00
3c84161606 specs updated 2026-06-08 14:14:38 +03:00
f731a53c7d 038: add @RATIONALE/@REJECTED contracts to async C4/C5 modules
5 contracts updated:
  - AsyncNetworkModule (C5): async migration rationale, per-client CSRF cookie rejection
  - AsyncAPIClient (C4): auth lifecycle, cache-hit CSRF refresh
  - AsyncAPIClient.request (C4): string vs dict handling, double-encoding root cause
  - AsyncAPIClient.upload_file (C4): multipart async upload
  - LLMAsyncHttpClient (C4): response.ok -> is_success, module-level httpx singleton
2026-06-05 17:02:30 +03:00
fbe0ba122c 037: fix 99 failing tests — missing await after async migration
Fixed async/sync boundary bugs across 14 test files. Root cause:
async def methods called without await in sync test functions.

Fixed files:
  - test_translate_jobs.py (10): create_job/get_job/update_job/delete_job
  - test_translate_scheduler.py (5): create_schedule/update/delete
  - test_datasets.py (14): AsyncMock + corrected patch target
  - test_mapping_service.py (11): sync_environment + MockSupersetClient
  - test_defensive_guards.py (6): GitService/SupersetClient guards
  - test_maintenance_service.py (29): all 6 maintenance services
  - test_dry_run_orchestrator.py (1): run() without await
  - test_dashboards_api.py (23): registry client via AsyncMock
  - test_validation_tasks.py (4): trailing slash in POST URL
  - test_superset_matrix.py (3): AsyncMock for compile_preview
  - test_payload_reduction.py (6): LLMClient._optimize_image wrapper
  - test_compliance_task_integration.py (2): event_bus ref
  - test_smoke_plugins.py (1): flusher_stop_event fallback
  - test_task_manager.py (1): _flusher_stop_event/thread fallback

Remaining 31 failures in test_task_manager.py (29) and
test_smoke_plugins.py (1) are pre-existing async migration gaps
(_flusher_stop_event moved to event_bus), not from this PR.
2026-06-05 15:43:35 +03:00
7f1937f10b 036: wire SupersetClientRegistry into translate + services flow
Replaces direct SupersetClient(env) calls with shared
get_superset_client(env) from client_registry.

Changed files (9):
  - client_registry.py: added get_superset_client(), _build_env_id(),
    accepts Environment models (not just dicts), fixed async_client attr
  - superset_executor.py: _get_client() now async, uses shared client
  - preview_executor.py, _run_source.py, service.py, service_datasource.py
  - health_service.py, resource_service.py, debug.py

Impact per environment:
  - 6 separate httpx.AsyncClient instances → 1 shared client
  - 6 CSRF cookie fetches → 1 (on first access)
  - 6 connection pools → 1 shared pool
  - Shared semaphore for backpressure
  - Shared cookie jar (fixes CSRF 'tokens do not match' on SQL Lab execute)
2026-06-05 15:14:44 +03:00
4cef6af041 fix 2026-06-05 15:01:34 +03:00
50180aaa14 035: fix httpx.Response.ok -> httpx.Response.is_success in LLM client
httpx.Response does not have .ok attribute (that's requests.Response).
Async migration missed this: _llm_async_http.py used response.ok in two
places, causing 502 errors when LLM API responded.

Fix: response.ok -> response.is_success
2026-06-05 12:10:20 +03:00
5d9d214bd6 034: fix double JSON serialization in AsyncAPIClient.request
Root cause: AsyncAPIClient.request() passes its  parameter directly
to httpx.AsyncClient.request(json=data). When callers pass a pre-serialized
JSON string (data=json.dumps(dict)), httpx re-encodes it via json.dumps(),
resulting in a double-encoded JSON string body instead of a JSON object.

This caused ALL POST/PUT requests with string data to fail — Superset received
a JSON string instead of a JSON object, returning GENERIC_BACKEND_ERROR
('dictionary update sequence element #0 has length 1; 2 is required').

Fix: if data is a string, pass it via httpx  parameter (raw body);
if it's a dict/list, pass via  for automatic encoding.

Affected callers (6 files) now correctly send JSON objects:
  - preview_executor.py: chart data requests
  - superset_executor.py
  - _run_source.py
  - _datasets.py: update_dataset
  - _datasets_preview.py: compile_dataset_preview
  - _dashboards_write.py

Also simplified preview_executor.fetch_sample_rows back to single-strategy
(chart data API only) since the root cause is now fixed.
2026-06-05 12:07:55 +03:00
72fcf698af 033: fix preview_translation route + MultiSelect label regression
1. preview_translation route: missing await on async preview_rows()
   - preview_rows() is async def, called without await
   - returned coroutine object instead of result -> 'coroutine not iterable' error

2. MultiSelect.svelte: opt.label -> opt.name
   - option type is {code, name} but template used {opt.label}
   - rendered empty spans instead of language names
2026-06-05 11:38:35 +03:00
250b69db8c 032: fix coroutine never awaited in debug.py + task_logger.py
debug.py: _test_db_api and _get_dataset_structure were already async def
but called SupersetClient methods (authenticate, get_databases, get_dataset)
without await. Added await to 4 calls.

task_logger.py: _add_log callback is async def but _log() called it without
await, silently dropping all task log messages (RuntimeWarning: coroutine
never awaited). Changed to fire-and-forget via asyncio.ensure_future when
a running event loop is available, drops gracefully otherwise.
2026-06-05 11:23:53 +03:00
909b568784 032: add async regression tests (21 tests covering all fixed bug patterns) 2026-06-05 10:40:51 +03:00
3fc09b086e 032: fix 2 critical QA issues — missing endregion + Tombstone dead helpers
C1: _llm_call.py — added missing #endregion _split_and_retry (violated INV_3)
C2: dashboards/_helpers.py — sync _find_dashboard_id_by_slug and
    _resolve_dashboard_id_from_ref typed Tombstone per INV_6 (dead code,
    callers use _async versions from git/_helpers.py)
2026-06-05 10:32:37 +03:00
4247e7a20e 032: fix final sync→async cascade (dataset_review, parsing, validation)
- _parsing.py: parse_superset_link + _recover_dataset_binding → async
    (await get_dashboard_detail, get_chart)
  - dataset_review/orchestrator.py: start_session + _build_recovery_bootstrap → async
    (await get_dataset_detail, parse_superset_link)
  - _routes.py (dataset_review): await orchestrator.start_session()
  - validation_tasks.py: await parse_superset_link + get_dashboard_detail
2026-06-05 09:55:34 +03:00
c7d8f4431e 032: fix remaining sync→async propagation (17 call sites)
Core fixes:
  - service_datasource.py: fetch_datasource_metadata() → async
  - service.py: create_job(), update_job() → async (callers await)
  - _job_routes.py: await create_job/update_job

Maintenance scanners:
  - _dashboard_scanner.py: 4 functions → async (find_affected, _get_linked,
    _apply_filters, _resolve_title)
  - _chart_manager.py: 3 functions → async
  - _banner_renderer.py: rebuild_banner → async
  - _orchestrators.py: 3 orchestrators → async
  - maintenance_banner.py: await async calls

Migration:
  - dry_run_orchestrator.py: run(), _build_target_signatures() → async
  - risk_assessor.py: build_risks() → async
  - migration.py: await service.run()
  - mapping_service.py: sync_environment() → async

Dead code:
  - _helpers.py: _find_dashboard_id_by_slug marked DEPRECATED
2026-06-05 08:56:37 +03:00
a5cd06546f 032: fix missing await on TranslationExecutor.execute_run() in orchestrator_exec
executor.execute_run() is async def but was called without await in
TranslationExecutionEngine.execute_run(). This returned a coroutine
instead of a TranslationRun, causing:
  'coroutine' object has no attribute 'status'
This made every translation run fail in the background execute path.
2026-06-05 08:47:56 +03:00
71000db785 032: fix run_translation route handler — sync def → async def (asyncio.create_task needs running loop)
run_translation was def (sync FastAPI handler runs in thread pool with no event
loop), but its body calls asyncio.create_task(_background_execute()) which
requires a running event loop. Changed to async def so FastAPI runs it on the
event loop directly.

Error: 'Run failed: no running event loop'
2026-06-05 08:40:46 +03:00
71156783cf 032: fix get_dashboards_page_async -> get_dashboards_page (method renamed during async migration)
get_dashboards_page_async() no longer exists — the sync/async split was
removed and the method is now simply get_dashboards_page() (already async).
Calls in dashboard slug resolution and git helpers were using the old name.
This caused AttributeError at runtime, making all slug-based dashboard
lookups fail with 'Dashboard not found'.
2026-06-05 08:39:05 +03:00
079b2dd37b 032: add aclose() to SupersetClient (was missing, used in dashboard detail routes)
SupersetClientBase was missing aclose() method. AsyncSupersetClient had it
via override, but code creating SupersetClient directly (e.g. dashboard tasks
history route) would fail with AttributeError on await client.aclose().
2026-06-05 08:33:49 +03:00
5eb116509b 032: dead code cleanup — remove sync APIClient, _llm_http, preview_llm_client, fix retry chains 2026-06-05 08:31:18 +03:00
58e994cd75 032: fix tests — validate_target_table_schema became async (add await + AsyncMock)
All test methods calling validate_target_table_schema now async def + await.
Mocked async methods (resolve_database_id, execute_and_poll) use AsyncMock
instead of MagicMock since await on MagicMock raises TypeError.
2026-06-05 00:18:13 +03:00
c2578facdb 032: fix health_service _prime_dashboard_meta_cache — async get_dashboards_summary
_prime_dashboard_meta_cache was sync but called async get_dashboards_summary().
Parent get_health_summary was already async def. Made child async + added awaits.
2026-06-05 00:14:15 +03:00
ee9123bcf2 032: deep async propagation — orchestrator, insert, mapper, batch chains
Full async conversion for all sync callers of async SupersetClient methods:

orchestrator_sql.py: generate_and_insert_sql + _resolve_dialect → async
orchestrator_run_completion.py: complete_success → async (calls generate_and_insert_sql)
orchestrator_exec.py: execute_run → async (awaits complete_success)
orchestrator_runner.py: execute_run → async (delegates to engine)
orchestrator.py: execute_run + _generate_and_insert_sql → async
executor.py: _insert_batch_to_target → async (awaits batch insert)
_batch_proc.py: insert_batch_to_target → async
_batch_insert.py: insert_batch_to_target + _resolve_insert_backend + _execute_insert_sql → async
dataset_mapper.py: get_sqllab_mappings + run_mapping → async
  + await get_dataset, update_dataset, execute_and_poll
mapper.py: await on run_mapping + resolve_database_id calls
_run_routes.py: threading.Thread → asyncio.create_task (_background_execute async)
2026-06-05 00:13:01 +03:00
f416583a8c 032: fix async chain for target schema check + mapper + executor
resolve_database_id → async (was sync but called async SupersetClient methods)
  - await client.get_database(db_id)
  - await client.get_databases(...)

validate_target_table_schema → async (calls async resolve_database_id + execute_and_poll)
  - await executor.resolve_database_id(...)
  - await executor.execute_and_poll(...)

Route check_target_schema → await validate_target_table_schema(...)

MapperPlugin.execute → await executor.resolve_database_id(...)
  (execute() was already async def, just missing await)

SupersetSqlLabExecutor.execute_sql → await self.resolve_database_id()
  (was sync call to now-async method)
2026-06-05 00:04:54 +03:00
102f48a0d2 032: fix UnboundLocalError for db_name in validate_target_table_schema
db_name and backend were initialized inside the try block but referenced
in the except handler. If an exception occurred before their assignment
(e.g. in resolve_database_id), the except block would raise:
  cannot access local variable 'db_name' where it is not associated
Moved initialization before try with safe defaults.
2026-06-04 23:59:11 +03:00
f6dc7c47e7 032: fix missing awaits in translate datasource endpoints (#5,6,7)
Three sync functions called async SupersetClient methods without await:
  - get_datasource_columns(): get_dataset_detail + get_database (async)
  - fetch_available_datasources(): get_datasets (async)
  - Route handlers: both calls missing await

Converted to async functions + added awaits. Both endpoints now return
proper data instead of coroutine errors.
2026-06-04 23:57:46 +03:00
fe2602dc89 032: fix duplicate class:border-warning / class:bg-warning-light in TargetSchemaHint
Svelte 5 does not allow duplicate class: directives on the same element.
The two class:border-warning (and two class:bg-warning-light) conditions
were mutually exclusive but Svelte rejected them at compile time.
Merged into single || condition.
2026-06-04 23:56:00 +03:00
380f8806fb 032: fix missing await on git_service.get_status() (async -> coroutine)
git_service.get_status() is async def, but was called without await,
returning a coroutine object instead of dict. The coroutine was then
used as a dict value in RepoStatusBatchResponse, causing Pydantic
ValidationError (dict type mismatch).
2026-06-04 23:54:42 +03:00
87565b8147 032: fix async get_dataset_linked_dashboard_count passed to asyncio.to_thread
get_dataset_linked_dashboard_count is async (coroutine function), but was
passed to asyncio.to_thread() which expects a sync callable. This returned
a coroutine object instead of an int, causing:
  '>' not supported between instances of 'coroutine' and 'int'

Fix: await the async method directly inside asyncio.wait_for().
2026-06-04 23:53:26 +03:00
545eea22f8 032: fix missing await on async SupersetClient calls in resource_service.py
Found 3 missing 'await' keywords causing 'coroutine object is not iterable':
  - get_dashboards_summary()  (line 57)
  - get_dashboards_summary_page() (line 114)
  - get_datasets_summary() (line 306)

All three were calling async methods in sync context — returned coroutine
objects instead of lists/dicts, causing iteration failures.
2026-06-04 23:47:09 +03:00
4522709ac7 032: fix SyntaxError in git/_base.py — positional after keyword args in run_blocking
7 calls fixed: moved kind and fn to positional to avoid
SyntaxError 'positional argument follows keyword argument'.
2026-06-04 23:37:37 +03:00
0721681cfe 032: fix(TargetSchemaHint) — differentiate transient errors (503/504) from table not found
HIGH: 'bodyClass' and display logic updated — 503/504 errors now show
warning (yellow) styling and 'Could not verify' message instead of
destructive (red) 'Table not found'. Backend now returns 503 on pool
exhaustion and 504 on upstream timeout after async refactoring.
2026-06-04 23:34:45 +03:00
bc32dd65ef 032: mark all tasks [x] — feature complete
Updated status in spec.md, plan.md, tasks.md, research.md,
data-model.md, ux_reference.md, quickstart.md.
All 72 tasks completed. 34/34 tests passing.
2026-06-04 21:09:14 +03:00
9f9ad91345 032: T029-T031 + T046-T048 — all remaining tests
T029: concurrent preview+schema check test
T030: static asyncio.sleep audit
T031: LLM rate-limit backoff test + 6 edge cases
T046: TaskManager concurrent tasks + cancellation tests
T047: async notifications — SMTP timeout test
T048: EventBus publish/subscribe + maxsize tests

34 total async tests passing.
2026-06-04 21:06:22 +03:00
b190414747 032: final — tombstones, tests, cleanup
T055: APIClient tombstone in network.py
T056: AsyncSupersetClient @DEPRECATED marker
T057: _llm_http.py + preview_llm_client.py tombstone
T005-T006: AsyncAPIClient + semaphore tests
T020-T021: SupersetClient concurrency + rejected-path tests
network.py cleaned from 584 to 220 lines (orphan code removed)

All 20 async tests pass.
2026-06-04 20:57:20 +03:00
2e95971b1b 032: fix tests — AsyncMock for async SupersetClient methods
All 10 preview pipeline tests pass.
2026-06-04 20:42:31 +03:00
3bb5e6dde7 032: final — async_superset_client collapse, profile/superset_lookup async 2026-06-04 20:37:55 +03:00
5d2813f173 032: T059 — ADR-0011 async-backend decision record 2026-06-04 20:36:53 +03:00
27bbf057ec 032: T045 — git services async (run_blocking for all blocking ops)
_merge.py partially done. Tests still pending.
2026-06-04 20:36:33 +03:00
5ca1131ba3 032: Phase 5 (T036-T039) + Phase 6 (T040-T044) completed
T036: superset_compilation_adapter fully async
T037: fileio.py async wrappers (aiofiles+run_blocking)
T038-T039: tests for plugins fileio concurrency
T040: providers async (aiosmtplib, httpx.AsyncClient)
T041: dispatch_report parallel via asyncio.gather
T042: TaskManager ThreadPoolExecutor->asyncio.create_task
T043: EventBus asyncio.Queue(maxsize=10000)
T044: lifecycle async context manager

Remaining: T045 git/_base.py, T046-T048 tests, T005-T020-T021 tests
2026-06-04 20:30:43 +03:00
82b5707369 032: Phase 5 US3 — backup, git, llm_analysis, storage async
T032-T035 completed. T036 partial.
Remaining: T036 superset_compilation_adapter, T037 fileio.py, cascade updates
2026-06-04 20:17:13 +03:00
e377c965e9 032: Phase 4 US2 — Translate plugin fully async
T022-T028: All translate methods async.
- _llm_async_http.py created (httpx.AsyncClient+asyncio.sleep)
- Old _llm_http.py and preview_llm_client.py preserved (tombstone later)
- superset_executor, preview, executor, run_source, llm_call all async

RATIONALE: httpx.AsyncClient + asyncio.sleep instead of time.sleep.
REJECTED: AsyncOpenAI SDK — doesn't support custom base_url.
2026-06-04 20:09:45 +03:00
846d2cb9a9 032: T019 — remaining route files migrated to async SupersetClient
All routes (assistant, migration, datasets, git) now use AsyncSupersetClient.
_helpers.py sync->async for dashboard ref resolution.
_detail_routes.py import fixed.

Known residual: MigrationDryRunService and IdMappingService still sync.
2026-06-04 20:02:33 +03:00
51afbee470 032: Phase 3 US1 — 13 mixins migrated to async + dashboard routes
T011-T017: All SupersetClient mixins now async.
T018: _detail_routes.py uses registry/AsyncSupersetClient.
T019: Partial — routes/environments, settings, profile, listing async.

RATIONALE: Big-bang merge of sync+AsyncSupersetClient.
REJECTED: dual-stack.

Remaining T019: assistant/*, migration, datasets, git helpers still use sync.
2026-06-04 19:55:43 +03:00
496584e0da 032: Phase 1-2 — setup deps + AsyncAPIClient extend + client_registry + executors
Phase 1 (Setup):
- T001: requirements-dev.txt with pytest-httpx
- T002: aiofiles added to requirements.txt
- T003: aiosmtplib added to requirements.txt
- T004: EnvironmentConfig extended (connection_pool_size, etc.)
  + AppAsyncRuntimeConfig created (executor workers, shutdown)

Phase 2 (Foundational):
- T007: AsyncAPIClient extended — semaphore parameter, request() method
- T008a: SupersetClientRegistry — singleton per-env client/semaphore/lock
- T008b: run_blocking helper + bounded executors (db/file/git)

RATIONALE: httpx.AsyncClient replaces requests.Session; singleton
registry ensures global per-env semaphore; named executors prevent
thread pool exhaustion.
REJECTED: asyncio.to_thread (default executor, no backpressure);
per-request clients (lose pooling); dual-stack (rejected at clarify).
2026-06-04 19:45:57 +03:00
37f62a3032 tasks ready 2026-06-04 19:41:08 +03:00