Discovery of virtual datasets now works, but a runtime blocker remained: any
virtual dataset whose SQL exceeds sqlparse's MAX_GROUPING_TOKENS (10000 tokens)
raised SQLParseError 'Maximum number of tokens exceeded (10000)' from
extract_tables_from_sql_span, which is called unguarded in the scan loop — one
oversized virtual dataset aborted the whole maintenance preview/start.
- extract_tables_from_sql_span now wraps sqlparse.parse + token walk in
try/except and falls back to regex-only extraction (keeping all schema.table
matches) instead of raising, so huge SQL no longer fails the scan.
- Tier-1 virtual filter uses value "" (not None) so the sql is_not_null filter
passes Superset's rison schema instead of always falling back to a full scan.
Tests: huge-SQL fallback (extractor) and huge-virtual-dataset scan resilience
(scanner). ADR-0020 updated with Decision 3.
Virtual (SQL) datasets were never matched, so maintenance discovery returned
0 affected dashboards. Two defects fixed:
- find_affected_dashboards filtered by is_sqllab_view, which is NOT a
filterable column in Superset's dataset list API (absent from search_columns),
so the query was rejected. Now discover virtual datasets via the filterable
sql column: primary server-side 'sql is_not_null' filter with a client-side
non-empty-sql scan as fallback (best-effort vs pagination cap), dedupe by id.
- AsyncAPIClient.request never called raise_for_status(), so rejected filters
(HTTP 400) were returned as bodies without a 'result' key and surfaced as
'Found 0 datasets', dead-coding the filtered->full-scan fallback. request()
now raises on non-2xx via the existing error mapper.
Tests cover both virtual-scan tiers, all fallback paths, the raise behavior,
and an end-to-end match with the real sql_table_extractor on production SQL.
Documented in ADR-0020.
Enhance the reliability and observability of the task execution engine
by introducing retry mechanisms, idempotency, and structured progress
tracking.
- Implement centralized retry logic with exponential backoff support
in `JobLifecycle`.
- Add `retry_task` API endpoint and `TaskManager` method for manual
task restarts.
- Introduce task idempotency using `_idempotency_key` to prevent
duplicate executions.
- Add `retry_count`, `max_retries`, `last_error`, and `progress` fields
to the `Task` model and ensure persistence via `TaskPersistenceService`.
- Upgrade `SchedulerService` to use differential synchronization with
the persistent `SQLAlchemyJobStore` for better job durability.
- Implement structured heartbeat logging to support real-time progress
updates.
- Update project documentation and ADRs to reflect the new plugin
runtime and task resilience patterns.
- Add comprehensive unit and integration tests for the new task
lifecycle features.
