sqlparse raises SQLParseError above MAX_GROUPING_TOKENS=10000 tokens
(~25KB of typical SQL). The try/except fallback already handled it, but paid
~1s per oversized SQL for a parse doomed to fail. Add _SQLPARSE_SKIP_THRESHOLD
(30k chars) to bypass sqlparse for oversized text (~15x faster, 1.2s->0.08s for
a 212KB SQL) while keeping literal filtering for SQL under the threshold.
Tests: oversized-SQL skip-threshold behavior.
Discovery of virtual datasets now works, but a runtime blocker remained: any
virtual dataset whose SQL exceeds sqlparse's MAX_GROUPING_TOKENS (10000 tokens)
raised SQLParseError 'Maximum number of tokens exceeded (10000)' from
extract_tables_from_sql_span, which is called unguarded in the scan loop — one
oversized virtual dataset aborted the whole maintenance preview/start.
- extract_tables_from_sql_span now wraps sqlparse.parse + token walk in
try/except and falls back to regex-only extraction (keeping all schema.table
matches) instead of raising, so huge SQL no longer fails the scan.
- Tier-1 virtual filter uses value "" (not None) so the sql is_not_null filter
passes Superset's rison schema instead of always falling back to a full scan.
Tests: huge-SQL fallback (extractor) and huge-virtual-dataset scan resilience
(scanner). ADR-0020 updated with Decision 3.
Virtual (SQL) datasets were never matched, so maintenance discovery returned
0 affected dashboards. Two defects fixed:
- find_affected_dashboards filtered by is_sqllab_view, which is NOT a
filterable column in Superset's dataset list API (absent from search_columns),
so the query was rejected. Now discover virtual datasets via the filterable
sql column: primary server-side 'sql is_not_null' filter with a client-side
non-empty-sql scan as fallback (best-effort vs pagination cap), dedupe by id.
- AsyncAPIClient.request never called raise_for_status(), so rejected filters
(HTTP 400) were returned as bodies without a 'result' key and surfaced as
'Found 0 datasets', dead-coding the filtered->full-scan fallback. request()
now raises on non-2xx via the existing error mapper.
Tests cover both virtual-scan tiers, all fallback paths, the raise behavior,
and an end-to-end match with the real sql_table_extractor on production SQL.
Documented in ADR-0020.
