ss-tools/docs/adr/ADR-0003-orchestrator-pattern.md

@{ Doc.Adr.ADR0003 [C:1] [TYPE ADR]

@STATUS ACCEPTED

@BRIEF Definitively establish superset-tools as a standalone orchestrator service operating above Apache Superset environments rather than integrating into them. This is a repo‑shaping architectural decision that governs deployment topology, security boundary, release cycle independence, and technology stack freedom.

@RELATION DEPENDS_ON -> [Doc.Adr.ADR0001]

@RELATION CALLS -> [Doc.Adr.ADR0016]

@RELATION CALLS -> [Doc.Adr.ADR0005]

@RATIONALE The core value proposition of superset-tools is multi‑environment orchestration: moving dashboards, datasets, and configurations between independent Superset instances (Dev → Stage → Prod). Embedding this logic inside any single Superset instance would immediately create a "master controller" dependency — if that instance fails, all environment management is lost. An external orchestrator treats all Superset instances as equal endpoints.

@RATIONALE Technology stack mismatch is irreversible: Superset uses Flask + React, superset-tools uses FastAPI + SvelteKit. Integration would require rewriting the entire Svelte frontend into React components and refactoring FastAPI dependency injection into Flask-AppBuilder patterns — a multi‑month effort with zero business value and high regression risk.

@RATIONALE Release cycle independence is non‑negotiable: Superset has a slow, complex release cadence. superset-tools must ship LLM plugins, Git integrations, and backup features on its own schedule without waiting for upstream Superset releases or maintaining a fork with perpetual merge conflicts.

@RATIONALE Security isolation: superset-tools manages DevOps privileges (deployment, backup, migration) while Superset manages BI privileges (data viewing, chart creation). Mixing these in one system expands the blast radius of any security incident.

@REJECTED Integration as a Superset plugin/extension — rejected for the four reasons above (orchestrator topology break, 100% frontend rewrite, Superset release cycle coupling, privilege boundary collapse).

@REJECTED Superset fork with superset-tools baked in — rejected because maintaining a fork requires rebase on every Superset release, creating perpetual merge conflicts and making security patch adoption dangerously slow.

@REJECTED Shared database with Superset — rejected because it couples superset-tools migrations to Superset's Alembic migration chain, preventing independent schema evolution and creating a single point of failure for both systems.

Decision

superset-tools operates as a standalone orchestrator microservice with these defined boundaries:

Deployment Topology

                       ┌─────────────┐
                       │  superset-tools   │  ← FastAPI + SvelteKit
                       │  PostgreSQL │  ← Own database
                       └──────┬──────┘
                              │ REST API calls
              ┌───────────────┼───────────────┐
              ▼               ▼               ▼
       ┌──────────┐   ┌──────────┐   ┌──────────┐
       │Superset  │   │Superset  │   │Superset  │
       │  (Dev)   │   │ (Stage)  │   │  (Prod)  │
       └──────────┘   └──────────┘   └──────────┘

Key Architectural Properties

Property	Decision
Topology	External orchestrator above all Superset instances
Communication	Superset REST API (bearer token auth per environment)
Database	Dedicated PostgreSQL 16 instance (not Superset metadata DB)
Frontend	SvelteKit SPA (not integrated into Superset UI)
Release cadence	Independent Docker image releases
Auth	Own RBAC system with optional ADFS SSO (see ADR-0005)
Plugins	Own plugin lifecycle (see ADR-0016)

Consequences

• Positive: Fast independent development, modern stack, no Superset coupling, clear security boundary.
• Negative: Users have two separate UIs (Superset for BI, superset-tools for DevOps). Mitigated by consistent UX design and cross‑linking.
• Risk: API compatibility if Superset changes its REST API. Mitigated by versioned API client with automated compatibility tests.

Migration from Existing Document

This ADR supersedes and formalizes docs/architecture_decision_superset_migration.md. The original document contained the same recommendation and rationale but lacked the legacy DEF contract anchor, making it invisible to the semantic index and decision‑memory audit chain.

@} Doc.Adr.ADR0003