busya
ec6421de35
- Replace all occurrences of 'ss-tools' with 'superset-tools' in 104 files - Rename git bundle file ss-tools.bundle → superset-tools.bundle - Update .gitignore pattern accordingly - Preserve variable names (hasSsTools etc.) and code identifiers
81 lines
4.6 KiB
Markdown
81 lines
4.6 KiB
Markdown
# [DEF:ADR-0004:ADR]
|
||
# @STATUS ACTIVE
|
||
# @PURPOSE Define the plugin architecture for superset-tools — the loading mechanism, lifecycle contract, isolation guarantees, and the boundary between core services and pluggable extensions.
|
||
# @RELATION DEPENDS_ON -> [ADR-0001:ADR]
|
||
# @RELATION DEPENDS_ON -> [ADR-0003:ADR]
|
||
# @RELATION CALLS -> [ADR-0005:ADR]
|
||
# @RATIONALE Extensibility is a core architectural value: the system must support LLM‑driven analysis, custom data transformations, and environment‑specific logic without modifying core code. A plugin system prevents the monolith from accumulating every domain‑specific feature and enables third‑party (or future‑self) contributions without forking.
|
||
# @RATIONALE Process isolation was chosen over in‑process imports because: (a) plugins may use incompatible library versions, (b) a crashing plugin must not take down the orchestrator, (c) security boundary — plugins should not access the orchestrator's database connection directly.
|
||
# @REJECTED In‑process Python `importlib` plugin loading — rejected because a misbehaving plugin can corrupt global state, exhaust memory, or crash the server. Process isolation provides a hard boundary.
|
||
# @REJECTED Docker‑container per plugin — rejected because it adds excessive orchestration complexity and startup latency for plugins that are mostly lightweight LLM prompt chains. Subprocess isolation is sufficient.
|
||
# @REJECTED WebAssembly (WASI) sandbox — rejected because the Python AI/LLM ecosystem (langchain, transformers) does not yet reliably compile to WASM. Premature optimization.
|
||
|
||
## Decision
|
||
|
||
### Plugin Architecture
|
||
|
||
```
|
||
superset-tools Core
|
||
├── core/plugin_loader.py # Plugin discovery, loading, lifecycle
|
||
├── core/plugin_executor.py # Subprocess execution, timeout, error boundary
|
||
├── core/plugin_registry.py # Registered plugins, metadata, health
|
||
└── plugins/ # Plugin packages (each = one directory)
|
||
├── llm_analysis/ # LLM‑driven Superset data analysis; v2 adds dual-path execution (Path A: Playwright screenshot + multimodal LLM; Path B: text-only API + dataset health checking)
|
||
├── dataset_orchestration/ # LLM dataset operations
|
||
└── git_integration/ # Git‑based version control for dashboards
|
||
```
|
||
|
||
### Plugin Contract
|
||
|
||
> **v2 update:** Task-based validation was introduced via `ValidationTaskService`, creating persistent validation policies with `ValidationRun` aggregates for grouping per-dashboard results. Each run is tracked as a `ValidationRun` record with aggregate pass/fail/warn counts, replacing the previous single-shot task result pattern. See `ValidationTaskService` in `services/validation_service.py`.
|
||
|
||
Every plugin MUST provide:
|
||
|
||
1. **`plugin.toml`** — metadata manifest at the plugin root
|
||
```toml
|
||
[plugin]
|
||
id = "llm_analysis"
|
||
name = "LLM Data Analysis"
|
||
version = "1.0.0"
|
||
entrypoint = "plugin.py"
|
||
timeout_sec = 300
|
||
max_memory_mb = 512
|
||
requires = ["superset-api>=1.0", "openai>=1.0"]
|
||
```
|
||
|
||
2. **`plugin.py`** — entrypoint with two required functions:
|
||
- `def register(registry: PluginRegistry) -> PluginInfo` — declare capabilities
|
||
- `def execute(task: TaskContext) -> TaskResult` — run the plugin
|
||
|
||
3. **Task context contract** (Pydantic `TaskContext`):
|
||
- `task_id: str`, `plugin_id: str`, `action: str`, `params: dict`, `superset_env: SupersetConnection`, `auth_token: str` (scoped, short‑lived)
|
||
|
||
4. **Result envelope** (Pydantic `TaskResult`):
|
||
- `status: Literal["success", "warning", "error"]`, `data: dict | None`, `error_message: str | None`, `execution_time_ms: int`, `artifacts: list[str]` (file paths to saved artifacts)
|
||
|
||
### Plugin Lifecycle
|
||
|
||
```
|
||
Discover → Validate → Register → [Execute] → Report
|
||
│ │ │ │
|
||
│ Check TOML Store in Spawn subprocess
|
||
│ schema, registry with timeout +
|
||
│ dependencies in DB memory limit
|
||
```
|
||
|
||
### Isolation Guarantees
|
||
|
||
- **Subprocess**: `subprocess.run(..., timeout=timeout_sec)`, killed on timeout.
|
||
- **Memory**: `resource.setrlimit(RLIMIT_AS, max_memory_mb * 1024 * 1024)` before exec.
|
||
- **No DB access**: Plugins receive only a scoped REST API token, never a database connection.
|
||
- **No filesystem writes outside allowed dirs**: Configurable artifact directory per plugin.
|
||
|
||
### RBAC Integration
|
||
|
||
Plugin access is governed by ADR-0005 (RBAC):
|
||
- Each plugin declares `required_roles: ["admin", "analyst"]` in `plugin.toml`.
|
||
- Plugin executor checks the user's role set before allowing execution.
|
||
- Forbidden access returns `403` with audit log entry.
|
||
|
||
# [/DEF:ADR-0004:ADR]
|