ss-tools/specs/011-git-integration-dashboard/research.md

Research & Decisions: Git Integration Plugin

Feature: Git Integration for Dashboard Development Date: 2026-01-22 Status: Finalized, amended by 2026-07-01 implementation audit

1. Unknowns & Clarifications

The following clarifications were resolved during the specification phase:

Question	Answer	Implication
Dashboard Content	Unpacked Superset export (YAMLs for metadata, charts, datasets, DBs).	We need to use superset_tool or internal logic to unzip exports before committing, and zip them before importing/deploying.
Repo Structure	1 Repository per Dashboard.	Simplifies conflict resolution (no cross-dashboard conflicts). Requires managing multiple local clones if multiple dashboards are edited.
Deployment	Import via Superset API.	Need to repackage the YAML files into a ZIP structure compatible with Superset Import API.
Conflicts	UI-based "Keep Mine / Keep Theirs".	We need a frontend diff viewer/resolver.
Auth	Personal Access Token (PAT).	Uniform auth mechanism for GitHub/GitLab/Gitea.
Primary UX audience	BI developer / Superset analyst.	UI must present a dashboard publishing workflow first and raw Git mechanics second.
Production branch naming	prod, not main.	New repositories and UI defaults use prod; main/master remain legacy-compatible only.

2. Technology Decisions

2.1 Git Interaction Library

Decision: GitPython Rationale:

• Mature and widely used Python wrapper for the git CLI.
• Supports all required operations (clone, fetch, pull, push, branch, commit, diff).
• Easier to handle output parsing compared to raw subprocess calls. Alternatives Considered:
• pygit2: Bindings for libgit2. Faster but harder to install (binary dependencies) and more complex API.
• subprocess.run(['git', ...]): Too manual, error-prone parsing.

2.2 Dashboard Serialization Format

Decision: Unpacked YAML structure (Superset Export format) Rationale:

• Superset exports dashboards as a ZIP containing YAML files.
• Unpacking this allows Git to track changes at a granular level (e.g., "changed x-axis label in chart A") rather than a binary blob change.
• Enables meaningful diffs and merge conflict resolution.

2.3 Repository Management Strategy

Decision: Local Clone Isolation Path: backend/data/git_repos/{dashboard_uuid}/ Rationale:

• Each dashboard corresponds to a remote repository.
• Isolating clones by dashboard UUID prevents collisions.
• The backend acts as a bridge between the Superset instance (source of truth for "current state" in UI) and the Git repo (source of truth for version control).

2.4 Authentication Storage

Decision: Encrypted storage in SQLite Rationale:

• PATs are sensitive credentials.
• They should not be stored in plain text.
• We will use the existing config_manager or a new secure storage utility if available, or standard encryption if not. (For MVP, standard storage in SQLite GitServerConfig table is acceptable per current project standards, assuming internal tool usage).

2.5 Branch Naming and Compatibility

Decision: dev -> preprod -> prod Rationale:

• Names match business environment stages used by BI teams.
• prod removes ambiguity between Git default branch naming and target production stage.
• The workflow is easier to explain in UI: develop, validate, publish. Compatibility:
• Existing main and master repositories remain operable.
• New repositories should use prod as default production branch.
• UI should show main/master as legacy or advanced refs, not as the primary production path.

2.6 BI-Oriented Git UX

Decision: Dashboard lifecycle UI over generic Git client UI. Rationale:

• BI developers and analysts usually think in terms of dashboard versions, validation, publishing, and rollback.
• Raw Git terms are still useful, but should not be the primary decision surface. Required UX shape:
• Show DEV -> PREPROD -> PROD pipeline state.
• Show recommended next action.
• Provide semantic change summary before YAML diff.
• Require explicit confirmation for PROD-affecting actions.

3. Architecture Patterns

3.1 The "Bridge" Workflow

1. Edit: User edits dashboard in Superset UI.
2. Sync/Stage: User clicks "Git" in our plugin. Plugin fetches current dashboard export from Superset API, unpacks it to the local git repo working directory.
3. Diff: git status / git diff shows changes between Superset state and last commit.
4. Commit: User selects files and commits.
5. Push: Pushes to remote.

3.3 BI Publishing Workflow

1. Develop in DEV: User edits the dashboard in Superset and syncs it into dev.
2. Save version: User commits a named dashboard version.
3. Validate in PREPROD: User promotes dev -> preprod through MR or approved direct merge.
4. Publish to PROD: User promotes preprod -> prod, then deploys from prod.
5. Recover: User inspects history and can rollback to a previous version.

3.2 Deployment Workflow

1. Select: User selects target environment (another Superset instance).
2. Package: Plugin zips the current HEAD (or selected commit) of the repo.
3. Upload: Plugin POSTs the ZIP to the target environment's Import API.

4. Risks & Mitigations

• Risk: Superset Export format changes.
  • Mitigation: We rely on the Superset version's export format. If it changes, the YAML structure changes, but Git will just see it as text changes. The Import API compatibility is the main constraint.
• Risk: Large repositories.
  • Mitigation: We are doing 1 repo per dashboard, which naturally limits size.
• Risk: Concurrent edits.
  • Mitigation: Git itself handles this via locking, but we should ensure our backend doesn't try to run parallel git ops on the same folder.
• Risk: Branch naming drift (main vs prod) confuses users and breaks release defaults.
  • Mitigation: Add remediation tasks, tests, and browser validation for backend defaults and frontend release flow.
• Risk: BI users are forced to reason about raw Git refs.
  • Mitigation: Group branches by business type, hide/de-emphasize remote tracking refs, and use dashboard lifecycle language in primary actions.
• Risk: YAML diff is too low-level for BI review.
  • Mitigation: Add semantic artifact summaries for charts, datasets, filters, and dashboard metadata before raw diff.