Superset Automation Tools

Overview

This repository contains Python scripts and a utility library (superset_tool) designed to automate common tasks with Apache Superset, such as:

• Backup: Exporting all dashboards from Superset instances to local storage.
• Migration: Transferring and transforming dashboards between different Superset environments (e.g., Development, Sandbox, Production).

Project Structure

• backup_script.py: Main script for performing scheduled backups of Superset dashboards.
• migration_script.py: Main script for migrating specific dashboards between environments, including database connection remapping.
• superset_tool/:
  • client.py: Python client for interacting with the Superset API.
  • exceptions.py: Custom exception classes for structured error handling.
  • models.py: Pydantic models for configuration data validation.
  • utils/:
    • fileio.py: Utilities for file system operations (archive handling, YAML parsing).
    • logger.py: Custom logger configuration for consistent logging across the project.
    • network.py: Low-level HTTP client for network requests, handling authentication and retries.

Setup

Prerequisites

• Python 3.9+
• pip for package management.
• keyring for secure password storage.

Installation

1. Clone the repository:

   git clone https://your-repo-link/superset-automation.git
   cd superset-automation
   

2. Install dependencies:

   pip install -r requirements.txt
   

   (You might need to create requirements.txt with pydantic, requests, keyring, PyYAML, urllib3)
3. Configure Passwords: Use keyring to store your Superset API user passwords. Example for backup_script.py:

   import keyring
   keyring.set_password("system", "dev migrate", "your_dev_password")
   keyring.set_password("system", "prod migrate", "your_prod_password")
   keyring.set_password("system", "sandbox migrate", "your_sandbox_password")
   

   Replace "system" with a suitable service name if preferred.

Usage

Backup Script (backup_script.py)

To back up dashboards from configured Superset environments:

python backup_script.py

Backup files will be saved to P:\Superset\010 Бекапы\ by default. Logs will be in P:\Superset\010 Бекапы\Logs.

Migration Script (migration_script.py)

To migrate a specific dashboard:

python migration_script.py

Note: This script is currently configured to migrate a hardcoded dashboard slug (FI0070) and uses a local .zip file for the dashboard source. For production use, you should modify the script to:

• Accept dashboard slug/ID as command-line arguments.
• Utilize the from_c.export_dashboard() method instead of reading from a local file.
• Potentially parameterize from_c and to_c environments.

Logging

Logs are configured to be written to a file within a Logs directory (e.g., P:\Superset\010 Бекапы\Logs for backups) and also output to the console. The log level is set to INFO by default.

Development & Contribution

• Follow the architectural patterns ([MODULE], [CONTRACT], [SECTION], [ANCHOR]) and logging guidelines for consistency.
• Ensure all new code adheres to the principles of "LLM-friendly" code generation.
• Use Pydantic models for data validation.
• Implement comprehensive error handling using custom exceptions.

---

[COHERENCE_CHECK_PASSED] README.md создан и согласован с модулями.