78 lines
3.5 KiB
Markdown
78 lines
3.5 KiB
Markdown
# Superset Automation Tools
|
||
|
||
## Overview
|
||
This repository contains Python scripts and a utility library (`superset_tool`) designed to automate common tasks with Apache Superset, such as:
|
||
- **Backup**: Exporting all dashboards from Superset instances to local storage.
|
||
- **Migration**: Transferring and transforming dashboards between different Superset environments (e.g., Development, Sandbox, Production).
|
||
|
||
## Project Structure
|
||
- `backup_script.py`: Main script for performing scheduled backups of Superset dashboards.
|
||
- `migration_script.py`: Main script for migrating specific dashboards between environments, including database connection remapping.
|
||
- `superset_tool/`:
|
||
- `client.py`: Python client for interacting with the Superset API.
|
||
- `exceptions.py`: Custom exception classes for structured error handling.
|
||
- `models.py`: Pydantic models for configuration data validation.
|
||
- `utils/`:
|
||
- `fileio.py`: Utilities for file system operations (archive handling, YAML parsing).
|
||
- `logger.py`: Custom logger configuration for consistent logging across the project.
|
||
- `network.py`: Low-level HTTP client for network requests, handling authentication and retries.
|
||
|
||
## Setup
|
||
|
||
### Prerequisites
|
||
- Python 3.9+
|
||
- `pip` for package management.
|
||
- `keyring` for secure password storage.
|
||
|
||
### Installation
|
||
1. **Clone the repository:**
|
||
```bash
|
||
git clone https://your-repo-link/superset-automation.git
|
||
cd superset-automation
|
||
```
|
||
2. **Install dependencies:**
|
||
```bash
|
||
pip install -r requirements.txt
|
||
```
|
||
(You might need to create `requirements.txt` with `pydantic`, `requests`, `keyring`, `PyYAML`, `urllib3`)
|
||
3. **Configure Passwords:**
|
||
Use `keyring` to store your Superset API user passwords.
|
||
Example for `backup_script.py`:
|
||
```python
|
||
import keyring
|
||
keyring.set_password("system", "dev migrate", "your_dev_password")
|
||
keyring.set_password("system", "prod migrate", "your_prod_password")
|
||
keyring.set_password("system", "sandbox migrate", "your_sandbox_password")
|
||
```
|
||
Replace `"system"` with a suitable service name if preferred.
|
||
|
||
## Usage
|
||
|
||
### Backup Script (`backup_script.py`)
|
||
To back up dashboards from configured Superset environments:
|
||
```bash
|
||
python backup_script.py
|
||
```
|
||
Backup files will be saved to `P:\Superset\010 Бекапы\` by default. Logs will be in `P:\Superset\010 Бекапы\Logs`.
|
||
|
||
### Migration Script (`migration_script.py`)
|
||
To migrate a specific dashboard:
|
||
```bash
|
||
python migration_script.py
|
||
```
|
||
**Note:** This script is currently configured to migrate a hardcoded dashboard slug (`FI0070`) and uses a local `.zip` file for the dashboard source. **For production use, you should modify the script to:**
|
||
- Accept dashboard slug/ID as command-line arguments.
|
||
- Utilize the `from_c.export_dashboard()` method instead of reading from a local file.
|
||
- Potentially parameterize `from_c` and `to_c` environments.
|
||
|
||
## Logging
|
||
Logs are configured to be written to a file within a `Logs` directory (e.g., `P:\Superset\010 Бекапы\Logs` for backups) and also output to the console. The log level is set to `INFO` by default.
|
||
|
||
## Development & Contribution
|
||
- Follow the architectural patterns (`[MODULE]`, `[CONTRACT]`, `[SECTION]`, `[ANCHOR]`) and logging guidelines for consistency.
|
||
- Ensure all new code adheres to the principles of "LLM-friendly" code generation.
|
||
- Use `Pydantic` models for data validation.
|
||
- Implement comprehensive error handling using custom exceptions.
|
||
|
||
---
|
||
[COHERENCE_CHECK_PASSED] README.md создан и согласован с модулями. |