feat: attention-optimized semantic protocol v2.7
Core changes: - Add @defgroup/@ingroup to 1791 C2+ contracts (555 files) for HCA 128× pre-training DSA grouping - Add §0.1 Pre-Training Frequency matrix to semantics-core - Add §VIII Attention Architecture rules (ATTN_1-4) with MLA/CSA/HCA/DSA mechanics - Add @defgroup/@ingroup to canonical syntax (§II) and all contract examples Agent prompts (5 files): - Add ZERO-STATE RATIONALE with MLA/CSA/HCA/DSA compression mechanics - Add pre-training note: @RATIONALE/@REJECTED are in-context learned tags - svelte-coder: add missing #region contract, fix Svelte rule violations - python-coder/fullstack-coder: honor function contracts from speckit plan - qa-tester: add attention compliance audit (P3 ATTN_1-4 checks) Skills (6 files): - Translate all axiom_config descriptions to English - Fix doc_dirs to index .opencode/ and .specify/ - Deduplicate 5× complexity_rules → single global_tags catalog - Reduce semantics-svelte 591→485 lines (remove duplicate code blocks) - Fix semantics-testing: 'Short IDs' → 'Short hierarchical IDs' - Fix all examples: flat IDs → hierarchical Domain.Name format - Fix Svelte examples: replace raw Tailwind + <button> with semantic tokens + /ui Speckit workflow (commands + templates): - speckit.plan: add Function-Level Contracts for C3+ with @PRE/@POST/@TEST_EDGE - speckit.plan: add Attention Compliance Gate (ATTN_1-4 before contract generation) - speckit.tasks: add function contract inlining format (constraints in task description) - speckit.specify: load semantics-core for spec density rules - spec-template: add #region contract, @SEMANTICS grouping, hierarchical IDs - ux-reference-template: add #region wrapper - plan-template: add attention gate, @defgroup/@ingroup guidance - tasks-template: add attention audit + rebuild + orphan check tasks - constitution.md: translate to English, add Principle VIII (attention-optimized contracts) Reference modules rewritten (hierarchical IDs + full contracts): - Auth.Jwt: 6 child contracts with @RATIONALE/@REJECTED/@TEST_EDGE - Api.Auth: 5 endpoints with @TEST_EDGE + molecular CoT markers - Migration.Model: @defgroup Migration with 18 @ACTION + 6 @INVARIANT Scripts: - add_defgroup_ingroup.py: zero-risk additive @ingroup migration (1791 insertions) - migrate_hierarchical.py: flat→hierarchical ID dry-run analysis (792 contracts) - merge_prompts.py: merge all prompts/skills/commands into one review file Config: - axiom_config.yaml: 749→395 lines (-47%), English, doc_dirs include prompts - Fix test_datasets.py import collision (rename → test_datasets_routes.py) - Fix test_preview.py: SupersetClient→get_superset_client, AsyncMock, logger f-string
This commit is contained in:
@@ -1,3 +1,4 @@
|
||||
# #region src.plugins [TYPE Package] [SEMANTICS plugin, package, discovery, runtime]
|
||||
# @ingroup Plugin
|
||||
# @BRIEF Plugin package root for dynamic discovery and runtime imports.
|
||||
# #endregion src.plugins
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region BackupPlugin [TYPE Module] [SEMANTICS backup, export, archive, superset, dashboard]
|
||||
# @defgroup Plugin Module group.
|
||||
# @BRIEF A plugin that provides functionality to back up Superset dashboards.
|
||||
# @LAYER App
|
||||
# @RELATION IMPLEMENTS -> PluginBase
|
||||
@@ -22,6 +23,7 @@ from ..dependencies import get_config_manager
|
||||
|
||||
|
||||
# #region BackupPlugin [TYPE Class]
|
||||
# @defgroup Plugin Module group.
|
||||
# @BRIEF Implementation of the backup plugin logic.
|
||||
class BackupPlugin(PluginBase):
|
||||
"""
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region DebugPluginModule [TYPE Module] [SEMANTICS debug, diagnostic, superset, api, system]
|
||||
# @defgroup Plugin Module group.
|
||||
# @BRIEF Implements a plugin for system diagnostics and debugging Superset API responses.
|
||||
# @LAYER Plugin
|
||||
# @BRIEF Plugin for diagnostics. Inherits PluginBase.
|
||||
@@ -13,6 +14,7 @@ from ..core.utils.client_registry import get_superset_client
|
||||
|
||||
|
||||
# #region DebugPlugin [TYPE Class]
|
||||
# @defgroup Plugin Module group.
|
||||
# @BRIEF Plugin for system diagnostics and debugging.
|
||||
class DebugPlugin(PluginBase):
|
||||
"""
|
||||
|
||||
@@ -1,3 +1,4 @@
|
||||
# #region GitPluginExt [TYPE Package] [SEMANTICS git, plugin, extension, package]
|
||||
# @ingroup Git
|
||||
# @BRIEF Git plugin extension package root.
|
||||
# #endregion GitPluginExt
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region GitLLMExtensionModule [C:3] [TYPE Module] [SEMANTICS git, llm, commit, message, generation]
|
||||
# @defgroup Git Module group.
|
||||
# @BRIEF LLM-based extensions for the Git plugin, specifically for commit message generation.
|
||||
# @LAYER Domain
|
||||
# @RELATION DEPENDS_ON -> [LLMClient]
|
||||
@@ -12,6 +13,7 @@ from ..llm_analysis.service import LLMClient
|
||||
|
||||
|
||||
# #region GitLLMExtension [TYPE Class]
|
||||
# @defgroup Git Module group.
|
||||
# @BRIEF Provides LLM capabilities to the Git plugin.
|
||||
class GitLLMExtension:
|
||||
def __init__(self, client: LLMClient):
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region GitPluginModule [C:4] [TYPE Module] [SEMANTICS git, versioning, deploy, sync, superset, backup, transactional]
|
||||
# @defgroup Plugin Module group.
|
||||
#
|
||||
# @BRIEF Предоставляет плагин для версионирования и развертывания дашбордов Superset.
|
||||
# @LAYER Plugin
|
||||
@@ -123,6 +124,7 @@ def _pack_deploy_zip(repo_path: Path, root_dir_name: str, zip_buffer: io.BytesIO
|
||||
|
||||
|
||||
# #region GitPlugin [TYPE Class]
|
||||
# @defgroup Plugin Module group.
|
||||
# @BRIEF Реализация плагина Git Integration для управления версиями дашбордов.
|
||||
class GitPlugin(PluginBase):
|
||||
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region LLMAnalysisPackage [TYPE Module] [SEMANTICS llm, analysis, plugin, package, export]
|
||||
# @defgroup LLMAnalysis Module group.
|
||||
|
||||
"""
|
||||
LLM Analysis Plugin for automated dashboard validation and dataset documentation.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region DashboardTopology [C:3] [TYPE Module] [SEMANTICS superset, topology, dashboard, layout, parsing]
|
||||
# @defgroup LLMAnalysis Module group.
|
||||
# @BRIEF Dashboard topology extraction — parses Superset position_json into human-readable
|
||||
# text descriptions of Tab → Row → Chart hierarchy for LLM prompt context.
|
||||
# @LAYER Plugin
|
||||
@@ -227,6 +228,7 @@ def _extract_topology_children(
|
||||
|
||||
|
||||
# #region build_dashboard_topology [C:2] [TYPE Function]
|
||||
# @ingroup LLMAnalysis
|
||||
# @BRIEF Build a structured text description of dashboard layout from position_json.
|
||||
# @PARAM dashboard (dict) - Dashboard metadata dict with position_json, dashboard_title.
|
||||
# @PARAM charts (list[dict]) - List of fetched chart detail dicts for name/viz lookup.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region V1ToV2Migration [C:3] [TYPE Module] [SEMANTICS migration,validation,data,idempotent]
|
||||
# @defgroup LLMAnalysis Module group.
|
||||
# @BRIEF Migration script: v1 → v2 ValidationPolicy data. Converts existing dashboard_ids
|
||||
# JSON arrays to ValidationSource relational rows.
|
||||
# @PRE Database has both validation_policies and validation_sources tables.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region LLMAnalysisModels [C:3] [TYPE Module] [SEMANTICS pydantic, llm, plugin, model, provider-type]
|
||||
# @defgroup LLMAnalysis Module group.
|
||||
# @BRIEF Define Pydantic models for LLM Analysis plugin.
|
||||
# @LAYER Domain
|
||||
# @RELATION DEPENDS_ON -> [EXT:Library:pydantic]
|
||||
@@ -10,6 +11,7 @@ from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
# #region LLMProviderType [TYPE Class]
|
||||
# @defgroup LLMAnalysis Module group.
|
||||
# @BRIEF Enum for supported LLM providers.
|
||||
class LLMProviderType(str, Enum):
|
||||
OPENAI = "openai"
|
||||
@@ -19,6 +21,7 @@ class LLMProviderType(str, Enum):
|
||||
# #endregion LLMProviderType
|
||||
|
||||
# #region LLMProviderConfig [TYPE Class]
|
||||
# @defgroup LLMAnalysis Module group.
|
||||
# @BRIEF Configuration for an LLM provider.
|
||||
class LLMProviderConfig(BaseModel):
|
||||
id: str | None = None
|
||||
@@ -41,6 +44,7 @@ class LLMProviderConfig(BaseModel):
|
||||
# #endregion LLMProviderConfig
|
||||
|
||||
# #region ValidationStatus [TYPE Class]
|
||||
# @defgroup LLMAnalysis Module group.
|
||||
# @BRIEF Enum for dashboard validation status.
|
||||
class ValidationStatus(str, Enum):
|
||||
PASS = "PASS"
|
||||
@@ -50,6 +54,7 @@ class ValidationStatus(str, Enum):
|
||||
# #endregion ValidationStatus
|
||||
|
||||
# #region DetectedIssue [TYPE Class]
|
||||
# @defgroup LLMAnalysis Module group.
|
||||
# @BRIEF Model for a single issue detected during validation.
|
||||
class DetectedIssue(BaseModel):
|
||||
severity: ValidationStatus
|
||||
@@ -58,6 +63,7 @@ class DetectedIssue(BaseModel):
|
||||
# #endregion DetectedIssue
|
||||
|
||||
# #region ValidationResult [TYPE Class]
|
||||
# @defgroup LLMAnalysis Module group.
|
||||
# @BRIEF Model for dashboard validation result.
|
||||
class ValidationResult(BaseModel):
|
||||
id: str | None = None
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region LLMAnalysisPlugin [C:5] [TYPE Module] [SEMANTICS llm, analysis, dashboard, validation, documentation]
|
||||
# @defgroup LLMAnalysis Module group.
|
||||
# @BRIEF Implements DashboardValidationPlugin and DocumentationPlugin.
|
||||
# @LAYER Plugin
|
||||
# @RELATION INHERITS -> [PluginBase]
|
||||
@@ -159,6 +160,7 @@ def _update_run_status(db, run_id: str | None) -> None:
|
||||
|
||||
|
||||
# #region DashboardValidationPlugin [TYPE Class]
|
||||
# @defgroup LLMAnalysis Module group.
|
||||
# @BRIEF Plugin for automated dashboard health analysis using LLMs.
|
||||
# @RELATION IMPLEMENTS -> [PluginBase]
|
||||
class DashboardValidationPlugin(PluginBase):
|
||||
@@ -846,6 +848,7 @@ class DashboardValidationPlugin(PluginBase):
|
||||
|
||||
|
||||
# #region DocumentationPlugin [TYPE Class]
|
||||
# @defgroup LLMAnalysis Module group.
|
||||
# @BRIEF Plugin for automated dataset documentation using LLMs.
|
||||
# @RELATION IMPLEMENTS -> [PluginBase]
|
||||
class DocumentationPlugin(PluginBase):
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region LLMAnalysisScheduler [C:3] [TYPE Module] [SEMANTICS scheduler, llm, validation, task, cron]
|
||||
# @defgroup LLMAnalysis Module group.
|
||||
# @BRIEF Provides helper functions to schedule LLM-based validation tasks.
|
||||
# @LAYER Domain
|
||||
# @RELATION DEPENDS_ON -> [SchedulerService]
|
||||
@@ -10,6 +11,7 @@ from ...dependencies import get_scheduler_service, get_task_manager
|
||||
|
||||
|
||||
# #region schedule_dashboard_validation [TYPE Function]
|
||||
# @ingroup LLMAnalysis
|
||||
# @BRIEF Schedules a recurring dashboard validation task.
|
||||
# @SIDE_EFFECT Adds a job to the scheduler service.
|
||||
def schedule_dashboard_validation(dashboard_id: str, cron_expression: str, params: dict[str, Any]):
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region LLMAnalysisService [C:5] [TYPE Module] [SEMANTICS llm, screenshot, playwright, openai, tenacity]
|
||||
# @defgroup LLMAnalysis Module group.
|
||||
# @BRIEF Services for LLM interaction and dashboard screenshots.
|
||||
# @LAYER Plugin
|
||||
# @RELATION DEPENDS_ON -> [EXT:Library:tenacity]
|
||||
@@ -37,6 +38,7 @@ LLM_HTTP_TIMEOUT_S = 120 # seconds (httpx client timeout)
|
||||
DEFAULT_USER_AGENT = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36"
|
||||
|
||||
# #region ScreenshotService [TYPE Class]
|
||||
# @defgroup LLMAnalysis Module group.
|
||||
# @BRIEF Handles capturing screenshots of Superset dashboards.
|
||||
class ScreenshotService:
|
||||
# region ScreenshotService.__init__ [TYPE Function]
|
||||
@@ -837,6 +839,7 @@ class ScreenshotService:
|
||||
# #endregion ScreenshotService
|
||||
|
||||
# #region LLMClient [TYPE Class]
|
||||
# @defgroup LLMAnalysis Module group.
|
||||
# @BRIEF Wrapper for LLM provider APIs.
|
||||
class LLMClient:
|
||||
# region LLMClient.__init__ [TYPE Function]
|
||||
@@ -1441,6 +1444,7 @@ class LLMClient:
|
||||
# #endregion LLMClient
|
||||
|
||||
# #region DatasetHealthChecker [C:3] [TYPE Class]
|
||||
# @defgroup LLMAnalysis Module group.
|
||||
# @BRIEF Checks dataset accessibility and KXD connectivity via Superset API.
|
||||
# @LAYER Service
|
||||
# @RELATION CALLS -> [SupersetClient]
|
||||
@@ -1641,6 +1645,7 @@ class DatasetHealthChecker:
|
||||
# #endregion DatasetHealthChecker
|
||||
|
||||
# #region RedactionService [C:2] [TYPE Module]
|
||||
# @defgroup LLMAnalysis Module group.
|
||||
# @BRIEF Redacts PII, credentials, and sensitive data from logs and LLM responses.
|
||||
# @LAYER Service
|
||||
# @RATIONALE FR-029: sensitive data must be filtered BEFORE external LLM send and BEFORE persistence.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region MaintenanceBannerPlugin [C:4] [TYPE Module] [SEMANTICS plugin, maintenance, banner, execution, task]
|
||||
# @defgroup Plugin Module group.
|
||||
# @BRIEF TaskManager plugin for executing maintenance banner operations (start, end, end-all).
|
||||
# Dispatches to MaintenanceService based on operation type in params.
|
||||
# @LAYER Plugin
|
||||
@@ -21,6 +22,7 @@ from ..services.maintenance import end_all_maintenance, end_maintenance, start_m
|
||||
|
||||
|
||||
# #region MaintenanceBannerPlugin [C:4] [TYPE Class]
|
||||
# @defgroup Plugin Module group.
|
||||
# @BRIEF Plugin for executing maintenance banner operations asynchronously via TaskManager.
|
||||
# @RELATION CALLS -> [start_maintenance]
|
||||
# @RELATION CALLS -> [end_maintenance]
|
||||
@@ -29,36 +31,42 @@ class MaintenanceBannerPlugin(PluginBase):
|
||||
"""TaskManager plugin for maintenance banner operations."""
|
||||
|
||||
# #region id [TYPE Property]
|
||||
# @ingroup Plugin
|
||||
@property
|
||||
def id(self) -> str:
|
||||
return "maintenance_banner_apply"
|
||||
# #endregion id
|
||||
|
||||
# #region name [TYPE Property]
|
||||
# @ingroup Plugin
|
||||
@property
|
||||
def name(self) -> str:
|
||||
return "Maintenance Banner"
|
||||
# #endregion name
|
||||
|
||||
# #region description [TYPE Property]
|
||||
# @ingroup Plugin
|
||||
@property
|
||||
def description(self) -> str:
|
||||
return "Applies or removes maintenance banners on Superset dashboards."
|
||||
# #endregion description
|
||||
|
||||
# #region version [TYPE Property]
|
||||
# @ingroup Plugin
|
||||
@property
|
||||
def version(self) -> str:
|
||||
return "1.0.0"
|
||||
# #endregion version
|
||||
|
||||
# #region ui_route [TYPE Property]
|
||||
# @ingroup Plugin
|
||||
@property
|
||||
def ui_route(self) -> str:
|
||||
return "" # No UI route — async-only
|
||||
# #endregion ui_route
|
||||
|
||||
# #region get_schema [TYPE Function]
|
||||
# @ingroup Plugin
|
||||
def get_schema(self) -> dict[str, Any]:
|
||||
return {
|
||||
"type": "object",
|
||||
@@ -82,6 +90,7 @@ class MaintenanceBannerPlugin(PluginBase):
|
||||
# #endregion get_schema
|
||||
|
||||
# #region execute [TYPE Function] [C:4]
|
||||
# @ingroup Plugin
|
||||
# @BRIEF Execute maintenance operation based on params.operation.
|
||||
# @PARAM params contains: operation (start|end|end_all), event_id, environment_id.
|
||||
# @PARAM context TaskContext for logging.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region MapperPluginModule [TYPE Module] [SEMANTICS mapper, dataset, column, sqllab, excel, mapping]
|
||||
# @defgroup Plugin Module group.
|
||||
# @BRIEF Implements a plugin for mapping dataset columns using Superset SQL Lab or Excel files.
|
||||
# @LAYER Plugin
|
||||
# @BRIEF Plugin for dataset column mapping. Inherits PluginBase.
|
||||
@@ -14,6 +15,7 @@ from ..core.utils.dataset_mapper import DatasetMapper
|
||||
|
||||
|
||||
# #region MapperPlugin [TYPE Class]
|
||||
# @defgroup Plugin Module group.
|
||||
# @BRIEF Plugin for mapping dataset columns verbose names.
|
||||
class MapperPlugin(PluginBase):
|
||||
"""
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region MigrationPlugin [C:5] [TYPE Module] [SEMANTICS migration, export, import, mapping, superset]
|
||||
# @defgroup Plugin Module group.
|
||||
# @BRIEF Orchestrates export, DB-mapping transformation, and import of Superset dashboards across environments.
|
||||
# @LAYER App
|
||||
# @RELATION IMPLEMENTS -> PluginBase
|
||||
@@ -28,6 +29,7 @@ from ..models.mapping import DatabaseMapping, Environment
|
||||
|
||||
|
||||
# #region MigrationPlugin [TYPE Class]
|
||||
# @defgroup Plugin Module group.
|
||||
# @BRIEF Implementation of the migration plugin workflow and transformation orchestration.
|
||||
# @PRE SupersetClient authenticated, database session active
|
||||
# @POST Returns MigrationResult with success/failure status and artifact list
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region SearchPluginModule [TYPE Module] [SEMANTICS search, dataset, text, pattern, superset]
|
||||
# @defgroup Plugin Module group.
|
||||
# @BRIEF Implements a plugin for searching text patterns across all datasets in a specific Superset environment.
|
||||
# @LAYER Plugin
|
||||
# @BRIEF Plugin for text search across datasets. Inherits PluginBase.
|
||||
@@ -14,6 +15,7 @@ from ..core.task_manager.context import TaskContext
|
||||
|
||||
|
||||
# #region SearchPlugin [TYPE Class]
|
||||
# @defgroup Plugin Module group.
|
||||
# @BRIEF Plugin for searching text patterns in Superset datasets.
|
||||
class SearchPlugin(PluginBase):
|
||||
"""
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region StoragePluginPackage [TYPE Package] [SEMANTICS storage, plugin, package, export]
|
||||
# @ingroup Storage
|
||||
from .plugin import StoragePlugin
|
||||
|
||||
__all__ = ["StoragePlugin"]
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region StoragePlugin [TYPE Module] [SEMANTICS fastapi, storage, filesystem, backup, archive]
|
||||
# @defgroup Storage Module group.
|
||||
# @BRIEF Provides core filesystem operations for managing backups and repositories.
|
||||
# @LAYER App
|
||||
# @RELATION DEPENDS_ON -> [TaskContext]
|
||||
@@ -34,6 +35,7 @@ def _copy_upload_to_disk(dest_path: Path, file_obj) -> None:
|
||||
|
||||
|
||||
# #region StoragePlugin [TYPE Class]
|
||||
# @defgroup Storage Module group.
|
||||
# @BRIEF Implementation of the storage management plugin.
|
||||
class StoragePlugin(PluginBase):
|
||||
"""
|
||||
|
||||
@@ -1,2 +1,3 @@
|
||||
# #region TranslatePluginPackage [TYPE Package] [SEMANTICS translate, plugin, package, sql, dashboard]
|
||||
# @ingroup Translate
|
||||
# #endregion TranslatePluginPackage
|
||||
|
||||
@@ -7,14 +7,15 @@
|
||||
from datetime import UTC, datetime, timedelta
|
||||
import json
|
||||
import pytest
|
||||
from unittest.mock import MagicMock, patch
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
|
||||
from src.models.translate import (
|
||||
TranslationJob,
|
||||
TranslationPreviewRecord,
|
||||
TranslationPreviewSession,
|
||||
)
|
||||
from src.plugins.translate.preview import TokenEstimator, TranslationPreview
|
||||
from src.plugins.translate.preview import TranslationPreview
|
||||
from src.plugins.translate.preview_token_estimator import TokenEstimator
|
||||
from src.plugins.translate.preview_executor import PreviewExecutor
|
||||
|
||||
|
||||
@@ -59,7 +60,8 @@ class TestTranslationPreview:
|
||||
|
||||
# region test_preview_valid_job [TYPE Function]
|
||||
# @PURPOSE: Verify preview creates session and records successfully.
|
||||
def test_preview_valid_job(self):
|
||||
@pytest.mark.asyncio
|
||||
async def test_preview_valid_job(self):
|
||||
"""Preview with valid job should create session and return records."""
|
||||
db = MagicMock()
|
||||
config_manager = MagicMock()
|
||||
@@ -79,21 +81,21 @@ class TestTranslationPreview:
|
||||
config_manager.get_environments.return_value = [env]
|
||||
|
||||
# Mock SupersetClient
|
||||
with patch("src.plugins.translate.preview_executor.SupersetClient") as MockClient:
|
||||
with patch("src.plugins.translate.preview_executor.get_superset_client") as MockClient:
|
||||
mock_client = MagicMock()
|
||||
MockClient.return_value = mock_client
|
||||
|
||||
# Mock dataset_detail
|
||||
mock_client.get_dataset_detail.return_value = {
|
||||
# Mock dataset_detail (async method)
|
||||
mock_client.get_dataset_detail = AsyncMock(return_value={
|
||||
"table_name": "test_table",
|
||||
"columns": [
|
||||
{"column_name": "title", "type": "VARCHAR"},
|
||||
{"column_name": "category", "type": "VARCHAR"},
|
||||
{"column_name": "description", "type": "TEXT"},
|
||||
],
|
||||
}
|
||||
})
|
||||
|
||||
# Mock build_dataset_preview_query_context
|
||||
# Mock build_dataset_preview_query_context (NOT async)
|
||||
mock_client.build_dataset_preview_query_context.return_value = {
|
||||
"datasource": {"id": 42, "type": "table"},
|
||||
"queries": [{
|
||||
@@ -107,8 +109,8 @@ class TestTranslationPreview:
|
||||
"force": True,
|
||||
}
|
||||
|
||||
# Mock network request to Superset
|
||||
mock_client.network.request.return_value = {
|
||||
# Mock network request to Superset (async)
|
||||
mock_client.network.request = AsyncMock(return_value={
|
||||
"result": [{
|
||||
"status": "success",
|
||||
"data": [
|
||||
@@ -116,7 +118,7 @@ class TestTranslationPreview:
|
||||
{"title": "Goodbye", "category": "farewell", "description": "A parting message"},
|
||||
],
|
||||
}]
|
||||
}
|
||||
})
|
||||
|
||||
# Mock LLM provider
|
||||
with patch("src.services.llm_provider.LLMProviderService") as MockProviderService:
|
||||
@@ -140,7 +142,7 @@ class TestTranslationPreview:
|
||||
mock_filter.return_value = []
|
||||
|
||||
preview_service = TranslationPreview(db, config_manager, "test-user")
|
||||
result = preview_service.preview_rows(job_id="job-123", sample_size=10)
|
||||
result = await preview_service.preview_rows(job_id="job-123", sample_size=10)
|
||||
|
||||
# Verify result structure
|
||||
assert result["job_id"] == "job-123"
|
||||
@@ -165,7 +167,8 @@ class TestTranslationPreview:
|
||||
|
||||
# region test_preview_with_dictionary [TYPE Function]
|
||||
# @PURPOSE: Verify glossary entries from dictionary are included in LLM prompt.
|
||||
def test_preview_with_dictionary(self):
|
||||
@pytest.mark.asyncio
|
||||
async def test_preview_with_dictionary(self):
|
||||
"""Preview should include dictionary glossary in the prompt sent to LLM."""
|
||||
db = MagicMock()
|
||||
config_manager = MagicMock()
|
||||
@@ -177,10 +180,10 @@ class TestTranslationPreview:
|
||||
env.id = "postgresql"
|
||||
config_manager.get_environments.return_value = [env]
|
||||
|
||||
with patch("src.plugins.translate.preview_executor.SupersetClient") as MockClient:
|
||||
with patch("src.plugins.translate.preview_executor.get_superset_client") as MockClient:
|
||||
mock_client = MagicMock()
|
||||
MockClient.return_value = mock_client
|
||||
mock_client.get_dataset_detail.return_value = {"table_name": "test", "columns": []}
|
||||
mock_client.get_dataset_detail = AsyncMock(return_value={"table_name": "test", "columns": []})
|
||||
mock_client.build_dataset_preview_query_context.return_value = {
|
||||
"datasource": {"id": 42, "type": "table"},
|
||||
"queries": [{"columns": [], "metrics": ["count"], "row_limit": 10}],
|
||||
@@ -189,9 +192,9 @@ class TestTranslationPreview:
|
||||
"result_type": "query",
|
||||
"force": True,
|
||||
}
|
||||
mock_client.network.request.return_value = {
|
||||
mock_client.network.request = AsyncMock(return_value={
|
||||
"result": [{"status": "success", "data": [{"title": "Hello World", "category": "greeting"}]}]
|
||||
}
|
||||
})
|
||||
|
||||
with patch("src.services.llm_provider.LLMProviderService") as MockProviderService:
|
||||
mock_provider_svc = MagicMock()
|
||||
@@ -215,7 +218,7 @@ class TestTranslationPreview:
|
||||
]
|
||||
|
||||
preview_service = TranslationPreview(db, config_manager, "test-user")
|
||||
result = preview_service.preview_rows(job_id="job-123", sample_size=5)
|
||||
result = await preview_service.preview_rows(job_id="job-123", sample_size=5)
|
||||
|
||||
# Verify LLM was called
|
||||
mock_llm.assert_called_once()
|
||||
@@ -508,7 +511,8 @@ class TestTranslationPreview:
|
||||
|
||||
# region test_preview_missing_datasource [TYPE Function]
|
||||
# @PURPOSE: Verify error when job has no datasource configured.
|
||||
def test_preview_missing_datasource(self):
|
||||
@pytest.mark.asyncio
|
||||
async def test_preview_missing_datasource(self):
|
||||
"""Preview should fail if job has no datasource."""
|
||||
db = MagicMock()
|
||||
config_manager = MagicMock()
|
||||
@@ -518,12 +522,13 @@ class TestTranslationPreview:
|
||||
|
||||
preview_service = TranslationPreview(db, config_manager, "test-user")
|
||||
with pytest.raises(ValueError, match="source datasource"):
|
||||
preview_service.preview_rows(job_id="job-123")
|
||||
await preview_service.preview_rows(job_id="job-123")
|
||||
# endregion test_preview_missing_datasource
|
||||
|
||||
# region test_preview_missing_translation_column [TYPE Function]
|
||||
# @PURPOSE: Verify error when job has no translation column.
|
||||
def test_preview_missing_translation_column(self):
|
||||
@pytest.mark.asyncio
|
||||
async def test_preview_missing_translation_column(self):
|
||||
"""Preview should fail if job has no translation column."""
|
||||
db = MagicMock()
|
||||
config_manager = MagicMock()
|
||||
@@ -533,7 +538,7 @@ class TestTranslationPreview:
|
||||
|
||||
preview_service = TranslationPreview(db, config_manager, "test-user")
|
||||
with pytest.raises(ValueError, match="translation column"):
|
||||
preview_service.preview_rows(job_id="job-123")
|
||||
await preview_service.preview_rows(job_id="job-123")
|
||||
# endregion test_preview_missing_translation_column
|
||||
|
||||
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region BatchInsertService [C:3] [TYPE Module] [SEMANTICS translate, insert, sqllab, upsert, target]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Insert successful translation records into target table via Superset SQL Lab.
|
||||
# Builds INSERT SQL (original + per-language rows), resolves backend dialect,
|
||||
# and executes via SupersetSqlLabExecutor.
|
||||
@@ -24,6 +25,7 @@ from .superset_executor import SupersetSqlLabExecutor
|
||||
|
||||
|
||||
# #region insert_batch_to_target [C:3] [TYPE Function] [SEMANTICS translate, insert, orchestrate]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Insert successful records from a single batch into the target table.
|
||||
async def insert_batch_to_target(
|
||||
db: Session, config_manager: ConfigManager,
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region BatchProcessingService [C:4] [TYPE Module] [SEMANTICS translate, batch, process, classify, cache]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Batch processing for translation: classify rows (same-language/cache/preview/LLM),
|
||||
# call LLM service, persist TranslationRecord/TranslationLanguage rows.
|
||||
# Local language detection (lingua) replaces LLM-based detection.
|
||||
@@ -32,6 +33,7 @@ from .dictionary import DictionaryManager
|
||||
|
||||
|
||||
# #region BatchProcessingService [C:4] [TYPE Class]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Create batch records, classify rows, process LLM calls, persist results.
|
||||
class BatchProcessingService:
|
||||
"""Process a batch: classify (cache/preview/LLM), persist, and insert to target."""
|
||||
@@ -42,6 +44,7 @@ class BatchProcessingService:
|
||||
self._llm_service = LLMTranslationService(db)
|
||||
|
||||
# #region process_batch [C:3] [TYPE Function]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Process a single batch: create record, classify rows, call LLM, persist.
|
||||
# @PRE job and batch_rows are valid.
|
||||
# @POST TranslationBatch and TranslationRecord rows are created.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region AdaptiveBatchSizer [C:3] [TYPE Module] [SEMANTICS translate, batch, sizing, token-budget]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Adaptive batch sizing for LLM translation — splits source rows into variable-sized
|
||||
# batches based on actual content length and token budget estimates.
|
||||
# Output budget is the PRIMARY constraint; input budget is secondary.
|
||||
@@ -29,6 +30,7 @@ from ._utils import estimate_row_tokens
|
||||
|
||||
|
||||
# #region AdaptiveBatchSizer [C:3] [TYPE Class]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Split source rows into auto-sized batches based on token budget estimates.
|
||||
class AdaptiveBatchSizer:
|
||||
def __init__(self, db: Session, config_manager: ConfigManager) -> None:
|
||||
@@ -36,6 +38,7 @@ class AdaptiveBatchSizer:
|
||||
self.config_manager = config_manager
|
||||
|
||||
# #region resolve_provider_config [C:2] [TYPE Function] [SEMANTICS llm, provider, model, token-config]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Resolve provider token config (model name + token limits) for budget estimation.
|
||||
# DB values (context_window, max_output_tokens) take priority over PROVIDER_DEFAULTS.
|
||||
# @POST Returns dict with model, context_window, max_output_tokens. None values = use fallback.
|
||||
@@ -52,6 +55,7 @@ class AdaptiveBatchSizer:
|
||||
# #endregion resolve_provider_config
|
||||
|
||||
# #region auto_size_batches [C:3] [TYPE Function] [SEMANTICS translate, batch, sizing]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Split source rows into variable-sized batches based on content length.
|
||||
# Output budget (max_rows_by_output) is the PRIMARY row-count constraint.
|
||||
# @PRE source_rows is non-empty. job has valid config.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region LanguageDetectService [C:2] [TYPE Module] [SEMANTICS translate, language, detection, heuristic]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Local language detection powered by lingua-language-detector (no LLM).
|
||||
# Provides fast, non-LLM language identification for source text rows,
|
||||
# reducing LLM token costs by pre-detecting same-language rows and
|
||||
@@ -64,6 +65,7 @@ def _detector_cache_key(target_languages: list[str] | None) -> str:
|
||||
|
||||
|
||||
# #region build_detector [C:2] [TYPE Function] [SEMANTICS language, detection, builder]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Build a LanguageDetector restricted to target + common source languages.
|
||||
# Restricting the language set improves detection speed (~5x faster than
|
||||
# all 87 languages). The detector includes both target languages (for
|
||||
@@ -95,6 +97,7 @@ def build_detector(target_languages: list[str] | None = None) -> LanguageDetecto
|
||||
|
||||
|
||||
# #region get_detector [C:2] [TYPE Function] [SEMANTICS language, detection, cache]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Get or create a cached detector for the given target languages.
|
||||
# Cache keyed by sorted set of language codes — same key returns same instance.
|
||||
def get_detector(target_languages: list[str] | None = None) -> LanguageDetector:
|
||||
@@ -109,6 +112,7 @@ def get_detector(target_languages: list[str] | None = None) -> LanguageDetector:
|
||||
|
||||
|
||||
# #region detect_language [C:2] [TYPE Function] [SEMANTICS language, detection]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Detect the language of a single text string. Returns BCP-47 code or "und".
|
||||
# @RAISES Does not raise — all exceptions caught internally and return "und".
|
||||
def detect_language(text: str, detector: LanguageDetector) -> str:
|
||||
@@ -178,6 +182,7 @@ def _character_block_fallback(
|
||||
|
||||
|
||||
# #region batch_detect [C:3] [TYPE Function] [SEMANTICS language, detection, batch]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Detect language for multiple texts in batch. Builds detector if not provided.
|
||||
# @RELATION DEPENDS_ON -> [detect_language]
|
||||
# @RELATION DEPENDS_ON -> [get_detector]
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region LLMAsyncHttpClient [C:4] [TYPE Module] [SEMANTICS translate, llm, http, openai, async, retry, rate-limit]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Async HTTP client for OpenAI-compatible LLM API calls using httpx.AsyncClient
|
||||
# with rate-limit handling and structured output fallback.
|
||||
# @LAYER Infrastructure
|
||||
@@ -61,6 +62,7 @@ async def _get_http_client() -> httpx.AsyncClient:
|
||||
|
||||
|
||||
# #region call_openai_compatible [C:3] [TYPE Function] [SEMANTICS translate, llm, http, openai, async]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Call OpenAI-compatible API asynchronously with rate-limit handling and structured output fallback.
|
||||
# @PRE Valid API endpoint, key, model, and prompt.
|
||||
# @POST Returns (response text, finish_reason).
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region LLMTranslationService [C:4] [TYPE Module] [SEMANTICS translate, llm, call, orchestrate, retry]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF LLM interaction for batch translation: call provider with retry, handle truncation
|
||||
# by recursive splitting, enforce dictionary post-processing. Orchestrates HTTP calls
|
||||
# (_llm_http) and response parsing (_llm_parse).
|
||||
@@ -38,6 +39,7 @@ MAX_RETRIES_PER_BATCH = 3
|
||||
|
||||
|
||||
# #region LLMTranslationService [C:4] [TYPE Class]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Call LLM, handle retry/truncation, parse response, persist records.
|
||||
class LLMTranslationService:
|
||||
|
||||
@@ -45,6 +47,7 @@ class LLMTranslationService:
|
||||
self.db = db
|
||||
|
||||
# #region call_llm_for_batch [C:3] [TYPE Function] [SEMANTICS translate, llm, batch, orchestrate]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Call LLM for a batch of rows requiring translation. Parse and persist results.
|
||||
# @PRE job has valid provider_id. batch_rows is non-empty.
|
||||
# @POST Returns dict with successful/failed/skipped counts.
|
||||
@@ -484,6 +487,7 @@ class LLMTranslationService:
|
||||
# #endregion _add_skipped
|
||||
|
||||
# #region call_llm [C:3] [TYPE Function] [SEMANTICS translate, llm, call]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Route to provider-specific LLM call implementation.
|
||||
async def call_llm(self, job: TranslationJob, prompt: str, max_tokens: int = 8192) -> tuple[str, str | None]:
|
||||
with belief_scope("LLMTranslationService.call_llm"):
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region LLMResponseParser [C:3] [TYPE Module] [SEMANTICS translate, llm, parse, json, recovery]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Parse LLM JSON response into per-row translations with support for markdown code
|
||||
# blocks, truncated JSON recovery, and multi-language format.
|
||||
# @LAYER Domain
|
||||
@@ -13,6 +14,7 @@ from ...core.logger import logger
|
||||
|
||||
|
||||
# #region parse_llm_response [C:3] [TYPE Function] [SEMANTICS translate, llm, parse, json]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Parse LLM JSON response into dict of row_id -> per-language translations.
|
||||
# @PRE response_text is valid JSON with {"rows": [...]} structure.
|
||||
# @POST Returns dict mapping row_id to dict with 'detected_source_language' and per-language codes.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region RunExecutionService [C:4] [TYPE Module] [SEMANTICS translate, run, execution, orchestration]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Full run lifecycle: prepare run, fetch source rows, filter new keys, orchestrate batches,
|
||||
# handle cancellation, update per-language stats, finalize run status.
|
||||
# @LAYER Domain
|
||||
@@ -23,6 +24,7 @@ from ._run_source import _extract_chart_data_rows, fetch_source_rows
|
||||
|
||||
|
||||
# #region RunExecutionService [C:4] [TYPE Class]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Orchestrate full translation run: fetch data, process batches, finalize.
|
||||
class RunExecutionService:
|
||||
"""Orchestrate a full translation run: prepare, process batches, finalize."""
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region RunSourceFetcher [C:3] [TYPE Module] [SEMANTICS translate, source, fetch, superset, preview]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Fetch source rows for translation runs from Superset datasource or preview session.
|
||||
# Extracted from RunExecutionService to comply with INV_7 (< 400 lines/module).
|
||||
# @LAYER Domain
|
||||
@@ -24,6 +25,7 @@ MAX_ROWS_PER_RUN = 10000
|
||||
|
||||
|
||||
# #region fetch_source_rows [C:3] [TYPE Function] [SEMANTICS translate, source, fetch]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Fetch source rows from Superset datasource or preview session fallback.
|
||||
async def fetch_source_rows(db: Session, config_manager: ConfigManager, job_id: str, run_id: str) -> list[dict[str, Any]]:
|
||||
"""Fetch source rows from Superset datasource or preview session fallback."""
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region TextCleaner [C:3] [TYPE Module] [SEMANTICS translate, text, cleaning, whitespace, truncation]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Text cleaning utilities for the translation pipeline — whitespace normalization
|
||||
# and character-length truncation. Used as a preprocessing step before LLM calls.
|
||||
# @LAYER Domain
|
||||
@@ -8,6 +9,7 @@
|
||||
# Normalizing whitespace before truncation ensures accurate character counts.
|
||||
|
||||
# #region normalize_whitespace [C:2] [TYPE Function] [SEMANTICS text, whitespace, normalize]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Collapse multiple spaces, tabs, and newlines into single spaces; trim leading/trailing
|
||||
# whitespace. Returns empty string for empty or whitespace-only input.
|
||||
# @PRE text is a string (empty string allowed).
|
||||
@@ -27,6 +29,7 @@ def normalize_whitespace(text: str) -> str:
|
||||
|
||||
|
||||
# #region truncate_text [C:2] [TYPE Function] [SEMANTICS text, truncation, length]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Truncate text to max_length characters, appending "..." if truncation occurs.
|
||||
# @PRE text is a string. max_length >= 0.
|
||||
# @POST When len(text) <= max_length, returns text unchanged.
|
||||
@@ -46,6 +49,7 @@ def truncate_text(text: str, max_length: int = 500) -> str:
|
||||
|
||||
|
||||
# #region clean_text [C:2] [TYPE Function] [SEMANTICS text, clean, normalize, truncate]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Combine whitespace normalization and truncation into one step.
|
||||
# Applies normalize_whitespace first, then truncate_text.
|
||||
# @PRE text is a string. max_length >= 0.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region estimate_token_budget [C:3] [TYPE Module] [SEMANTICS translate, token, budget, estimation, llm]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Calculate safe batch_size and max_tokens for LLM translation calls based on actual content length and model context window limits.
|
||||
# Output budget is now the PRIMARY constraint — finish_reason=length is usually output exceeding max_tokens, not input overflowing context_window.
|
||||
# @LAYER Domain
|
||||
@@ -14,20 +15,24 @@
|
||||
# Input-only batch sizing — output budget is the primary truncation cause (finish_reason=length).
|
||||
|
||||
# #region DEFAULT_CONTEXT_WINDOW [TYPE Constant]
|
||||
# @ingroup Translate
|
||||
|
||||
DEFAULT_CONTEXT_WINDOW = 64000
|
||||
# #endregion DEFAULT_CONTEXT_WINDOW
|
||||
# #region DEFAULT_MAX_OUTPUT_TOKENS [TYPE Constant]
|
||||
# @ingroup Translate
|
||||
|
||||
DEFAULT_MAX_OUTPUT_TOKENS = 16384
|
||||
# #endregion DEFAULT_MAX_OUTPUT_TOKENS
|
||||
# @BRIEF CoT reasoning overhead tokens for DeepSeek models (~2000 tokens for chain-of-thought).
|
||||
# #region REASONING_OVERHEAD [TYPE Constant]
|
||||
# @ingroup Translate
|
||||
|
||||
REASONING_OVERHEAD = 2000
|
||||
# #endregion REASONING_OVERHEAD
|
||||
|
||||
# #region PROVIDER_DEFAULTS [TYPE Constant]
|
||||
# @ingroup Translate
|
||||
|
||||
# Maps model name (or "default" fallback) to capacity limits.
|
||||
# @RATIONALE Different providers have drastically different context windows and
|
||||
@@ -44,35 +49,42 @@ PROVIDER_DEFAULTS: dict[str, dict[str, int]] = {
|
||||
# #endregion PROVIDER_DEFAULTS
|
||||
# Increased from 60 to 120 because SQL/dashboard text and JSON structure need more.
|
||||
# #region OUTPUT_PER_ROW_PER_LANG [TYPE Constant]
|
||||
# @ingroup Translate
|
||||
|
||||
# Increased from 60 to 120 because SQL/dashboard text and JSON structure need more.
|
||||
OUTPUT_PER_ROW_PER_LANG = 120
|
||||
# #endregion OUTPUT_PER_ROW_PER_LANG
|
||||
# #region JSON_OVERHEAD_PER_ROW [TYPE Constant]
|
||||
# @ingroup Translate
|
||||
|
||||
JSON_OVERHEAD_PER_ROW = 50
|
||||
# #endregion JSON_OVERHEAD_PER_ROW
|
||||
# #region PROMPT_BASE_TOKENS [TYPE Constant]
|
||||
# @ingroup Translate
|
||||
|
||||
# Increased from 300 to 600 to account for longer template, system msg, and dict preamble.
|
||||
PROMPT_BASE_TOKENS = 600
|
||||
# #endregion PROMPT_BASE_TOKENS
|
||||
# @BRIEF Cap for dictionary tokens in estimation to avoid overestimating.
|
||||
# #region DICT_TOKENS_PER_ENTRY [TYPE Constant]
|
||||
# @ingroup Translate
|
||||
|
||||
DICT_TOKENS_PER_ENTRY = 20
|
||||
# #endregion DICT_TOKENS_PER_ENTRY
|
||||
|
||||
# #region DICT_TOKENS_MAX [TYPE Constant]
|
||||
# @ingroup Translate
|
||||
|
||||
DICT_TOKENS_MAX = 5000
|
||||
# #endregion DICT_TOKENS_MAX
|
||||
# #region CHARS_PER_TOKEN_MIXED [TYPE Constant]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Deprecated — use CJK_RATIO + OTHER_RATIO instead. Kept for backward compat in estimate_row_tokens.
|
||||
CHARS_PER_TOKEN_MIXED = 2.2
|
||||
# #endregion CHARS_PER_TOKEN_MIXED
|
||||
|
||||
# #region CJK_RATIO [TYPE Constant]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Conservative CJK chars/token ratio. Modern models (Qwen, DeepSeek) tokenize at ~1.0-1.3.
|
||||
# @RATIONALE Lowered from 1.5 to 1.0 to produce more conservative (higher) token estimates.
|
||||
# Actual token density varies by model tokenizer; this is intentionally pessimistic.
|
||||
@@ -80,11 +92,13 @@ CJK_RATIO = 1.0
|
||||
# #endregion CJK_RATIO
|
||||
|
||||
# #region OTHER_RATIO [TYPE Constant]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Conservative ratio for non-CJK text. cl100k_base averages ~1.8-2.5.
|
||||
OTHER_RATIO = 1.8
|
||||
# #endregion OTHER_RATIO
|
||||
|
||||
# #region INPUT_SAFETY_FACTOR [TYPE Constant]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Multiplier applied to per-batch input budget in batch sizer.
|
||||
# Accounts for CJK estimation variance across different tokenizers.
|
||||
# @RATIONALE Single centralised factor instead of scattered margins in multiple functions.
|
||||
@@ -93,15 +107,18 @@ INPUT_SAFETY_FACTOR = 0.75
|
||||
# #endregion INPUT_SAFETY_FACTOR
|
||||
|
||||
# #region OUTPUT_SAFETY_FACTOR [TYPE Constant]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Multiplier applied to output budget when computing max rows per batch.
|
||||
OUTPUT_SAFETY_FACTOR = 0.70
|
||||
# #endregion OUTPUT_SAFETY_FACTOR
|
||||
|
||||
# #region MIN_MAX_TOKENS [TYPE Constant]
|
||||
# @ingroup Translate
|
||||
|
||||
MIN_MAX_TOKENS = 4096
|
||||
# #endregion MIN_MAX_TOKENS
|
||||
# #region MAX_OUTPUT_HEADROOM [TYPE Constant]
|
||||
# @ingroup Translate
|
||||
|
||||
# Increased from 1000 to 3000 because SQL/dashboard text output varies significantly.
|
||||
MAX_OUTPUT_HEADROOM = 3000
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region TranslationUtils [C:3] [TYPE Module] [SEMANTICS translate, utils, hash, dictionary, cache]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Shared utility functions for the translation plugin — dictionary enforcement,
|
||||
# source hashing, cache lookup. Extracted from executor.py to break circular imports.
|
||||
# @LAYER Domain
|
||||
@@ -197,6 +198,7 @@ def _compute_key_hash(source_data: dict) -> str:
|
||||
|
||||
|
||||
# #region estimate_row_tokens [C:2] [TYPE Function] [SEMANTICS translate, token, estimation]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Estimate token count for a single source row including context fields.
|
||||
# @PRE source_text is a string.
|
||||
# @POST Returns estimated token count >= 1.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region DictionaryManagerModule [C:4] [TYPE Module] [SEMANTICS sqlalchemy, translate, dictionary, batch, term]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Business logic for terminology dictionary management, entry CRUD, CSV/TSV import with conflict detection, and per-batch filtering.
|
||||
# @LAYER Domain
|
||||
# @RELATION DEPENDS_ON -> [TranslationJob]
|
||||
@@ -22,6 +23,7 @@ from .dictionary_import_export import DictionaryImportExport
|
||||
|
||||
|
||||
# #region DictionaryManager [C:4] [TYPE Class]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Facade for terminology dictionaries: delegates to DictionaryCRUD, DictionaryEntryCRUD, DictionaryImportExport, DictionaryBatchFilter, DictionaryCorrectionService.
|
||||
# @PRE Database session is open and valid.
|
||||
# @POST Dictionary and entry mutations are persisted with conflict detection.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region DictionaryCorrectionService [C:3] [TYPE Class] [SEMANTICS dictionary, correction, bulk]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Submit term corrections and bulk corrections to dictionaries with conflict detection.
|
||||
# @RELATION DEPENDS_ON -> [DictionaryEntry]
|
||||
# @RELATION DEPENDS_ON -> [TerminologyDictionary]
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region DictionaryCRUD [C:3] [TYPE Class] [SEMANTICS dictionary, crud, terminology]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF CRUD operations for TerminologyDictionary records.
|
||||
# @RELATION DEPENDS_ON -> [TerminologyDictionary]
|
||||
# @RELATION DEPENDS_ON -> [TranslationJob]
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region DictionaryEntryCRUD [C:3] [TYPE Class] [SEMANTICS dictionary, entries, crud]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF CRUD operations for DictionaryEntry records.
|
||||
# @RELATION DEPENDS_ON -> [DictionaryEntry]
|
||||
# @RELATION DEPENDS_ON -> [TerminologyDictionary]
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region DictionaryBatchFilter [C:3] [TYPE Class] [SEMANTICS dictionary, filter, batch, matching]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Scan batch texts for case-insensitive, word-boundary-aware matches against dictionaries.
|
||||
# @RELATION DEPENDS_ON -> [DictionaryEntry]
|
||||
# @RELATION DEPENDS_ON -> [TranslationJobDictionary]
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region DictionaryImportExport [C:3] [TYPE Class] [SEMANTICS dictionary, import, export, csv, migration]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Import/export entries as CSV/TSV with conflict detection, and migration of old entries.
|
||||
# @RELATION DEPENDS_ON -> [DictionaryEntry]
|
||||
# @RELATION DEPENDS_ON -> [TerminologyDictionary]
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region TranslationEventLog [C:5] [TYPE Module] [SEMANTICS sqlalchemy, translate, event, log, audit]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Structured event logging for translation operations with terminal event invariant enforcement.
|
||||
# @LAYER Domain
|
||||
# @RELATION DEPENDS_ON -> [TranslationEvent]
|
||||
@@ -38,6 +39,7 @@ DEFAULT_RETENTION_DAYS = 90
|
||||
|
||||
|
||||
# #region TranslationEventLog [C:5] [TYPE Class]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Structured event logging for translation operations with terminal event invariant enforcement.
|
||||
# @PRE Database session is available.
|
||||
# @POST Events are written immutably; terminal events enforced per run.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region TranslationExecutor [C:4] [TYPE Module] [SEMANTICS sqlalchemy, translate, orchestration]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Process translation in batches: fetch source rows, call LLM, persist TranslationBatch
|
||||
# and TranslationRecord rows. Thin orchestrator — delegates to focused sub-services.
|
||||
# @LAYER Domain
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region TranslationMetrics [C:3] [TYPE Module] [SEMANTICS sqlalchemy, translate, metrics, job, statistics]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Aggregate translation metrics from live TranslationEvent + MetricSnapshot for per-job reporting.
|
||||
# @LAYER Domain
|
||||
# @RELATION DEPENDS_ON -> [TranslationSchedule]
|
||||
@@ -23,6 +24,7 @@ from ...models.translate import (
|
||||
|
||||
|
||||
# #region TranslationMetrics [C:3] [TYPE Class]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Aggregate translation metrics from live events and MetricSnapshot.
|
||||
class TranslationMetrics:
|
||||
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region TranslationOrchestrator [C:5] [TYPE Module] [SEMANTICS sqlalchemy, tenacity, translate, orchestration, batch]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Run lifecycle coordination: validate preconditions, dispatch executor, generate SQL, submit to Superset, record events.
|
||||
# @LAYER Domain
|
||||
# @RELATION DEPENDS_ON -> [TranslationRun]
|
||||
@@ -34,6 +35,7 @@ from .orchestrator_runner import TranslationStageRunner
|
||||
|
||||
|
||||
# #region TranslationOrchestrator [C:5] [TYPE Class]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Coordinates full translation run lifecycle: validation, execution, SQL generation, Superset submission, event logging.
|
||||
# @PRE DB session and config manager are available.
|
||||
# @POST Runs are created, executed, and finalized with event records.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region TranslationResultAggregator [C:4] [TYPE Class] [SEMANTICS sqlalchemy, translate, query, history, records]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Query translation run status, records, and history.
|
||||
# @PRE Database session is available.
|
||||
# @POST Returns structured run data with pagination.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region orchestrator_cancel [C:3] [TYPE Module] [SEMANTICS translate, cancel, retry_insert]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Cancel and retry-insert operations for translation runs.
|
||||
# @RELATION DEPENDS_ON -> [TranslationRun]
|
||||
# @RELATION DEPENDS_ON -> [TranslationJob]
|
||||
@@ -37,6 +38,7 @@ def _fallback_cancel_request(db: Session, run_id: str) -> TranslationRun:
|
||||
|
||||
|
||||
# #region cancel_run [C:3] [TYPE Function] [SEMANTICS translate, cancel, run]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Cancel a running translation run.
|
||||
# @SIDE_EFFECT DB writes; event log.
|
||||
# @RELATION DEPENDS_ON -> [TranslationRun]
|
||||
@@ -80,6 +82,7 @@ def cancel_run(
|
||||
|
||||
|
||||
# #region retry_insert [C:3] [TYPE Function] [SEMANTICS translate, retry, insert]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Retry the SQL insert phase for a completed run.
|
||||
# @SIDE_EFFECT Superset API call; DB writes.
|
||||
# @RELATION DEPENDS_ON -> [TranslationRun]
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region orchestrator_config [C:3] [TYPE Module] [SEMANTICS hash, config, snapshot, translate]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Config hash and dictionary snapshot hash utilities for translation planning.
|
||||
# @RELATION DEPENDS_ON -> [TranslationJob]
|
||||
# @RELATION DEPENDS_ON -> [TranslationJobDictionary]
|
||||
@@ -8,6 +9,7 @@ import json
|
||||
|
||||
|
||||
# #region compute_config_hash [C:2] [TYPE Function] [SEMANTICS config, hash, deterministic]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Compute a deterministic hash of job configuration for snapshot comparison.
|
||||
# @PRE job has valid configuration attributes.
|
||||
# @POST Returns 16-char hex SHA-256 hash.
|
||||
@@ -29,6 +31,7 @@ def compute_config_hash(job) -> str:
|
||||
|
||||
|
||||
# #region compute_dict_snapshot_hash [C:2] [TYPE Function] [SEMANTICS dictionary, hash, snapshot]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Compute a hash of dictionary state for snapshot comparison.
|
||||
# Includes dictionary IDs, entry count, and max entry updated_at
|
||||
# to invalidate cache when dictionary entries are modified.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region TranslationExecutionEngine [C:4] [TYPE Class] [SEMANTICS translate, execution, run, orchestration]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Execute translation runs: dispatch executor, manage completion and failure paths.
|
||||
# @PRE Database session and config manager are available. Run is in valid state.
|
||||
# @POST Run is executed with SQL generated; events recorded.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region update_language_stats [C:3] [TYPE Function] [SEMANTICS translate, language, stats, aggregation]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Aggregate TranslationLanguage entries by language_code and update TranslationRunLanguageStats.
|
||||
# @SIDE_EFFECT DB writes on language_stats objects.
|
||||
# @RELATION DEPENDS_ON -> [TranslationRecord]
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region TranslationPlanner [C:4] [TYPE Class] [SEMANTICS sqlalchemy, translate, orchestration, planning]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Handle translation run planning: validation, config hashing, dictionary snapshot.
|
||||
# @PRE Database session and config manager are available.
|
||||
# @POST TranslationRun is planned with hashes and config snapshot; events recorded.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region orchestrator_query [C:3] [TYPE Module] [SEMANTICS translate, query, records, history]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Query translation run records and history with pagination.
|
||||
# @RELATION DEPENDS_ON -> [TranslationRecord]
|
||||
# @RELATION DEPENDS_ON -> [TranslationRun]
|
||||
@@ -11,6 +12,7 @@ from ...models.translate import TranslationRecord, TranslationRun
|
||||
|
||||
|
||||
# #region get_run_records [C:2] [TYPE Function] [SEMANTICS records, query, paginated]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Get paginated records for a run. Supports deduplicate=true to return unique source_hash rows.
|
||||
def get_run_records(
|
||||
db: Session,
|
||||
@@ -106,6 +108,7 @@ def get_run_records(
|
||||
# #endregion get_run_records
|
||||
|
||||
# #region get_run_history [C:2] [TYPE Function] [SEMANTICS history, query, paginated]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Get run history for a job with pagination. Returns trigger_type, cache_hits for enhanced run list display.
|
||||
def get_run_history(
|
||||
db: Session,
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region TranslationRunRetryManager [C:4] [TYPE Class] [SEMANTICS translate, retry]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Manage retry of translation run batches.
|
||||
# @PRE Database session and config manager are available.
|
||||
# @POST Retried batches are re-executed.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region orchestrator_run_completion [C:3] [TYPE Module] [SEMANTICS translate, run, completion, failure]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Post-execution run completion handlers: cancelled, success-with-insert, and failure paths.
|
||||
# @RELATION DEPENDS_ON -> [TranslationRun]
|
||||
# @RELATION DEPENDS_ON -> [TranslationJob]
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region TranslationStageRunner [C:4] [TYPE Class] [SEMANTICS sqlalchemy, translate, execution, superset]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Coordinate translation run execution, retry, and cancellation via delegated components.
|
||||
# @PRE Database session and config manager are available. Run is in valid state.
|
||||
# @POST Run is executed; events recorded.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region SQLInsertService [C:4] [TYPE Class] [SEMANTICS sql, insert, superset, translate]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Generate INSERT SQL from translation records and submit to Superset SQL Lab.
|
||||
# @PRE Job has target table configured. Run has successful records.
|
||||
# @POST SQL is generated and submitted to Superset; returns execution result.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region orchestrator_sql_rows [C:3] [TYPE Module] [SEMANTICS sql, insert, row, column, builders]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Column and row building utilities for SQL insertion in translate plugin.
|
||||
# @RELATION DEPENDS_ON -> [TranslationRecord]
|
||||
# @RELATION DEPENDS_ON -> [TranslationJob]
|
||||
@@ -10,6 +11,7 @@ from .sql_generator import _normalize_timestamp_value
|
||||
|
||||
|
||||
# #region build_columns [C:2] [TYPE Function] [SEMANTICS sql, columns, build]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Build list of target columns for SQL INSERT from job configuration.
|
||||
def build_columns(job: TranslationJob, effective_target: str | None) -> list[str]:
|
||||
"""Build column list for SQL INSERT from job config."""
|
||||
@@ -48,6 +50,7 @@ def build_context_keys(job: TranslationJob, effective_target: str | None) -> lis
|
||||
|
||||
|
||||
# #region build_rows [C:2] [TYPE Function] [SEMANTICS sql, rows, build]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Build row data for SQL INSERT with per-language expansion.
|
||||
# @SIDE_EFFECT Reads translation record language entries.
|
||||
def build_rows(
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region validate_job_preconditions [C:3] [TYPE Function] [SEMANTICS validation, preconditions, translate]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Validate preconditions before starting a translation run.
|
||||
# @PRE Job exists and db session is valid.
|
||||
# @POST Raises ValueError if any precondition fails.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region TranslatePlugin [C:2] [TYPE Module] [SEMANTICS clickhouse, translate, sql, dashboard, dialect]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF TranslatePlugin skeleton for cross-dialect SQL and dashboard translation.
|
||||
# @LAYER Domain
|
||||
# @RELATION INHERITS -> [PluginBase]
|
||||
@@ -11,6 +12,7 @@ from ...core.plugin_base import PluginBase
|
||||
|
||||
|
||||
# #region TranslatePlugin [TYPE Class]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Plugin for translating SQL queries and dashboard definitions across database dialects.
|
||||
# @RELATION IMPLEMENTS -> [PluginBase]
|
||||
class TranslatePlugin(PluginBase):
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region TranslationPreview [C:4] [TYPE Module] [SEMANTICS sqlalchemy, translate, preview, llm, review]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Preview session management: fetch sample rows from Superset, send to LLM with context + filtered dictionary, return side-by-side results.
|
||||
# @LAYER Domain
|
||||
# @RELATION DEPENDS_ON -> [TranslationJob]
|
||||
@@ -43,6 +44,7 @@ from .preview_review import PreviewSessionManager
|
||||
|
||||
|
||||
# #region TranslationPreview [C:4] [TYPE Class]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Manages preview lifecycle: fetch sample rows, call LLM, manage row-level approve/edit/reject, accept gate.
|
||||
# @PRE Database session and config manager are available.
|
||||
# @POST Preview sessions created with persisted records; full execution gates on accepted session.
|
||||
@@ -131,7 +133,7 @@ class TranslationPreview:
|
||||
llm_response = await self._executor.call_llm(
|
||||
job=job, prompt=prompt_data["prompt"], max_tokens=token_budget["max_output_needed"],
|
||||
)
|
||||
logger.reason(f"TIMING: LLM call: {time.monotonic() - t_llm:.2f}s", {})
|
||||
logger.reason(f"TIMING: LLM call: {time.monotonic() - t_llm:.2f}s")
|
||||
|
||||
translations = self._executor.parse_llm_response(
|
||||
llm_response, prompt_data["actual_row_count"], target_languages=target_languages,
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region PreviewExecutor [C:4] [TYPE Class] [SEMANTICS sqlalchemy, superset, llm, preview]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Fetch sample data from Superset and call LLM provider for preview.
|
||||
# @PRE Database session, config manager, and Superset client are available.
|
||||
# @POST Sample rows fetched, LLM called.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region PreviewPromptBuilder [C:3] [TYPE Class] [SEMANTICS prompt, dictionary, preview]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Build LLM prompts for preview sessions including dictionary glossary and row context.
|
||||
# @RELATION DEPENDS_ON -> [DictionaryManager]
|
||||
# @RELATION DEPENDS_ON -> [render_prompt]
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region preview_prompt_helpers [C:2] [TYPE Module] [SEMANTICS token, budget, metadata, estimate]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Token estimation metadata and budget helpers for preview prompt building.
|
||||
# @RELATION DEPENDS_ON -> [estimate_token_budget]
|
||||
# @RELATION DEPENDS_ON -> [TokenEstimator]
|
||||
@@ -12,6 +13,7 @@ from .preview_token_estimator import TokenEstimator
|
||||
|
||||
|
||||
# #region estimate_token_budget_for_rows [C:2] [TYPE Function] [SEMANTICS token, budget, estimate, truncate]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Check token budget and optionally truncate source rows for preview.
|
||||
# @POST Returns adjusted source_rows, actual_row_count, and token_budget dict.
|
||||
def estimate_token_budget_for_rows(
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region resolve_provider_model [C:2] [TYPE Function] [SEMANTICS llm, provider, model, resolve]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Resolve the LLM provider model name for a translation job.
|
||||
# @RELATION DEPENDS_ON -> [LLMProviderService]
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region preview_response_parser [C:3] [TYPE Module] [SEMANTICS llm, parse, json, superset, hash]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Parse LLM JSON responses and Superset data; compute config/dict hashes for preview.
|
||||
# @RELATION DEPENDS_ON -> [TranslationJob]
|
||||
# @RELATION DEPENDS_ON -> [TranslationJobDictionary]
|
||||
@@ -14,6 +15,7 @@ from ...models.translate import TranslationJob, TranslationJobDictionary
|
||||
|
||||
|
||||
# #region parse_llm_response [C:3] [TYPE Function] [SEMANTICS llm, parse, json, translate]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Parse LLM JSON response into structured translations dict with per-language support.
|
||||
# @PRE response_text is a valid JSON string (possibly wrapped in markdown code block).
|
||||
# @POST Returns dict mapping row_id -> {detected_source_language, lang_code: translation, ...}.
|
||||
@@ -122,6 +124,7 @@ def compute_config_hash(job: TranslationJob) -> str:
|
||||
|
||||
|
||||
# #region compute_dict_snapshot_hash [C:2] [TYPE Function] [SEMANTICS dictionary, hash, snapshot]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Compute a hash of dictionary state for snapshot comparison.
|
||||
# Includes dictionary IDs, entry count, and max entry updated_at.
|
||||
def compute_dict_snapshot_hash(db: Session, job_id: str) -> str:
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region PreviewSessionManager [C:3] [TYPE Class] [SEMANTICS preview, session, review, approve]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Manage preview session row-level actions: approve/edit/reject rows.
|
||||
# @RELATION DEPENDS_ON -> [TranslationPreviewSession]
|
||||
# @RELATION DEPENDS_ON -> [TranslationPreviewRecord]
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region preview_session_ops [C:3] [TYPE Module] [SEMANTICS preview, session, accept, get]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Preview session lifecycle operations: accept and query preview sessions.
|
||||
# @RELATION DEPENDS_ON -> [TranslationPreviewSession]
|
||||
# @RELATION DEPENDS_ON -> [TranslationPreviewRecord]
|
||||
@@ -21,6 +22,7 @@ from .preview_session_serializer import (
|
||||
|
||||
|
||||
# #region accept_preview_session [C:2] [TYPE Function] [SEMANTICS preview, session, accept]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Mark a preview session as accepted, which gates full execution.
|
||||
# @SIDE_EFFECT DB writes on session status.
|
||||
def accept_preview_session(db: Session, job_id: str, _current_user: str | None = None) -> dict[str, Any]:
|
||||
@@ -60,6 +62,7 @@ def accept_preview_session(db: Session, job_id: str, _current_user: str | None =
|
||||
|
||||
|
||||
# #region get_preview_session [C:2] [TYPE Function] [SEMANTICS preview, session, query]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Get the latest preview session for a job with its records.
|
||||
def get_preview_session(db: Session, job_id: str) -> dict[str, Any]:
|
||||
"""Get the latest preview session for a job with its records."""
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region preview_session_serializer [C:3] [TYPE Module] [SEMANTICS preview, record, serialize, action]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Serialization and action helpers for preview sessions: record serialization and per-row approve/reject/edit actions.
|
||||
# @RELATION DEPENDS_ON -> [TranslationPreviewRecord]
|
||||
# @RELATION DEPENDS_ON -> [TranslationPreviewLanguage]
|
||||
@@ -45,6 +46,7 @@ def serialize_preview_record(r: TranslationPreviewRecord) -> dict[str, Any]:
|
||||
|
||||
|
||||
# #region apply_language_action [C:2] [TYPE Function] [SEMANTICS preview, language, action]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Apply approve/reject/edit action to a TranslationPreviewLanguage entry.
|
||||
def apply_language_action(
|
||||
lang_entry: TranslationPreviewLanguage,
|
||||
@@ -68,6 +70,7 @@ def apply_language_action(
|
||||
|
||||
|
||||
# #region apply_record_action [C:2] [TYPE Function] [SEMANTICS preview, record, action]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Apply approve/reject/edit action to a TranslationPreviewRecord and its language entries.
|
||||
def apply_record_action(
|
||||
record: TranslationPreviewRecord,
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region TokenEstimator [C:2] [TYPE Class] [SEMANTICS tokens, cost, estimation]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Estimate token counts and costs for LLM translation operations.
|
||||
# @RATIONALE Token estimation uses a heuristic (chars/token ratio) since exact tokenization depends on the LLM model.
|
||||
# @REJECTED Using an external tokenizer library would introduce a heavy dependency for estimation only.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region ContextAwarePromptBuilder [C:2] [TYPE Module] [SEMANTICS translate, prompt, context, dictionary]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Pure-function prompt builder that enhances dictionary entries with context annotations.
|
||||
# @LAYER Domain
|
||||
# @RELATION DEPENDS_ON -> [DictionaryEntry]
|
||||
@@ -15,6 +16,7 @@ import json
|
||||
from typing import Any
|
||||
|
||||
# #region ContextAwarePromptBuilder [C:2] [TYPE Class]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Build LLM prompts with context-aware dictionary entries and similarity-based priority.
|
||||
|
||||
class ContextAwarePromptBuilder:
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region TranslationScheduler [C:4] [TYPE Module] [SEMANTICS sqlalchemy, translate, schedule, cron, job]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Manage TranslationSchedule rows and register them with core SchedulerService.
|
||||
# @LAYER Domain
|
||||
# @RELATION DEPENDS_ON -> [TranslationSchedule]
|
||||
@@ -26,6 +27,7 @@ from .events import TranslationEventLog
|
||||
|
||||
|
||||
# #region TranslationScheduler [C:4] [TYPE Class]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF CRUD for TranslationSchedule rows + APScheduler registration wrappers.
|
||||
class TranslationScheduler:
|
||||
|
||||
@@ -241,6 +243,7 @@ class TranslationScheduler:
|
||||
|
||||
|
||||
# #region execute_scheduled_translation [C:4] [TYPE Function]
|
||||
# @ingroup Translate
|
||||
# @BRIEF APScheduler job handler — runs scheduled translation with concurrency check and new-key-only fallback.
|
||||
# @PRE schedule_id is valid.
|
||||
# @POST Translation run created and executed if no concurrent run exists.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region TranslateJobService [C:4] [TYPE Module] [SEMANTICS sqlalchemy, translate, job, datasource, dialect]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Service layer for translation job CRUD with datasource column validation and database dialect detection.
|
||||
# @LAYER Domain
|
||||
# @RELATION DEPENDS_ON -> [TranslationJob]
|
||||
@@ -26,6 +27,7 @@ from .service_utils import _extract_dialect
|
||||
|
||||
|
||||
# #region TranslateJobService [TYPE Class]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Service for translation job CRUD with validation and Superset integration.
|
||||
class TranslateJobService:
|
||||
"""CRUD service for translation jobs with Superset datasource integration."""
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region BulkFindReplaceService [C:3] [TYPE Class] [SEMANTICS translate, bulk, find, replace, regex]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Service for bulk find-and-replace operations on translated values.
|
||||
# @RELATION DEPENDS_ON -> [TranslationLanguage]
|
||||
# @RELATION DEPENDS_ON -> [TranslationRecord]
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region DatasourceMetadataService [C:4] [TYPE Module] [SEMANTICS superset, datasource, columns, dialect]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Fetch datasource column metadata and database dialect from Superset.
|
||||
# @LAYER Domain
|
||||
# @RELATION DEPENDS_ON -> [SupersetClient]
|
||||
@@ -22,6 +23,7 @@ SUPPORTED_DIALECTS = {
|
||||
|
||||
|
||||
# #region get_dialect_from_database [C:2] [TYPE Function]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Extract normalized dialect string from a Superset database record.
|
||||
def get_dialect_from_database(database_record: dict[str, Any]) -> str:
|
||||
"""Extract and validate dialect from Superset database record."""
|
||||
@@ -56,6 +58,7 @@ def get_dialect_from_database(database_record: dict[str, Any]) -> str:
|
||||
|
||||
|
||||
# #region fetch_datasource_metadata [C:3] [TYPE Function]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Fetch datasource columns and database dialect from Superset.
|
||||
async def fetch_datasource_metadata(
|
||||
dataset_id: int,
|
||||
@@ -105,6 +108,7 @@ async def fetch_datasource_metadata(
|
||||
|
||||
|
||||
# #region detect_virtual_columns [C:2] [TYPE Function]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Identify virtual (calculated) columns from column metadata.
|
||||
def detect_virtual_columns(columns: list[dict[str, Any]]) -> list[str]:
|
||||
"""Return names of columns that are virtual (not physical)."""
|
||||
@@ -113,6 +117,7 @@ def detect_virtual_columns(columns: list[dict[str, Any]]) -> list[str]:
|
||||
|
||||
|
||||
# #region get_datasource_columns [C:3] [TYPE Function]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Fetch datasource column metadata from Superset and return structured response.
|
||||
async def get_datasource_columns(
|
||||
datasource_id: int,
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region InlineCorrectionService [C:4] [TYPE Class] [SEMANTICS translate, correction, inline, dictionary]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Service for inline editing translated values and submitting corrections to dictionaries.
|
||||
# @PRE Database session is available.
|
||||
# @POST Inline edits applied with optional dictionary submission.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region TargetSchemaValidation [C:4] [TYPE Module] [SEMANTICS translate, schema, validation, target-table]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Проверка схемы целевой таблицы: запрос колонок через Superset SQL Lab,
|
||||
# сравнение с ожидаемыми (из build_columns), возврат diff.
|
||||
# @LAYER Service
|
||||
@@ -181,6 +182,7 @@ def _parse_sqllab_result(result: dict[str, Any]) -> tuple[list[dict[str, Any]],
|
||||
|
||||
|
||||
# #region validate_target_table_schema [C:4] [TYPE Function] [SEMANTICS translate, schema, validate, orchestrate]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Основная функция: проверяет схему целевой таблицы через Superset SQL Lab.
|
||||
# @PRE Superset окружение и target_database_id валидны.
|
||||
# @POST Возвращает TargetSchemaValidationResponse с diff-анализом.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region SQLGenerator [C:3] [TYPE Module] [SEMANTICS clickhouse, translate, sql, insert, generate]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Dialect-aware safe SQL generation for INSERT/UPSERT operations.
|
||||
# @LAYER Domain
|
||||
# @RELATION DEPENDS_ON -> [TranslationJob]
|
||||
@@ -142,6 +143,7 @@ def _build_values_clause(columns: list[str], rows: list[dict[str, Any]], dialect
|
||||
|
||||
|
||||
# #region generate_insert_sql [C:3] [TYPE Function] [SEMANTICS translate,sql,insert]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Generate a dialect-aware plain INSERT SQL statement for the given table, columns, and rows.
|
||||
# @RELATION DEPENDS_ON -> [_quote_identifier]
|
||||
# @RELATION DEPENDS_ON -> [_build_values_clause]
|
||||
@@ -174,6 +176,7 @@ def generate_insert_sql(
|
||||
|
||||
|
||||
# #region generate_upsert_sql [C:4] [TYPE Function]
|
||||
# @ingroup Translate
|
||||
# @BRIEF Generate PostgreSQL dialect UPSERT SQL with ON CONFLICT DO UPDATE.
|
||||
# @PRE dialect is postgresql-compatible. target_table, columns, key_columns are non-empty.
|
||||
# @POST Returns UPSERT SQL string or raises ValueError.
|
||||
@@ -229,6 +232,7 @@ def generate_upsert_sql(
|
||||
|
||||
|
||||
# #region SQLGenerator [C:3] [TYPE Class]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Generate safe, dialect-appropriate SQL INSERT/UPSERT statements.
|
||||
# @PRE Job has target_schema, target_table, key columns configured.
|
||||
# @POST Returns generated SQL string for the target dialect.
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
# #region SupersetSqlLabExecutor [C:4] [TYPE Module] [SEMANTICS translate, superset, sqllab, execute, query]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Submit SQL to Superset SQL Lab API and poll execution status.
|
||||
# @LAYER Infrastructure
|
||||
# @RELATION DEPENDS_ON -> [ConfigManager]
|
||||
@@ -21,6 +22,7 @@ from ...core.utils.client_registry import get_superset_client
|
||||
|
||||
|
||||
# #region SupersetSqlLabExecutor [C:4] [TYPE Class]
|
||||
# @defgroup Translate Module group.
|
||||
# @BRIEF Submit SQL to Superset SQL Lab API with polling and status tracking.
|
||||
# @PRE Valid environment ID and ConfigManager.
|
||||
# @POST SQL submitted to Superset; execution reference recorded.
|
||||
|
||||
Reference in New Issue
Block a user