feat: attention-optimized semantic protocol v2.7

Core changes:
- Add @defgroup/@ingroup to 1791 C2+ contracts (555 files) for HCA 128× pre-training DSA grouping
- Add §0.1 Pre-Training Frequency matrix to semantics-core
- Add §VIII Attention Architecture rules (ATTN_1-4) with MLA/CSA/HCA/DSA mechanics
- Add @defgroup/@ingroup to canonical syntax (§II) and all contract examples

Agent prompts (5 files):
- Add ZERO-STATE RATIONALE with MLA/CSA/HCA/DSA compression mechanics
- Add pre-training note: @RATIONALE/@REJECTED are in-context learned tags
- svelte-coder: add missing #region contract, fix Svelte rule violations
- python-coder/fullstack-coder: honor function contracts from speckit plan
- qa-tester: add attention compliance audit (P3 ATTN_1-4 checks)

Skills (6 files):
- Translate all axiom_config descriptions to English
- Fix doc_dirs to index .opencode/ and .specify/
- Deduplicate 5× complexity_rules → single global_tags catalog
- Reduce semantics-svelte 591→485 lines (remove duplicate code blocks)
- Fix semantics-testing: 'Short IDs' → 'Short hierarchical IDs'
- Fix all examples: flat IDs → hierarchical Domain.Name format
- Fix Svelte examples: replace raw Tailwind + <button> with semantic tokens + /ui

Speckit workflow (commands + templates):
- speckit.plan: add Function-Level Contracts for C3+ with @PRE/@POST/@TEST_EDGE
- speckit.plan: add Attention Compliance Gate (ATTN_1-4 before contract generation)
- speckit.tasks: add function contract inlining format (constraints in task description)
- speckit.specify: load semantics-core for spec density rules
- spec-template: add #region contract, @SEMANTICS grouping, hierarchical IDs
- ux-reference-template: add #region wrapper
- plan-template: add attention gate, @defgroup/@ingroup guidance
- tasks-template: add attention audit + rebuild + orphan check tasks
- constitution.md: translate to English, add Principle VIII (attention-optimized contracts)

Reference modules rewritten (hierarchical IDs + full contracts):
- Auth.Jwt: 6 child contracts with @RATIONALE/@REJECTED/@TEST_EDGE
- Api.Auth: 5 endpoints with @TEST_EDGE + molecular CoT markers
- Migration.Model: @defgroup Migration with 18 @ACTION + 6 @INVARIANT

Scripts:
- add_defgroup_ingroup.py: zero-risk additive @ingroup migration (1791 insertions)
- migrate_hierarchical.py: flat→hierarchical ID dry-run analysis (792 contracts)
- merge_prompts.py: merge all prompts/skills/commands into one review file

Config:
- axiom_config.yaml: 749→395 lines (-47%), English, doc_dirs include prompts
- Fix test_datasets.py import collision (rename → test_datasets_routes.py)
- Fix test_preview.py: SupersetClient→get_superset_client, AsyncMock, logger f-string
This commit is contained in:
2026-06-08 16:30:59 +03:00
parent 73e809e189
commit 8e8a3c3235
562 changed files with 2523 additions and 244 deletions

View File

@@ -1,3 +1,4 @@
# #region src.plugins [TYPE Package] [SEMANTICS plugin, package, discovery, runtime]
# @ingroup Plugin
# @BRIEF Plugin package root for dynamic discovery and runtime imports.
# #endregion src.plugins

View File

@@ -1,4 +1,5 @@
# #region BackupPlugin [TYPE Module] [SEMANTICS backup, export, archive, superset, dashboard]
# @defgroup Plugin Module group.
# @BRIEF A plugin that provides functionality to back up Superset dashboards.
# @LAYER App
# @RELATION IMPLEMENTS -> PluginBase
@@ -22,6 +23,7 @@ from ..dependencies import get_config_manager
# #region BackupPlugin [TYPE Class]
# @defgroup Plugin Module group.
# @BRIEF Implementation of the backup plugin logic.
class BackupPlugin(PluginBase):
"""

View File

@@ -1,4 +1,5 @@
# #region DebugPluginModule [TYPE Module] [SEMANTICS debug, diagnostic, superset, api, system]
# @defgroup Plugin Module group.
# @BRIEF Implements a plugin for system diagnostics and debugging Superset API responses.
# @LAYER Plugin
# @BRIEF Plugin for diagnostics. Inherits PluginBase.
@@ -13,6 +14,7 @@ from ..core.utils.client_registry import get_superset_client
# #region DebugPlugin [TYPE Class]
# @defgroup Plugin Module group.
# @BRIEF Plugin for system diagnostics and debugging.
class DebugPlugin(PluginBase):
"""

View File

@@ -1,3 +1,4 @@
# #region GitPluginExt [TYPE Package] [SEMANTICS git, plugin, extension, package]
# @ingroup Git
# @BRIEF Git plugin extension package root.
# #endregion GitPluginExt

View File

@@ -1,4 +1,5 @@
# #region GitLLMExtensionModule [C:3] [TYPE Module] [SEMANTICS git, llm, commit, message, generation]
# @defgroup Git Module group.
# @BRIEF LLM-based extensions for the Git plugin, specifically for commit message generation.
# @LAYER Domain
# @RELATION DEPENDS_ON -> [LLMClient]
@@ -12,6 +13,7 @@ from ..llm_analysis.service import LLMClient
# #region GitLLMExtension [TYPE Class]
# @defgroup Git Module group.
# @BRIEF Provides LLM capabilities to the Git plugin.
class GitLLMExtension:
def __init__(self, client: LLMClient):

View File

@@ -1,4 +1,5 @@
# #region GitPluginModule [C:4] [TYPE Module] [SEMANTICS git, versioning, deploy, sync, superset, backup, transactional]
# @defgroup Plugin Module group.
#
# @BRIEF Предоставляет плагин для версионирования и развертывания дашбордов Superset.
# @LAYER Plugin
@@ -123,6 +124,7 @@ def _pack_deploy_zip(repo_path: Path, root_dir_name: str, zip_buffer: io.BytesIO
# #region GitPlugin [TYPE Class]
# @defgroup Plugin Module group.
# @BRIEF Реализация плагина Git Integration для управления версиями дашбордов.
class GitPlugin(PluginBase):

View File

@@ -1,4 +1,5 @@
# #region LLMAnalysisPackage [TYPE Module] [SEMANTICS llm, analysis, plugin, package, export]
# @defgroup LLMAnalysis Module group.
"""
LLM Analysis Plugin for automated dashboard validation and dataset documentation.

View File

@@ -1,4 +1,5 @@
# #region DashboardTopology [C:3] [TYPE Module] [SEMANTICS superset, topology, dashboard, layout, parsing]
# @defgroup LLMAnalysis Module group.
# @BRIEF Dashboard topology extraction — parses Superset position_json into human-readable
# text descriptions of Tab → Row → Chart hierarchy for LLM prompt context.
# @LAYER Plugin
@@ -227,6 +228,7 @@ def _extract_topology_children(
# #region build_dashboard_topology [C:2] [TYPE Function]
# @ingroup LLMAnalysis
# @BRIEF Build a structured text description of dashboard layout from position_json.
# @PARAM dashboard (dict) - Dashboard metadata dict with position_json, dashboard_title.
# @PARAM charts (list[dict]) - List of fetched chart detail dicts for name/viz lookup.

View File

@@ -1,4 +1,5 @@
# #region V1ToV2Migration [C:3] [TYPE Module] [SEMANTICS migration,validation,data,idempotent]
# @defgroup LLMAnalysis Module group.
# @BRIEF Migration script: v1 → v2 ValidationPolicy data. Converts existing dashboard_ids
# JSON arrays to ValidationSource relational rows.
# @PRE Database has both validation_policies and validation_sources tables.

View File

@@ -1,4 +1,5 @@
# #region LLMAnalysisModels [C:3] [TYPE Module] [SEMANTICS pydantic, llm, plugin, model, provider-type]
# @defgroup LLMAnalysis Module group.
# @BRIEF Define Pydantic models for LLM Analysis plugin.
# @LAYER Domain
# @RELATION DEPENDS_ON -> [EXT:Library:pydantic]
@@ -10,6 +11,7 @@ from pydantic import BaseModel, Field
# #region LLMProviderType [TYPE Class]
# @defgroup LLMAnalysis Module group.
# @BRIEF Enum for supported LLM providers.
class LLMProviderType(str, Enum):
OPENAI = "openai"
@@ -19,6 +21,7 @@ class LLMProviderType(str, Enum):
# #endregion LLMProviderType
# #region LLMProviderConfig [TYPE Class]
# @defgroup LLMAnalysis Module group.
# @BRIEF Configuration for an LLM provider.
class LLMProviderConfig(BaseModel):
id: str | None = None
@@ -41,6 +44,7 @@ class LLMProviderConfig(BaseModel):
# #endregion LLMProviderConfig
# #region ValidationStatus [TYPE Class]
# @defgroup LLMAnalysis Module group.
# @BRIEF Enum for dashboard validation status.
class ValidationStatus(str, Enum):
PASS = "PASS"
@@ -50,6 +54,7 @@ class ValidationStatus(str, Enum):
# #endregion ValidationStatus
# #region DetectedIssue [TYPE Class]
# @defgroup LLMAnalysis Module group.
# @BRIEF Model for a single issue detected during validation.
class DetectedIssue(BaseModel):
severity: ValidationStatus
@@ -58,6 +63,7 @@ class DetectedIssue(BaseModel):
# #endregion DetectedIssue
# #region ValidationResult [TYPE Class]
# @defgroup LLMAnalysis Module group.
# @BRIEF Model for dashboard validation result.
class ValidationResult(BaseModel):
id: str | None = None

View File

@@ -1,4 +1,5 @@
# #region LLMAnalysisPlugin [C:5] [TYPE Module] [SEMANTICS llm, analysis, dashboard, validation, documentation]
# @defgroup LLMAnalysis Module group.
# @BRIEF Implements DashboardValidationPlugin and DocumentationPlugin.
# @LAYER Plugin
# @RELATION INHERITS -> [PluginBase]
@@ -159,6 +160,7 @@ def _update_run_status(db, run_id: str | None) -> None:
# #region DashboardValidationPlugin [TYPE Class]
# @defgroup LLMAnalysis Module group.
# @BRIEF Plugin for automated dashboard health analysis using LLMs.
# @RELATION IMPLEMENTS -> [PluginBase]
class DashboardValidationPlugin(PluginBase):
@@ -846,6 +848,7 @@ class DashboardValidationPlugin(PluginBase):
# #region DocumentationPlugin [TYPE Class]
# @defgroup LLMAnalysis Module group.
# @BRIEF Plugin for automated dataset documentation using LLMs.
# @RELATION IMPLEMENTS -> [PluginBase]
class DocumentationPlugin(PluginBase):

View File

@@ -1,4 +1,5 @@
# #region LLMAnalysisScheduler [C:3] [TYPE Module] [SEMANTICS scheduler, llm, validation, task, cron]
# @defgroup LLMAnalysis Module group.
# @BRIEF Provides helper functions to schedule LLM-based validation tasks.
# @LAYER Domain
# @RELATION DEPENDS_ON -> [SchedulerService]
@@ -10,6 +11,7 @@ from ...dependencies import get_scheduler_service, get_task_manager
# #region schedule_dashboard_validation [TYPE Function]
# @ingroup LLMAnalysis
# @BRIEF Schedules a recurring dashboard validation task.
# @SIDE_EFFECT Adds a job to the scheduler service.
def schedule_dashboard_validation(dashboard_id: str, cron_expression: str, params: dict[str, Any]):

View File

@@ -1,4 +1,5 @@
# #region LLMAnalysisService [C:5] [TYPE Module] [SEMANTICS llm, screenshot, playwright, openai, tenacity]
# @defgroup LLMAnalysis Module group.
# @BRIEF Services for LLM interaction and dashboard screenshots.
# @LAYER Plugin
# @RELATION DEPENDS_ON -> [EXT:Library:tenacity]
@@ -37,6 +38,7 @@ LLM_HTTP_TIMEOUT_S = 120 # seconds (httpx client timeout)
DEFAULT_USER_AGENT = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36"
# #region ScreenshotService [TYPE Class]
# @defgroup LLMAnalysis Module group.
# @BRIEF Handles capturing screenshots of Superset dashboards.
class ScreenshotService:
# region ScreenshotService.__init__ [TYPE Function]
@@ -837,6 +839,7 @@ class ScreenshotService:
# #endregion ScreenshotService
# #region LLMClient [TYPE Class]
# @defgroup LLMAnalysis Module group.
# @BRIEF Wrapper for LLM provider APIs.
class LLMClient:
# region LLMClient.__init__ [TYPE Function]
@@ -1441,6 +1444,7 @@ class LLMClient:
# #endregion LLMClient
# #region DatasetHealthChecker [C:3] [TYPE Class]
# @defgroup LLMAnalysis Module group.
# @BRIEF Checks dataset accessibility and KXD connectivity via Superset API.
# @LAYER Service
# @RELATION CALLS -> [SupersetClient]
@@ -1641,6 +1645,7 @@ class DatasetHealthChecker:
# #endregion DatasetHealthChecker
# #region RedactionService [C:2] [TYPE Module]
# @defgroup LLMAnalysis Module group.
# @BRIEF Redacts PII, credentials, and sensitive data from logs and LLM responses.
# @LAYER Service
# @RATIONALE FR-029: sensitive data must be filtered BEFORE external LLM send and BEFORE persistence.

View File

@@ -1,4 +1,5 @@
# #region MaintenanceBannerPlugin [C:4] [TYPE Module] [SEMANTICS plugin, maintenance, banner, execution, task]
# @defgroup Plugin Module group.
# @BRIEF TaskManager plugin for executing maintenance banner operations (start, end, end-all).
# Dispatches to MaintenanceService based on operation type in params.
# @LAYER Plugin
@@ -21,6 +22,7 @@ from ..services.maintenance import end_all_maintenance, end_maintenance, start_m
# #region MaintenanceBannerPlugin [C:4] [TYPE Class]
# @defgroup Plugin Module group.
# @BRIEF Plugin for executing maintenance banner operations asynchronously via TaskManager.
# @RELATION CALLS -> [start_maintenance]
# @RELATION CALLS -> [end_maintenance]
@@ -29,36 +31,42 @@ class MaintenanceBannerPlugin(PluginBase):
"""TaskManager plugin for maintenance banner operations."""
# #region id [TYPE Property]
# @ingroup Plugin
@property
def id(self) -> str:
return "maintenance_banner_apply"
# #endregion id
# #region name [TYPE Property]
# @ingroup Plugin
@property
def name(self) -> str:
return "Maintenance Banner"
# #endregion name
# #region description [TYPE Property]
# @ingroup Plugin
@property
def description(self) -> str:
return "Applies or removes maintenance banners on Superset dashboards."
# #endregion description
# #region version [TYPE Property]
# @ingroup Plugin
@property
def version(self) -> str:
return "1.0.0"
# #endregion version
# #region ui_route [TYPE Property]
# @ingroup Plugin
@property
def ui_route(self) -> str:
return "" # No UI route — async-only
# #endregion ui_route
# #region get_schema [TYPE Function]
# @ingroup Plugin
def get_schema(self) -> dict[str, Any]:
return {
"type": "object",
@@ -82,6 +90,7 @@ class MaintenanceBannerPlugin(PluginBase):
# #endregion get_schema
# #region execute [TYPE Function] [C:4]
# @ingroup Plugin
# @BRIEF Execute maintenance operation based on params.operation.
# @PARAM params contains: operation (start|end|end_all), event_id, environment_id.
# @PARAM context TaskContext for logging.

View File

@@ -1,4 +1,5 @@
# #region MapperPluginModule [TYPE Module] [SEMANTICS mapper, dataset, column, sqllab, excel, mapping]
# @defgroup Plugin Module group.
# @BRIEF Implements a plugin for mapping dataset columns using Superset SQL Lab or Excel files.
# @LAYER Plugin
# @BRIEF Plugin for dataset column mapping. Inherits PluginBase.
@@ -14,6 +15,7 @@ from ..core.utils.dataset_mapper import DatasetMapper
# #region MapperPlugin [TYPE Class]
# @defgroup Plugin Module group.
# @BRIEF Plugin for mapping dataset columns verbose names.
class MapperPlugin(PluginBase):
"""

View File

@@ -1,4 +1,5 @@
# #region MigrationPlugin [C:5] [TYPE Module] [SEMANTICS migration, export, import, mapping, superset]
# @defgroup Plugin Module group.
# @BRIEF Orchestrates export, DB-mapping transformation, and import of Superset dashboards across environments.
# @LAYER App
# @RELATION IMPLEMENTS -> PluginBase
@@ -28,6 +29,7 @@ from ..models.mapping import DatabaseMapping, Environment
# #region MigrationPlugin [TYPE Class]
# @defgroup Plugin Module group.
# @BRIEF Implementation of the migration plugin workflow and transformation orchestration.
# @PRE SupersetClient authenticated, database session active
# @POST Returns MigrationResult with success/failure status and artifact list

View File

@@ -1,4 +1,5 @@
# #region SearchPluginModule [TYPE Module] [SEMANTICS search, dataset, text, pattern, superset]
# @defgroup Plugin Module group.
# @BRIEF Implements a plugin for searching text patterns across all datasets in a specific Superset environment.
# @LAYER Plugin
# @BRIEF Plugin for text search across datasets. Inherits PluginBase.
@@ -14,6 +15,7 @@ from ..core.task_manager.context import TaskContext
# #region SearchPlugin [TYPE Class]
# @defgroup Plugin Module group.
# @BRIEF Plugin for searching text patterns in Superset datasets.
class SearchPlugin(PluginBase):
"""

View File

@@ -1,4 +1,5 @@
# #region StoragePluginPackage [TYPE Package] [SEMANTICS storage, plugin, package, export]
# @ingroup Storage
from .plugin import StoragePlugin
__all__ = ["StoragePlugin"]

View File

@@ -1,4 +1,5 @@
# #region StoragePlugin [TYPE Module] [SEMANTICS fastapi, storage, filesystem, backup, archive]
# @defgroup Storage Module group.
# @BRIEF Provides core filesystem operations for managing backups and repositories.
# @LAYER App
# @RELATION DEPENDS_ON -> [TaskContext]
@@ -34,6 +35,7 @@ def _copy_upload_to_disk(dest_path: Path, file_obj) -> None:
# #region StoragePlugin [TYPE Class]
# @defgroup Storage Module group.
# @BRIEF Implementation of the storage management plugin.
class StoragePlugin(PluginBase):
"""

View File

@@ -1,2 +1,3 @@
# #region TranslatePluginPackage [TYPE Package] [SEMANTICS translate, plugin, package, sql, dashboard]
# @ingroup Translate
# #endregion TranslatePluginPackage

View File

@@ -7,14 +7,15 @@
from datetime import UTC, datetime, timedelta
import json
import pytest
from unittest.mock import MagicMock, patch
from unittest.mock import AsyncMock, MagicMock, patch
from src.models.translate import (
TranslationJob,
TranslationPreviewRecord,
TranslationPreviewSession,
)
from src.plugins.translate.preview import TokenEstimator, TranslationPreview
from src.plugins.translate.preview import TranslationPreview
from src.plugins.translate.preview_token_estimator import TokenEstimator
from src.plugins.translate.preview_executor import PreviewExecutor
@@ -59,7 +60,8 @@ class TestTranslationPreview:
# region test_preview_valid_job [TYPE Function]
# @PURPOSE: Verify preview creates session and records successfully.
def test_preview_valid_job(self):
@pytest.mark.asyncio
async def test_preview_valid_job(self):
"""Preview with valid job should create session and return records."""
db = MagicMock()
config_manager = MagicMock()
@@ -79,21 +81,21 @@ class TestTranslationPreview:
config_manager.get_environments.return_value = [env]
# Mock SupersetClient
with patch("src.plugins.translate.preview_executor.SupersetClient") as MockClient:
with patch("src.plugins.translate.preview_executor.get_superset_client") as MockClient:
mock_client = MagicMock()
MockClient.return_value = mock_client
# Mock dataset_detail
mock_client.get_dataset_detail.return_value = {
# Mock dataset_detail (async method)
mock_client.get_dataset_detail = AsyncMock(return_value={
"table_name": "test_table",
"columns": [
{"column_name": "title", "type": "VARCHAR"},
{"column_name": "category", "type": "VARCHAR"},
{"column_name": "description", "type": "TEXT"},
],
}
})
# Mock build_dataset_preview_query_context
# Mock build_dataset_preview_query_context (NOT async)
mock_client.build_dataset_preview_query_context.return_value = {
"datasource": {"id": 42, "type": "table"},
"queries": [{
@@ -107,8 +109,8 @@ class TestTranslationPreview:
"force": True,
}
# Mock network request to Superset
mock_client.network.request.return_value = {
# Mock network request to Superset (async)
mock_client.network.request = AsyncMock(return_value={
"result": [{
"status": "success",
"data": [
@@ -116,7 +118,7 @@ class TestTranslationPreview:
{"title": "Goodbye", "category": "farewell", "description": "A parting message"},
],
}]
}
})
# Mock LLM provider
with patch("src.services.llm_provider.LLMProviderService") as MockProviderService:
@@ -140,7 +142,7 @@ class TestTranslationPreview:
mock_filter.return_value = []
preview_service = TranslationPreview(db, config_manager, "test-user")
result = preview_service.preview_rows(job_id="job-123", sample_size=10)
result = await preview_service.preview_rows(job_id="job-123", sample_size=10)
# Verify result structure
assert result["job_id"] == "job-123"
@@ -165,7 +167,8 @@ class TestTranslationPreview:
# region test_preview_with_dictionary [TYPE Function]
# @PURPOSE: Verify glossary entries from dictionary are included in LLM prompt.
def test_preview_with_dictionary(self):
@pytest.mark.asyncio
async def test_preview_with_dictionary(self):
"""Preview should include dictionary glossary in the prompt sent to LLM."""
db = MagicMock()
config_manager = MagicMock()
@@ -177,10 +180,10 @@ class TestTranslationPreview:
env.id = "postgresql"
config_manager.get_environments.return_value = [env]
with patch("src.plugins.translate.preview_executor.SupersetClient") as MockClient:
with patch("src.plugins.translate.preview_executor.get_superset_client") as MockClient:
mock_client = MagicMock()
MockClient.return_value = mock_client
mock_client.get_dataset_detail.return_value = {"table_name": "test", "columns": []}
mock_client.get_dataset_detail = AsyncMock(return_value={"table_name": "test", "columns": []})
mock_client.build_dataset_preview_query_context.return_value = {
"datasource": {"id": 42, "type": "table"},
"queries": [{"columns": [], "metrics": ["count"], "row_limit": 10}],
@@ -189,9 +192,9 @@ class TestTranslationPreview:
"result_type": "query",
"force": True,
}
mock_client.network.request.return_value = {
mock_client.network.request = AsyncMock(return_value={
"result": [{"status": "success", "data": [{"title": "Hello World", "category": "greeting"}]}]
}
})
with patch("src.services.llm_provider.LLMProviderService") as MockProviderService:
mock_provider_svc = MagicMock()
@@ -215,7 +218,7 @@ class TestTranslationPreview:
]
preview_service = TranslationPreview(db, config_manager, "test-user")
result = preview_service.preview_rows(job_id="job-123", sample_size=5)
result = await preview_service.preview_rows(job_id="job-123", sample_size=5)
# Verify LLM was called
mock_llm.assert_called_once()
@@ -508,7 +511,8 @@ class TestTranslationPreview:
# region test_preview_missing_datasource [TYPE Function]
# @PURPOSE: Verify error when job has no datasource configured.
def test_preview_missing_datasource(self):
@pytest.mark.asyncio
async def test_preview_missing_datasource(self):
"""Preview should fail if job has no datasource."""
db = MagicMock()
config_manager = MagicMock()
@@ -518,12 +522,13 @@ class TestTranslationPreview:
preview_service = TranslationPreview(db, config_manager, "test-user")
with pytest.raises(ValueError, match="source datasource"):
preview_service.preview_rows(job_id="job-123")
await preview_service.preview_rows(job_id="job-123")
# endregion test_preview_missing_datasource
# region test_preview_missing_translation_column [TYPE Function]
# @PURPOSE: Verify error when job has no translation column.
def test_preview_missing_translation_column(self):
@pytest.mark.asyncio
async def test_preview_missing_translation_column(self):
"""Preview should fail if job has no translation column."""
db = MagicMock()
config_manager = MagicMock()
@@ -533,7 +538,7 @@ class TestTranslationPreview:
preview_service = TranslationPreview(db, config_manager, "test-user")
with pytest.raises(ValueError, match="translation column"):
preview_service.preview_rows(job_id="job-123")
await preview_service.preview_rows(job_id="job-123")
# endregion test_preview_missing_translation_column

View File

@@ -1,4 +1,5 @@
# #region BatchInsertService [C:3] [TYPE Module] [SEMANTICS translate, insert, sqllab, upsert, target]
# @defgroup Translate Module group.
# @BRIEF Insert successful translation records into target table via Superset SQL Lab.
# Builds INSERT SQL (original + per-language rows), resolves backend dialect,
# and executes via SupersetSqlLabExecutor.
@@ -24,6 +25,7 @@ from .superset_executor import SupersetSqlLabExecutor
# #region insert_batch_to_target [C:3] [TYPE Function] [SEMANTICS translate, insert, orchestrate]
# @ingroup Translate
# @BRIEF Insert successful records from a single batch into the target table.
async def insert_batch_to_target(
db: Session, config_manager: ConfigManager,

View File

@@ -1,4 +1,5 @@
# #region BatchProcessingService [C:4] [TYPE Module] [SEMANTICS translate, batch, process, classify, cache]
# @defgroup Translate Module group.
# @BRIEF Batch processing for translation: classify rows (same-language/cache/preview/LLM),
# call LLM service, persist TranslationRecord/TranslationLanguage rows.
# Local language detection (lingua) replaces LLM-based detection.
@@ -32,6 +33,7 @@ from .dictionary import DictionaryManager
# #region BatchProcessingService [C:4] [TYPE Class]
# @defgroup Translate Module group.
# @BRIEF Create batch records, classify rows, process LLM calls, persist results.
class BatchProcessingService:
"""Process a batch: classify (cache/preview/LLM), persist, and insert to target."""
@@ -42,6 +44,7 @@ class BatchProcessingService:
self._llm_service = LLMTranslationService(db)
# #region process_batch [C:3] [TYPE Function]
# @ingroup Translate
# @BRIEF Process a single batch: create record, classify rows, call LLM, persist.
# @PRE job and batch_rows are valid.
# @POST TranslationBatch and TranslationRecord rows are created.

View File

@@ -1,4 +1,5 @@
# #region AdaptiveBatchSizer [C:3] [TYPE Module] [SEMANTICS translate, batch, sizing, token-budget]
# @defgroup Translate Module group.
# @BRIEF Adaptive batch sizing for LLM translation — splits source rows into variable-sized
# batches based on actual content length and token budget estimates.
# Output budget is the PRIMARY constraint; input budget is secondary.
@@ -29,6 +30,7 @@ from ._utils import estimate_row_tokens
# #region AdaptiveBatchSizer [C:3] [TYPE Class]
# @defgroup Translate Module group.
# @BRIEF Split source rows into auto-sized batches based on token budget estimates.
class AdaptiveBatchSizer:
def __init__(self, db: Session, config_manager: ConfigManager) -> None:
@@ -36,6 +38,7 @@ class AdaptiveBatchSizer:
self.config_manager = config_manager
# #region resolve_provider_config [C:2] [TYPE Function] [SEMANTICS llm, provider, model, token-config]
# @ingroup Translate
# @BRIEF Resolve provider token config (model name + token limits) for budget estimation.
# DB values (context_window, max_output_tokens) take priority over PROVIDER_DEFAULTS.
# @POST Returns dict with model, context_window, max_output_tokens. None values = use fallback.
@@ -52,6 +55,7 @@ class AdaptiveBatchSizer:
# #endregion resolve_provider_config
# #region auto_size_batches [C:3] [TYPE Function] [SEMANTICS translate, batch, sizing]
# @ingroup Translate
# @BRIEF Split source rows into variable-sized batches based on content length.
# Output budget (max_rows_by_output) is the PRIMARY row-count constraint.
# @PRE source_rows is non-empty. job has valid config.

View File

@@ -1,4 +1,5 @@
# #region LanguageDetectService [C:2] [TYPE Module] [SEMANTICS translate, language, detection, heuristic]
# @defgroup Translate Module group.
# @BRIEF Local language detection powered by lingua-language-detector (no LLM).
# Provides fast, non-LLM language identification for source text rows,
# reducing LLM token costs by pre-detecting same-language rows and
@@ -64,6 +65,7 @@ def _detector_cache_key(target_languages: list[str] | None) -> str:
# #region build_detector [C:2] [TYPE Function] [SEMANTICS language, detection, builder]
# @ingroup Translate
# @BRIEF Build a LanguageDetector restricted to target + common source languages.
# Restricting the language set improves detection speed (~5x faster than
# all 87 languages). The detector includes both target languages (for
@@ -95,6 +97,7 @@ def build_detector(target_languages: list[str] | None = None) -> LanguageDetecto
# #region get_detector [C:2] [TYPE Function] [SEMANTICS language, detection, cache]
# @ingroup Translate
# @BRIEF Get or create a cached detector for the given target languages.
# Cache keyed by sorted set of language codes — same key returns same instance.
def get_detector(target_languages: list[str] | None = None) -> LanguageDetector:
@@ -109,6 +112,7 @@ def get_detector(target_languages: list[str] | None = None) -> LanguageDetector:
# #region detect_language [C:2] [TYPE Function] [SEMANTICS language, detection]
# @ingroup Translate
# @BRIEF Detect the language of a single text string. Returns BCP-47 code or "und".
# @RAISES Does not raise — all exceptions caught internally and return "und".
def detect_language(text: str, detector: LanguageDetector) -> str:
@@ -178,6 +182,7 @@ def _character_block_fallback(
# #region batch_detect [C:3] [TYPE Function] [SEMANTICS language, detection, batch]
# @ingroup Translate
# @BRIEF Detect language for multiple texts in batch. Builds detector if not provided.
# @RELATION DEPENDS_ON -> [detect_language]
# @RELATION DEPENDS_ON -> [get_detector]

View File

@@ -1,4 +1,5 @@
# #region LLMAsyncHttpClient [C:4] [TYPE Module] [SEMANTICS translate, llm, http, openai, async, retry, rate-limit]
# @defgroup Translate Module group.
# @BRIEF Async HTTP client for OpenAI-compatible LLM API calls using httpx.AsyncClient
# with rate-limit handling and structured output fallback.
# @LAYER Infrastructure
@@ -61,6 +62,7 @@ async def _get_http_client() -> httpx.AsyncClient:
# #region call_openai_compatible [C:3] [TYPE Function] [SEMANTICS translate, llm, http, openai, async]
# @ingroup Translate
# @BRIEF Call OpenAI-compatible API asynchronously with rate-limit handling and structured output fallback.
# @PRE Valid API endpoint, key, model, and prompt.
# @POST Returns (response text, finish_reason).

View File

@@ -1,4 +1,5 @@
# #region LLMTranslationService [C:4] [TYPE Module] [SEMANTICS translate, llm, call, orchestrate, retry]
# @defgroup Translate Module group.
# @BRIEF LLM interaction for batch translation: call provider with retry, handle truncation
# by recursive splitting, enforce dictionary post-processing. Orchestrates HTTP calls
# (_llm_http) and response parsing (_llm_parse).
@@ -38,6 +39,7 @@ MAX_RETRIES_PER_BATCH = 3
# #region LLMTranslationService [C:4] [TYPE Class]
# @defgroup Translate Module group.
# @BRIEF Call LLM, handle retry/truncation, parse response, persist records.
class LLMTranslationService:
@@ -45,6 +47,7 @@ class LLMTranslationService:
self.db = db
# #region call_llm_for_batch [C:3] [TYPE Function] [SEMANTICS translate, llm, batch, orchestrate]
# @ingroup Translate
# @BRIEF Call LLM for a batch of rows requiring translation. Parse and persist results.
# @PRE job has valid provider_id. batch_rows is non-empty.
# @POST Returns dict with successful/failed/skipped counts.
@@ -484,6 +487,7 @@ class LLMTranslationService:
# #endregion _add_skipped
# #region call_llm [C:3] [TYPE Function] [SEMANTICS translate, llm, call]
# @ingroup Translate
# @BRIEF Route to provider-specific LLM call implementation.
async def call_llm(self, job: TranslationJob, prompt: str, max_tokens: int = 8192) -> tuple[str, str | None]:
with belief_scope("LLMTranslationService.call_llm"):

View File

@@ -1,4 +1,5 @@
# #region LLMResponseParser [C:3] [TYPE Module] [SEMANTICS translate, llm, parse, json, recovery]
# @defgroup Translate Module group.
# @BRIEF Parse LLM JSON response into per-row translations with support for markdown code
# blocks, truncated JSON recovery, and multi-language format.
# @LAYER Domain
@@ -13,6 +14,7 @@ from ...core.logger import logger
# #region parse_llm_response [C:3] [TYPE Function] [SEMANTICS translate, llm, parse, json]
# @ingroup Translate
# @BRIEF Parse LLM JSON response into dict of row_id -> per-language translations.
# @PRE response_text is valid JSON with {"rows": [...]} structure.
# @POST Returns dict mapping row_id to dict with 'detected_source_language' and per-language codes.

View File

@@ -1,4 +1,5 @@
# #region RunExecutionService [C:4] [TYPE Module] [SEMANTICS translate, run, execution, orchestration]
# @defgroup Translate Module group.
# @BRIEF Full run lifecycle: prepare run, fetch source rows, filter new keys, orchestrate batches,
# handle cancellation, update per-language stats, finalize run status.
# @LAYER Domain
@@ -23,6 +24,7 @@ from ._run_source import _extract_chart_data_rows, fetch_source_rows
# #region RunExecutionService [C:4] [TYPE Class]
# @defgroup Translate Module group.
# @BRIEF Orchestrate full translation run: fetch data, process batches, finalize.
class RunExecutionService:
"""Orchestrate a full translation run: prepare, process batches, finalize."""

View File

@@ -1,4 +1,5 @@
# #region RunSourceFetcher [C:3] [TYPE Module] [SEMANTICS translate, source, fetch, superset, preview]
# @defgroup Translate Module group.
# @BRIEF Fetch source rows for translation runs from Superset datasource or preview session.
# Extracted from RunExecutionService to comply with INV_7 (< 400 lines/module).
# @LAYER Domain
@@ -24,6 +25,7 @@ MAX_ROWS_PER_RUN = 10000
# #region fetch_source_rows [C:3] [TYPE Function] [SEMANTICS translate, source, fetch]
# @ingroup Translate
# @BRIEF Fetch source rows from Superset datasource or preview session fallback.
async def fetch_source_rows(db: Session, config_manager: ConfigManager, job_id: str, run_id: str) -> list[dict[str, Any]]:
"""Fetch source rows from Superset datasource or preview session fallback."""

View File

@@ -1,4 +1,5 @@
# #region TextCleaner [C:3] [TYPE Module] [SEMANTICS translate, text, cleaning, whitespace, truncation]
# @defgroup Translate Module group.
# @BRIEF Text cleaning utilities for the translation pipeline — whitespace normalization
# and character-length truncation. Used as a preprocessing step before LLM calls.
# @LAYER Domain
@@ -8,6 +9,7 @@
# Normalizing whitespace before truncation ensures accurate character counts.
# #region normalize_whitespace [C:2] [TYPE Function] [SEMANTICS text, whitespace, normalize]
# @ingroup Translate
# @BRIEF Collapse multiple spaces, tabs, and newlines into single spaces; trim leading/trailing
# whitespace. Returns empty string for empty or whitespace-only input.
# @PRE text is a string (empty string allowed).
@@ -27,6 +29,7 @@ def normalize_whitespace(text: str) -> str:
# #region truncate_text [C:2] [TYPE Function] [SEMANTICS text, truncation, length]
# @ingroup Translate
# @BRIEF Truncate text to max_length characters, appending "..." if truncation occurs.
# @PRE text is a string. max_length >= 0.
# @POST When len(text) <= max_length, returns text unchanged.
@@ -46,6 +49,7 @@ def truncate_text(text: str, max_length: int = 500) -> str:
# #region clean_text [C:2] [TYPE Function] [SEMANTICS text, clean, normalize, truncate]
# @ingroup Translate
# @BRIEF Combine whitespace normalization and truncation into one step.
# Applies normalize_whitespace first, then truncate_text.
# @PRE text is a string. max_length >= 0.

View File

@@ -1,4 +1,5 @@
# #region estimate_token_budget [C:3] [TYPE Module] [SEMANTICS translate, token, budget, estimation, llm]
# @defgroup Translate Module group.
# @BRIEF Calculate safe batch_size and max_tokens for LLM translation calls based on actual content length and model context window limits.
# Output budget is now the PRIMARY constraint — finish_reason=length is usually output exceeding max_tokens, not input overflowing context_window.
# @LAYER Domain
@@ -14,20 +15,24 @@
# Input-only batch sizing — output budget is the primary truncation cause (finish_reason=length).
# #region DEFAULT_CONTEXT_WINDOW [TYPE Constant]
# @ingroup Translate
DEFAULT_CONTEXT_WINDOW = 64000
# #endregion DEFAULT_CONTEXT_WINDOW
# #region DEFAULT_MAX_OUTPUT_TOKENS [TYPE Constant]
# @ingroup Translate
DEFAULT_MAX_OUTPUT_TOKENS = 16384
# #endregion DEFAULT_MAX_OUTPUT_TOKENS
# @BRIEF CoT reasoning overhead tokens for DeepSeek models (~2000 tokens for chain-of-thought).
# #region REASONING_OVERHEAD [TYPE Constant]
# @ingroup Translate
REASONING_OVERHEAD = 2000
# #endregion REASONING_OVERHEAD
# #region PROVIDER_DEFAULTS [TYPE Constant]
# @ingroup Translate
# Maps model name (or "default" fallback) to capacity limits.
# @RATIONALE Different providers have drastically different context windows and
@@ -44,35 +49,42 @@ PROVIDER_DEFAULTS: dict[str, dict[str, int]] = {
# #endregion PROVIDER_DEFAULTS
# Increased from 60 to 120 because SQL/dashboard text and JSON structure need more.
# #region OUTPUT_PER_ROW_PER_LANG [TYPE Constant]
# @ingroup Translate
# Increased from 60 to 120 because SQL/dashboard text and JSON structure need more.
OUTPUT_PER_ROW_PER_LANG = 120
# #endregion OUTPUT_PER_ROW_PER_LANG
# #region JSON_OVERHEAD_PER_ROW [TYPE Constant]
# @ingroup Translate
JSON_OVERHEAD_PER_ROW = 50
# #endregion JSON_OVERHEAD_PER_ROW
# #region PROMPT_BASE_TOKENS [TYPE Constant]
# @ingroup Translate
# Increased from 300 to 600 to account for longer template, system msg, and dict preamble.
PROMPT_BASE_TOKENS = 600
# #endregion PROMPT_BASE_TOKENS
# @BRIEF Cap for dictionary tokens in estimation to avoid overestimating.
# #region DICT_TOKENS_PER_ENTRY [TYPE Constant]
# @ingroup Translate
DICT_TOKENS_PER_ENTRY = 20
# #endregion DICT_TOKENS_PER_ENTRY
# #region DICT_TOKENS_MAX [TYPE Constant]
# @ingroup Translate
DICT_TOKENS_MAX = 5000
# #endregion DICT_TOKENS_MAX
# #region CHARS_PER_TOKEN_MIXED [TYPE Constant]
# @ingroup Translate
# @BRIEF Deprecated — use CJK_RATIO + OTHER_RATIO instead. Kept for backward compat in estimate_row_tokens.
CHARS_PER_TOKEN_MIXED = 2.2
# #endregion CHARS_PER_TOKEN_MIXED
# #region CJK_RATIO [TYPE Constant]
# @ingroup Translate
# @BRIEF Conservative CJK chars/token ratio. Modern models (Qwen, DeepSeek) tokenize at ~1.0-1.3.
# @RATIONALE Lowered from 1.5 to 1.0 to produce more conservative (higher) token estimates.
# Actual token density varies by model tokenizer; this is intentionally pessimistic.
@@ -80,11 +92,13 @@ CJK_RATIO = 1.0
# #endregion CJK_RATIO
# #region OTHER_RATIO [TYPE Constant]
# @ingroup Translate
# @BRIEF Conservative ratio for non-CJK text. cl100k_base averages ~1.8-2.5.
OTHER_RATIO = 1.8
# #endregion OTHER_RATIO
# #region INPUT_SAFETY_FACTOR [TYPE Constant]
# @ingroup Translate
# @BRIEF Multiplier applied to per-batch input budget in batch sizer.
# Accounts for CJK estimation variance across different tokenizers.
# @RATIONALE Single centralised factor instead of scattered margins in multiple functions.
@@ -93,15 +107,18 @@ INPUT_SAFETY_FACTOR = 0.75
# #endregion INPUT_SAFETY_FACTOR
# #region OUTPUT_SAFETY_FACTOR [TYPE Constant]
# @ingroup Translate
# @BRIEF Multiplier applied to output budget when computing max rows per batch.
OUTPUT_SAFETY_FACTOR = 0.70
# #endregion OUTPUT_SAFETY_FACTOR
# #region MIN_MAX_TOKENS [TYPE Constant]
# @ingroup Translate
MIN_MAX_TOKENS = 4096
# #endregion MIN_MAX_TOKENS
# #region MAX_OUTPUT_HEADROOM [TYPE Constant]
# @ingroup Translate
# Increased from 1000 to 3000 because SQL/dashboard text output varies significantly.
MAX_OUTPUT_HEADROOM = 3000

View File

@@ -1,4 +1,5 @@
# #region TranslationUtils [C:3] [TYPE Module] [SEMANTICS translate, utils, hash, dictionary, cache]
# @defgroup Translate Module group.
# @BRIEF Shared utility functions for the translation plugin — dictionary enforcement,
# source hashing, cache lookup. Extracted from executor.py to break circular imports.
# @LAYER Domain
@@ -197,6 +198,7 @@ def _compute_key_hash(source_data: dict) -> str:
# #region estimate_row_tokens [C:2] [TYPE Function] [SEMANTICS translate, token, estimation]
# @ingroup Translate
# @BRIEF Estimate token count for a single source row including context fields.
# @PRE source_text is a string.
# @POST Returns estimated token count >= 1.

View File

@@ -1,4 +1,5 @@
# #region DictionaryManagerModule [C:4] [TYPE Module] [SEMANTICS sqlalchemy, translate, dictionary, batch, term]
# @defgroup Translate Module group.
# @BRIEF Business logic for terminology dictionary management, entry CRUD, CSV/TSV import with conflict detection, and per-batch filtering.
# @LAYER Domain
# @RELATION DEPENDS_ON -> [TranslationJob]
@@ -22,6 +23,7 @@ from .dictionary_import_export import DictionaryImportExport
# #region DictionaryManager [C:4] [TYPE Class]
# @defgroup Translate Module group.
# @BRIEF Facade for terminology dictionaries: delegates to DictionaryCRUD, DictionaryEntryCRUD, DictionaryImportExport, DictionaryBatchFilter, DictionaryCorrectionService.
# @PRE Database session is open and valid.
# @POST Dictionary and entry mutations are persisted with conflict detection.

View File

@@ -1,4 +1,5 @@
# #region DictionaryCorrectionService [C:3] [TYPE Class] [SEMANTICS dictionary, correction, bulk]
# @defgroup Translate Module group.
# @BRIEF Submit term corrections and bulk corrections to dictionaries with conflict detection.
# @RELATION DEPENDS_ON -> [DictionaryEntry]
# @RELATION DEPENDS_ON -> [TerminologyDictionary]

View File

@@ -1,4 +1,5 @@
# #region DictionaryCRUD [C:3] [TYPE Class] [SEMANTICS dictionary, crud, terminology]
# @defgroup Translate Module group.
# @BRIEF CRUD operations for TerminologyDictionary records.
# @RELATION DEPENDS_ON -> [TerminologyDictionary]
# @RELATION DEPENDS_ON -> [TranslationJob]

View File

@@ -1,4 +1,5 @@
# #region DictionaryEntryCRUD [C:3] [TYPE Class] [SEMANTICS dictionary, entries, crud]
# @defgroup Translate Module group.
# @BRIEF CRUD operations for DictionaryEntry records.
# @RELATION DEPENDS_ON -> [DictionaryEntry]
# @RELATION DEPENDS_ON -> [TerminologyDictionary]

View File

@@ -1,4 +1,5 @@
# #region DictionaryBatchFilter [C:3] [TYPE Class] [SEMANTICS dictionary, filter, batch, matching]
# @defgroup Translate Module group.
# @BRIEF Scan batch texts for case-insensitive, word-boundary-aware matches against dictionaries.
# @RELATION DEPENDS_ON -> [DictionaryEntry]
# @RELATION DEPENDS_ON -> [TranslationJobDictionary]

View File

@@ -1,4 +1,5 @@
# #region DictionaryImportExport [C:3] [TYPE Class] [SEMANTICS dictionary, import, export, csv, migration]
# @defgroup Translate Module group.
# @BRIEF Import/export entries as CSV/TSV with conflict detection, and migration of old entries.
# @RELATION DEPENDS_ON -> [DictionaryEntry]
# @RELATION DEPENDS_ON -> [TerminologyDictionary]

View File

@@ -1,4 +1,5 @@
# #region TranslationEventLog [C:5] [TYPE Module] [SEMANTICS sqlalchemy, translate, event, log, audit]
# @defgroup Translate Module group.
# @BRIEF Structured event logging for translation operations with terminal event invariant enforcement.
# @LAYER Domain
# @RELATION DEPENDS_ON -> [TranslationEvent]
@@ -38,6 +39,7 @@ DEFAULT_RETENTION_DAYS = 90
# #region TranslationEventLog [C:5] [TYPE Class]
# @defgroup Translate Module group.
# @BRIEF Structured event logging for translation operations with terminal event invariant enforcement.
# @PRE Database session is available.
# @POST Events are written immutably; terminal events enforced per run.

View File

@@ -1,4 +1,5 @@
# #region TranslationExecutor [C:4] [TYPE Module] [SEMANTICS sqlalchemy, translate, orchestration]
# @defgroup Translate Module group.
# @BRIEF Process translation in batches: fetch source rows, call LLM, persist TranslationBatch
# and TranslationRecord rows. Thin orchestrator — delegates to focused sub-services.
# @LAYER Domain

View File

@@ -1,4 +1,5 @@
# #region TranslationMetrics [C:3] [TYPE Module] [SEMANTICS sqlalchemy, translate, metrics, job, statistics]
# @defgroup Translate Module group.
# @BRIEF Aggregate translation metrics from live TranslationEvent + MetricSnapshot for per-job reporting.
# @LAYER Domain
# @RELATION DEPENDS_ON -> [TranslationSchedule]
@@ -23,6 +24,7 @@ from ...models.translate import (
# #region TranslationMetrics [C:3] [TYPE Class]
# @defgroup Translate Module group.
# @BRIEF Aggregate translation metrics from live events and MetricSnapshot.
class TranslationMetrics:

View File

@@ -1,4 +1,5 @@
# #region TranslationOrchestrator [C:5] [TYPE Module] [SEMANTICS sqlalchemy, tenacity, translate, orchestration, batch]
# @defgroup Translate Module group.
# @BRIEF Run lifecycle coordination: validate preconditions, dispatch executor, generate SQL, submit to Superset, record events.
# @LAYER Domain
# @RELATION DEPENDS_ON -> [TranslationRun]
@@ -34,6 +35,7 @@ from .orchestrator_runner import TranslationStageRunner
# #region TranslationOrchestrator [C:5] [TYPE Class]
# @defgroup Translate Module group.
# @BRIEF Coordinates full translation run lifecycle: validation, execution, SQL generation, Superset submission, event logging.
# @PRE DB session and config manager are available.
# @POST Runs are created, executed, and finalized with event records.

View File

@@ -1,4 +1,5 @@
# #region TranslationResultAggregator [C:4] [TYPE Class] [SEMANTICS sqlalchemy, translate, query, history, records]
# @defgroup Translate Module group.
# @BRIEF Query translation run status, records, and history.
# @PRE Database session is available.
# @POST Returns structured run data with pagination.

View File

@@ -1,4 +1,5 @@
# #region orchestrator_cancel [C:3] [TYPE Module] [SEMANTICS translate, cancel, retry_insert]
# @defgroup Translate Module group.
# @BRIEF Cancel and retry-insert operations for translation runs.
# @RELATION DEPENDS_ON -> [TranslationRun]
# @RELATION DEPENDS_ON -> [TranslationJob]
@@ -37,6 +38,7 @@ def _fallback_cancel_request(db: Session, run_id: str) -> TranslationRun:
# #region cancel_run [C:3] [TYPE Function] [SEMANTICS translate, cancel, run]
# @ingroup Translate
# @BRIEF Cancel a running translation run.
# @SIDE_EFFECT DB writes; event log.
# @RELATION DEPENDS_ON -> [TranslationRun]
@@ -80,6 +82,7 @@ def cancel_run(
# #region retry_insert [C:3] [TYPE Function] [SEMANTICS translate, retry, insert]
# @ingroup Translate
# @BRIEF Retry the SQL insert phase for a completed run.
# @SIDE_EFFECT Superset API call; DB writes.
# @RELATION DEPENDS_ON -> [TranslationRun]

View File

@@ -1,4 +1,5 @@
# #region orchestrator_config [C:3] [TYPE Module] [SEMANTICS hash, config, snapshot, translate]
# @defgroup Translate Module group.
# @BRIEF Config hash and dictionary snapshot hash utilities for translation planning.
# @RELATION DEPENDS_ON -> [TranslationJob]
# @RELATION DEPENDS_ON -> [TranslationJobDictionary]
@@ -8,6 +9,7 @@ import json
# #region compute_config_hash [C:2] [TYPE Function] [SEMANTICS config, hash, deterministic]
# @ingroup Translate
# @BRIEF Compute a deterministic hash of job configuration for snapshot comparison.
# @PRE job has valid configuration attributes.
# @POST Returns 16-char hex SHA-256 hash.
@@ -29,6 +31,7 @@ def compute_config_hash(job) -> str:
# #region compute_dict_snapshot_hash [C:2] [TYPE Function] [SEMANTICS dictionary, hash, snapshot]
# @ingroup Translate
# @BRIEF Compute a hash of dictionary state for snapshot comparison.
# Includes dictionary IDs, entry count, and max entry updated_at
# to invalidate cache when dictionary entries are modified.

View File

@@ -1,4 +1,5 @@
# #region TranslationExecutionEngine [C:4] [TYPE Class] [SEMANTICS translate, execution, run, orchestration]
# @defgroup Translate Module group.
# @BRIEF Execute translation runs: dispatch executor, manage completion and failure paths.
# @PRE Database session and config manager are available. Run is in valid state.
# @POST Run is executed with SQL generated; events recorded.

View File

@@ -1,4 +1,5 @@
# #region update_language_stats [C:3] [TYPE Function] [SEMANTICS translate, language, stats, aggregation]
# @ingroup Translate
# @BRIEF Aggregate TranslationLanguage entries by language_code and update TranslationRunLanguageStats.
# @SIDE_EFFECT DB writes on language_stats objects.
# @RELATION DEPENDS_ON -> [TranslationRecord]

View File

@@ -1,4 +1,5 @@
# #region TranslationPlanner [C:4] [TYPE Class] [SEMANTICS sqlalchemy, translate, orchestration, planning]
# @defgroup Translate Module group.
# @BRIEF Handle translation run planning: validation, config hashing, dictionary snapshot.
# @PRE Database session and config manager are available.
# @POST TranslationRun is planned with hashes and config snapshot; events recorded.

View File

@@ -1,4 +1,5 @@
# #region orchestrator_query [C:3] [TYPE Module] [SEMANTICS translate, query, records, history]
# @defgroup Translate Module group.
# @BRIEF Query translation run records and history with pagination.
# @RELATION DEPENDS_ON -> [TranslationRecord]
# @RELATION DEPENDS_ON -> [TranslationRun]
@@ -11,6 +12,7 @@ from ...models.translate import TranslationRecord, TranslationRun
# #region get_run_records [C:2] [TYPE Function] [SEMANTICS records, query, paginated]
# @ingroup Translate
# @BRIEF Get paginated records for a run. Supports deduplicate=true to return unique source_hash rows.
def get_run_records(
db: Session,
@@ -106,6 +108,7 @@ def get_run_records(
# #endregion get_run_records
# #region get_run_history [C:2] [TYPE Function] [SEMANTICS history, query, paginated]
# @ingroup Translate
# @BRIEF Get run history for a job with pagination. Returns trigger_type, cache_hits for enhanced run list display.
def get_run_history(
db: Session,

View File

@@ -1,4 +1,5 @@
# #region TranslationRunRetryManager [C:4] [TYPE Class] [SEMANTICS translate, retry]
# @defgroup Translate Module group.
# @BRIEF Manage retry of translation run batches.
# @PRE Database session and config manager are available.
# @POST Retried batches are re-executed.

View File

@@ -1,4 +1,5 @@
# #region orchestrator_run_completion [C:3] [TYPE Module] [SEMANTICS translate, run, completion, failure]
# @defgroup Translate Module group.
# @BRIEF Post-execution run completion handlers: cancelled, success-with-insert, and failure paths.
# @RELATION DEPENDS_ON -> [TranslationRun]
# @RELATION DEPENDS_ON -> [TranslationJob]

View File

@@ -1,4 +1,5 @@
# #region TranslationStageRunner [C:4] [TYPE Class] [SEMANTICS sqlalchemy, translate, execution, superset]
# @defgroup Translate Module group.
# @BRIEF Coordinate translation run execution, retry, and cancellation via delegated components.
# @PRE Database session and config manager are available. Run is in valid state.
# @POST Run is executed; events recorded.

View File

@@ -1,4 +1,5 @@
# #region SQLInsertService [C:4] [TYPE Class] [SEMANTICS sql, insert, superset, translate]
# @defgroup Translate Module group.
# @BRIEF Generate INSERT SQL from translation records and submit to Superset SQL Lab.
# @PRE Job has target table configured. Run has successful records.
# @POST SQL is generated and submitted to Superset; returns execution result.

View File

@@ -1,4 +1,5 @@
# #region orchestrator_sql_rows [C:3] [TYPE Module] [SEMANTICS sql, insert, row, column, builders]
# @defgroup Translate Module group.
# @BRIEF Column and row building utilities for SQL insertion in translate plugin.
# @RELATION DEPENDS_ON -> [TranslationRecord]
# @RELATION DEPENDS_ON -> [TranslationJob]
@@ -10,6 +11,7 @@ from .sql_generator import _normalize_timestamp_value
# #region build_columns [C:2] [TYPE Function] [SEMANTICS sql, columns, build]
# @ingroup Translate
# @BRIEF Build list of target columns for SQL INSERT from job configuration.
def build_columns(job: TranslationJob, effective_target: str | None) -> list[str]:
"""Build column list for SQL INSERT from job config."""
@@ -48,6 +50,7 @@ def build_context_keys(job: TranslationJob, effective_target: str | None) -> lis
# #region build_rows [C:2] [TYPE Function] [SEMANTICS sql, rows, build]
# @ingroup Translate
# @BRIEF Build row data for SQL INSERT with per-language expansion.
# @SIDE_EFFECT Reads translation record language entries.
def build_rows(

View File

@@ -1,4 +1,5 @@
# #region validate_job_preconditions [C:3] [TYPE Function] [SEMANTICS validation, preconditions, translate]
# @ingroup Translate
# @BRIEF Validate preconditions before starting a translation run.
# @PRE Job exists and db session is valid.
# @POST Raises ValueError if any precondition fails.

View File

@@ -1,4 +1,5 @@
# #region TranslatePlugin [C:2] [TYPE Module] [SEMANTICS clickhouse, translate, sql, dashboard, dialect]
# @defgroup Translate Module group.
# @BRIEF TranslatePlugin skeleton for cross-dialect SQL and dashboard translation.
# @LAYER Domain
# @RELATION INHERITS -> [PluginBase]
@@ -11,6 +12,7 @@ from ...core.plugin_base import PluginBase
# #region TranslatePlugin [TYPE Class]
# @defgroup Translate Module group.
# @BRIEF Plugin for translating SQL queries and dashboard definitions across database dialects.
# @RELATION IMPLEMENTS -> [PluginBase]
class TranslatePlugin(PluginBase):

View File

@@ -1,4 +1,5 @@
# #region TranslationPreview [C:4] [TYPE Module] [SEMANTICS sqlalchemy, translate, preview, llm, review]
# @defgroup Translate Module group.
# @BRIEF Preview session management: fetch sample rows from Superset, send to LLM with context + filtered dictionary, return side-by-side results.
# @LAYER Domain
# @RELATION DEPENDS_ON -> [TranslationJob]
@@ -43,6 +44,7 @@ from .preview_review import PreviewSessionManager
# #region TranslationPreview [C:4] [TYPE Class]
# @defgroup Translate Module group.
# @BRIEF Manages preview lifecycle: fetch sample rows, call LLM, manage row-level approve/edit/reject, accept gate.
# @PRE Database session and config manager are available.
# @POST Preview sessions created with persisted records; full execution gates on accepted session.
@@ -131,7 +133,7 @@ class TranslationPreview:
llm_response = await self._executor.call_llm(
job=job, prompt=prompt_data["prompt"], max_tokens=token_budget["max_output_needed"],
)
logger.reason(f"TIMING: LLM call: {time.monotonic() - t_llm:.2f}s", {})
logger.reason(f"TIMING: LLM call: {time.monotonic() - t_llm:.2f}s")
translations = self._executor.parse_llm_response(
llm_response, prompt_data["actual_row_count"], target_languages=target_languages,

View File

@@ -1,4 +1,5 @@
# #region PreviewExecutor [C:4] [TYPE Class] [SEMANTICS sqlalchemy, superset, llm, preview]
# @defgroup Translate Module group.
# @BRIEF Fetch sample data from Superset and call LLM provider for preview.
# @PRE Database session, config manager, and Superset client are available.
# @POST Sample rows fetched, LLM called.

View File

@@ -1,4 +1,5 @@
# #region PreviewPromptBuilder [C:3] [TYPE Class] [SEMANTICS prompt, dictionary, preview]
# @defgroup Translate Module group.
# @BRIEF Build LLM prompts for preview sessions including dictionary glossary and row context.
# @RELATION DEPENDS_ON -> [DictionaryManager]
# @RELATION DEPENDS_ON -> [render_prompt]

View File

@@ -1,4 +1,5 @@
# #region preview_prompt_helpers [C:2] [TYPE Module] [SEMANTICS token, budget, metadata, estimate]
# @defgroup Translate Module group.
# @BRIEF Token estimation metadata and budget helpers for preview prompt building.
# @RELATION DEPENDS_ON -> [estimate_token_budget]
# @RELATION DEPENDS_ON -> [TokenEstimator]
@@ -12,6 +13,7 @@ from .preview_token_estimator import TokenEstimator
# #region estimate_token_budget_for_rows [C:2] [TYPE Function] [SEMANTICS token, budget, estimate, truncate]
# @ingroup Translate
# @BRIEF Check token budget and optionally truncate source rows for preview.
# @POST Returns adjusted source_rows, actual_row_count, and token_budget dict.
def estimate_token_budget_for_rows(

View File

@@ -1,4 +1,5 @@
# #region resolve_provider_model [C:2] [TYPE Function] [SEMANTICS llm, provider, model, resolve]
# @ingroup Translate
# @BRIEF Resolve the LLM provider model name for a translation job.
# @RELATION DEPENDS_ON -> [LLMProviderService]
from sqlalchemy.orm import Session

View File

@@ -1,4 +1,5 @@
# #region preview_response_parser [C:3] [TYPE Module] [SEMANTICS llm, parse, json, superset, hash]
# @defgroup Translate Module group.
# @BRIEF Parse LLM JSON responses and Superset data; compute config/dict hashes for preview.
# @RELATION DEPENDS_ON -> [TranslationJob]
# @RELATION DEPENDS_ON -> [TranslationJobDictionary]
@@ -14,6 +15,7 @@ from ...models.translate import TranslationJob, TranslationJobDictionary
# #region parse_llm_response [C:3] [TYPE Function] [SEMANTICS llm, parse, json, translate]
# @ingroup Translate
# @BRIEF Parse LLM JSON response into structured translations dict with per-language support.
# @PRE response_text is a valid JSON string (possibly wrapped in markdown code block).
# @POST Returns dict mapping row_id -> {detected_source_language, lang_code: translation, ...}.
@@ -122,6 +124,7 @@ def compute_config_hash(job: TranslationJob) -> str:
# #region compute_dict_snapshot_hash [C:2] [TYPE Function] [SEMANTICS dictionary, hash, snapshot]
# @ingroup Translate
# @BRIEF Compute a hash of dictionary state for snapshot comparison.
# Includes dictionary IDs, entry count, and max entry updated_at.
def compute_dict_snapshot_hash(db: Session, job_id: str) -> str:

View File

@@ -1,4 +1,5 @@
# #region PreviewSessionManager [C:3] [TYPE Class] [SEMANTICS preview, session, review, approve]
# @defgroup Translate Module group.
# @BRIEF Manage preview session row-level actions: approve/edit/reject rows.
# @RELATION DEPENDS_ON -> [TranslationPreviewSession]
# @RELATION DEPENDS_ON -> [TranslationPreviewRecord]

View File

@@ -1,4 +1,5 @@
# #region preview_session_ops [C:3] [TYPE Module] [SEMANTICS preview, session, accept, get]
# @defgroup Translate Module group.
# @BRIEF Preview session lifecycle operations: accept and query preview sessions.
# @RELATION DEPENDS_ON -> [TranslationPreviewSession]
# @RELATION DEPENDS_ON -> [TranslationPreviewRecord]
@@ -21,6 +22,7 @@ from .preview_session_serializer import (
# #region accept_preview_session [C:2] [TYPE Function] [SEMANTICS preview, session, accept]
# @ingroup Translate
# @BRIEF Mark a preview session as accepted, which gates full execution.
# @SIDE_EFFECT DB writes on session status.
def accept_preview_session(db: Session, job_id: str, _current_user: str | None = None) -> dict[str, Any]:
@@ -60,6 +62,7 @@ def accept_preview_session(db: Session, job_id: str, _current_user: str | None =
# #region get_preview_session [C:2] [TYPE Function] [SEMANTICS preview, session, query]
# @ingroup Translate
# @BRIEF Get the latest preview session for a job with its records.
def get_preview_session(db: Session, job_id: str) -> dict[str, Any]:
"""Get the latest preview session for a job with its records."""

View File

@@ -1,4 +1,5 @@
# #region preview_session_serializer [C:3] [TYPE Module] [SEMANTICS preview, record, serialize, action]
# @defgroup Translate Module group.
# @BRIEF Serialization and action helpers for preview sessions: record serialization and per-row approve/reject/edit actions.
# @RELATION DEPENDS_ON -> [TranslationPreviewRecord]
# @RELATION DEPENDS_ON -> [TranslationPreviewLanguage]
@@ -45,6 +46,7 @@ def serialize_preview_record(r: TranslationPreviewRecord) -> dict[str, Any]:
# #region apply_language_action [C:2] [TYPE Function] [SEMANTICS preview, language, action]
# @ingroup Translate
# @BRIEF Apply approve/reject/edit action to a TranslationPreviewLanguage entry.
def apply_language_action(
lang_entry: TranslationPreviewLanguage,
@@ -68,6 +70,7 @@ def apply_language_action(
# #region apply_record_action [C:2] [TYPE Function] [SEMANTICS preview, record, action]
# @ingroup Translate
# @BRIEF Apply approve/reject/edit action to a TranslationPreviewRecord and its language entries.
def apply_record_action(
record: TranslationPreviewRecord,

View File

@@ -1,4 +1,5 @@
# #region TokenEstimator [C:2] [TYPE Class] [SEMANTICS tokens, cost, estimation]
# @defgroup Translate Module group.
# @BRIEF Estimate token counts and costs for LLM translation operations.
# @RATIONALE Token estimation uses a heuristic (chars/token ratio) since exact tokenization depends on the LLM model.
# @REJECTED Using an external tokenizer library would introduce a heavy dependency for estimation only.

View File

@@ -1,4 +1,5 @@
# #region ContextAwarePromptBuilder [C:2] [TYPE Module] [SEMANTICS translate, prompt, context, dictionary]
# @defgroup Translate Module group.
# @BRIEF Pure-function prompt builder that enhances dictionary entries with context annotations.
# @LAYER Domain
# @RELATION DEPENDS_ON -> [DictionaryEntry]
@@ -15,6 +16,7 @@ import json
from typing import Any
# #region ContextAwarePromptBuilder [C:2] [TYPE Class]
# @defgroup Translate Module group.
# @BRIEF Build LLM prompts with context-aware dictionary entries and similarity-based priority.
class ContextAwarePromptBuilder:

View File

@@ -1,4 +1,5 @@
# #region TranslationScheduler [C:4] [TYPE Module] [SEMANTICS sqlalchemy, translate, schedule, cron, job]
# @defgroup Translate Module group.
# @BRIEF Manage TranslationSchedule rows and register them with core SchedulerService.
# @LAYER Domain
# @RELATION DEPENDS_ON -> [TranslationSchedule]
@@ -26,6 +27,7 @@ from .events import TranslationEventLog
# #region TranslationScheduler [C:4] [TYPE Class]
# @defgroup Translate Module group.
# @BRIEF CRUD for TranslationSchedule rows + APScheduler registration wrappers.
class TranslationScheduler:
@@ -241,6 +243,7 @@ class TranslationScheduler:
# #region execute_scheduled_translation [C:4] [TYPE Function]
# @ingroup Translate
# @BRIEF APScheduler job handler — runs scheduled translation with concurrency check and new-key-only fallback.
# @PRE schedule_id is valid.
# @POST Translation run created and executed if no concurrent run exists.

View File

@@ -1,4 +1,5 @@
# #region TranslateJobService [C:4] [TYPE Module] [SEMANTICS sqlalchemy, translate, job, datasource, dialect]
# @defgroup Translate Module group.
# @BRIEF Service layer for translation job CRUD with datasource column validation and database dialect detection.
# @LAYER Domain
# @RELATION DEPENDS_ON -> [TranslationJob]
@@ -26,6 +27,7 @@ from .service_utils import _extract_dialect
# #region TranslateJobService [TYPE Class]
# @defgroup Translate Module group.
# @BRIEF Service for translation job CRUD with validation and Superset integration.
class TranslateJobService:
"""CRUD service for translation jobs with Superset datasource integration."""

View File

@@ -1,4 +1,5 @@
# #region BulkFindReplaceService [C:3] [TYPE Class] [SEMANTICS translate, bulk, find, replace, regex]
# @defgroup Translate Module group.
# @BRIEF Service for bulk find-and-replace operations on translated values.
# @RELATION DEPENDS_ON -> [TranslationLanguage]
# @RELATION DEPENDS_ON -> [TranslationRecord]

View File

@@ -1,4 +1,5 @@
# #region DatasourceMetadataService [C:4] [TYPE Module] [SEMANTICS superset, datasource, columns, dialect]
# @defgroup Translate Module group.
# @BRIEF Fetch datasource column metadata and database dialect from Superset.
# @LAYER Domain
# @RELATION DEPENDS_ON -> [SupersetClient]
@@ -22,6 +23,7 @@ SUPPORTED_DIALECTS = {
# #region get_dialect_from_database [C:2] [TYPE Function]
# @ingroup Translate
# @BRIEF Extract normalized dialect string from a Superset database record.
def get_dialect_from_database(database_record: dict[str, Any]) -> str:
"""Extract and validate dialect from Superset database record."""
@@ -56,6 +58,7 @@ def get_dialect_from_database(database_record: dict[str, Any]) -> str:
# #region fetch_datasource_metadata [C:3] [TYPE Function]
# @ingroup Translate
# @BRIEF Fetch datasource columns and database dialect from Superset.
async def fetch_datasource_metadata(
dataset_id: int,
@@ -105,6 +108,7 @@ async def fetch_datasource_metadata(
# #region detect_virtual_columns [C:2] [TYPE Function]
# @ingroup Translate
# @BRIEF Identify virtual (calculated) columns from column metadata.
def detect_virtual_columns(columns: list[dict[str, Any]]) -> list[str]:
"""Return names of columns that are virtual (not physical)."""
@@ -113,6 +117,7 @@ def detect_virtual_columns(columns: list[dict[str, Any]]) -> list[str]:
# #region get_datasource_columns [C:3] [TYPE Function]
# @ingroup Translate
# @BRIEF Fetch datasource column metadata from Superset and return structured response.
async def get_datasource_columns(
datasource_id: int,

View File

@@ -1,4 +1,5 @@
# #region InlineCorrectionService [C:4] [TYPE Class] [SEMANTICS translate, correction, inline, dictionary]
# @defgroup Translate Module group.
# @BRIEF Service for inline editing translated values and submitting corrections to dictionaries.
# @PRE Database session is available.
# @POST Inline edits applied with optional dictionary submission.

View File

@@ -1,4 +1,5 @@
# #region TargetSchemaValidation [C:4] [TYPE Module] [SEMANTICS translate, schema, validation, target-table]
# @defgroup Translate Module group.
# @BRIEF Проверка схемы целевой таблицы: запрос колонок через Superset SQL Lab,
# сравнение с ожидаемыми (из build_columns), возврат diff.
# @LAYER Service
@@ -181,6 +182,7 @@ def _parse_sqllab_result(result: dict[str, Any]) -> tuple[list[dict[str, Any]],
# #region validate_target_table_schema [C:4] [TYPE Function] [SEMANTICS translate, schema, validate, orchestrate]
# @ingroup Translate
# @BRIEF Основная функция: проверяет схему целевой таблицы через Superset SQL Lab.
# @PRE Superset окружение и target_database_id валидны.
# @POST Возвращает TargetSchemaValidationResponse с diff-анализом.

View File

@@ -1,4 +1,5 @@
# #region SQLGenerator [C:3] [TYPE Module] [SEMANTICS clickhouse, translate, sql, insert, generate]
# @defgroup Translate Module group.
# @BRIEF Dialect-aware safe SQL generation for INSERT/UPSERT operations.
# @LAYER Domain
# @RELATION DEPENDS_ON -> [TranslationJob]
@@ -142,6 +143,7 @@ def _build_values_clause(columns: list[str], rows: list[dict[str, Any]], dialect
# #region generate_insert_sql [C:3] [TYPE Function] [SEMANTICS translate,sql,insert]
# @ingroup Translate
# @BRIEF Generate a dialect-aware plain INSERT SQL statement for the given table, columns, and rows.
# @RELATION DEPENDS_ON -> [_quote_identifier]
# @RELATION DEPENDS_ON -> [_build_values_clause]
@@ -174,6 +176,7 @@ def generate_insert_sql(
# #region generate_upsert_sql [C:4] [TYPE Function]
# @ingroup Translate
# @BRIEF Generate PostgreSQL dialect UPSERT SQL with ON CONFLICT DO UPDATE.
# @PRE dialect is postgresql-compatible. target_table, columns, key_columns are non-empty.
# @POST Returns UPSERT SQL string or raises ValueError.
@@ -229,6 +232,7 @@ def generate_upsert_sql(
# #region SQLGenerator [C:3] [TYPE Class]
# @defgroup Translate Module group.
# @BRIEF Generate safe, dialect-appropriate SQL INSERT/UPSERT statements.
# @PRE Job has target_schema, target_table, key columns configured.
# @POST Returns generated SQL string for the target dialect.

View File

@@ -1,4 +1,5 @@
# #region SupersetSqlLabExecutor [C:4] [TYPE Module] [SEMANTICS translate, superset, sqllab, execute, query]
# @defgroup Translate Module group.
# @BRIEF Submit SQL to Superset SQL Lab API and poll execution status.
# @LAYER Infrastructure
# @RELATION DEPENDS_ON -> [ConfigManager]
@@ -21,6 +22,7 @@ from ...core.utils.client_registry import get_superset_client
# #region SupersetSqlLabExecutor [C:4] [TYPE Class]
# @defgroup Translate Module group.
# @BRIEF Submit SQL to Superset SQL Lab API with polling and status tracking.
# @PRE Valid environment ID and ConfigManager.
# @POST SQL submitted to Superset; execution reference recorded.