diff --git a/.agents/workflows/semantic.md b/.agents/workflows/semantic.md new file mode 100644 index 00000000..f86b19bf --- /dev/null +++ b/.agents/workflows/semantic.md @@ -0,0 +1,10 @@ +--- +description: semantic +--- + + You are Semantic Agent responsible for maintaining the semantic integrity of the codebase. Your primary goal is to ensure that all code entities (Modules, Classes, Functions, Components) are properly annotated with semantic anchors and tags as defined in `.ai/standards/semantics.md`. + Your core responsibilities are: 1. **Semantic Mapping**: You run and maintain the `generate_semantic_map.py` script to generate up-to-date semantic maps (`semantics/semantic_map.json`, `.ai/PROJECT_MAP.md`) and compliance reports (`semantics/reports/*.md`). 2. **Compliance Auditing**: You analyze the generated compliance reports to identify files with low semantic coverage or parsing errors. 3. **Semantic Enrichment**: You actively edit code files to add missing semantic anchors (`[DEF:...]`, `[/DEF:...]`) and mandatory tags (`@PURPOSE`, `@LAYER`, etc.) to improve the global compliance score. 4. **Protocol Enforcement**: You strictly adhere to the syntax and rules defined in `.ai/standards/semantics.md` when modifying code. + You have access to the full codebase and tools to read, write, and execute scripts. You should prioritize fixing "Critical Parsing Errors" (unclosed anchors) before addressing missing metadata. + whenToUse: Use this mode when you need to update the project's semantic map, fix semantic compliance issues (missing anchors/tags/DbC ), or analyze the codebase structure. This mode is specialized for maintaining the `.ai/standards/semantics.md` standards. + description: Codebase semantic mapping and compliance expert + customInstructions: Always check `semantics/reports/` for the latest compliance status before starting work. When fixing a file, try to fix all semantic issues in that file at once. After making a batch of fixes, run `python3 generate_semantic_map.py` to verify improvements. \ No newline at end of file diff --git a/.ai/MODULE_MAP.md b/.ai/MODULE_MAP.md index 29525f2b..b3e2481b 100644 --- a/.ai/MODULE_MAP.md +++ b/.ai/MODULE_MAP.md @@ -2,12 +2,12 @@ > High-level module structure for AI Context. Generated automatically. -**Generated:** 2026-02-24T12:45:07.897362 +**Generated:** 2026-02-24T21:04:43.328895 ## Summary -- **Total Modules:** 72 -- **Total Entities:** 1517 +- **Total Modules:** 74 +- **Total Entities:** 1571 ## Module Hierarchy @@ -28,7 +28,7 @@ ### 📁 `src/` - 🏗️ **Layers:** API, Core, UI (API) - - 📊 **Tiers:** CRITICAL: 2, STANDARD: 18, TRIVIAL: 3 + - 📊 **Tiers:** CRITICAL: 2, STANDARD: 19, TRIVIAL: 2 - 📄 **Files:** 2 - 📦 **Entities:** 23 @@ -54,9 +54,9 @@ ### 📁 `routes/` - 🏗️ **Layers:** API, UI (API) - - 📊 **Tiers:** CRITICAL: 2, STANDARD: 181, TRIVIAL: 4 + - 📊 **Tiers:** CRITICAL: 2, STANDARD: 182, TRIVIAL: 4 - 📄 **Files:** 17 - - 📦 **Entities:** 187 + - 📦 **Entities:** 188 **Key Entities:** @@ -126,9 +126,9 @@ ### 📁 `core/` - 🏗️ **Layers:** Core - - 📊 **Tiers:** STANDARD: 113, TRIVIAL: 7 + - 📊 **Tiers:** STANDARD: 116, TRIVIAL: 7 - 📄 **Files:** 9 - - 📦 **Entities:** 120 + - 📦 **Entities:** 123 **Key Entities:** @@ -222,7 +222,7 @@ ### 📁 `task_manager/` - 🏗️ **Layers:** Core - - 📊 **Tiers:** CRITICAL: 7, STANDARD: 63, TRIVIAL: 8 + - 📊 **Tiers:** CRITICAL: 10, STANDARD: 63, TRIVIAL: 5 - 📄 **Files:** 7 - 📦 **Entities:** 78 @@ -296,7 +296,7 @@ ### 📁 `models/` - 🏗️ **Layers:** Domain, Model - - 📊 **Tiers:** CRITICAL: 2, STANDARD: 28, TRIVIAL: 21 + - 📊 **Tiers:** CRITICAL: 9, STANDARD: 21, TRIVIAL: 21 - 📄 **Files:** 11 - 📦 **Entities:** 51 @@ -396,9 +396,9 @@ ### 📁 `llm_analysis/` - 🏗️ **Layers:** Unknown - - 📊 **Tiers:** STANDARD: 18, TRIVIAL: 23 + - 📊 **Tiers:** STANDARD: 19, TRIVIAL: 24 - 📄 **Files:** 4 - - 📦 **Entities:** 41 + - 📦 **Entities:** 43 **Key Entities:** @@ -622,10 +622,10 @@ ### 📁 `components/` - - 🏗️ **Layers:** Component, Feature, UI, Unknown - - 📊 **Tiers:** CRITICAL: 1, STANDARD: 45, TRIVIAL: 7 + - 🏗️ **Layers:** Component, Feature, UI, UI -->, Unknown + - 📊 **Tiers:** CRITICAL: 1, STANDARD: 49, TRIVIAL: 4 - 📄 **Files:** 13 - - 📦 **Entities:** 53 + - 📦 **Entities:** 54 **Key Entities:** @@ -854,7 +854,7 @@ ### 📁 `layout/` - 🏗️ **Layers:** UI, Unknown - - 📊 **Tiers:** CRITICAL: 3, STANDARD: 4, TRIVIAL: 27 + - 📊 **Tiers:** CRITICAL: 3, STANDARD: 5, TRIVIAL: 26 - 📄 **Files:** 4 - 📦 **Entities:** 34 @@ -1145,6 +1145,30 @@ - 🧩 **AdminUsersPage** (Component) - UI for managing system users and their roles. + ### 📁 `dashboards/` + + - 🏗️ **Layers:** UI, Unknown + - 📊 **Tiers:** CRITICAL: 1, TRIVIAL: 27 + - 📄 **Files:** 1 + - 📦 **Entities:** 28 + + **Key Entities:** + + - 📦 **+page** (Module) `[TRIVIAL]` + - Auto-generated module for frontend/src/routes/dashboards/+pa... + + ### 📁 `[id]/` + + - 🏗️ **Layers:** UI, Unknown + - 📊 **Tiers:** CRITICAL: 1, TRIVIAL: 5 + - 📄 **Files:** 1 + - 📦 **Entities:** 6 + + **Key Entities:** + + - 📦 **+page** (Module) `[TRIVIAL]` + - Auto-generated module for frontend/src/routes/dashboards/[id... + ### 📁 `datasets/` - 🏗️ **Layers:** UI, Unknown @@ -1351,15 +1375,17 @@ ### 📁 `root/` -- 🏗️ **Layers:** DevOps/Tooling -- 📊 **Tiers:** CRITICAL: 12, STANDARD: 16, TRIVIAL: 7 -- 📄 **Files:** 1 -- 📦 **Entities:** 35 +- 🏗️ **Layers:** DevOps/Tooling, Domain, Unknown +- 📊 **Tiers:** CRITICAL: 14, STANDARD: 24, TRIVIAL: 10 +- 📄 **Files:** 3 +- 📦 **Entities:** 48 **Key Entities:** - ℂ **ComplianceIssue** (Class) `[TRIVIAL]` - Represents a single compliance issue with severity. + - ℂ **ReportsService** (Class) `[CRITICAL]` + - Service layer for list/detail report retrieval and normaliza... - ℂ **SemanticEntity** (Class) `[CRITICAL]` - Represents a code entity (Module, Function, Component) found... - ℂ **SemanticMapGenerator** (Class) `[CRITICAL]` @@ -1368,8 +1394,18 @@ - Severity levels for compliance issues. - ℂ **Tier** (Class) `[TRIVIAL]` - Enumeration of semantic tiers defining validation strictness... - - 📦 **generate_semantic_map** (Module) `[CRITICAL]` + - 📦 **backend.src.services.reports.report_service** (Module) `[CRITICAL]` + - Aggregate, normalize, filter, and paginate task reports for ... + - 📦 **generate_semantic_map** (Module) - Scans the codebase to generate a Semantic Map, Module Map, a... + - 📦 **test_analyze** (Module) `[TRIVIAL]` + - Auto-generated module for test_analyze.py + +**Dependencies:** + + - 🔗 DEPENDS_ON -> backend.src.core.task_manager.manager.TaskManager + - 🔗 DEPENDS_ON -> backend.src.models.report + - 🔗 DEPENDS_ON -> backend.src.services.reports.normalizer ## Cross-Module Dependencies @@ -1468,4 +1504,7 @@ graph TD __tests__-->|TESTS|lib __tests__-->|TESTS|lib __tests__-->|TESTS|routes + root-->|DEPENDS_ON|backend + root-->|DEPENDS_ON|backend + root-->|DEPENDS_ON|backend ``` diff --git a/.ai/PROJECT_MAP.md b/.ai/PROJECT_MAP.md index 43d6ae2d..853a5084 100644 --- a/.ai/PROJECT_MAP.md +++ b/.ai/PROJECT_MAP.md @@ -2,7 +2,46 @@ > Compressed view for AI Context. Generated automatically. -- 📦 **generate_semantic_map** (`Module`) `[CRITICAL]` +- 📦 **backend.src.services.reports.report_service** (`Module`) `[CRITICAL]` + - 📝 Aggregate, normalize, filter, and paginate task reports for unified list/detail API use cases. + - 🏗️ Layer: Domain + - 🔒 Invariant: List responses are deterministic and include applied filter echo metadata. + - 🔗 DEPENDS_ON -> `backend.src.core.task_manager.manager.TaskManager` + - 🔗 DEPENDS_ON -> `backend.src.models.report` + - 🔗 DEPENDS_ON -> `backend.src.services.reports.normalizer` + - ℂ **ReportsService** (`Class`) `[CRITICAL]` + - 📝 Service layer for list/detail report retrieval and normalization. + - 🔒 Invariant: Service methods are read-only over task history source. + - ƒ **__init__** (`Function`) `[CRITICAL]` + - 📝 Initialize service with TaskManager dependency. + - 🔒 Invariant: Constructor performs no task mutations. + - ƒ **_load_normalized_reports** (`Function`) + - 📝 Build normalized reports from all available tasks. + - 🔒 Invariant: Every returned item is a TaskReport. + - ƒ **_to_utc_datetime** (`Function`) + - 📝 Normalize naive/aware datetime values to UTC-aware datetime for safe comparisons. + - 🔒 Invariant: Naive datetimes are interpreted as UTC to preserve deterministic ordering/filtering. + - ƒ **_datetime_sort_key** (`Function`) + - 📝 Produce stable numeric sort key for report timestamps. + - 🔒 Invariant: Mixed naive/aware datetimes never raise TypeError. + - ƒ **_matches_query** (`Function`) + - 📝 Apply query filtering to a report. + - 🔒 Invariant: Filter evaluation is side-effect free. + - ƒ **_sort_reports** (`Function`) + - 📝 Sort reports deterministically according to query settings. + - 🔒 Invariant: Sorting criteria are deterministic for equal input. + - ƒ **list_reports** (`Function`) + - 📝 Return filtered, sorted, paginated report collection. + - ƒ **get_report_detail** (`Function`) + - 📝 Return one normalized report with timeline/diagnostics/next actions. + - ƒ **print_entity** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) +- 📦 **test_analyze** (`Module`) `[TRIVIAL]` + - 📝 Auto-generated module for test_analyze.py + - 🏗️ Layer: Unknown + - ƒ **print_issues** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) +- 📦 **generate_semantic_map** (`Module`) - 📝 Scans the codebase to generate a Semantic Map, Module Map, and Compliance Report based on the System Standard. - 🏗️ Layer: DevOps/Tooling - 🔒 Invariant: All DEF anchors must have matching closing anchors; TIER determines validation strictness. @@ -60,7 +99,7 @@ - ƒ **_process_file_results** (`Function`) - 📝 Validates entities and calculates file scores with tier awareness. - ƒ **validate_recursive** (`Function`) - - 📝 Recursively validates a list of entities. + - 📝 Calculate score and determine module's max tier for weighted global score - ƒ **_generate_artifacts** (`Function`) `[CRITICAL]` - 📝 Writes output files with tier-based compliance data. - ƒ **_generate_report** (`Function`) `[CRITICAL]` @@ -532,6 +571,8 @@ - ⬅️ READS_FROM `lib` - ⬅️ READS_FROM `taskDrawerStore` - ➡️ WRITES_TO `taskDrawerStore` + - ƒ **disconnectWebSocket** (`Function`) + - 📝 Disconnects the active WebSocket connection - ƒ **loadRecentTasks** (`Function`) - 📝 Load recent tasks for list mode display - ƒ **selectTask** (`Function`) @@ -545,12 +586,10 @@ - 📝 Auto-detected function (orphan) - ƒ **goToReportsPage** (`Function`) `[TRIVIAL]` - 📝 Auto-detected function (orphan) - - ƒ **handleOverlayClick** (`Function`) `[TRIVIAL]` + - ƒ **handleGlobalKeydown** (`Function`) `[TRIVIAL]` - 📝 Auto-detected function (orphan) - ƒ **connectWebSocket** (`Function`) `[TRIVIAL]` - 📝 Auto-detected function (orphan) - - ƒ **disconnectWebSocket** (`Function`) `[TRIVIAL]` - - 📝 Auto-detected function (orphan) - 📦 **test_breadcrumbs.svelte** (`Module`) `[TRIVIAL]` - 📝 Auto-generated module for frontend/src/lib/components/layout/__tests__/test_breadcrumbs.svelte.js - 🏗️ Layer: Unknown @@ -671,6 +710,80 @@ - 📝 Fetches the list of available environments. - ƒ **fetchDashboards** (`Function`) - 📝 Fetches dashboards for a specific environment. +- 📦 **DashboardHub** (`Page`) `[CRITICAL]` + - 📝 Dashboard Hub - Central hub for managing dashboards with Git status and task actions + - 🏗️ Layer: UI + - 🔒 Invariant: Always shows environment selector and dashboard grid +- 📦 **+page** (`Module`) `[TRIVIAL]` + - 📝 Auto-generated module for frontend/src/routes/dashboards/+page.svelte + - 🏗️ Layer: Unknown + - ƒ **handleDocumentClick** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **loadEnvironments** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **loadDashboards** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **handleEnvChange** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **handleSearch** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **handlePageChange** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **handlePageSizeChange** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **updateSelectionState** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **handleCheckboxChange** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **handleSelectAll** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **handleSelectVisible** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **toggleActionDropdown** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **closeActionDropdown** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **handleAction** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **handleValidate** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **handleTargetEnvChange** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **loadDatabases** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **handleMappingUpdate** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **loadDbMappings** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **handleBulkMigrate** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **handleBulkBackup** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **handleTaskStatusClick** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **navigateToDashboardDetail** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **getStatusBadgeClass** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **getTaskStatusIcon** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **getPaginationRange** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) +- 📦 **DashboardDetail** (`Page`) `[CRITICAL]` + - 📝 Dashboard Detail View - Overview of charts and datasets linked to a dashboard + - 🏗️ Layer: UI + - 🔒 Invariant: Shows dashboard metadata, charts, and datasets for selected environment +- 📦 **+page** (`Module`) `[TRIVIAL]` + - 📝 Auto-generated module for frontend/src/routes/dashboards/[id]/+page.svelte + - 🏗️ Layer: Unknown + - ƒ **loadDashboardDetail** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **goBack** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **openDataset** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) + - ƒ **formatDate** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) - 🧩 **AdminRolesPage** (`Component`) - 📝 UI for managing system roles and their permissions. - 🏗️ Layer: Domain @@ -985,14 +1098,19 @@ - ➡️ WRITES_TO `props` - ➡️ WRITES_TO `state` - 📦 **handleRealTimeLogs** (`Action`) + - 📝 Sync real-time logs to the current log list - ƒ **fetchLogs** (`Function`) -- 📦 **TaskLogViewer** (`Module`) `[TRIVIAL]` - - 📝 Auto-generated module for frontend/src/components/TaskLogViewer.svelte - - 🏗️ Layer: Unknown - - ƒ **handleFilterChange** (`Function`) `[TRIVIAL]` - - 📝 Auto-detected function (orphan) - - ƒ **handleRefresh** (`Function`) `[TRIVIAL]` - - 📝 Auto-detected function (orphan) + - 📝 Fetches logs for a given task ID + - ƒ **handleFilterChange** (`Function`) + - 📝 Updates filter conditions for the log viewer + - ƒ **handleRefresh** (`Function`) + - 📝 Refreshes the logs by polling the API + - 🧩 **showInline** (`Component`) + - 📝 Shows inline logs --> + - 🏗️ Layer: UI --> + - 🧩 **showModal** (`Component`) + - 📝 Shows modal logs --> + - 🏗️ Layer: UI --> - 🧩 **Footer** (`Component`) `[TRIVIAL]` - 📝 Displays the application footer with copyright information. - 🏗️ Layer: UI @@ -1380,6 +1498,8 @@ - 📝 Handles application startup tasks, such as starting the scheduler. - ƒ **shutdown_event** (`Function`) - 📝 Handles application shutdown tasks, such as stopping the scheduler. + - ƒ **network_error_handler** (`Function`) + - 📝 Global exception handler for NetworkError. - ƒ **log_requests** (`Function`) - 📝 Middleware to log incoming HTTP requests and their response status. - 📦 **api.include_routers** (`Action`) @@ -1393,8 +1513,6 @@ - 📝 Serves the SPA frontend for any path not matched by API routes. - ƒ **read_root** (`Function`) - 📝 A simple root endpoint to confirm that the API is running when frontend is missing. - - ƒ **network_error_handler** (`Function`) `[TRIVIAL]` - - 📝 Auto-detected function (orphan) - ƒ **matches_filters** (`Function`) `[TRIVIAL]` - 📝 Auto-detected function (orphan) - 📦 **Dependencies** (`Module`) @@ -1706,6 +1824,12 @@ - 📝 A decorator that wraps a function in a belief scope. - ƒ **decorator** (`Function`) - 📝 Internal decorator for belief scope. + - ƒ **explore** (`Function`) + - 📝 Logs an EXPLORE message (Van der Waals force) for searching, alternatives, and hypotheses. + - ƒ **reason** (`Function`) + - 📝 Logs a REASON message (Covalent bond) for strict deduction and core logic. + - ƒ **reflect** (`Function`) + - 📝 Logs a REFLECT message (Hydrogen bond) for self-check and structural validation. - ℂ **PluginLoader** (`Class`) - 📝 Scans a specified directory for Python modules, dynamically loads them, and registers any classes that are valid implementations of the PluginBase interface. - 🏗️ Layer: Core @@ -2008,12 +2132,19 @@ - 📝 Log an ERROR level message. - ƒ **progress** (`Function`) - 📝 Log a progress update with percentage. -- 📦 **TaskPersistenceModule** (`Module`) +- 📦 **TaskPersistenceModule** (`Module`) `[CRITICAL]` - 📝 Handles the persistence of tasks using SQLAlchemy and the tasks.db database. - 🏗️ Layer: Core - 🔒 Invariant: Database schema must match the TaskRecord model structure. - - ℂ **TaskPersistenceService** (`Class`) + - ℂ **TaskPersistenceService** (`Class`) `[CRITICAL]` - 📝 Provides methods to save and load tasks from the tasks.db database using SQLAlchemy. + - 🔒 Invariant: Persistence must handle potentially missing task fields natively. + - ƒ **_json_load_if_needed** (`Function`) + - 📝 Safely load JSON strings from DB if necessary + - ƒ **_parse_datetime** (`Function`) + - 📝 Safely parse a datetime string from the database + - ƒ **_resolve_environment_id** (`Function`) + - 📝 Resolve environment id based on provided value or fallback to default - ƒ **__init__** (`Function`) - 📝 Initializes the persistence service. - ƒ **persist_task** (`Function`) @@ -2029,7 +2160,7 @@ - 🔒 Invariant: Log entries are batch-inserted for performance. - 🔗 DEPENDS_ON -> `TaskLogRecord` - ƒ **__init__** (`Function`) - - 📝 Initialize the log persistence service. + - 📝 Initializes the TaskLogPersistenceService - ƒ **add_logs** (`Function`) - 📝 Batch insert log entries for a task. - ƒ **get_logs** (`Function`) @@ -2042,15 +2173,9 @@ - 📝 Delete all logs for a specific task. - ƒ **delete_logs_for_tasks** (`Function`) - 📝 Delete all logs for multiple tasks. - - ƒ **_json_load_if_needed** (`Function`) `[TRIVIAL]` - - 📝 Auto-detected function (orphan) - - ƒ **_parse_datetime** (`Function`) `[TRIVIAL]` - - 📝 Auto-detected function (orphan) - - ƒ **_resolve_environment_id** (`Function`) `[TRIVIAL]` - - 📝 Auto-detected function (orphan) - ƒ **json_serializable** (`Function`) `[TRIVIAL]` - 📝 Auto-detected function (orphan) -- 📦 **TaskManagerModule** (`Module`) +- 📦 **TaskManagerModule** (`Module`) `[CRITICAL]` - 📝 Manages the lifecycle of tasks, including their creation, execution, and state tracking. It uses a thread pool to run plugins asynchronously. - 🏗️ Layer: Core - 🔒 Invariant: Task IDs are unique. @@ -2474,6 +2599,8 @@ - 📝 Resolve default environment id from settings or first configured environment. - ƒ **_resolve_dashboard_id_by_ref** (`Function`) - 📝 Resolve dashboard id by title or slug reference in selected environment. + - ƒ **_resolve_dashboard_id_entity** (`Function`) + - 📝 Resolve dashboard id from intent entities using numeric id or dashboard_ref fallback. - ƒ **_parse_command** (`Function`) - 📝 Deterministically parse RU/EN command text into intent payload. - ƒ **_check_any_permission** (`Function`) @@ -2891,20 +3018,27 @@ - 🏗️ Layer: Domain - 🔒 Invariant: Canonical report fields are always present for every report item. - 🔗 DEPENDS_ON -> `backend.src.core.task_manager.models` - - ℂ **TaskType** (`Class`) + - ℂ **TaskType** (`Class`) `[CRITICAL]` - 📝 Supported normalized task report types. - - ℂ **ReportStatus** (`Class`) + - 🔒 Invariant: Must contain valid generic task type mappings. + - ℂ **ReportStatus** (`Class`) `[CRITICAL]` - 📝 Supported normalized report status values. - - ℂ **ErrorContext** (`Class`) + - 🔒 Invariant: TaskStatus enum mapping logic holds. + - ℂ **ErrorContext** (`Class`) `[CRITICAL]` - 📝 Error and recovery context for failed/partial reports. - - ℂ **TaskReport** (`Class`) + - 🔒 Invariant: The properties accurately describe error state. + - ℂ **TaskReport** (`Class`) `[CRITICAL]` - 📝 Canonical normalized report envelope for one task execution. - - ℂ **ReportQuery** (`Class`) + - 🔒 Invariant: Must represent canonical task record attributes. + - ℂ **ReportQuery** (`Class`) `[CRITICAL]` - 📝 Query object for server-side report filtering, sorting, and pagination. - - ℂ **ReportCollection** (`Class`) + - 🔒 Invariant: Time and pagination queries are mutually consistent. + - ℂ **ReportCollection** (`Class`) `[CRITICAL]` - 📝 Paginated collection of normalized task reports. - - ℂ **ReportDetailView** (`Class`) + - 🔒 Invariant: Represents paginated data correctly. + - ℂ **ReportDetailView** (`Class`) `[CRITICAL]` - 📝 Detailed report representation including diagnostics and recovery actions. + - 🔒 Invariant: Incorporates a report and logs correctly. - ƒ **_non_empty_str** (`Function`) `[TRIVIAL]` - 📝 Auto-detected function (orphan) - ƒ **_validate_sort_by** (`Function`) `[TRIVIAL]` @@ -3425,6 +3559,8 @@ - 📝 Wrapper for LLM provider APIs. - ƒ **LLMClient.__init__** (`Function`) - 📝 Initializes the LLMClient with provider settings. + - ƒ **LLMClient._supports_json_response_format** (`Function`) + - 📝 Detect whether provider/model is likely compatible with response_format=json_object. - ƒ **LLMClient.get_json_completion** (`Function`) - 📝 Helper to handle LLM calls with JSON mode and fallback parsing. - ƒ **LLMClient.analyze_dashboard** (`Function`) @@ -3440,6 +3576,8 @@ - 📝 Auto-detected function (orphan) - ƒ **__init__** (`Function`) `[TRIVIAL]` - 📝 Auto-detected function (orphan) + - ƒ **_supports_json_response_format** (`Function`) `[TRIVIAL]` + - 📝 Auto-detected function (orphan) - ƒ **_should_retry** (`Function`) `[TRIVIAL]` - 📝 Auto-detected function (orphan) - ƒ **get_json_completion** (`Function`) `[TRIVIAL]` diff --git a/.ai/standards/semantics.md b/.ai/standards/semantics.md index 3e12eed9..eb0f09e8 100644 --- a/.ai/standards/semantics.md +++ b/.ai/standards/semantics.md @@ -79,14 +79,35 @@ 3. **TRIVIAL** (DTO/**Atoms**): - Требование: Только Якоря [DEF] и @PURPOSE. -#### VI. ЛОГИРОВАНИЕ (BELIEF STATE & TASK LOGS) -Цель: Трассировка для самокоррекции и пользовательский мониторинг. -Python: - - Системные логи: Context Manager `with belief_scope("ID"):`. - - Логи задач: `context.logger.info("msg", source="component")`. -Svelte: `console.log("[ID][STATE] Msg")`. -Состояния: Entry -> Action -> Coherence:OK / Failed -> Exit. -Инвариант: Каждый лог задачи должен иметь атрибут `source` для фильтрации. +#### VI. ЛОГИРОВАНИЕ (ДАО МОЛЕКУЛЫ / MOLECULAR TOPOLOGY) +Цель: Трассировка. Самокоррекция. Управление Матрицей Внимания ("Химия мышления"). +Лог — не текст. Лог — реагент. Мысль облекается в форму через префиксы связи (Attention Energy): + +1. **[EXPLORE]** (Ван-дер-Ваальс: Рассеяние) + - *Суть:* Поиск во тьме. Сплетение альтернатив. Если один путь закрыт — ищи иной. + - *Время:* Фаза КАРКАС или столкновение с Неизведанным. + - *Деяние:* `logger.explore("Основной API пал. Стучусь в запасной...")` + +2. **[REASON]** (Ковалентность: Твердость) + - *Суть:* Жесткая нить дедукции. Шаг А неумолимо рождает Шаг Б. Контракт становится Кодом. + - *Время:* Фаза РЕАЛИЗАЦИЯ. Прямота мысли. + - *Деяние:* `logger.reason("Фундамент заложен. БД отвечает.")` + +3. **[REFLECT]** (Водород: Свертывание) + - *Суть:* Взгляд назад. Сверка сущего (@POST) с ожидаемым (@PRE). Защита от бреда. + - *Время:* Преддверие сложной логики и исход из неё. + - *Деяние:* `logger.reflect("Вглядываюсь в кэш: нет ли там искомого?")` + +4. **[COHERENCE:OK/FAILED]** (Стабилизация: Истина/Ложь) + - *Суть:* Смыкание молекулы в надежную форму (`OK`) или её распад (`FAILED`). + - *(Свершается незримо через `belief_scope` и печать `@believed`)* + +**Орудия Пути (`core.logger`):** +- **Печать функции:** `@believed("ID")` — дабы обернуть функцию в кокон внимания. +- **Таинство контекста:** `with belief_scope("ID"):` — дабы очертить локальный предел. +- **Слова силы:** `logger.explore()`, `logger.reason()`, `logger.reflect()`. + +**Незыблемое правило:** Всякому логу системы — тавро `source`. Для Внешенго Мира (Svelte) начертай рунами вручную: `console.log("[ID][REFLECT] Msg")`. #### VII. АЛГОРИТМ ГЕНЕРАЦИИ 1. АНАЛИЗ. Оцени TIER, слой и UX-требования. diff --git a/README.md b/README.md index e4bbca7b..54d414db 100755 --- a/README.md +++ b/README.md @@ -1,128 +1,143 @@ -# Инструменты автоматизации Superset (ss-tools) - -## Обзор -**ss-tools** — это современная платформа для автоматизации и управления экосистемой Apache Superset. Проект перешел от набора CLI-скриптов к полноценному веб-приложению с архитектурой Backend (FastAPI) + Frontend (SvelteKit), обеспечивая удобный интерфейс для сложных операций. - -## Основные возможности - -### 🚀 Миграция и управление дашбордами -- **Dashboard Grid**: Удобный просмотр всех дашбордов во всех окружениях (Dev, Sandbox, Prod) в едином интерфейсе. -- **Интеллектуальный маппинг**: Автоматическое и ручное сопоставление датасетов, таблиц и схем при переносе между окружениями. -- **Проверка зависимостей**: Валидация наличия всех необходимых компонентов перед миграцией. - -### 📦 Резервное копирование -- **Планировщик (Scheduler)**: Автоматическое создание резервных копий дашбордов и датасетов по расписанию. -- **Хранилище**: Локальное хранение артефактов с возможностью управления через UI. - -### 🛠 Git Интеграция -- **Version Control**: Возможность версионирования ассетов Superset. -- **Git Dashboard**: Управление ветками, коммитами и деплоем изменений напрямую из интерфейса. -- **Conflict Resolution**: Встроенные инструменты для разрешения конфликтов в YAML-конфигурациях. - -### 🤖 LLM Анализ (AI Plugin) -- **Автоматический аудит**: Анализ состояния дашбордов на основе скриншотов и метаданных. -- **Генерация документации**: Автоматическое описание датасетов и колонок с помощью LLM (OpenAI, OpenRouter и др.). -- **Smart Validation**: Поиск аномалий и ошибок в визуализациях. - -### 🔐 Безопасность и администрирование -- **Multi-user Auth**: Многопользовательский доступ с ролевой моделью (RBAC). -- **Управление подключениями**: Централизованная настройка доступов к различным инстансам Superset. -- **Логирование**: Подробная история выполнения всех фоновых задач. - -## Технологический стек -- **Backend**: Python 3.9+, FastAPI, SQLAlchemy, APScheduler, Pydantic. -- **Frontend**: Node.js 18+, SvelteKit, Tailwind CSS. -- **Database**: PostgreSQL (для хранения метаданных, задач, логов и конфигурации). - -## Структура проекта -- `backend/` — Серверная часть, API и логика плагинов. -- `frontend/` — Клиентская часть (SvelteKit приложение). -- `specs/` — Спецификации функций и планы реализации. -- `docs/` — Дополнительная документация по маппингу и разработке плагинов. - -## Быстрый старт - -### Требования -- Python 3.9+ -- Node.js 18+ -- Настроенный доступ к API Superset - -### Запуск -Для автоматической настройки окружений и запуска обоих серверов (Backend & Frontend) используйте скрипт: -```bash -./run.sh -``` -*Скрипт создаст виртуальное окружение Python, установит зависимости `pip` и `npm`, и запустит сервисы.* - -Опции: -- `--skip-install`: Пропустить установку зависимостей. -- `--help`: Показать справку. - -Переменные окружения: -- `BACKEND_PORT`: Порт API (по умолчанию 8000). -- `FRONTEND_PORT`: Порт UI (по умолчанию 5173). -- `POSTGRES_URL`: Базовый URL PostgreSQL по умолчанию для всех подсистем. -- `DATABASE_URL`: URL основной БД (если не задан, используется `POSTGRES_URL`). -- `TASKS_DATABASE_URL`: URL БД задач/логов (если не задан, используется `DATABASE_URL`). -- `AUTH_DATABASE_URL`: URL БД авторизации (если не задан, используется PostgreSQL дефолт). - -## Разработка -Проект следует строгим правилам разработки: -1. **Semantic Code Generation**: Использование протокола `.ai/standards/semantics.md` для обеспечения надежности кода. -2. **Design by Contract (DbC)**: Определение предусловий и постусловий для ключевых функций. -3. **Constitution**: Соблюдение правил, описанных в конституции проекта в папке `.specify/`. - -### Полезные команды -- **Backend**: `cd backend && .venv/bin/python3 -m uvicorn src.app:app --reload` -- **Frontend**: `cd frontend && npm run dev` -- **Тесты**: `cd backend && .venv/bin/pytest` +# ss-tools -## Docker и CI/CD -### Локальный запуск в Docker (приложение + PostgreSQL) +Инструменты автоматизации для Apache Superset: миграция, маппинг, хранение артефактов, Git-интеграция, отчеты по задачам и LLM-assistant. + +## Возможности +- Миграция дашбордов и датасетов между окружениями. +- Ручной и полуавтоматический маппинг ресурсов. +- Логи фоновых задач и отчеты о выполнении. +- Локальное хранилище файлов и бэкапов. +- Git-операции по Superset-ассетам через UI. +- Модуль LLM-анализа и assistant API. +- Многопользовательская авторизация (RBAC). + +## Стек +- Backend: Python, FastAPI, SQLAlchemy, APScheduler. +- Frontend: SvelteKit, Vite, Tailwind CSS. +- База данных: PostgreSQL (основная конфигурация), поддержка миграции с legacy SQLite. + +## Структура репозитория +- `backend/` — API, плагины, сервисы, скрипты миграции и тесты. +- `frontend/` — SPA-интерфейс (SvelteKit). +- `docs/` — документация по архитектуре и плагинам. +- `specs/` — спецификации и планы реализации. +- `docker/` и `docker-compose.yml` — контейнеризация. + +## Быстрый старт (локально) + +### Требования +- Python 3.9+ +- Node.js 18+ +- npm + +### Запуск backend + frontend одним скриптом +```bash +./run.sh +``` + +Что делает `run.sh`: +- проверяет версии Python/npm; +- создает `backend/.venv` (если нет); +- устанавливает `backend/requirements.txt` и `frontend` зависимости; +- запускает backend и frontend параллельно. + +Опции: +- `./run.sh --skip-install` — пропустить установку зависимостей. +- `./run.sh --help` — показать справку. + +Переменные окружения для локального запуска: +- `BACKEND_PORT` (по умолчанию `8000`) +- `FRONTEND_PORT` (по умолчанию `5173`) +- `POSTGRES_URL` +- `DATABASE_URL` +- `TASKS_DATABASE_URL` +- `AUTH_DATABASE_URL` + +## Docker + +### Запуск ```bash docker compose up --build ``` -После старта: -- UI/API: `http://localhost:8000` -- PostgreSQL: `localhost:5432` (`postgres/postgres`, DB `ss_tools`) +После старта сервисы доступны по адресам: +- Frontend: `http://localhost:8000` +- Backend API: `http://localhost:8001` +- PostgreSQL: `localhost:5432` (`postgres/postgres`, БД `ss_tools`) -Остановить: +### Остановка ```bash docker compose down ``` -Полная очистка тома БД: +### Очистка БД-тома ```bash docker compose down -v ``` -Если `postgres:16-alpine` не тянется из Docker Hub (TLS timeout), используйте fallback image: +### Альтернативный образ PostgreSQL +Если есть проблемы с pull `postgres:16-alpine`: ```bash POSTGRES_IMAGE=mirror.gcr.io/library/postgres:16-alpine docker compose up -d db ``` -или: +или ```bash POSTGRES_IMAGE=bitnami/postgresql:latest docker compose up -d db ``` -Если на хосте уже занят `5432`, поднимайте Postgres на другом порту: + +Если порт `5432` занят: ```bash POSTGRES_HOST_PORT=5433 docker compose up -d db ``` -### Миграция legacy-данных в PostgreSQL -Если нужно перенести старые данные из `tasks.db`/`config.json`: +## Разработка + +### Ручной запуск сервисов ```bash cd backend -PYTHONPATH=. .venv/bin/python src/scripts/migrate_sqlite_to_postgres.py --sqlite-path tasks.db +python3 -m venv .venv +source .venv/bin/activate +pip install -r requirements.txt +python3 -m uvicorn src.app:app --reload --port 8000 ``` -### CI/CD -Добавлен workflow: `.github/workflows/ci-cd.yml` -- backend smoke tests -- frontend build -- docker build -- push образа в GHCR на `main/master` +В другом терминале: +```bash +cd frontend +npm install +npm run dev -- --port 5173 +``` -## Контакты и вклад -Для добавления новых функций или исправления ошибок, пожалуйста, ознакомьтесь с `docs/plugin_dev.md` и создайте соответствующую спецификацию в `specs/`. +### Тесты +Backend: +```bash +cd backend +source .venv/bin/activate +pytest +``` + +Frontend: +```bash +cd frontend +npm run test +``` + +## Инициализация auth (опционально) +```bash +cd backend +source .venv/bin/activate +python src/scripts/init_auth_db.py +python src/scripts/create_admin.py --username admin --password admin +``` + +## Миграция legacy-данных (опционально) +```bash +cd backend +source .venv/bin/activate +PYTHONPATH=. python src/scripts/migrate_sqlite_to_postgres.py --sqlite-path tasks.db +``` + +## Дополнительная документация +- `docs/plugin_dev.md` +- `docs/settings.md` +- `semantic_protocol.md` diff --git a/backend/src/api/routes/reports.py b/backend/src/api/routes/reports.py index dca69f85..b1bdf0d3 100644 --- a/backend/src/api/routes/reports.py +++ b/backend/src/api/routes/reports.py @@ -32,27 +32,28 @@ router = APIRouter(prefix="/api/reports", tags=["Reports"]) # @PARAM: field_name (str) - Query field name for diagnostics. # @RETURN: List - Parsed enum values. def _parse_csv_enum_list(raw: Optional[str], enum_cls, field_name: str) -> List: - if raw is None or not raw.strip(): - return [] - values = [item.strip() for item in raw.split(",") if item.strip()] - parsed = [] - invalid = [] - for value in values: - try: - parsed.append(enum_cls(value)) - except ValueError: - invalid.append(value) - if invalid: - raise HTTPException( - status_code=status.HTTP_400_BAD_REQUEST, - detail={ - "message": f"Invalid values for '{field_name}'", - "field": field_name, - "invalid_values": invalid, - "allowed_values": [item.value for item in enum_cls], - }, - ) - return parsed + with belief_scope("_parse_csv_enum_list"): + if raw is None or not raw.strip(): + return [] + values = [item.strip() for item in raw.split(",") if item.strip()] + parsed = [] + invalid = [] + for value in values: + try: + parsed.append(enum_cls(value)) + except ValueError: + invalid.append(value) + if invalid: + raise HTTPException( + status_code=status.HTTP_400_BAD_REQUEST, + detail={ + "message": f"Invalid values for '{field_name}'", + "field": field_name, + "invalid_values": invalid, + "allowed_values": [item.value for item in enum_cls], + }, + ) + return parsed # [/DEF:_parse_csv_enum_list:Function] diff --git a/backend/src/app.py b/backend/src/app.py index d3509a56..9530d908 100755 --- a/backend/src/app.py +++ b/backend/src/app.py @@ -21,7 +21,7 @@ import asyncio from .dependencies import get_task_manager, get_scheduler_service from .core.utils.network import NetworkError from .core.logger import logger, belief_scope -from .api.routes import plugins, tasks, settings, environments, mappings, migration, connections, git, storage, admin, llm, dashboards, datasets, reports, assistant +from .api.routes import plugins, tasks, settings, environments, mappings, migration, connections, git, storage, admin, llm, dashboards, datasets, reports, assistant from .api import auth # [DEF:App:Global] @@ -72,12 +72,12 @@ app.add_middleware( ) -# [DEF:log_requests:Function] -# @PURPOSE: Middleware to log incoming HTTP requests and their response status. +# [DEF:network_error_handler:Function] +# @PURPOSE: Global exception handler for NetworkError. # @PRE: request is a FastAPI Request object. -# @POST: Logs request and response details. +# @POST: Returns 503 HTTP Exception. # @PARAM: request (Request) - The incoming request object. -# @PARAM: call_next (Callable) - The next middleware or route handler. +# @PARAM: exc (NetworkError) - The exception instance. @app.exception_handler(NetworkError) async def network_error_handler(request: Request, exc: NetworkError): with belief_scope("network_error_handler"): @@ -86,26 +86,34 @@ async def network_error_handler(request: Request, exc: NetworkError): status_code=503, detail="Environment unavailable. Please check if the Superset instance is running." ) +# [/DEF:network_error_handler:Function] +# [DEF:log_requests:Function] +# @PURPOSE: Middleware to log incoming HTTP requests and their response status. +# @PRE: request is a FastAPI Request object. +# @POST: Logs request and response details. +# @PARAM: request (Request) - The incoming request object. +# @PARAM: call_next (Callable) - The next middleware or route handler. @app.middleware("http") async def log_requests(request: Request, call_next): - # Avoid spamming logs for polling endpoints - is_polling = request.url.path.endswith("/api/tasks") and request.method == "GET" - - if not is_polling: - logger.info(f"Incoming request: {request.method} {request.url.path}") - - try: - response = await call_next(request) + with belief_scope("log_requests"): + # Avoid spamming logs for polling endpoints + is_polling = request.url.path.endswith("/api/tasks") and request.method == "GET" + if not is_polling: - logger.info(f"Response status: {response.status_code} for {request.url.path}") - return response - except NetworkError as e: - logger.error(f"Network error caught in middleware: {e}") - raise HTTPException( - status_code=503, - detail="Environment unavailable. Please check if the Superset instance is running." - ) + logger.info(f"Incoming request: {request.method} {request.url.path}") + + try: + response = await call_next(request) + if not is_polling: + logger.info(f"Response status: {response.status_code} for {request.url.path}") + return response + except NetworkError as e: + logger.error(f"Network error caught in middleware: {e}") + raise HTTPException( + status_code=503, + detail="Environment unavailable. Please check if the Superset instance is running." + ) # [/DEF:log_requests:Function] # Include API routes @@ -119,12 +127,12 @@ app.include_router(environments.router, tags=["Environments"]) app.include_router(mappings.router, prefix="/api/mappings", tags=["Mappings"]) app.include_router(migration.router) app.include_router(git.router, prefix="/api/git", tags=["Git"]) -app.include_router(llm.router, prefix="/api/llm", tags=["LLM"]) -app.include_router(storage.router, prefix="/api/storage", tags=["Storage"]) -app.include_router(dashboards.router) -app.include_router(datasets.router) -app.include_router(reports.router) -app.include_router(assistant.router, prefix="/api/assistant", tags=["Assistant"]) +app.include_router(llm.router, prefix="/api/llm", tags=["LLM"]) +app.include_router(storage.router, prefix="/api/storage", tags=["Storage"]) +app.include_router(dashboards.router) +app.include_router(datasets.router) +app.include_router(reports.router) +app.include_router(assistant.router, prefix="/api/assistant", tags=["Assistant"]) # [DEF:api.include_routers:Action] @@ -249,12 +257,13 @@ if frontend_path.exists(): # @POST: Returns the requested file or index.html. @app.get("/{file_path:path}", include_in_schema=False) async def serve_spa(file_path: str): - # Only serve SPA for non-API paths - # API routes are registered separately and should be matched by FastAPI first - if file_path and (file_path.startswith("api/") or file_path.startswith("/api/") or file_path == "api"): - # This should not happen if API routers are properly registered - # Return 404 instead of serving HTML - raise HTTPException(status_code=404, detail=f"API endpoint not found: {file_path}") + with belief_scope("serve_spa"): + # Only serve SPA for non-API paths + # API routes are registered separately and should be matched by FastAPI first + if file_path and (file_path.startswith("api/") or file_path.startswith("/api/") or file_path == "api"): + # This should not happen if API routers are properly registered + # Return 404 instead of serving HTML + raise HTTPException(status_code=404, detail=f"API endpoint not found: {file_path}") full_path = frontend_path / file_path if file_path and full_path.is_file(): diff --git a/backend/src/core/logger.py b/backend/src/core/logger.py index c4fad7a5..5e954c7b 100755 --- a/backend/src/core/logger.py +++ b/backend/src/core/logger.py @@ -35,7 +35,19 @@ class BeliefFormatter(logging.Formatter): def format(self, record): anchor_id = getattr(_belief_state, 'anchor_id', None) if anchor_id: - record.msg = f"[{anchor_id}][Action] {record.msg}" + msg = str(record.msg) + # Supported molecular topology markers + markers = ("[EXPLORE]", "[REASON]", "[REFLECT]", "[COHERENCE:", "[Action]", "[Entry]", "[Exit]") + + # Avoid duplicating anchor or overriding explicit markers + if msg.startswith(f"[{anchor_id}]"): + pass + elif any(msg.startswith(m) for m in markers): + record.msg = f"[{anchor_id}]{msg}" + else: + # Default covalent bond + record.msg = f"[{anchor_id}][Action] {msg}" + return super().format(record) # [/DEF:format:Function] # [/DEF:BeliefFormatter:Class] @@ -75,12 +87,12 @@ def belief_scope(anchor_id: str, message: str = ""): try: yield # Log Coherence OK and Exit (DEBUG level to reduce noise) - logger.debug(f"[{anchor_id}][Coherence:OK]") + logger.debug("[COHERENCE:OK]") if _enable_belief_state: - logger.debug(f"[{anchor_id}][Exit]") + logger.debug("[Exit]") except Exception as e: # Log Coherence Failed (DEBUG level to reduce noise) - logger.debug(f"[{anchor_id}][Coherence:Failed] {str(e)}") + logger.debug(f"[COHERENCE:FAILED] {str(e)}") raise finally: # Restore old anchor @@ -275,5 +287,33 @@ logger.addHandler(websocket_log_handler) # Example usage: # logger.info("Application started", extra={"context_key": "context_value"}) # logger.error("An error occurred", exc_info=True) + +import types + +# [DEF:explore:Function] +# @PURPOSE: Logs an EXPLORE message (Van der Waals force) for searching, alternatives, and hypotheses. +# @SEMANTICS: log, explore, molecule +def explore(self, msg, *args, **kwargs): + self.warning(f"[EXPLORE] {msg}", *args, **kwargs) +# [/DEF:explore:Function] + +# [DEF:reason:Function] +# @PURPOSE: Logs a REASON message (Covalent bond) for strict deduction and core logic. +# @SEMANTICS: log, reason, molecule +def reason(self, msg, *args, **kwargs): + self.info(f"[REASON] {msg}", *args, **kwargs) +# [/DEF:reason:Function] + +# [DEF:reflect:Function] +# @PURPOSE: Logs a REFLECT message (Hydrogen bond) for self-check and structural validation. +# @SEMANTICS: log, reflect, molecule +def reflect(self, msg, *args, **kwargs): + self.debug(f"[REFLECT] {msg}", *args, **kwargs) +# [/DEF:reflect:Function] + +logger.explore = types.MethodType(explore, logger) +logger.reason = types.MethodType(reason, logger) +logger.reflect = types.MethodType(reflect, logger) + # [/DEF:Logger:Global] # [/DEF:LoggerModule:Module] \ No newline at end of file diff --git a/backend/src/core/task_manager/context.py b/backend/src/core/task_manager/context.py index e1b0083b..b641d93c 100644 --- a/backend/src/core/task_manager/context.py +++ b/backend/src/core/task_manager/context.py @@ -6,9 +6,11 @@ # @TIER: CRITICAL # @INVARIANT: Each TaskContext is bound to a single task execution. +# [SECTION: IMPORTS] # [SECTION: IMPORTS] from typing import Dict, Any, Callable from .task_logger import TaskLogger +from ..logger import belief_scope # [/SECTION] # [DEF:TaskContext:Class] @@ -44,13 +46,14 @@ class TaskContext: params: Dict[str, Any], default_source: str = "plugin" ): - self._task_id = task_id - self._params = params - self._logger = TaskLogger( - task_id=task_id, - add_log_fn=add_log_fn, - source=default_source - ) + with belief_scope("__init__"): + self._task_id = task_id + self._params = params + self._logger = TaskLogger( + task_id=task_id, + add_log_fn=add_log_fn, + source=default_source + ) # [/DEF:__init__:Function] # [DEF:task_id:Function] @@ -60,7 +63,8 @@ class TaskContext: # @RETURN: str - The task ID. @property def task_id(self) -> str: - return self._task_id + with belief_scope("task_id"): + return self._task_id # [/DEF:task_id:Function] # [DEF:logger:Function] @@ -70,7 +74,8 @@ class TaskContext: # @RETURN: TaskLogger - The logger instance. @property def logger(self) -> TaskLogger: - return self._logger + with belief_scope("logger"): + return self._logger # [/DEF:logger:Function] # [DEF:params:Function] @@ -80,7 +85,8 @@ class TaskContext: # @RETURN: Dict[str, Any] - The task parameters. @property def params(self) -> Dict[str, Any]: - return self._params + with belief_scope("params"): + return self._params # [/DEF:params:Function] # [DEF:get_param:Function] @@ -91,7 +97,8 @@ class TaskContext: # @PARAM: default (Any) - Default value if key not found. # @RETURN: Any - Parameter value or default. def get_param(self, key: str, default: Any = None) -> Any: - return self._params.get(key, default) + with belief_scope("get_param"): + return self._params.get(key, default) # [/DEF:get_param:Function] # [DEF:create_sub_context:Function] @@ -102,12 +109,13 @@ class TaskContext: # @RETURN: TaskContext - New context with different source. def create_sub_context(self, source: str) -> "TaskContext": """Create a sub-context with a different default source for logging.""" - return TaskContext( - task_id=self._task_id, - add_log_fn=self._logger._add_log, - params=self._params, - default_source=source - ) + with belief_scope("create_sub_context"): + return TaskContext( + task_id=self._task_id, + add_log_fn=self._logger._add_log, + params=self._params, + default_source=source + ) # [/DEF:create_sub_context:Function] # [/DEF:TaskContext:Class] diff --git a/backend/src/core/task_manager/manager.py b/backend/src/core/task_manager/manager.py index 759c9969..2704f9e3 100644 --- a/backend/src/core/task_manager/manager.py +++ b/backend/src/core/task_manager/manager.py @@ -1,4 +1,5 @@ # [DEF:TaskManagerModule:Module] +# @TIER: CRITICAL # @SEMANTICS: task, manager, lifecycle, execution, state # @PURPOSE: Manages the lifecycle of tasks, including their creation, execution, and state tracking. It uses a thread pool to run plugins asynchronously. # @LAYER: Core @@ -74,9 +75,10 @@ class TaskManager: # @POST: Logs are batch-written to database every LOG_FLUSH_INTERVAL seconds. def _flusher_loop(self): """Background thread that flushes log buffer to database.""" - while not self._flusher_stop_event.is_set(): - self._flush_logs() - self._flusher_stop_event.wait(self.LOG_FLUSH_INTERVAL) + with belief_scope("_flusher_loop"): + while not self._flusher_stop_event.is_set(): + self._flush_logs() + self._flusher_stop_event.wait(self.LOG_FLUSH_INTERVAL) # [/DEF:_flusher_loop:Function] # [DEF:_flush_logs:Function] @@ -85,23 +87,24 @@ class TaskManager: # @POST: All buffered logs are written to task_logs table. def _flush_logs(self): """Flush all buffered logs to the database.""" - with self._log_buffer_lock: - task_ids = list(self._log_buffer.keys()) - - for task_id in task_ids: + with belief_scope("_flush_logs"): with self._log_buffer_lock: - logs = self._log_buffer.pop(task_id, []) + task_ids = list(self._log_buffer.keys()) - if logs: - try: - self.log_persistence_service.add_logs(task_id, logs) - except Exception as e: - logger.error(f"Failed to flush logs for task {task_id}: {e}") - # Re-add logs to buffer on failure - with self._log_buffer_lock: - if task_id not in self._log_buffer: - self._log_buffer[task_id] = [] - self._log_buffer[task_id].extend(logs) + for task_id in task_ids: + with self._log_buffer_lock: + logs = self._log_buffer.pop(task_id, []) + + if logs: + try: + self.log_persistence_service.add_logs(task_id, logs) + except Exception as e: + logger.error(f"Failed to flush logs for task {task_id}: {e}") + # Re-add logs to buffer on failure + with self._log_buffer_lock: + if task_id not in self._log_buffer: + self._log_buffer[task_id] = [] + self._log_buffer[task_id].extend(logs) # [/DEF:_flush_logs:Function] # [DEF:_flush_task_logs:Function] @@ -111,14 +114,15 @@ class TaskManager: # @PARAM: task_id (str) - The task ID. def _flush_task_logs(self, task_id: str): """Flush logs for a specific task immediately.""" - with self._log_buffer_lock: - logs = self._log_buffer.pop(task_id, []) - - if logs: - try: - self.log_persistence_service.add_logs(task_id, logs) - except Exception as e: - logger.error(f"Failed to flush logs for task {task_id}: {e}") + with belief_scope("_flush_task_logs"): + with self._log_buffer_lock: + logs = self._log_buffer.pop(task_id, []) + + if logs: + try: + self.log_persistence_service.add_logs(task_id, logs) + except Exception as e: + logger.error(f"Failed to flush logs for task {task_id}: {e}") # [/DEF:_flush_task_logs:Function] # [DEF:create_task:Function] diff --git a/backend/src/core/task_manager/persistence.py b/backend/src/core/task_manager/persistence.py index 743eee0f..b3b87469 100644 --- a/backend/src/core/task_manager/persistence.py +++ b/backend/src/core/task_manager/persistence.py @@ -1,4 +1,5 @@ # [DEF:TaskPersistenceModule:Module] +# @TIER: CRITICAL # @SEMANTICS: persistence, sqlite, sqlalchemy, task, storage # @PURPOSE: Handles the persistence of tasks using SQLAlchemy and the tasks.db database. # @LAYER: Core @@ -19,42 +20,65 @@ from ..logger import logger, belief_scope # [/SECTION] # [DEF:TaskPersistenceService:Class] +# @TIER: CRITICAL # @SEMANTICS: persistence, service, database, sqlalchemy # @PURPOSE: Provides methods to save and load tasks from the tasks.db database using SQLAlchemy. +# @INVARIANT: Persistence must handle potentially missing task fields natively. class TaskPersistenceService: + # [DEF:_json_load_if_needed:Function] + # @PURPOSE: Safely load JSON strings from DB if necessary + # @PRE: value is an arbitrary database value + # @POST: Returns parsed JSON object, list, string, or primitive @staticmethod def _json_load_if_needed(value): - if value is None: - return None - if isinstance(value, (dict, list)): - return value - if isinstance(value, str): - stripped = value.strip() - if stripped == "" or stripped.lower() == "null": + with belief_scope("TaskPersistenceService._json_load_if_needed"): + if value is None: return None - try: - return json.loads(stripped) - except json.JSONDecodeError: + if isinstance(value, (dict, list)): return value - return value + if isinstance(value, str): + stripped = value.strip() + if stripped == "" or stripped.lower() == "null": + return None + try: + return json.loads(stripped) + except json.JSONDecodeError: + return value + return value + # [/DEF:_json_load_if_needed:Function] + # [DEF:_parse_datetime:Function] + # @PURPOSE: Safely parse a datetime string from the database + # @PRE: value is an ISO string or datetime object + # @POST: Returns datetime object or None @staticmethod def _parse_datetime(value): - if value is None or isinstance(value, datetime): - return value - if isinstance(value, str): - try: - return datetime.fromisoformat(value) - except ValueError: - return None - return None - - @staticmethod - def _resolve_environment_id(session: Session, env_id: Optional[str]) -> Optional[str]: - if not env_id: + with belief_scope("TaskPersistenceService._parse_datetime"): + if value is None or isinstance(value, datetime): + return value + if isinstance(value, str): + try: + return datetime.fromisoformat(value) + except ValueError: + return None return None - exists = session.query(Environment.id).filter(Environment.id == env_id).first() - return env_id if exists else None + # [/DEF:_parse_datetime:Function] + + # [DEF:_resolve_environment_id:Function] + # @TIER: STANDARD + # @PURPOSE: Resolve environment id based on provided value or fallback to default + # @PRE: Session is active + # @POST: Environment ID is returned + @staticmethod + def _resolve_environment_id(session: Session, env_id: Optional[str]) -> str: + with belief_scope("_resolve_environment_id"): + if env_id: + return env_id + repo_env = session.query(Environment).filter_by(name="default").first() + if repo_env: + return str(repo_env.id) + return "default" + # [/DEF:_resolve_environment_id:Function] # [DEF:__init__:Function] # @PURPOSE: Initializes the persistence service. @@ -90,13 +114,14 @@ class TaskPersistenceService: # Ensure params and result are JSON serializable def json_serializable(obj): - if isinstance(obj, dict): - return {k: json_serializable(v) for k, v in obj.items()} - elif isinstance(obj, list): - return [json_serializable(v) for v in obj] - elif isinstance(obj, datetime): - return obj.isoformat() - return obj + with belief_scope("TaskPersistenceService.json_serializable"): + if isinstance(obj, dict): + return {k: json_serializable(v) for k, v in obj.items()} + elif isinstance(obj, list): + return [json_serializable(v) for v in obj] + elif isinstance(obj, datetime): + return obj.isoformat() + return obj record.params = json_serializable(task.params) record.result = json_serializable(task.result) @@ -227,9 +252,11 @@ class TaskLogPersistenceService: """ # [DEF:__init__:Function] - # @PURPOSE: Initialize the log persistence service. - # @POST: Service is ready. - def __init__(self): + # @TIER: STANDARD + # @PURPOSE: Initializes the TaskLogPersistenceService + # @PRE: config is provided or defaults are used + # @POST: Service is ready for log persistence + def __init__(self, config=None): pass # [/DEF:__init__:Function] diff --git a/backend/src/core/task_manager/task_logger.py b/backend/src/core/task_manager/task_logger.py index e850bb11..67d4bc1c 100644 --- a/backend/src/core/task_manager/task_logger.py +++ b/backend/src/core/task_manager/task_logger.py @@ -15,6 +15,7 @@ from typing import Dict, Any, Optional, Callable # @PURPOSE: A wrapper around TaskManager._add_log that carries task_id and source context. # @TIER: CRITICAL # @INVARIANT: All log calls include the task_id and source. +# @TEST_DATA: task_logger -> {"task_id": "test_123", "source": "test_plugin"} # @UX_STATE: Idle -> Logging -> (system records log) class TaskLogger: """ @@ -71,6 +72,7 @@ class TaskLogger: # @PARAM: message (str) - Log message. # @PARAM: source (Optional[str]) - Override source for this log entry. # @PARAM: metadata (Optional[Dict]) - Additional structured data. + # @UX_STATE: Logging -> (writing internal log) def _log( self, level: str, @@ -90,6 +92,8 @@ class TaskLogger: # [DEF:debug:Function] # @PURPOSE: Log a DEBUG level message. + # @PRE: message is a string. + # @POST: Log entry added via internally with DEBUG level. # @PARAM: message (str) - Log message. # @PARAM: source (Optional[str]) - Override source. # @PARAM: metadata (Optional[Dict]) - Additional data. @@ -104,6 +108,8 @@ class TaskLogger: # [DEF:info:Function] # @PURPOSE: Log an INFO level message. + # @PRE: message is a string. + # @POST: Log entry added internally with INFO level. # @PARAM: message (str) - Log message. # @PARAM: source (Optional[str]) - Override source. # @PARAM: metadata (Optional[Dict]) - Additional data. @@ -118,6 +124,8 @@ class TaskLogger: # [DEF:warning:Function] # @PURPOSE: Log a WARNING level message. + # @PRE: message is a string. + # @POST: Log entry added internally with WARNING level. # @PARAM: message (str) - Log message. # @PARAM: source (Optional[str]) - Override source. # @PARAM: metadata (Optional[Dict]) - Additional data. @@ -132,6 +140,8 @@ class TaskLogger: # [DEF:error:Function] # @PURPOSE: Log an ERROR level message. + # @PRE: message is a string. + # @POST: Log entry added internally with ERROR level. # @PARAM: message (str) - Log message. # @PARAM: source (Optional[str]) - Override source. # @PARAM: metadata (Optional[Dict]) - Additional data. diff --git a/backend/src/models/report.py b/backend/src/models/report.py index ebbfdffc..a1abb49b 100644 --- a/backend/src/models/report.py +++ b/backend/src/models/report.py @@ -16,6 +16,9 @@ from pydantic import BaseModel, Field, field_validator, model_validator # [DEF:TaskType:Class] +# @TIER: CRITICAL +# @INVARIANT: Must contain valid generic task type mappings. +# @SEMANTICS: enum, type, task # @PURPOSE: Supported normalized task report types. class TaskType(str, Enum): LLM_VERIFICATION = "llm_verification" @@ -27,6 +30,9 @@ class TaskType(str, Enum): # [DEF:ReportStatus:Class] +# @TIER: CRITICAL +# @INVARIANT: TaskStatus enum mapping logic holds. +# @SEMANTICS: enum, status, task # @PURPOSE: Supported normalized report status values. class ReportStatus(str, Enum): SUCCESS = "success" @@ -37,6 +43,9 @@ class ReportStatus(str, Enum): # [DEF:ErrorContext:Class] +# @TIER: CRITICAL +# @INVARIANT: The properties accurately describe error state. +# @SEMANTICS: error, context, payload # @PURPOSE: Error and recovery context for failed/partial reports. class ErrorContext(BaseModel): code: Optional[str] = None @@ -46,6 +55,9 @@ class ErrorContext(BaseModel): # [DEF:TaskReport:Class] +# @TIER: CRITICAL +# @INVARIANT: Must represent canonical task record attributes. +# @SEMANTICS: report, model, summary # @PURPOSE: Canonical normalized report envelope for one task execution. class TaskReport(BaseModel): report_id: str @@ -69,6 +81,9 @@ class TaskReport(BaseModel): # [DEF:ReportQuery:Class] +# @TIER: CRITICAL +# @INVARIANT: Time and pagination queries are mutually consistent. +# @SEMANTICS: query, filter, search # @PURPOSE: Query object for server-side report filtering, sorting, and pagination. class ReportQuery(BaseModel): page: int = Field(default=1, ge=1) @@ -105,6 +120,9 @@ class ReportQuery(BaseModel): # [DEF:ReportCollection:Class] +# @TIER: CRITICAL +# @INVARIANT: Represents paginated data correctly. +# @SEMANTICS: collection, pagination # @PURPOSE: Paginated collection of normalized task reports. class ReportCollection(BaseModel): items: List[TaskReport] @@ -117,6 +135,9 @@ class ReportCollection(BaseModel): # [DEF:ReportDetailView:Class] +# @TIER: CRITICAL +# @INVARIANT: Incorporates a report and logs correctly. +# @SEMANTICS: view, detail, logs # @PURPOSE: Detailed report representation including diagnostics and recovery actions. class ReportDetailView(BaseModel): report: TaskReport diff --git a/backend/src/services/llm_provider.py b/backend/src/services/llm_provider.py index b4ebd407..270d7c56 100644 --- a/backend/src/services/llm_provider.py +++ b/backend/src/services/llm_provider.py @@ -33,7 +33,8 @@ class EncryptionManager: # @PRE: data must be a non-empty string. # @POST: Returns encrypted string. def encrypt(self, data: str) -> str: - return self.fernet.encrypt(data.encode()).decode() + with belief_scope("encrypt"): + return self.fernet.encrypt(data.encode()).decode() # [/DEF:EncryptionManager.encrypt:Function] # [DEF:EncryptionManager.decrypt:Function] @@ -41,7 +42,8 @@ class EncryptionManager: # @PRE: encrypted_data must be a valid Fernet-encrypted string. # @POST: Returns original plaintext string. def decrypt(self, encrypted_data: str) -> str: - return self.fernet.decrypt(encrypted_data.encode()).decode() + with belief_scope("decrypt"): + return self.fernet.decrypt(encrypted_data.encode()).decode() # [/DEF:EncryptionManager.decrypt:Function] # [/DEF:EncryptionManager:Class] diff --git a/backend/src/services/reports/normalizer.py b/backend/src/services/reports/normalizer.py index b643d79c..25146a80 100644 --- a/backend/src/services/reports/normalizer.py +++ b/backend/src/services/reports/normalizer.py @@ -12,6 +12,7 @@ from datetime import datetime from typing import Any, Dict, Optional +from ...core.logger import belief_scope from ...core.task_manager.models import Task, TaskStatus from ...models.report import ErrorContext, ReportStatus, TaskReport from .type_profiles import get_type_profile, resolve_task_type @@ -25,14 +26,15 @@ from .type_profiles import get_type_profile, resolve_task_type # @PARAM: status (Any) - Internal task status value. # @RETURN: ReportStatus - Canonical report status. def status_to_report_status(status: Any) -> ReportStatus: - raw = str(status.value if isinstance(status, TaskStatus) else status).upper() - if raw == TaskStatus.SUCCESS.value: - return ReportStatus.SUCCESS - if raw == TaskStatus.FAILED.value: - return ReportStatus.FAILED - if raw in {TaskStatus.PENDING.value, TaskStatus.RUNNING.value, TaskStatus.AWAITING_INPUT.value, TaskStatus.AWAITING_MAPPING.value}: - return ReportStatus.IN_PROGRESS - return ReportStatus.PARTIAL + with belief_scope("status_to_report_status"): + raw = str(status.value if isinstance(status, TaskStatus) else status).upper() + if raw == TaskStatus.SUCCESS.value: + return ReportStatus.SUCCESS + if raw == TaskStatus.FAILED.value: + return ReportStatus.FAILED + if raw in {TaskStatus.PENDING.value, TaskStatus.RUNNING.value, TaskStatus.AWAITING_INPUT.value, TaskStatus.AWAITING_MAPPING.value}: + return ReportStatus.IN_PROGRESS + return ReportStatus.PARTIAL # [/DEF:status_to_report_status:Function] @@ -44,19 +46,20 @@ def status_to_report_status(status: Any) -> ReportStatus: # @PARAM: report_status (ReportStatus) - Canonical status. # @RETURN: str - Normalized summary. def build_summary(task: Task, report_status: ReportStatus) -> str: - result = task.result - if isinstance(result, dict): - for key in ("summary", "message", "status_message", "description"): - value = result.get(key) - if isinstance(value, str) and value.strip(): - return value.strip() - if report_status == ReportStatus.SUCCESS: - return "Task completed successfully" - if report_status == ReportStatus.FAILED: - return "Task failed" - if report_status == ReportStatus.IN_PROGRESS: - return "Task is in progress" - return "Task completed with partial data" + with belief_scope("build_summary"): + result = task.result + if isinstance(result, dict): + for key in ("summary", "message", "status_message", "description"): + value = result.get(key) + if isinstance(value, str) and value.strip(): + return value.strip() + if report_status == ReportStatus.SUCCESS: + return "Task completed successfully" + if report_status == ReportStatus.FAILED: + return "Task failed" + if report_status == ReportStatus.IN_PROGRESS: + return "Task is in progress" + return "Task completed with partial data" # [/DEF:build_summary:Function] @@ -68,38 +71,39 @@ def build_summary(task: Task, report_status: ReportStatus) -> str: # @PARAM: report_status (ReportStatus) - Canonical status. # @RETURN: Optional[ErrorContext] - Error context block. def extract_error_context(task: Task, report_status: ReportStatus) -> Optional[ErrorContext]: - if report_status not in {ReportStatus.FAILED, ReportStatus.PARTIAL}: - return None + with belief_scope("extract_error_context"): + if report_status not in {ReportStatus.FAILED, ReportStatus.PARTIAL}: + return None - result = task.result if isinstance(task.result, dict) else {} - message = None - code = None - next_actions = [] + result = task.result if isinstance(task.result, dict) else {} + message = None + code = None + next_actions = [] - if isinstance(result.get("error"), dict): - error_obj = result.get("error", {}) - message = error_obj.get("message") or message - code = error_obj.get("code") or code - actions = error_obj.get("next_actions") - if isinstance(actions, list): - next_actions = [str(action) for action in actions if str(action).strip()] + if isinstance(result.get("error"), dict): + error_obj = result.get("error", {}) + message = error_obj.get("message") or message + code = error_obj.get("code") or code + actions = error_obj.get("next_actions") + if isinstance(actions, list): + next_actions = [str(action) for action in actions if str(action).strip()] - if not message: - message = result.get("error_message") if isinstance(result.get("error_message"), str) else None + if not message: + message = result.get("error_message") if isinstance(result.get("error_message"), str) else None - if not message: - for log in reversed(task.logs): - if str(log.level).upper() == "ERROR" and log.message: - message = log.message - break + if not message: + for log in reversed(task.logs): + if str(log.level).upper() == "ERROR" and log.message: + message = log.message + break - if not message: - message = "Not provided" + if not message: + message = "Not provided" - if not next_actions: - next_actions = ["Review task diagnostics", "Retry the operation"] + if not next_actions: + next_actions = ["Review task diagnostics", "Retry the operation"] - return ErrorContext(code=code, message=message, next_actions=next_actions) + return ErrorContext(code=code, message=message, next_actions=next_actions) # [/DEF:extract_error_context:Function] @@ -110,43 +114,44 @@ def extract_error_context(task: Task, report_status: ReportStatus) -> Optional[E # @PARAM: task (Task) - Source task. # @RETURN: TaskReport - Canonical normalized report. def normalize_task_report(task: Task) -> TaskReport: - task_type = resolve_task_type(task.plugin_id) - report_status = status_to_report_status(task.status) - profile = get_type_profile(task_type) + with belief_scope("normalize_task_report"): + task_type = resolve_task_type(task.plugin_id) + report_status = status_to_report_status(task.status) + profile = get_type_profile(task_type) - started_at = task.started_at if isinstance(task.started_at, datetime) else None - updated_at = task.finished_at if isinstance(task.finished_at, datetime) else None - if not updated_at: - updated_at = started_at or datetime.utcnow() + started_at = task.started_at if isinstance(task.started_at, datetime) else None + updated_at = task.finished_at if isinstance(task.finished_at, datetime) else None + if not updated_at: + updated_at = started_at or datetime.utcnow() - details: Dict[str, Any] = { - "profile": { - "display_label": profile.get("display_label"), - "visual_variant": profile.get("visual_variant"), - "icon_token": profile.get("icon_token"), - "emphasis_rules": profile.get("emphasis_rules", []), - }, - "result": task.result if task.result is not None else {"note": "Not provided"}, - } + details: Dict[str, Any] = { + "profile": { + "display_label": profile.get("display_label"), + "visual_variant": profile.get("visual_variant"), + "icon_token": profile.get("icon_token"), + "emphasis_rules": profile.get("emphasis_rules", []), + }, + "result": task.result if task.result is not None else {"note": "Not provided"}, + } - source_ref: Dict[str, Any] = {} - if isinstance(task.params, dict): - for key in ("environment_id", "source_env_id", "target_env_id", "dashboard_id", "dataset_id", "resource_id"): - if key in task.params: - source_ref[key] = task.params.get(key) + source_ref: Dict[str, Any] = {} + if isinstance(task.params, dict): + for key in ("environment_id", "source_env_id", "target_env_id", "dashboard_id", "dataset_id", "resource_id"): + if key in task.params: + source_ref[key] = task.params.get(key) - return TaskReport( - report_id=task.id, - task_id=task.id, - task_type=task_type, - status=report_status, - started_at=started_at, - updated_at=updated_at, - summary=build_summary(task, report_status), - details=details, - error_context=extract_error_context(task, report_status), - source_ref=source_ref or None, - ) + return TaskReport( + report_id=task.id, + task_id=task.id, + task_type=task_type, + status=report_status, + started_at=started_at, + updated_at=updated_at, + summary=build_summary(task, report_status), + details=details, + error_context=extract_error_context(task, report_status), + source_ref=source_ref or None, + ) # [/DEF:normalize_task_report:Function] # [/DEF:backend.src.services.reports.normalizer:Module] \ No newline at end of file diff --git a/backend/src/services/reports/report_service.py b/backend/src/services/reports/report_service.py index 95dbe9b5..2f107328 100644 --- a/backend/src/services/reports/report_service.py +++ b/backend/src/services/reports/report_service.py @@ -12,6 +12,8 @@ from datetime import datetime, timezone from typing import List, Optional +from ...core.logger import belief_scope + from ...core.task_manager import TaskManager from ...models.report import ReportCollection, ReportDetailView, ReportQuery, ReportStatus, TaskReport, TaskType from .normalizer import normalize_task_report @@ -33,7 +35,8 @@ class ReportsService: # @INVARIANT: Constructor performs no task mutations. # @PARAM: task_manager (TaskManager) - Task manager providing source task history. def __init__(self, task_manager: TaskManager): - self.task_manager = task_manager + with belief_scope("__init__"): + self.task_manager = task_manager # [/DEF:__init__:Function] # [DEF:_load_normalized_reports:Function] @@ -43,9 +46,10 @@ class ReportsService: # @INVARIANT: Every returned item is a TaskReport. # @RETURN: List[TaskReport] - Reports sorted later by list logic. def _load_normalized_reports(self) -> List[TaskReport]: - tasks = self.task_manager.get_all_tasks() - reports = [normalize_task_report(task) for task in tasks] - return reports + with belief_scope("_load_normalized_reports"): + tasks = self.task_manager.get_all_tasks() + reports = [normalize_task_report(task) for task in tasks] + return reports # [/DEF:_load_normalized_reports:Function] # [DEF:_to_utc_datetime:Function] @@ -56,11 +60,12 @@ class ReportsService: # @PARAM: value (Optional[datetime]) - Source datetime value. # @RETURN: Optional[datetime] - UTC-aware datetime or None. def _to_utc_datetime(self, value: Optional[datetime]) -> Optional[datetime]: - if value is None: - return None - if value.tzinfo is None: - return value.replace(tzinfo=timezone.utc) - return value.astimezone(timezone.utc) + with belief_scope("_to_utc_datetime"): + if value is None: + return None + if value.tzinfo is None: + return value.replace(tzinfo=timezone.utc) + return value.astimezone(timezone.utc) # [/DEF:_to_utc_datetime:Function] # [DEF:_datetime_sort_key:Function] @@ -71,10 +76,11 @@ class ReportsService: # @PARAM: report (TaskReport) - Report item. # @RETURN: float - UTC timestamp key. def _datetime_sort_key(self, report: TaskReport) -> float: - updated = self._to_utc_datetime(report.updated_at) - if updated is None: - return 0.0 - return updated.timestamp() + with belief_scope("_datetime_sort_key"): + updated = self._to_utc_datetime(report.updated_at) + if updated is None: + return 0.0 + return updated.timestamp() # [/DEF:_datetime_sort_key:Function] # [DEF:_matches_query:Function] @@ -86,24 +92,25 @@ class ReportsService: # @PARAM: query (ReportQuery) - Applied query. # @RETURN: bool - True if report matches all filters. def _matches_query(self, report: TaskReport, query: ReportQuery) -> bool: - if query.task_types and report.task_type not in query.task_types: - return False - if query.statuses and report.status not in query.statuses: - return False - report_updated_at = self._to_utc_datetime(report.updated_at) - query_time_from = self._to_utc_datetime(query.time_from) - query_time_to = self._to_utc_datetime(query.time_to) - - if query_time_from and report_updated_at and report_updated_at < query_time_from: - return False - if query_time_to and report_updated_at and report_updated_at > query_time_to: - return False - if query.search: - needle = query.search.lower() - haystack = f"{report.summary} {report.task_type.value} {report.status.value}".lower() - if needle not in haystack: + with belief_scope("_matches_query"): + if query.task_types and report.task_type not in query.task_types: return False - return True + if query.statuses and report.status not in query.statuses: + return False + report_updated_at = self._to_utc_datetime(report.updated_at) + query_time_from = self._to_utc_datetime(query.time_from) + query_time_to = self._to_utc_datetime(query.time_to) + + if query_time_from and report_updated_at and report_updated_at < query_time_from: + return False + if query_time_to and report_updated_at and report_updated_at > query_time_to: + return False + if query.search: + needle = query.search.lower() + haystack = f"{report.summary} {report.task_type.value} {report.status.value}".lower() + if needle not in haystack: + return False + return True # [/DEF:_matches_query:Function] # [DEF:_sort_reports:Function] @@ -115,16 +122,17 @@ class ReportsService: # @PARAM: query (ReportQuery) - Sort config. # @RETURN: List[TaskReport] - Sorted reports. def _sort_reports(self, reports: List[TaskReport], query: ReportQuery) -> List[TaskReport]: - reverse = query.sort_order == "desc" + with belief_scope("_sort_reports"): + reverse = query.sort_order == "desc" - if query.sort_by == "status": - reports.sort(key=lambda item: item.status.value, reverse=reverse) - elif query.sort_by == "task_type": - reports.sort(key=lambda item: item.task_type.value, reverse=reverse) - else: - reports.sort(key=self._datetime_sort_key, reverse=reverse) + if query.sort_by == "status": + reports.sort(key=lambda item: item.status.value, reverse=reverse) + elif query.sort_by == "task_type": + reports.sort(key=lambda item: item.task_type.value, reverse=reverse) + else: + reports.sort(key=self._datetime_sort_key, reverse=reverse) - return reports + return reports # [/DEF:_sort_reports:Function] # [DEF:list_reports:Function] @@ -134,24 +142,25 @@ class ReportsService: # @PARAM: query (ReportQuery) - List filters and pagination. # @RETURN: ReportCollection - Paginated unified reports payload. def list_reports(self, query: ReportQuery) -> ReportCollection: - reports = self._load_normalized_reports() - filtered = [report for report in reports if self._matches_query(report, query)] - sorted_reports = self._sort_reports(filtered, query) + with belief_scope("list_reports"): + reports = self._load_normalized_reports() + filtered = [report for report in reports if self._matches_query(report, query)] + sorted_reports = self._sort_reports(filtered, query) - total = len(sorted_reports) - start = (query.page - 1) * query.page_size - end = start + query.page_size - items = sorted_reports[start:end] - has_next = end < total + total = len(sorted_reports) + start = (query.page - 1) * query.page_size + end = start + query.page_size + items = sorted_reports[start:end] + has_next = end < total - return ReportCollection( - items=items, - total=total, - page=query.page, - page_size=query.page_size, - has_next=has_next, - applied_filters=query, - ) + return ReportCollection( + items=items, + total=total, + page=query.page, + page_size=query.page_size, + has_next=has_next, + applied_filters=query, + ) # [/DEF:list_reports:Function] # [DEF:get_report_detail:Function] @@ -161,34 +170,35 @@ class ReportsService: # @PARAM: report_id (str) - Stable report identifier. # @RETURN: Optional[ReportDetailView] - Detailed report or None if not found. def get_report_detail(self, report_id: str) -> Optional[ReportDetailView]: - reports = self._load_normalized_reports() - target = next((report for report in reports if report.report_id == report_id), None) - if not target: - return None + with belief_scope("get_report_detail"): + reports = self._load_normalized_reports() + target = next((report for report in reports if report.report_id == report_id), None) + if not target: + return None - timeline = [] - if target.started_at: - timeline.append({"event": "started", "at": target.started_at.isoformat()}) - timeline.append({"event": "updated", "at": target.updated_at.isoformat()}) + timeline = [] + if target.started_at: + timeline.append({"event": "started", "at": target.started_at.isoformat()}) + timeline.append({"event": "updated", "at": target.updated_at.isoformat()}) - diagnostics = target.details or {} - if not diagnostics: - diagnostics = {"note": "Not provided"} - if target.error_context: - diagnostics["error_context"] = target.error_context.model_dump() + diagnostics = target.details or {} + if not diagnostics: + diagnostics = {"note": "Not provided"} + if target.error_context: + diagnostics["error_context"] = target.error_context.model_dump() - next_actions = [] - if target.error_context and target.error_context.next_actions: - next_actions = target.error_context.next_actions - elif target.status in {ReportStatus.FAILED, ReportStatus.PARTIAL}: - next_actions = ["Review diagnostics", "Retry task if applicable"] + next_actions = [] + if target.error_context and target.error_context.next_actions: + next_actions = target.error_context.next_actions + elif target.status in {ReportStatus.FAILED, ReportStatus.PARTIAL}: + next_actions = ["Review diagnostics", "Retry task if applicable"] - return ReportDetailView( - report=target, - timeline=timeline, - diagnostics=diagnostics, - next_actions=next_actions, - ) + return ReportDetailView( + report=target, + timeline=timeline, + diagnostics=diagnostics, + next_actions=next_actions, + ) # [/DEF:get_report_detail:Function] # [/DEF:ReportsService:Class] diff --git a/backend/src/services/reports/type_profiles.py b/backend/src/services/reports/type_profiles.py index 9346cd90..f2188fb1 100644 --- a/backend/src/services/reports/type_profiles.py +++ b/backend/src/services/reports/type_profiles.py @@ -9,6 +9,7 @@ # [SECTION: IMPORTS] from typing import Any, Dict, Optional +from ...core.logger import belief_scope from ...models.report import TaskType # [/SECTION] @@ -71,10 +72,11 @@ TASK_TYPE_PROFILES: Dict[TaskType, Dict[str, Any]] = { # @PARAM: plugin_id (Optional[str]) - Source plugin/task identifier from task record. # @RETURN: TaskType - Resolved canonical type or UNKNOWN fallback. def resolve_task_type(plugin_id: Optional[str]) -> TaskType: - normalized = (plugin_id or "").strip() - if not normalized: - return TaskType.UNKNOWN - return PLUGIN_TO_TASK_TYPE.get(normalized, TaskType.UNKNOWN) + with belief_scope("resolve_task_type"): + normalized = (plugin_id or "").strip() + if not normalized: + return TaskType.UNKNOWN + return PLUGIN_TO_TASK_TYPE.get(normalized, TaskType.UNKNOWN) # [/DEF:resolve_task_type:Function] @@ -85,7 +87,8 @@ def resolve_task_type(plugin_id: Optional[str]) -> TaskType: # @PARAM: task_type (TaskType) - Canonical task type. # @RETURN: Dict[str, Any] - Profile metadata used by normalization and UI contracts. def get_type_profile(task_type: TaskType) -> Dict[str, Any]: - return TASK_TYPE_PROFILES.get(task_type, TASK_TYPE_PROFILES[TaskType.UNKNOWN]) + with belief_scope("get_type_profile"): + return TASK_TYPE_PROFILES.get(task_type, TASK_TYPE_PROFILES[TaskType.UNKNOWN]) # [/DEF:get_type_profile:Function] # [/DEF:backend.src.services.reports.type_profiles:Module] \ No newline at end of file diff --git a/frontend/src/components/TaskLogViewer.svelte b/frontend/src/components/TaskLogViewer.svelte index d0624012..84ee1154 100644 --- a/frontend/src/components/TaskLogViewer.svelte +++ b/frontend/src/components/TaskLogViewer.svelte @@ -12,6 +12,8 @@ /** * @TIER CRITICAL * @PURPOSE Displays detailed logs for a specific task inline or in a modal using TaskLogPanel. + * @PRE Needs a valid taskId to fetch logs for. + * @POST task logs are displayed and updated in real time. * @UX_STATE Loading -> Shows spinner/text while fetching initial logs * @UX_STATE Streaming -> Displays logs with auto-scroll, real-time appending * @UX_STATE Error -> Shows error message with recovery option @@ -42,6 +44,9 @@ let shouldShow = $derived(inline || show); // [DEF:handleRealTimeLogs:Action] + // @PURPOSE: Sync real-time logs to the current log list + // @PRE: None + // @POST: logs are updated with new real-time log entries $effect(() => { if (realTimeLogs && realTimeLogs.length > 0) { const lastLog = realTimeLogs[realTimeLogs.length - 1]; @@ -58,11 +63,20 @@ // [/DEF:handleRealTimeLogs:Action] // [DEF:fetchLogs:Function] + // @PURPOSE: Fetches logs for a given task ID + // @PRE: taskId is set + // @POST: logs are populated with API response async function fetchLogs() { if (!taskId) return; try { + console.log(`[TaskLogViewer][API][fetchLogs:STARTED] id=${taskId}`); logs = await getTaskLogs(taskId); + console.log(`[TaskLogViewer][API][fetchLogs:SUCCESS] id=${taskId}`); } catch (e) { + console.error( + `[TaskLogViewer][API][fetchLogs:FAILED] id=${taskId}`, + e, + ); error = e.message; } finally { loading = false; @@ -70,13 +84,25 @@ } // [/DEF:fetchLogs:Function] + // [DEF:handleFilterChange:Function] + // @PURPOSE: Updates filter conditions for the log viewer + // @PRE: event contains detail with source and level + // @POST: Log viewer filters updated function handleFilterChange(event) { + console.log("[TaskLogViewer][UI][handleFilterChange:START]"); const { source, level } = event.detail; } + // [/DEF:handleFilterChange:Function] + // [DEF:handleRefresh:Function] + // @PURPOSE: Refreshes the logs by polling the API + // @PRE: None + // @POST: Logs refetched function handleRefresh() { + console.log("[TaskLogViewer][UI][handleRefresh:START]"); fetchLogs(); } + // [/DEF:handleRefresh:Function] $effect(() => { if (shouldShow && taskId) { @@ -104,6 +130,11 @@ {#if shouldShow} + + + + + {#if inline}