167 lines
8.2 KiB
Markdown
167 lines
8.2 KiB
Markdown
# DuckDB Rebuild Timeout — Investigation for Axiom Developer
|
||
|
||
## Symptom
|
||
|
||
`axiom_semantic_index rebuild rebuild_mode="full" use_duckdb=true` consistently times out via MCP transport regardless of timeout value.
|
||
|
||
## Environment
|
||
|
||
- Binary: `/home/busya/dev/axiom-mcp-rust-port/target/release/axiom-mcp-server-rs`
|
||
- Workspace: `/home/busya/dev/ss-tools` (3185 contracts, 2149 edges, 690 files)
|
||
- DuckDB path: `.axiom/semantic_index/graph.duckdb`
|
||
- Expected DuckDB size: ~225 MB (final, from earlier successful rebuild)
|
||
- OS: Linux, 64-bit
|
||
|
||
## Investigation Log
|
||
|
||
### Attempt 1: Timeout scaling test (old binary)
|
||
|
||
| use_duckdb | Timeout | Result | Notes |
|
||
|:----------:|:-------:|:------:|-------|
|
||
| false | 120s | ✅ Success | 3185 contracts, 2149 edges |
|
||
| **true** | **120s** | ❌ **Timeout** | |
|
||
| **true** | **300s** | ❌ **Timeout** | |
|
||
| **true** | **600s** | ❌ **Timeout** | |
|
||
| **true** | **900s** | ❌ **Timeout** | |
|
||
|
||
### Attempt 2: After chunking fix + error logging (new binary)
|
||
|
||
| Step | Result | Notes |
|
||
|------|:------:|-------|
|
||
| `rebuild use_duckdb=true` | ❌ `"Failed to populate DuckDB store"` | **Timeout побеждён!** Код не зависает |
|
||
| Очистка DuckDB + retry | ❌ `"Failed to populate DuckDB store"` | Та же логическая ошибка |
|
||
| Удалён `index.duckdb` (394MB legacy) + retry | ❌ `"Failed to populate DuckDB store"` | Не помогло |
|
||
| `rebuild (no DuckDB)` + сразу `rebuild use_duckdb=true` | ❌ `"Failed to populate DuckDB store"` | Даже после успешного non-DuckDB rebuild |
|
||
| `rebuild use_duckdb=false` (контрольный) | ✅ **Always success** | 3185 contracts, 2149 edges |
|
||
|
||
**Итого:** `"Failed to populate DuckDB store"` воспроизводится всегда, стабильно, на чистой DuckDB. Ошибка в Rust-коде популяции, не в данных.
|
||
|
||
### Attempt 3: Clean database
|
||
|
||
1. Moved old `graph.duckdb` (225MB) → `graph.duckdb.bak`
|
||
2. Tried `rebuild use_duckdb=true` — still ❌ timeout
|
||
3. **Partial write observed**: new `graph.duckdb` grew 0→38MB before timeout
|
||
4. After retry: grew 38→63MB — then **stalled completely**
|
||
5. WAL file (`graph.duckdb.wal`) appeared and disappeared, suggesting incomplete transactions
|
||
|
||
### Attempt 4: Non-DuckDB works flawlessly
|
||
|
||
```
|
||
reindex → 120s ✅ (3185 contracts, 0 edges, instant)
|
||
rebuild (no DuckDB) → 300s ✅ (3185 contracts, 2149 edges)
|
||
audit_contracts after rebuild → responds instantly
|
||
```
|
||
|
||
## Key observations
|
||
|
||
1. **DuckDB write starts but never finishes**. First attempt wrote 38MB, second wrote 63MB, then stopped making progress. Expected final size is ~225MB.
|
||
2. **Non-DuckDB operations are fast**. The contract parsing, relation resolution, and JSON index writing all complete within 180-300s.
|
||
3. **MCP transport times out before DuckDB finishes**. The MCP protocol returns `-32001: Request timed out` at the transport layer. The server process may still be running but the client disconnected.
|
||
4. **DuckDB file is 63MB, WAL is empty**. The partial file suggests the DuckDB transaction started but never committed.
|
||
5. **DuckDB WAL appeared then disappeared** between attempts, suggesting a rollback occurred.
|
||
|
||
## Root cause found! Duplicate @RELATION edges in source code
|
||
|
||
После логирования удалось выявить точную причину: **PRIMARY KEY constraint violation** при вставке в таблицу edges. DuckDB использует первичный ключ `(source_id, relation_type, target_id)`, а в коде есть дублирующиеся `@RELATION` строки.
|
||
|
||
### Примеры дубликатов (подтверждено)
|
||
|
||
```
|
||
DatasetReviewEntryUxTests:DEPENDS_ON:DatasetReviewWorkspaceEntry (2 файла, одинаковый contract_id)
|
||
BackupsPage:IMPLEMENTS:RoutePages (2 файла, одинаковый contract_id)
|
||
DeleteRunningTasksUtil:DEPENDS_ON:TaskRecord (2 строки в одном файле)
|
||
AuthJwtModule:DEPENDS_ON:auth_config (2 строки в одном файле)
|
||
```
|
||
|
||
### Масштаб проблемы
|
||
|
||
Скрипт поиска (`grep -r @RELATION | sort | uniq -d`) показал **десятки файлов** с дубликатами: profile.py (7 копий), database.py (2), async_network.py (2), security.py (2), и много тестовых файлов.
|
||
|
||
### Варианты фикса
|
||
|
||
#### Вариант A: Rust-код (рекомендуется)
|
||
|
||
Изменить вставку в DuckDB с `INSERT` на `INSERT OR REPLACE` или `INSERT OR IGNORE`:
|
||
|
||
```rust
|
||
// Текущий код (падает при дубликатах):
|
||
conn.execute("INSERT INTO edges (source, type, target) VALUES (?, ?, ?)", ...)?;
|
||
|
||
// Фикс 1: INSERT OR IGNORE — пропускает дубликаты
|
||
conn.execute("INSERT OR IGNORE INTO edges (source, type, target) VALUES (?, ?, ?)", ...)?;
|
||
|
||
// Фикс 2: INSERT OR REPLACE — перезаписывает
|
||
conn.execute("INSERT OR REPLACE INTO edges (source, type, target) VALUES (?, ?, ?)", ...)?;
|
||
```
|
||
|
||
**Преимущество:** не требует правки кода на проекте, устойчив к дубликатам в будущем.
|
||
|
||
#### Вариант B: Правка кода (трудоёмко)
|
||
|
||
Найти и удалить все дублирующиеся `@RELATION` строки в проекте. Оценка: ~50 файлов, ~200 дубликатов.
|
||
|
||
### Текущий статус (после ручного фикса 4 дубликатов)
|
||
|
||
| Попытка | Результат |
|
||
|---------|:---------:|
|
||
| `rebuild use_duckdb=true` | ❌ Всё ещё падает — есть ещё дубликаты (всего ~50 файлов) |
|
||
| `rebuild use_duckdb=false` | ✅ Всегда работает (JSON index не имеет unique constraint) |
|
||
|
||
### Рекомендация
|
||
|
||
Разработчику Axiom: **реализовать Вариант A** (`INSERT OR IGNORE`). Это:
|
||
1. Решает проблему мгновенно
|
||
2. Не требует правки проекта
|
||
3. Устойчив к регрессиям
|
||
4. Не теряет данные — дубликаты @RELATION это одна и та же связь, записанная дважды
|
||
|
||
## Suggested fix approach
|
||
|
||
```rust
|
||
// Before (current — single large write blocks MCP):
|
||
pub async fn rebuild_with_duckdb() -> Result<()> {
|
||
let contracts = parse_all_files()?;
|
||
let conn = DuckDbConnection::open(path)?;
|
||
conn.execute("BEGIN TRANSACTION")?; // single giant tx
|
||
for c in &contracts {
|
||
conn.execute("INSERT INTO contracts ...", params!(c))?;
|
||
}
|
||
conn.execute("COMMIT")?; // may take minutes
|
||
Ok(())
|
||
}
|
||
|
||
// After — chunked with MCP yield:
|
||
pub async fn rebuild_with_duckdb(stream: &McpStream) -> Result<()> {
|
||
let contracts = parse_all_files()?;
|
||
let conn = DuckDbConnection::open(path)?;
|
||
// Smaller batches to prevent transport timeout
|
||
for chunk in contracts.chunks(500) {
|
||
conn.execute("BEGIN TRANSACTION")?;
|
||
for c in chunk {
|
||
conn.execute("INSERT INTO contracts ...", params!(c))?;
|
||
}
|
||
conn.execute("COMMIT")?;
|
||
// Signal progress to keep MCP connection alive
|
||
stream.send(Progress { done: chunk_end, total: contracts.len() }).await?;
|
||
// Yield to allow other MCP requests
|
||
tokio::task::yield_now().await;
|
||
}
|
||
Ok(())
|
||
}
|
||
```
|
||
|
||
## Workaround (for current version)
|
||
|
||
Until the DuckDB rebuild is fixed, use the two-step process:
|
||
1. `axiom_semantic_index reindex` (in-memory, instant)
|
||
2. `axiom_semantic_index rebuild rebuild_mode="full" use_duckdb=false` (JSON index, ~180-300s)
|
||
|
||
The JSON index at `.axiom/semantic_index/index.json` is fully functional for `search_contracts`, `read_outline`, and `local_context`. Only `audit_contracts` with DuckDB-dependent schema validation may show stale tag validation results.
|
||
|
||
## Files referenced
|
||
|
||
- DuckDB binary path: `.axiom/semantic_index/graph.duckdb` (63 MB partial)
|
||
- JSON index path: `.axiom/semantic_index/index.json` (fresh, 3185 contracts, 2149 edges)
|
||
- Old DuckDB backup: `.axiom/semantic_index/graph.duckdb.bak` (225 MB)
|
||
- Axiom config: `.axiom/axiom_config.yaml` (all tag/permission fixes applied)
|