375 lines
15 KiB
Markdown
375 lines
15 KiB
Markdown
# [DEF:Report:Vectorization:Root:Module]
|
|
# @COMPLEXITY 5
|
|
# @PURPOSE Explain the current vectorization technology used by the Rust semantic index, step by step, in a contract-oriented format suitable for downstream LLM analysis.
|
|
# @RELATION DEPENDS_ON -> [Axiom:Embedding:VSS:EmbedText]
|
|
# @RELATION DEPENDS_ON -> [Axiom:Embedding:VSS:Normalize]
|
|
# @RELATION DEPENDS_ON -> [Axiom:Embedding:VSS:JsonSerialize]
|
|
# @RELATION DEPENDS_ON -> [Axiom:Embedding:VSS:JsonDeserialize]
|
|
# @RELATION DEPENDS_ON -> [Axiom:DB:Store:UpsertEmbedding]
|
|
# @RELATION DEPENDS_ON -> [Axiom:Services:Contract:Rebuild:SemanticIndex]
|
|
# @RATIONALE The report is structured as semantic contracts so another LLM can reason about the implementation without reverse-engineering code first.
|
|
# @REJECTED Free-form prose without @PRE/@POST was rejected because it weakens machine analysis and obscures invariants.
|
|
|
|
# Vectorization Technology Report
|
|
|
|
## 1. Executive Summary
|
|
|
|
The current system uses a **deterministic local fallback embedding pipeline**.
|
|
|
|
It is **not model-based** and **does not call any external embedding provider**. Instead, it computes a **128-dimensional vector** from raw text using **character-frequency hashing**, then **L2-normalizes** the vector and stores it in DuckDB as a **JSON array string** in the `embeddings` table.
|
|
|
|
This design is optimized for:
|
|
- deterministic rebuilds
|
|
- offline operation
|
|
- zero external dependencies at inference time
|
|
- reproducible semantic indexing across agent sessions
|
|
|
|
It is intentionally simpler than transformer embeddings.
|
|
|
|
---
|
|
|
|
## 2. Primary Production Contracts
|
|
|
|
### [DEF:Report:Vectorization:ContractMap:Block]
|
|
### @COMPLEXITY 4
|
|
### @PURPOSE Map the production contracts that implement the vectorization pipeline.
|
|
### @PRE Reader needs direct traceability from report steps to repository anchors.
|
|
### @POST Each critical stage is linked to a concrete production contract.
|
|
### @SIDE_EFFECT None.
|
|
|
|
| Stage | Contract ID | Responsibility |
|
|
|---|---|---|
|
|
| Vector generation | `Axiom:Embedding:VSS:EmbedText` | Build a 128-dim vector from text via character hashing |
|
|
| Normalization | `Axiom:Embedding:VSS:Normalize` | L2-normalize the vector |
|
|
| Similarity | `Axiom:Embedding:VSS:CosineSimilarity` | Compute cosine similarity between normalized vectors |
|
|
| Serialization | `Axiom:Embedding:VSS:JsonSerialize` | Encode vector as JSON string |
|
|
| Deserialization | `Axiom:Embedding:VSS:JsonDeserialize` | Decode JSON string back to `[f64; 128]` |
|
|
| Persistence | `Axiom:DB:Store:UpsertEmbedding` | Store embedding row in DuckDB |
|
|
| Retrieval | `Axiom:DB:Store:GetEmbedding` | Load embedding row from DuckDB |
|
|
| Rebuild orchestration | `Axiom:Services:Contract:Rebuild:SemanticIndex` | Trigger workspace reindex and optionally persist to DuckDB |
|
|
|
|
---
|
|
|
|
## 3. Step-by-Step Technology Flow
|
|
|
|
### [DEF:Report:Vectorization:Step1:Block]
|
|
### @COMPLEXITY 5
|
|
### @PURPOSE Define the text source that becomes embedding input.
|
|
### @PRE A semantic contract has already been parsed from workspace source and its `body` is available.
|
|
### @POST The system has a deterministic text payload suitable for embedding generation.
|
|
### @SIDE_EFFECT None directly; this step only defines input selection.
|
|
### @DATA_CONTRACT `ContractNode.body -> embed_text(text)`
|
|
### @INVARIANT The embedding source text is the contract body persisted by the indexer, not an external summary.
|
|
|
|
**Implementation reality**
|
|
- During rebuild, the system iterates over indexed contracts.
|
|
- For each contract, it passes `contract.body` into `embed_text(&contract.body)`.
|
|
- Therefore the vector represents the lexical content of the full `[DEF]...[/DEF]` body, including header metadata and body text.
|
|
|
|
**Important consequence**
|
|
- Similarity is influenced by both semantic tags (`@PURPOSE`, `@RELATION`, etc.) and implementation text.
|
|
|
|
---
|
|
|
|
### [DEF:Report:Vectorization:Step2:Block]
|
|
### @COMPLEXITY 5
|
|
### @PURPOSE Describe the deterministic vector construction algorithm.
|
|
### @PRE Input text is available as UTF-8 Rust `&str`.
|
|
### @POST A dense 128-dimensional floating-point vector is produced before normalization.
|
|
### @SIDE_EFFECT None.
|
|
### @DATA_CONTRACT `&str -> [f64; 128]`
|
|
### @INVARIANT No network, no stochastic model weights, and no external provider are involved.
|
|
### @RATIONALE Deterministic hashing is fast, portable, and reproducible.
|
|
### @REJECTED Transformer-based embeddings were rejected due to runtime cost and external dependency coupling.
|
|
|
|
**Algorithm**
|
|
1. Initialize `vector = [0.0; 128]`.
|
|
2. Iterate through `text.chars().take(2048)`.
|
|
3. For each character `ch`, compute `idx = (ch as usize) % 128`.
|
|
4. Increment `vector[idx] += 1.0`.
|
|
|
|
**Interpretation**
|
|
- This is a **character-bucket frequency sketch**.
|
|
- It is closer to a hashed lexical fingerprint than a learned semantic embedding.
|
|
|
|
**Strengths**
|
|
- deterministic
|
|
- cheap to compute
|
|
- stable across platforms
|
|
- robust enough for coarse lexical similarity
|
|
|
|
**Weaknesses**
|
|
- collisions are guaranteed because all characters map into 128 buckets
|
|
- no contextual semantics beyond lexical distribution
|
|
- weak synonym/generalization behavior compared with learned embeddings
|
|
|
|
---
|
|
|
|
### [DEF:Report:Vectorization:Step3:Block]
|
|
### @COMPLEXITY 4
|
|
### @PURPOSE Explain input bounding and its effect on reproducibility.
|
|
### @PRE Raw contract body may be arbitrarily long.
|
|
### @POST Embedding computation uses at most the first 2048 characters.
|
|
### @SIDE_EFFECT Truncates effective semantic coverage for long contracts.
|
|
### @INVARIANT Runtime cost remains bounded and reproducible for every rebuild.
|
|
|
|
**Mechanism**
|
|
- The generator uses `text.chars().take(2048)`.
|
|
|
|
**Why it exists**
|
|
- keeps rebuild cost bounded
|
|
- prevents very large contracts from dominating runtime
|
|
- ensures deterministic maximum work per contract
|
|
|
|
**Trade-off**
|
|
- content after the first 2048 characters does not affect the vector
|
|
|
|
---
|
|
|
|
### [DEF:Report:Vectorization:Step4:Block]
|
|
### @COMPLEXITY 5
|
|
### @PURPOSE Define the normalization stage that converts raw counts into a unit vector.
|
|
### @PRE Raw 128-dim vector has non-negative frequency counts.
|
|
### @POST Output vector has unit Euclidean norm unless the raw vector is all zeros.
|
|
### @SIDE_EFFECT Mutates the vector in place.
|
|
### @DATA_CONTRACT `[f64; 128] -> normalized [f64; 128]`
|
|
### @INVARIANT Similarity scoring assumes normalized vectors.
|
|
|
|
**Algorithm**
|
|
1. Compute `sum_sq = Σ(x_i^2)`.
|
|
2. Compute `norm = sqrt(sum_sq)`.
|
|
3. If `norm > 0.0`, divide each component by `norm`.
|
|
|
|
**Why normalization matters**
|
|
- removes bias from absolute text length
|
|
- enables cosine similarity as a direct dot product
|
|
|
|
**Operational note**
|
|
- for non-empty textual contracts, the vector should normally be non-zero and therefore normalized successfully
|
|
|
|
---
|
|
|
|
### [DEF:Report:Vectorization:Step5:Block]
|
|
### @COMPLEXITY 4
|
|
### @PURPOSE Explain persistence encoding for DuckDB storage.
|
|
### @PRE A normalized `[f64; 128]` vector exists in memory.
|
|
### @POST The vector is serialized into a compact JSON array string.
|
|
### @SIDE_EFFECT None.
|
|
### @DATA_CONTRACT `[f64; 128] -> String(vector_json)`
|
|
### @INVARIANT Stored vectors must remain length-128 after round-trip decoding.
|
|
|
|
**Mechanism**
|
|
- `vector_to_json` uses `serde_json::to_string(&vector.to_vec())`.
|
|
- Result is stored in DuckDB column `embeddings.vector_json TEXT`.
|
|
|
|
**Why JSON was chosen**
|
|
- simple and portable
|
|
- easy to inspect manually
|
|
- no custom binary format needed
|
|
|
|
**Cost**
|
|
- larger on disk than binary
|
|
- slower than native vector column types
|
|
|
|
---
|
|
|
|
### [DEF:Report:Vectorization:Step6:Block]
|
|
### @COMPLEXITY 5
|
|
### @PURPOSE Describe how vectors are written to DuckDB during rebuild.
|
|
### @PRE Rebuild runs with `use_duckdb=true`; schema bootstrap has succeeded; contracts are available in memory.
|
|
### @POST Each indexed contract receives an embedding row in `embeddings` when `refresh_embeddings=true`.
|
|
### @SIDE_EFFECT Inserts or replaces rows in DuckDB.
|
|
### @DATA_CONTRACT `ContractNode -> embeddings(contract_id, provider_id, vector_json, source_text)`
|
|
### @INVARIANT Embedding row identity is keyed by `contract_id`.
|
|
|
|
**Implementation path**
|
|
1. `rebuild_semantic_index(...)` reindexes the workspace.
|
|
2. If `use_duckdb=true`, it opens `graph.duckdb`.
|
|
3. `DuckDbIndexStore::populate_from_index(...)` clears/repopulates tables.
|
|
4. If `refresh_embeddings=true`, each contract body is embedded.
|
|
5. `upsert_embedding(...)` stores:
|
|
- `contract_id`
|
|
- `provider_id` (currently `local-fallback`)
|
|
- `vector_json`
|
|
- `source_text`
|
|
|
|
**Current provider identity**
|
|
- storage path marks the provider as `local-fallback`
|
|
- rebuild response payload separately reports `embedding_provider_id = lexical-graph`
|
|
|
|
**Interpretation for downstream analysis**
|
|
- both labels refer to the same local deterministic embedding strategy, but naming is currently inconsistent across layers
|
|
|
|
---
|
|
|
|
### [DEF:Report:Vectorization:Step7:Block]
|
|
### @COMPLEXITY 4
|
|
### @PURPOSE Explain how stored vectors are loaded back from DuckDB.
|
|
### @PRE A row exists in `embeddings` for the target `contract_id`.
|
|
### @POST The vector round-trips back into Rust as `[f64; 128]`.
|
|
### @SIDE_EFFECT Reads DuckDB state.
|
|
### @DATA_CONTRACT `contract_id -> Option<[f64; 128]>`
|
|
### @INVARIANT Invalid JSON or non-128 vectors are treated as errors, not silently accepted.
|
|
|
|
**Mechanism**
|
|
- `get_embedding(contract_id)` loads `vector_json`
|
|
- `vector_from_json(json_str)` parses `Vec<f64>`
|
|
- parser enforces exact length `128`
|
|
|
|
**Safety property**
|
|
- malformed stored vectors fail loudly instead of contaminating similarity logic
|
|
|
|
---
|
|
|
|
### [DEF:Report:Vectorization:Step8:Block]
|
|
### @COMPLEXITY 4
|
|
### @PURPOSE Define the similarity metric expected by the vector system.
|
|
### @PRE Both vectors are already L2-normalized and lengths are equal.
|
|
### @POST Cosine similarity is computed as a dot product in `[-1, 1]`.
|
|
### @SIDE_EFFECT None.
|
|
### @DATA_CONTRACT `[f64; 128] x [f64; 128] -> f64`
|
|
### @INVARIANT The similarity function assumes normalized inputs and does not renormalize them itself.
|
|
|
|
**Mechanism**
|
|
- `cosine_similarity(left, right) = Σ(left_i * right_i)`
|
|
|
|
**Important note**
|
|
- the primitive exists and is correct for the current representation
|
|
- but a full production similarity-search API over DuckDB embeddings is still minimal and not yet a rich ANN/vector-index system
|
|
|
|
---
|
|
|
|
## 4. Storage Schema Relevant to Vectorization
|
|
|
|
### [DEF:Report:Vectorization:Schema:Block]
|
|
### @COMPLEXITY 4
|
|
### @PURPOSE Describe the DuckDB schema fields directly involved in vectorization.
|
|
### @PRE Reader needs storage-level understanding for independent analysis.
|
|
### @POST The embedding persistence surface is explicitly documented.
|
|
### @SIDE_EFFECT None.
|
|
|
|
Relevant table:
|
|
|
|
```sql
|
|
CREATE TABLE IF NOT EXISTS embeddings (
|
|
contract_id TEXT PRIMARY KEY,
|
|
provider_id TEXT,
|
|
vector_json TEXT NOT NULL,
|
|
source_text TEXT
|
|
);
|
|
```
|
|
|
|
Field meaning:
|
|
- `contract_id`: stable logical owner of the vector
|
|
- `provider_id`: identifier of embedding strategy
|
|
- `vector_json`: serialized 128-dim normalized vector
|
|
- `source_text`: source body used to create the vector
|
|
|
|
---
|
|
|
|
## 5. Rebuild-Time Operational Sequence
|
|
|
|
### [DEF:Report:Vectorization:RebuildSequence:Block]
|
|
### @COMPLEXITY 5
|
|
### @PURPOSE Provide the operational sequence for vector generation during semantic index rebuild.
|
|
### @PRE Workspace root and policy are valid; operational directories are creatable.
|
|
### @POST JSON snapshot is updated and DuckDB embeddings are refreshed when enabled.
|
|
### @SIDE_EFFECT Reads source files, writes JSON snapshot, writes DuckDB tables.
|
|
### @INVARIANT JSON snapshot and DuckDB content are derived from the same in-memory index rebuild.
|
|
|
|
Sequence:
|
|
1. Reindex workspace contracts and edges in memory.
|
|
2. Persist JSON semantic snapshot.
|
|
3. If `use_duckdb=true`, open/create `graph.duckdb`.
|
|
4. Bootstrap schema if absent.
|
|
5. Clear/repopulate files, contracts, edges.
|
|
6. If `refresh_embeddings=true`, compute a 128-dim vector for each contract body.
|
|
7. Serialize vectors to JSON.
|
|
8. Upsert rows into `embeddings`.
|
|
9. Save `IndexStatus` with embedding counts.
|
|
|
|
---
|
|
|
|
## 6. Design Characteristics for Another LLM to Evaluate
|
|
|
|
### [DEF:Report:Vectorization:EvaluationHints:Block]
|
|
### @COMPLEXITY 4
|
|
### @PURPOSE Highlight the key architectural properties another LLM should evaluate.
|
|
### @PRE Reader wants actionable analysis targets rather than raw implementation trivia.
|
|
### @POST The main trade-offs and audit points are explicit.
|
|
### @SIDE_EFFECT None.
|
|
|
|
### Strengths
|
|
- deterministic and reproducible
|
|
- offline-safe
|
|
- cheap rebuild cost
|
|
- no model-serving dependency
|
|
- transparent storage format
|
|
|
|
### Weaknesses
|
|
- not semantically deep like transformer embeddings
|
|
- collisions from modulo-128 hashing
|
|
- truncation at 2048 characters
|
|
- JSON storage instead of typed vector columns
|
|
- provider naming inconsistency (`local-fallback` vs `lexical-graph`)
|
|
|
|
### Questions worth analyzing
|
|
1. Should metadata and code body be embedded together or separately?
|
|
2. Should bucket count remain 128 or be increased?
|
|
3. Should similarity search be exposed as a first-class tool/API?
|
|
4. Should `provider_id` naming be normalized across rebuild response and storage?
|
|
5. Should long contracts use chunking instead of hard truncation at 2048 chars?
|
|
|
|
---
|
|
|
|
## 7. Exact Minimal Pseudocode
|
|
|
|
### [DEF:Report:Vectorization:Pseudocode:Block]
|
|
### @COMPLEXITY 3
|
|
### @PURPOSE Give another LLM a language-agnostic reproduction of the current embedding pipeline.
|
|
### @PRE Reader needs a faithful abstract form of the implementation.
|
|
### @POST The algorithm can be reimplemented without inspecting Rust syntax.
|
|
### @SIDE_EFFECT None.
|
|
|
|
```text
|
|
function embed_text(text):
|
|
vector = [0.0] * 128
|
|
for ch in first_2048_characters(text):
|
|
idx = ord(ch) mod 128
|
|
vector[idx] += 1.0
|
|
|
|
norm = sqrt(sum(x*x for x in vector))
|
|
if norm > 0:
|
|
for i in range(128):
|
|
vector[i] /= norm
|
|
|
|
return vector
|
|
|
|
function store_embedding(contract_id, text):
|
|
vector = embed_text(text)
|
|
vector_json = json_encode(vector)
|
|
upsert into embeddings(contract_id, provider_id, vector_json, source_text)
|
|
```
|
|
|
|
---
|
|
|
|
## 8. Current Truth Statement
|
|
|
|
### [DEF:Report:Vectorization:CurrentTruth:Block]
|
|
### @COMPLEXITY 4
|
|
### @PURPOSE Provide a final machine-readable summary of what is true today.
|
|
### @PRE All previous sections have been read or can be ignored for a compact summary.
|
|
### @POST Another LLM can extract the operative facts in one pass.
|
|
### @SIDE_EFFECT None.
|
|
|
|
- Vectorization technology: **deterministic character-frequency hashing**
|
|
- Embedding dimensionality: **128**
|
|
- Input cap: **first 2048 characters**
|
|
- Normalization: **L2 normalization**
|
|
- Storage encoding: **JSON array in DuckDB `embeddings.vector_json`**
|
|
- Similarity metric: **cosine similarity via dot product of normalized vectors**
|
|
- External model/provider dependency: **none**
|
|
- Primary objective: **cheap, deterministic, offline lexical-semantic approximation**
|
|
|
|
# [/DEF:Report:Vectorization:Root:Module]
|