Vector Search – Embedding Queries & Similarity Search

Codifies the project’s dual vector search systems (Memory Store for agent domain knowledge, RAG Pipeline for document retrieval), the multi-provider embedding abstraction, pgvector indexing, hybrid search scoring, and chunking strategies. All patterns are built on Supabase/PostgreSQL with pgvector.

Description

Codifies pgvector embedding queries, similarity search, hybrid search, and multi-provider embedding generation for NodeJS-Starter-V1’s Supabase/PostgreSQL stack, covering the Memory Store and RAG Pipeline vector infrastructure, indexing strategies, and chunking patterns.

When to Apply

Positive Triggers

• Adding semantic search to new data types
• Creating or modifying embedding generation logic
• Implementing similarity queries or nearest-neighbour lookups
• Configuring chunking strategies for document ingestion
• Tuning search relevance (thresholds, weights, reranking)
• Adding new embedding providers
• User mentions: “vector”, “embedding”, “semantic search”, “similarity”, “RAG”, “pgvector”, “cosine”

Negative Triggers

• Building dashboard UI for search results (use dashboard-patterns instead)
• Adding full-text keyword search only (use PostgreSQL tsvector directly)
• Instrumenting search latency metrics (use metrics-collector instead)
• Logging search queries (use structured-logging instead)

Core Directives

The Three Laws of Vector Search

1. Provider-agnostic: All embedding generation goes through EmbeddingProvider abstraction. Never call OpenAI/Ollama directly.
2. Hybrid by default: Combine vector similarity with keyword matching. Pure vector search misses exact terms; pure keyword misses semantics.
3. Server-side scoring: Similarity computation happens in PostgreSQL via RPC functions. Never download all vectors to Python for client-side comparison.

Existing Project Infrastructure

Two Vector Search Systems

Both share the same EmbeddingProvider abstraction from src/memory/embeddings.py.

Embedding Providers

Selection via get_embedding_provider() — checks OPENAI_API_KEY, then ANTHROPIC_API_KEY, then falls back to SimpleEmbeddingProvider.

API Routes

Database

Embedding Provider Pattern

The EmbeddingProvider abstract base class defines a single method:

class EmbeddingProvider(ABC):
    @abstractmethod
    async def get_embedding(self, text: str) -> list[float]:
        """Generate embedding vector for text."""
        pass

Three implementations: OpenAIEmbeddingProvider (calls /v1/embeddings via httpx), OllamaEmbeddingProvider (local /api/embeddings), SimpleEmbeddingProvider (hash-based, testing only).

Adding a New Provider

1. Subclass EmbeddingProvider
2. Implement get_embedding() returning a fixed-dimension vector
3. Add selection logic in get_embedding_provider()
4. Match the dimension to existing index (1536 for OpenAI compatibility, or create a separate index)

Dimension Consistency Rule

All vectors in a table MUST share the same dimension. If mixing providers with different dimensions (e.g., OpenAI 1536 vs Ollama 768), either:

• Pad/truncate to a standard dimension, OR
• Use separate columns per dimension, OR
• Standardise on one dimension and re-embed when switching providers

The project currently standardises on 1536 dimensions (OpenAI).

Search Patterns

Similarity Search (Memory Store)

MemoryStore.find_similar() generates a query embedding and calls the find_similar_memories PostgreSQL RPC:

async def find_similar(self, query_text: str, domain: MemoryDomain | None = None,
    user_id: str | None = None, similarity_threshold: float = 0.7, limit: int = 10,
) -> list[dict[str, Any]]:
    query_embedding = await self.embedding_provider.get_embedding(query_text)
    result = self.client.rpc("find_similar_memories", {
        "query_embedding": json.dumps(query_embedding),
        "match_threshold": similarity_threshold,
        "match_count": limit,
        "filter_domain": domain.value if domain else None,
        "filter_user_id": user_id,
    }).execute()
    return result.data or []

Key parameters: match_threshold (0.0–1.0, cosine similarity minimum), match_count (max results). Domain and user filters are applied server-side in the RPC function.

Hybrid Search (RAG Pipeline)

RAGStore.hybrid_search() combines vector similarity with keyword matching using configurable weights:

async def hybrid_search(self, query: str, project_id: str,
    vector_weight: float = 0.6, keyword_weight: float = 0.4,
    limit: int = 10, threshold: float = 0.5,
) -> list[dict[str, Any]]:
    query_embedding = await self.embedding_provider.get_embedding(query)
    result = self.client.rpc("hybrid_search", {
        "query_text": query,
        "query_embedding": query_embedding,
        "project_id_filter": project_id,
        "vector_weight": vector_weight,
        "keyword_weight": keyword_weight,
        "match_threshold": threshold,
        "match_count": limit,
    }).execute()
    return result.data or []

Default weights: 60% vector + 40% keyword. Adjust for domain:

• Technical docs: 70/30 (semantics matter more)
• Exact match scenarios (IDs, codes): 30/70 (keywords matter more)
• General content: 60/40 (balanced)

Full-Text Search (PostgreSQL tsvector)

The /api/search route uses native PostgreSQL full-text search with ts_rank:

func.ts_rank(
    func.to_tsvector("english", Document.title + " " + Document.content),
    func.plainto_tsquery("english", query_text),
    32,  # RANK_CD normalisation flag
).label("relevance")

This is independent of vector search and uses the documents table directly via SQLAlchemy.

Indexing Patterns

IVFFlat Index (Current)

The project uses IVFFlat for approximate nearest-neighbour search:

CREATE INDEX idx_documents_embedding
  ON documents USING ivfflat (embedding vector_cosine_ops);

IVFFlat partitions vectors into lists (clusters). Query searches only the nearest cluster(s), trading recall for speed.

Tuning parameters:

• lists (build-time): Number of clusters. Rule of thumb: sqrt(row_count) for < 1M rows
• probes (query-time): Number of clusters to search. Higher = better recall, slower. Default: 1

-- Set probes for a session (higher = more accurate, slower)
SET ivfflat.probes = 10;

HNSW Index (Recommended for Production)

For datasets > 10K rows, prefer HNSW (Hierarchical Navigable Small World):

CREATE INDEX idx_documents_embedding_hnsw
  ON documents USING hnsw (embedding vector_cosine_ops)
  WITH (m = 16, ef_construction = 64);

HNSW provides better recall than IVFFlat without manual tuning. Higher m and ef_construction improve quality at the cost of build time and memory.

Distance Functions

The project uses cosine similarity (vector_cosine_ops) throughout.

Chunking Strategies

The RAG pipeline supports five chunking strategies via ChunkingStrategy enum:

Default: PARENT_CHILD with 512-token children and 2048-token parents. Search matches children; context retrieval includes the parent chunk.

Pipeline Config

PipelineConfig(
    chunking_strategy=ChunkingStrategy.PARENT_CHILD,
    chunk_size=512,
    chunk_overlap=50,
    parent_chunk_size=2048,
    generate_embeddings=True,
    generate_keywords=True,
)

Relevance & Scoring

Threshold Guidelines

The Memory Store defaults to similarity_threshold=0.7. The RAG Pipeline defaults to min_score=0.5.

Relevance Decay

MemoryStore.update_relevance() adjusts memory relevance based on feedback:

• Positive feedback (+0.1 per point, capped at 1.0)
• Negative feedback (configurable decay_rate, default 0.1, floored at 0.0)

Stale Memory Pruning

MemoryStore.prune_stale() removes memories below min_relevance=0.3 or older than max_age_days=90 via the prune_stale_memories RPC.

Pydantic Models

Memory System

RAG System

Database Schema

documents Table (Legacy)

CREATE TABLE documents (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  title VARCHAR(500) NOT NULL,
  content TEXT NOT NULL,
  embedding VECTOR(1536),
  -- ... other columns
);
CREATE INDEX idx_documents_embedding ON documents USING ivfflat (embedding vector_cosine_ops);

domain_memories Table

Stores agent memories with embeddings for semantic retrieval. Accessed via MemoryStore class.

document_chunks Table

Stores RAG pipeline chunks with embeddings. Accessed via RAGStore class. Includes heading_hierarchy, summary, entities, keywords, and classification_tags for enriched retrieval.

RPC Functions

Anti-Patterns

Checklist for New Vector Search Features

Embedding

• Uses EmbeddingProvider abstraction (never direct API calls)
• Dimension matches existing index (1536 default)
• Handles provider unavailability (fallback or graceful error)

Search

• Hybrid search by default (vector + keyword)
• Similarity threshold configured (not unbounded)
• Server-side computation via PostgreSQL RPC
• Results include similarity scores for transparency

Indexing

• pgvector index created on embedding column
• Distance function matches query pattern (cosine for normalised)
• Index type appropriate for dataset size (IVFFlat < 10K, HNSW >= 10K)

Data Quality

• Chunking strategy matches content type
• Chunk overlap prevents boundary information loss
• Stale/expired entries have pruning mechanism

Integration

• Search latency instrumented via metrics-collector
• Errors use error-taxonomy codes
• Queries logged via structured-logging

Response Format

[AGENT_ACTIVATED]: Vector Search
[PHASE]: {Design | Implementation | Review}
[STATUS]: {in_progress | complete}

{vector search analysis or implementation guidance}

[NEXT_ACTION]: {what to do next}

Integration Points

Council of Logic

• Turing: Verify search is O(log n) via index, not O(n) sequential scan
• Shannon: Embedding dimension and chunk size tuned for information density

Metrics Collector

• search_query_duration_ms histogram for search latency
• search_result_count gauge for average results per query
• embedding_generation_duration_ms histogram for provider latency

Structured Logging

• Debug-level embedding generation logs (model, dimensions, text length)
• Info-level search execution logs (query, domain, result count)

Error Taxonomy

• DATA_VECTOR_PROVIDER_UNAVAILABLE (503) — embedding provider down
• DATA_VECTOR_DIMENSION_MISMATCH (422) — wrong embedding dimension
• DATA_VECTOR_THRESHOLD_INVALID (422) — threshold out of [0, 1] range

Data Validation

• SearchRequest validated via Pydantic (query non-empty, threshold in range, limit bounded)
• PipelineConfig validates chunk sizes and strategy enum

Dashboard Patterns

• Search results displayed via DataStrip for aggregate metrics
• Real-time search activity via Supabase Realtime on document_chunks table

Australian Localisation (en-AU)

• Spelling: neighbour, optimise, normalise, analyse, behaviour, colour
• Date: ISO 8601 in storage; DD/MM/YYYY in UI display
• Timezone: AEST/AEDT — timestamps stored as UTC, converted for display