Building a Semantic Search Engine from Scratch

Semantic search goes beyond keyword matching to understand the meaning and context of queries. Instead of looking for exact text matches, it uses vector embeddings to find conceptually similar content. For example, searching for 'dog training' would also return results about 'puppy obedience' or 'canine behavior modification' even if those exact terms don't appear in your query.

Traditional search engines like Elasticsearch excel at keyword matching but struggle with synonyms, context, and conceptual similarity. Semantic search addresses these limitations by representing both documents and queries as high-dimensional vectors in a shared embedding space.

Building your own semantic search engine gives you complete control over the indexing strategy, embedding models, and ranking algorithms. This is essential for specialized domains, proprietary content, or when you need to integrate tightly with existing systems. Companies like Pinecone have built entire businesses around managed vector search infrastructure.

The core advantage is understanding intent rather than matching words. A user searching for 'machine learning jobs' should also see results for 'AI engineer positions' or 'data scientist careers' - concepts your keyword-based system would miss entirely. This capability has made semantic search a cornerstone of modern AI applications and RAG systems.

Every semantic search system consists of four essential components that work together to transform text into searchable meaning:

1. Embedding Model: Converts text into numerical vectors that capture semantic meaning. Popular choices include OpenAI's text-embedding-ada-002, Google's Universal Sentence Encoder, or open-source alternatives like sentence-transformers.
2. Vector Database: Stores and indexes embeddings for fast similarity search. Options range from purpose-built solutions like Pinecone and Weaviate to database extensions like pgvector for PostgreSQL.
3. Chunking Strategy: Breaks large documents into searchable segments. The goal is preserving context while maintaining granular retrieval. Typical chunk sizes range from 200-500 tokens with 10-20% overlap.
4. Similarity Search: Finds the most relevant chunks using distance metrics like cosine similarity or dot product. Advanced systems combine multiple ranking signals including semantic similarity, keyword relevance, and metadata filters.

The pipeline flows from indexing (embed and store documents) to querying (embed user query, find similar vectors, return ranked results). Understanding how embeddings work is crucial for optimizing each component of this architecture.

The embedding model is the heart of your semantic search system. It determines how well your system understands language, handles domain-specific terminology, and performs across different query types. Here's how to choose:

OpenAI text-embedding-ada-002 remains the gold standard for general-purpose applications. At 1536 dimensions and $0.0001 per 1K tokens, it offers excellent quality-to-cost ratio. It handles multiple languages, understands context well, and works out-of-the-box for most domains. Use this unless you have specific requirements that demand alternatives.

Open-source alternatives like sentence-transformers provide cost savings and data privacy. Models like all-MiniLM-L6-v2 (384 dimensions) offer decent quality for simple applications, while larger models like all-mpnet-base-v2 (768 dimensions) approach commercial quality. The trade-off is typically lower accuracy and the need for self-hosting.

Domain-specific models excel in specialized fields. For code search, models like CodeBERT or specialized variants trained on programming languages significantly outperform general models. For legal documents, models trained on legal corpora understand terminology and concepts that general models miss.

Fine-tuning considerations: If you have sufficient domain-specific data (10k+ document pairs), fine-tuning a base model can improve performance by 15-25%. However, this requires ML expertise and significant computational resources. Most applications should start with pre-trained models.

Your choice of vector database significantly impacts performance, scalability, and operational complexity. Here's a practical guide to the major options:

For prototyping and small datasets (under 1M vectors), ChromaDB offers the simplest setup. It's Python-native, requires no separate infrastructure, and integrates seamlessly with Jupyter notebooks. Perfect for proof-of-concepts and early-stage development where you need to iterate quickly.

For production workloads, Pinecone provides managed infrastructure that scales to billions of vectors. It handles indexing, replication, and performance optimization automatically. The serverless tier starts free and scales with usage, making it ideal for most production applications. The main trade-off is vendor lock-in and ongoing costs.

For data sovereignty or hybrid deployments, Weaviate offers powerful features with full control over hosting. It supports multi-tenancy, complex queries, and hybrid search out-of-the-box. The learning curve is steeper, but you retain complete data control and can optimize for specific use cases.

For existing PostgreSQL users, pgvector extends your familiar database with vector capabilities. This reduces operational complexity if you already manage Postgres infrastructure. Performance is adequate for moderate scale (millions of vectors) and you benefit from ACID transactions and mature tooling.

Let's build a complete semantic search system using Python, OpenAI embeddings, and ChromaDB. This example demonstrates the core concepts while remaining production-ready with minor modifications.

Step 1: Environment Setup

pip install chromadb openai tiktoken python-dotenv

# Create .env file with your OpenAI API key
echo "OPENAI_API_KEY=your_key_here" > .env

Step 2: Initialize the Embedding Client

import os
import openai
import chromadb
from dotenv import load_dotenv
import tiktoken

load_dotenv()
openai.api_key = os.getenv("OPENAI_API_KEY")

class SemanticSearch:
    def __init__(self, collection_name="documents"):
        self.client = chromadb.Client()
        self.collection = self.client.get_or_create_collection(
            name=collection_name,
            metadata={"hnsw:space": "cosine"}
        )
        self.encoding = tiktoken.get_encoding("cl100k_base")
    
    def get_embedding(self, text):
        """Get embedding for a single text"""
        response = openai.Embedding.create(
            model="text-embedding-ada-002",
            input=text
        )
        return response['data'][0]['embedding']
    
    def count_tokens(self, text):
        """Count tokens for cost estimation"""
        return len(self.encoding.encode(text))

Step 3: Document Indexing with Smart Chunking

def chunk_document(self, text, max_tokens=400, overlap_tokens=50):
    """Split document into overlapping chunks"""
    tokens = self.encoding.encode(text)
    chunks = []
    
    start = 0
    while start < len(tokens):
        end = min(start + max_tokens, len(tokens))
        chunk_tokens = tokens[start:end]
        chunk_text = self.encoding.decode(chunk_tokens)
        chunks.append(chunk_text)
        
        if end >= len(tokens):
            break
        start = end - overlap_tokens
    
    return chunks

def add_document(self, doc_id, text, metadata=None):
    """Add a document to the search index"""
    chunks = self.chunk_document(text)
    
    for i, chunk in enumerate(chunks):
        chunk_id = f"{doc_id}_chunk_{i}"
        embedding = self.get_embedding(chunk)
        
        chunk_metadata = {
            "doc_id": doc_id,
            "chunk_index": i,
            "text": chunk,
            **(metadata or {})
        }
        
        self.collection.add(
            ids=[chunk_id],
            embeddings=[embedding],
            metadatas=[chunk_metadata]
        )
    
    print(f"Added {len(chunks)} chunks for document {doc_id}")

Step 4: Search Implementation with Ranking

def search(self, query, k=5, score_threshold=0.7):
    """Search for relevant documents"""
    query_embedding = self.get_embedding(query)
    
    results = self.collection.query(
        query_embeddings=[query_embedding],
        n_results=k * 2,  # Get more results for reranking
    )
    
    # Process and rank results
    ranked_results = []
    for i, (doc_id, distance, metadata) in enumerate(zip(
        results['ids'][0],
        results['distances'][0], 
        results['metadatas'][0]
    )):
        similarity_score = 1 - distance  # Convert distance to similarity
        
        if similarity_score >= score_threshold:
            ranked_results.append({
                'doc_id': metadata['doc_id'],
                'chunk_id': doc_id,
                'text': metadata['text'],
                'score': similarity_score,
                'metadata': metadata
            })
    
    # Remove duplicates and return top k
    seen_docs = set()
    unique_results = []
    
    for result in sorted(ranked_results, key=lambda x: x['score'], reverse=True):
        if result['doc_id'] not in seen_docs:
            unique_results.append(result)
            seen_docs.add(result['doc_id'])
        
        if len(unique_results) >= k:
            break
    
    return unique_results

Chunking strategy directly impacts search quality and retrieval performance. Poor chunking leads to incomplete results, missing context, or irrelevant matches. Here are proven approaches for different content types:

Fixed-size chunking with overlap works well for general content. Use 200-500 tokens per chunk with 10-20% overlap. Smaller chunks improve precision but may lose context. Larger chunks preserve context but can include irrelevant information. The overlap ensures important concepts aren't split across boundaries.

Semantic chunking splits content at natural boundaries like paragraphs, sections, or topic changes. This preserves meaning better than arbitrary splits but requires content structure analysis. Libraries like LangChain provide semantic splitters for common document types.

Hierarchical chunking creates multiple granularity levels - document summaries, section overviews, and detailed chunks. During search, you first match broad topics, then drill down to specific details. This approach works exceptionally well for technical documentation and academic papers.

Sliding window chunking creates overlapping windows that move through the text with small steps. This maximizes coverage but increases storage requirements. Use for critical applications where missing relevant content is unacceptable, such as legal or medical document search.

Raw user queries often need preprocessing to improve search accuracy. Real users write incomplete sentences, use colloquialisms, make typos, or ask complex multi-part questions. Effective query processing can improve search results by 20-40%.

Query expansion adds related terms to improve recall. For technical searches, expand acronyms (AI → artificial intelligence), include synonyms (bug → defect → issue), and add domain-specific terminology. Use embedding-based expansion by finding similar terms in your vector space.

Multi-part query handling breaks complex questions into components. A query like 'How do I deploy a Python web app with SSL on AWS?' contains multiple intents: deployment, Python, web applications, SSL configuration, and AWS. Process each component separately and combine results.

Intent classification helps route queries to appropriate search strategies. Factual questions ('What is Docker?') need different handling than procedural queries ('How to containerize an app?') or comparative requests ('Docker vs Kubernetes'). Simple classification models or LLM-based routing work well.

def process_query(self, query):
    """Enhanced query processing"""
    # Basic preprocessing
    processed_query = query.lower().strip()
    
    # Expand common abbreviations
    expansions = {
        'ai': 'artificial intelligence',
        'ml': 'machine learning',
        'dl': 'deep learning',
        'nlp': 'natural language processing'
    }
    
    for abbrev, expansion in expansions.items():
        if abbrev in processed_query.split():
            processed_query = processed_query.replace(abbrev, f"{abbrev} {expansion}")
    
    # Add domain context if query is too short
    if len(processed_query.split()) < 3:
        processed_query += " tutorial guide example"
    
    return processed_query

Pure semantic similarity often isn't enough for optimal search results. Production systems combine multiple signals to rank results more effectively than single-metric approaches. The key is balancing semantic relevance with other quality indicators.

Hybrid scoring combines semantic similarity with keyword matching, recency, authority, and user engagement signals. A simple weighted approach: `final_score = 0.6 semantic_score + 0.2 keyword_score + 0.1 recency_score + 0.1 authority_score`. Tune weights based on your domain and user feedback.

Metadata-based boosting leverages document characteristics. Boost results from authoritative sources, recent content, highly-rated documents, or content matching user preferences. For technical documentation, boost official docs over community posts. For news, prioritize recent articles.

Learning to rank approaches use machine learning to optimize ranking based on user behavior. Collect implicit feedback (clicks, time on page, downloads) and explicit feedback (ratings, bookmarks) to train ranking models. This requires significant data but can improve relevance substantially.

def calculate_hybrid_score(self, semantic_score, text, metadata, query):
    """Combine multiple ranking signals"""
    # Keyword matching boost
    query_terms = set(query.lower().split())
    text_terms = set(text.lower().split())
    keyword_overlap = len(query_terms & text_terms) / len(query_terms)
    
    # Recency boost (if published_date available)
    recency_score = 1.0
    if 'published_date' in metadata:
        days_old = (datetime.now() - metadata['published_date']).days
        recency_score = max(0.1, 1.0 - (days_old / 365))
    
    # Authority boost
    authority_score = metadata.get('authority_score', 0.5)
    
    # Combine signals
    final_score = (
        0.6 * semantic_score +
        0.2 * keyword_overlap +
        0.1 * recency_score +
        0.1 * authority_score
    )
    
    return final_score

Production semantic search systems must handle thousands of queries per second with sub-100ms latency. Performance optimization focuses on three areas: indexing efficiency, query speed, and cost management.

Indexing optimization reduces storage costs and improves query speed. Use dimensionality reduction techniques like PCA to compress embeddings without significant accuracy loss. Quantization can reduce memory usage by 50-75% with minimal quality impact. Consider approximate indexing algorithms like HNSW for massive datasets.

Caching strategies dramatically improve response times for common queries. Cache embedding computations for frequent queries, maintain hot storage for popular documents, and implement intelligent prefetching. Redis or Memcached work well for embedding caches, while CDNs can serve static search interfaces.

Batch processing reduces API costs and improves throughput. Batch embed documents during indexing, process multiple queries simultaneously, and use asynchronous processing for non-critical operations. OpenAI's batch endpoints offer 50% cost savings for non-realtime operations.

Resource optimization balances cost with performance. Monitor embedding API usage, implement request rate limiting, and use model quantization for self-hosted embeddings. Consider edge deployment for latency-sensitive applications or regulatory compliance.

Deploying semantic search to production requires careful attention to reliability, observability, and scalability. Unlike traditional databases, vector systems have unique failure modes and performance characteristics that need monitoring.

Infrastructure considerations include high-memory requirements for vector storage, CPU optimization for similarity calculations, and network bandwidth for large embedding transfers. Plan for 2-4GB RAM per million 1536-dimensional vectors, plus overhead for query processing and caching.

Monitoring and alerting should track both system metrics and search quality. Monitor query latency, embedding API costs, index freshness, and search result quality over time. Set alerts for unusual query patterns, API failures, or degraded search performance.

A/B testing framework enables safe iteration on ranking algorithms, embedding models, and search features. Test changes on small traffic percentages while measuring both technical metrics (latency, cost) and business metrics (user satisfaction, conversion rates).

Disaster recovery planning includes embedding backup strategies, index reconstruction procedures, and fallback search mechanisms. Vector indexes can take hours to rebuild, so maintain hot standby systems for critical applications.