Production RAG Architectures: Hybrid Search, Reranking & PGVector in Python
A battle-tested blueprint for moving beyond fragile prototype tutorials to resilient, enterprise-grade retrieval pipelines.
Key Architectural Takeaways
- Why naive vector retrieval fails on dense enterprise documents (loss of structural hierarchy and chunk fragmentation).
- Designing semantic boundaries: Recursive character splitting with metadata inheritance (parent-child retrieval).
- Building hybrid search combining dense embeddings (pgvector HNSW) and sparse lexical search (PostgreSQL tsvector).
- Applying Reciprocal Rank Fusion (RRF) and Cross-Encoder re-ranking to achieve >92% top-3 retrieval accuracy.
- End-to-end evaluation using RAGAS for context recall, context precision, and hallucination reduction.
Guide Contents
1. The Naive RAG Trap: Why 5-Line Tutorials Break in Production
Every beginner tutorial on Retrieval-Augmented Generation follows the identical naive formula: load a PDF with PyPDFLoader, split text into arbitrary 500-token chunks with 50-token overlap, embed them using an OpenAI API call, dump them into an in-memory vector database, and perform cosine similarity search against the user prompt.
While this setup yields an impressive demo on clean 2-page blog posts, it disintegrates when deployed against real-world enterprise documents such as quarterly financial filings, technical specifications, and legal agreements. Real documents contain hierarchical headers, structured tables, and dispersed contextual dependencies.
When an arbitrary 500-token boundary cuts through a table or severs a clause from its governing condition, the embedding vector captures incomplete semantics. In production, naive cosine search frequently returns irrelevant fragments, causing large language models to hallucinate plausible yet incorrect answers.
2. Ingestion Pipeline: Semantic Boundaries & Parent-Child Retrieval
Production RAG begins with intelligent document parsing. Instead of naive fixed-size character splitting, high-precision ingestion relies on two structural techniques: Semantic Markdown Boundary Splitting and Parent-Child (Hierarchical) Chunking.
In Parent-Child retrieval, we separate the retrieval unit from the synthesis unit. Documents are indexed at two granularities: small child chunks (150–200 tokens) optimized for vector matching, and larger parent chunks (800–1200 tokens) that preserve complete paragraphs and surrounding context.
When a vector match fires on a child chunk, the ingestion pipeline resolves the parent chunk ID and injects the comprehensive parent context into the LLM prompt. This ensures high search precision without sacrificing contextual continuity.
import uuid
from typing import List, Dict
from pydantic import BaseModel
class ChildChunk(BaseModel):
id: str
parent_id: str
content: str
embedding: List[float]
class ParentChunk(BaseModel):
id: str
document_id: str
content: str
section_title: str
def create_hierarchical_chunks(document_id: str, text: str) -> tuple[List[ParentChunk], List[ChildChunk]]:
# 1. Parse document into sections by Markdown headers
sections = split_by_headers(text)
parents: List[ParentChunk] = []
children: List[ChildChunk] = []
for sec in sections:
p_id = str(uuid.uuid4())
parents.append(ParentChunk(id=p_id, document_id=document_id, content=sec.body, section_title=sec.title))
# 2. Slice parent section into compact child passages for dense indexing
sub_passages = recursive_char_splitter(sec.body, chunk_size=200, chunk_overlap=30)
for passage in sub_passages:
emb = compute_dense_embedding(passage)
children.append(ChildChunk(id=str(uuid.uuid4()), parent_id=p_id, content=passage, embedding=emb))
return parents, children3. Implementing Hybrid Search with PostgreSQL pgvector and Full-Text Search
Rather than managing a separate vector database (like Pinecone or Milvus) and synchronizing transactions with your primary database, modern production stacks leverage PostgreSQL with the `pgvector` extension.
Hybrid search executes two queries simultaneously: a dense vector similarity query using HNSW (Hierarchical Navigable Small World) index over vector embeddings, and a sparse lexical search using PostgreSQL's native `tsvector` with BM25-like full-text ranking.
Dense vector search captures conceptual meaning (e.g., matching 'automobile' when the user asks for 'car'), while sparse search captures exact keywords, IDs, SKU numbers, and technical abbreviations that embeddings often blur.
-- Create HNSW index for high-speed sub-millisecond approximate nearest neighbor search
CREATE INDEX ON document_chunks USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);
-- Execute Hybrid Search with Reciprocal Rank Fusion in a single SQL CTE
WITH vector_matches AS (
SELECT id, parent_id, content,
ROW_NUMBER() OVER (ORDER BY embedding <=> :query_vector) AS rank_dense
FROM document_chunks
ORDER BY embedding <=> :query_vector
LIMIT 25
),
lexical_matches AS (
SELECT id, parent_id, content,
ROW_NUMBER() OVER (ORDER BY ts_rank_cd(search_tsv, plainto_tsquery('english', :search_query)) DESC) AS rank_sparse
FROM document_chunks
WHERE search_tsv @@ plainto_tsquery('english', :search_query)
LIMIT 25
)
SELECT COALESCE(v.id, l.id) AS chunk_id,
COALESCE(v.parent_id, l.parent_id) AS parent_id,
COALESCE(v.content, l.content) AS content,
(COALESCE(1.0 / (60 + v.rank_dense), 0.0) +
COALESCE(1.0 / (60 + l.rank_sparse), 0.0)) AS rrf_score
FROM vector_matches v
FULL OUTER JOIN lexical_matches l ON v.id = l.id
ORDER BY rrf_score DESC
LIMIT 10;4. Cross-Encoder Re-Ranking & RAGAS Evaluation
Hybrid search narrows 100,000 document chunks down to the top 20 candidates in ~15ms. However, passing 20 chunks directly to your LLM introduces 'lost-in-the-middle' attention degradation and wastes token budget.
The final step before prompt synthesis is Cross-Encoder Re-ranking. Unlike bi-encoders (which encode prompt and passage separately into single vectors), a cross-encoder evaluates the prompt and candidate passage together through full transformer cross-attention layers, outputting a precise relevance score between 0.0 and 1.0.
We rerank the top 20 candidates with `bge-reranker-large` or Cohere Rerank v3, retaining only the top 3–5 highest-scoring contexts for the final LLM completion.
Finally, verify your pipeline using the RAGAS framework. Track Faithfulness (ensuring the model output does not contradict retrieved passages) and Context Recall (ensuring all ground-truth facts were retrieved). A production RAG system should sustain >90% faithfulness prior to release.
Frequently Asked Questions
Why use pgvector instead of dedicated vector databases like Pinecone?
pgvector keeps your relational data, user authorization tables, and vector embeddings in a single ACID-compliant database. You avoid cross-system synchronization bugs, separate security perimeters, and extra subscription bills.
What is Reciprocal Rank Fusion (RRF)?
RRF is an algorithm that combines rankings from different search mechanisms (such as dense vector similarity and keyword BM25 search) by scoring each item as 1 / (k + rank). It normalizes scale differences without requiring complex hyperparameter tuning.
Master this architecture with real code reviews
Build, evaluate, and deploy multi-modal RAG systems and autonomous agent workflows in our live online batch.