The RAG system I built for financial document analysis looked great in demos. You'd ask it a question, it would retrieve a few document chunks, feed them to the LLM, and return a confident, well-formatted answer. The problem: I had no idea if the retrieved chunks were actually the right ones. I was measuring output quality by reading the answers myself — which is exactly the kind of vibe-check that lets broken systems ship to production. When I finally added RAGAS evaluation, I discovered context precision sitting at 0.58. That means for every 10 chunks my retriever was pulling, more than 4 were noise. The LLM was doing its best with bad inputs and producing answers that sounded right but weren't reliably grounded. This post is about how I found that, fixed it, and why measurement should come before optimization — not after. I've deployed the system as a live demo at ragora.base44.app if you want to query it yourself.
TL;DR — I ran RAGAS on my RAG system and found context precision at 0.58 — over 40% of retrieved chunks were pure noise. Hybrid retrieval (pgvector + BM25) pushed it to 0.94. Here is exactly how I diagnosed and fixed it.
Want to see this in action? I built a live demo at ragora.base44.app — you can query a real RAG system and inspect the retrieved chunks, relevancy scores, and RAGAS metrics in real time.
How Ragora Is Architected
Ragora isn't just a single RAG pipeline — it's a full educational platform covering every layer of a production RAG system, from ingestion to observability. Here's what each part of the app does and why it exists.
/ — Landing & Naive vs Advanced RAG
The home page opens with a side-by-side comparison panel: Naive RAG (single vector lookup, no reranking, no query transformation) vs Advanced RAG (hybrid retrieval + cross-encoder reranking + query decomposition). The delta in retrieval quality is the entire motivation for everything else in the app.
/architecture — Full Pipeline Diagram
A visual walkthrough of the complete RAG pipeline: document ingestion → semantic chunking → embedding → hybrid retrieval → reranking → generation. Built with custom ArchNode and Arrow components so every stage is labelled with what's happening and why, not just a generic flowchart.
/chunking — Semantic Chunking Strategies
Two strategies are compared side by side. Sliding-window cosine-distance splitting detects semantic boundaries by measuring embedding similarity between adjacent sentences — a split happens when similarity drops below a threshold. Small-to-Big parent-document retrieval stores small chunks for precise matching but returns the full parent document as context, avoiding the truncation problem that hurts naive fixed-size chunking.
/hybrid-search — Vector + BM25 with RRF Merging
Vector search (Bi-Encoder) and BM25 (lexical) run in parallel. Results are merged using Reciprocal Rank Fusion — each document's final score is the sum of 1/(k + rank) across both retrievers, where k=60 is the standard smoothing constant. The interactive demo lets you see how the two rank lists merge into a single unified ranking.
/reranking — Cross-Encoder Re-ranking
After hybrid retrieval returns ~20 candidates, a cross-encoder scores each (query, chunk) pair jointly — unlike bi-encoders that embed query and document separately, cross-encoders see the full concatenation, which makes them far more accurate but too slow to run over the whole corpus. The page also visualises the 'Lost in the Middle' phenomenon: LLMs reliably recall content at the start and end of their context window but miss information buried in the middle. Reranking pushes the most relevant chunks to position 1–3.
/query-transform — Query Expansion & Decomposition
A single vague query retrieves poorly. This page covers two fixes: query expansion (generating alternative phrasings of the same question to broaden recall) and query decomposition (breaking a complex multi-part question into atomic sub-queries, each retrieved independently then synthesised). A live decomposition example shows the LLM breaking a compound question into four retrievable sub-questions in real time.
/decision-matrix — Advanced RAG vs Long-Context LLMs
Gemini 2.0 Flash has a 2M token context window. So does RAG still make sense? This page builds an economic comparison: at what corpus size and query volume does RAG (with its per-chunk infrastructure cost) become cheaper than stuffing everything into a long-context prompt on every request? The cost-estimator grid lets you input your own numbers.
/demo — Live End-to-End Pipeline
The interactive demo is the core of the app. Upload a document, the LLM chunks it semantically, then ask questions. Every query runs the full pipeline: query transform → hybrid search → cross-encoder rerank → GPT-4o generation. A real-time retrieval trace panel shows which chunks were retrieved, their rerank scores, and which ones made it into the final context window. Traces are persisted to a RAGTrace entity so they're available in observability.
/observability — Quality Dashboard
Pulls all RAGTrace records in real time and renders latency charts, candidate counts per query, rerank score distributions, and a query history log. The low-relevance chunk identification panel flags retrievals where the top rerank score fell below 0.4 — those are the queries most likely to produce hallucinations, and they're the first ones I investigate when a system degrades in production.
The Problem With Eyeballing RAG Quality
When you build a RAG system and test it manually, you're doing two things at once: evaluating retrieval quality and evaluating generation quality. The LLM is so good at sounding coherent that it masks retrieval failures. It takes bad chunks and produces fluent, confident text anyway. This is not a feature — it's the most dangerous failure mode in RAG systems.
A retriever returning irrelevant chunks is not obvious from the final answer. You only see it when the answer is factually wrong and you happen to know enough to catch it. For domain-specific data — financial documents, legal text, technical manuals — you often can't catch it because you're building the system precisely because you don't want to read every document yourself.
The LLM's ability to sound coherent with bad context is not a safety net. It's a liability. It turns retrieval failures into confident hallucinations.
What RAGAS Actually Measures
RAGAS (Retrieval Augmented Generation Assessment) is an evaluation framework that breaks RAG quality into four distinct metrics, each measuring a different failure mode. Understanding what each metric catches is what makes it useful.
- Context Precision — of the chunks retrieved, what fraction are actually relevant to the question? Low precision means your retriever is pulling noise.
- Context Recall — of all the relevant information that exists, what fraction did your retriever actually find? Low recall means you're missing things.
- Faithfulness — does the generated answer stick to what was in the retrieved context? Low faithfulness means the LLM is going off-script and hallucinating.
- Answer Relevancy — does the answer actually address what was asked? Low relevancy means the response is vague or drifting off-topic.
Each metric maps to a specific problem. If context precision is low, fix your retriever. If faithfulness is low, fix your prompt or add stricter grounding instructions. If answer relevancy is low, check your query understanding step. RAGAS tells you exactly where to look — instead of guessing.
Setting Up RAGAS: What You Actually Need
To run RAGAS you need a test dataset: a list of questions, the ground-truth answers, and optionally the ground-truth context. Building this dataset is the part most people skip because it takes time. It's also the most important step — garbage test data gives you metrics that mean nothing.
For financial documents I built a test set of 40 questions manually — real questions a user would ask, with answers I verified against the source documents. This took about three hours. Those three hours saved me weeks of debugging production issues later.
from ragas import evaluate
from ragas.metrics import (
context_precision,
context_recall,
faithfulness,
answer_relevancy,
)
from datasets import Dataset
# Your test dataset — questions + ground truth
test_data = {
"question": [
"What was the total revenue in Q3?",
"Which product line had the highest margin?",
# ... more questions
],
"ground_truth": [
"Total revenue in Q3 was $2.8M, up 12% from Q2.",
"The enterprise tier had the highest margin at 68%.",
# ... matching answers
],
"answer": [], # filled by your RAG pipeline
"contexts": [], # filled by your retriever
}
# Run your RAG pipeline to fill answers and contexts
for question in test_data["question"]:
result = rag_pipeline.invoke(question)
test_data["answer"].append(result["answer"])
test_data["contexts"].append(result["source_documents"])
dataset = Dataset.from_dict(test_data)
results = evaluate(
dataset,
metrics=[context_precision, context_recall, faithfulness, answer_relevancy],
)
print(results)My Baseline Results — And What They Revealed
The first time I ran RAGAS against my system, the results were humbling. This was a system I had been demoing confidently for weeks.
Baseline evaluation (pure vector search, 500-char chunks):
context_precision: 0.58
context_recall: 0.61
faithfulness: 0.71
answer_relevancy: 0.68Context precision at 0.58 was the critical number. Out of every 6 chunks retrieved, about 2.5 were irrelevant. The LLM was working with over 40% noise in its context window on average. The faithfulness score of 0.71 was a downstream symptom of that — when the model gets bad context, it starts filling gaps with what it knows from training rather than what's in the documents.
The root cause was clear once I looked at what was actually being retrieved. Numbers don't embed the way prose does. 'Revenue was $4.2M' and 'Revenue was $42M' produce similar embedding vectors because the semantic meaning is nearly identical. Pure cosine similarity can't distinguish them. For financial documents where the exact number is the answer, this is a serious problem.
The Fix: Hybrid Retrieval with pgvector + BM25
Hybrid retrieval combines dense vector search (semantic similarity) with sparse BM25 (term frequency matching). The two approaches catch completely different failure modes. Vector search excels at semantic questions — 'what were the main cost drivers this quarter.' BM25 excels at exact term and number matching — 'what was the Q3 EBITDA.' Together they cover the full range of queries a user actually asks.
from langchain.retrievers import EnsembleRetriever
from langchain_community.retrievers import BM25Retriever
from langchain_postgres.vectorstores import PGVector
from langchain_openai import OpenAIEmbeddings
embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
# Dense retriever — semantic similarity via pgvector
vector_store = PGVector(
embeddings=embeddings,
collection_name="financial_docs",
connection=connection_string,
)
dense_retriever = vector_store.as_retriever(
search_type="similarity",
search_kwargs={"k": 6},
)
# Sparse retriever — exact term matching via BM25
bm25_retriever = BM25Retriever.from_documents(all_documents)
bm25_retriever.k = 6
# Ensemble: weight vector slightly higher than BM25
# Tune these weights for your domain
hybrid_retriever = EnsembleRetriever(
retrievers=[dense_retriever, bm25_retriever],
weights=[0.6, 0.4],
)The 60/40 weight split between vector and BM25 was arrived at empirically — I ran RAGAS at 50/50, 60/40, and 70/30 and compared context precision scores. For financial documents, 60/40 consistently outperformed the others. For more conversational or semantic queries you'd likely want to tip the balance further toward vector. There's no universal answer — this is why evaluation exists.
Fixing the Chunking Strategy
Hybrid retrieval alone wasn't enough. The 500-character fixed chunks were destroying the structure of financial tables. A balance sheet row would get split mid-way through, separating the account name from its value. The retrieved chunk would have numbers with no labels, or labels with no numbers — equally useless to the LLM.
I switched to larger chunks with overlap, and added a pre-processing step that converts tables to a row-per-line text format before chunking — so each row stays intact as a retrievable unit with its full context.
from langchain.text_splitter import RecursiveCharacterTextSplitter
# Larger chunks, more overlap — keeps context coherent
splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200,
separators=["\n\n", "\n", ". ", " "],
)
def preprocess_table(table_text: str) -> str:
"""Convert table to row-per-line format before chunking.
Each row gets the header prepended so it is self-contained."""
lines = table_text.strip().split("\n")
if len(lines) < 2:
return table_text
headers = lines[0]
return "\n".join(
f"{headers} | {row}" for row in lines[1:] if row.strip()
)Results After Hybrid Retrieval + Better Chunking
I re-ran the same 40-question RAGAS evaluation after deploying both changes.
After hybrid retrieval + improved chunking:
Before After Delta
context_precision: 0.58 → 0.94 +62%
context_recall: 0.61 → 0.91 +49%
faithfulness: 0.71 → 0.89 +25%
answer_relevancy: 0.68 → 0.86 +26%Context precision from 0.58 to 0.94. The system went from returning roughly 2.5 irrelevant chunks per query to fewer than 1 on average. The faithfulness improvement followed directly — better context means the LLM has less reason to fill gaps from its training data. The system was now reliably grounded in the source documents.
The jump from 0.58 to 0.94 context precision didn't come from changing the LLM, tweaking the prompt, or upgrading the embedding model. It came from fixing what the retriever was actually fetching. That's where most RAG systems fail — not in generation.
Making Evaluation Part of the Pipeline
The biggest mistake I made initially was treating evaluation as a one-off diagnostic. Running RAGAS once, fixing the problem, and moving on means you'll ship a regression the next time you change your chunking logic, swap embedding models, or update your vector index.
Now RAGAS runs as part of the development workflow. Before any retrieval change is merged, the evaluation scores have to meet or exceed the current baseline. It's the same mental model as a test suite — you don't skip tests when you refactor, you don't skip evaluation when you change retrieval.
# Evaluation gate — run before merging retrieval changes
def assert_retrieval_quality(pipeline, test_dataset, baselines):
results = evaluate(
test_dataset,
metrics=[context_precision, faithfulness],
)
assert results["context_precision"] >= baselines["context_precision"], (
f"Context precision regression: {results['context_precision']:.2f} "
f"< baseline {baselines['context_precision']:.2f}"
)
assert results["faithfulness"] >= baselines["faithfulness"], (
f"Faithfulness regression: {results['faithfulness']:.2f} "
f"< baseline {baselines['faithfulness']:.2f}"
)
return results
BASELINES = {"context_precision": 0.90, "faithfulness": 0.85}
assert_retrieval_quality(rag_pipeline, test_dataset, BASELINES)What I'd Do Differently From the Start
- Build the evaluation dataset before writing the first line of retrieval code — even 20 questions is enough to catch major failures early.
- Start with hybrid retrieval by default for any structured or numerical data — pure vector search is optimized for semantic prose, not tables or numbers.
- Treat chunk size and overlap as hyperparameters to tune with RAGAS, not constants to pick once and forget.
- Set RAGAS baselines on day one and treat a drop in context precision as a failing test, not a known issue to revisit later.
- When faithfulness drops, look at the retriever first — most of the time a faithfulness problem is a context precision problem upstream.
Need an AI engineer for your project? I build RAG systems, LLM integrations, AI agents, and SaaS backends. Reach me at prayagbagthariya.vercel.app/hire-me or prayag129787@gmail.com.
Frequently Asked Questions
What is RAGAS and how is it different from other LLM evaluation tools?
RAGAS (Retrieval Augmented Generation Assessment) is an evaluation framework specifically designed for RAG pipelines. Unlike general LLM eval tools that only measure output quality, RAGAS decomposes quality into four metrics — context precision, context recall, faithfulness, and answer relevancy — each isolating a specific failure mode. This lets you pinpoint whether a problem is in your retriever, your chunking, or your generation step. It's available as an open-source Python library.
When should I use hybrid retrieval instead of pure vector search?
Use hybrid retrieval (vector + BM25) any time your documents contain structured data, numbers, codes, or proper nouns that need exact matching. Financial documents, legal contracts, product catalogs, and technical specifications all benefit from BM25's term-frequency component. Pure vector search is better suited for semantic, conversational queries over unstructured prose. In practice, hybrid retrieval rarely hurts even for prose-heavy corpora, so defaulting to it is a reasonable starting point.
How do I build a RAGAS test dataset without ground-truth answers?
RAGAS can generate synthetic test datasets from your documents using the TestsetGenerator class, which uses an LLM to create question-answer pairs from your source text. This is not as reliable as manually curated test data, but far better than no evaluation at all. For critical use cases, manually verify at least a subset of the generated questions — especially for any domain where factual accuracy matters.
Does improving context precision always improve faithfulness?
Usually yes, but not always. Context precision and faithfulness are correlated because the LLM is less likely to hallucinate when given clean, relevant context. However, faithfulness can still be low even with good retrieval if your prompt doesn't explicitly instruct the model to stay grounded in the provided context. Adding an explicit instruction like 'Only use information from the provided context — do not use external knowledge' significantly improves faithfulness scores independently of retrieval quality.
What pgvector index type works best for production RAG?
For most production RAG systems, HNSW (Hierarchical Navigable Small World) is the better choice over IVFFlat. HNSW has faster query performance with better recall at the cost of slightly higher memory usage and build time. IVFFlat requires specifying the number of lists upfront and can have lower recall if the number of probes is not tuned properly. Use IVFFlat if memory is severely constrained; use HNSW for most production workloads where query speed and recall matter more than index build time.
Continue reading
Get updates directly in your Google Search feed
Add this publication as a preferred source to highlight articles in Top Stories and AI Overviews.


