I Built the Reranker Every RAG Tutorial Recommends. Then I Measured It.

+0.024

That is the precision improvement I got from adding Cohere's reranker to my retrieval pipeline. Forty-two production queries from a dental clinic chatbot, five information retrieval metrics, one afternoon of measurement.

I built it because every RAG tutorial said I should. I measured it because I had built the eval harness first. And I left it off because +0.024 is not worth a recurring API bill for a system handling fewer than 500 conversations a month.

The conventional wisdom is clear: if you are building retrieval-augmented generation, add a reranker. Chunk your documents, embed them, retrieve the top candidates, then pass them through a cross-encoder to reorder by relevance. Every serious RAG tutorial includes this step. Pinecone, LangChain, LlamaIndex — they all recommend it.

They are not wrong. They are solving a different problem than mine.

The Reranker Was Built for a Problem I Did Not Have

Rerankers exist because dense retrieval at scale is noisy. When you search across hundreds of thousands of documents, your vector similarity search returns a rough top-100. Many of those results are near-misses — semantically close but not what the user asked. A cross-encoder rescores that top-100 with deeper attention to the query-document pair and pushes the real answers to the top.

Pinecone's benchmarks on BEIR datasets show this clearly: reranking improves NDCG@10 by 12% on average, up to 48% on TREC datasets. Those numbers are real. They are also measured on corpora of hundreds of thousands to millions of documents.

My dental clinic chatbot has about 50 knowledge base entries. Pricing pages, service descriptions, FAQs, doctor profiles. The entire knowledge base is roughly 15,000 tokens.

A reranker improves retrieval when the initial search returns a noisy candidate set from a large corpus — hundreds of thousands of documents where the top-100 includes many near-misses. Below a few hundred documents, vector search plus hybrid RRF already surfaces the right content most of the time. The two-stage retrieval architecture collapses into one stage, and the precision gain disappears. As of Q2 2026.

At that scale, the two-stage architecture collapses. My hybrid search — full-text search plus vector embeddings merged with Reciprocal Rank Fusion at k=60 — already returns the right answers in the top 3. There is no noisy candidate pool to clean up. The reranker has nothing to fix.

Anthropic's own guidance on contextual retrieval makes this explicit: if your knowledge base is smaller than 200,000 tokens, you can skip RAG entirely and load everything into the prompt. My 15,000-token clinic KB is an order of magnitude below that threshold. I use RAG because it fits the multi-tenant architecture of the AI reception system I deploy to clinics — not because the corpus demands it.

The Part Every Tutorial Skips

Before I touched Cohere, I built the evaluation harness. This is the step most RAG tutorials leave out. They tell you what to add. They do not tell you how to know whether it helped.

I exported 42 real queries from production conversations — patients asking about pricing, services, doctor availability, and general clinic information in Lithuanian. Not synthetic test data. Real questions from real patients, logged over four months.

Each query got a ground truth annotation: which URLs should appear in the results, and what text snippet the answer should contain. Four categories — pricing (11 queries), service (12), FAQ (16), doctor (3) — covering the actual distribution of what patients ask.

The harness measures five metrics at two cutoff points:

Build a golden dataset from production queries — not synthetic data. Export real user questions, annotate which documents should appear, and measure Precision@k, Recall@k, MRR, HitRate, and ContentContains before adding any optimization. Forty-two annotated test cases across four query categories took one afternoon to build and created a permanent regression gate for every future pipeline change. As of Q2 2026.

The RAGAS framework (Shahul Es et al., EACL 2024) formalized this evaluation-first approach as the academic standard for RAG quality measurement. The core principle is the same one I learned from building a multi-model dispatch system and from debugging a cold-email pipeline four times: you cannot optimize what you have not measured. A baseline tells you where you are. An eval harness tells you whether the next change helped or hurt.

Building the harness took one afternoon. It runs against the production database and produces a JSON report. Every change to the search pipeline gets measured against the same 42 queries before it ships.

What the Numbers Said

With the baseline established, I integrated Cohere's Rerank API. Full implementation: async service with circuit breaker, enterprise tier gating, cost logging to a billing table, graceful fallback to base results if the API goes down. Then I ran the eval with two Cohere models against the same 42 queries.

The rerank-v3.5 lifted Precision@3 by 0.024. That is roughly one more relevant result appearing in the top 3 across every 42 queries. Noticeable in a spreadsheet. Invisible to a patient asking about tooth whitening prices.

The surprise: rerank-v4.0-fast — the newer, faster model — made retrieval worse. Precision dropped by the same amount the multilingual model gained. On Lithuanian dental terminology, the "fast" model lost context that v3.5 preserved. If I had followed tutorials and picked the latest version without measuring, I would have shipped a regression.

On a 42-document dental clinic knowledge base, Cohere rerank-v3.5 improved Precision@3 by +0.024 (from 0.444 to 0.468). The newer rerank-v4.0-fast regressed by the same amount (−0.024). At fewer than 500 queries per month and Cohere's $2 per 1,000 searches pricing, the marginal gain did not justify the recurring cost, latency overhead, or new API dependency. As of Q2 2026.

At Cohere's pricing of $2 per 1,000 searches, a chatbot handling 500 conversations a month pays about $1 for reranking. The dollar amount is small. But you are adding an external API dependency, a circuit breaker, retry logic, latency (+100-200ms per query), and a billing table — all for a 2.4% precision improvement that no user will notice. In a world where AI costs are shifting to usage-based billing, every component needs to earn its place.

I left the feature off.

When a Reranker Earns Its Place

This is not an argument against rerankers. It is an argument against adding them without measuring.

A reranker earns its complexity when three conditions hold:

Large candidate pool. Your initial retrieval returns 50-100 candidates from a corpus of thousands or more. The top results include noise — semantically close but not quite right — that a cross-encoder can filter. At 42 documents, the entire corpus fits in the candidate pool. There is nothing to filter.

High query volume. The per-query cost and latency are justified when you run thousands of queries per day. At 500 per month, you are paying for infrastructure you cannot amortize.

Measurable quality gap. Your eval harness shows a meaningful precision improvement — not +0.024, but +0.05 or more. If you do not have an eval harness, you cannot answer this question. And that is the real problem.

Add a reranker when three conditions hold: your corpus has thousands of documents so initial retrieval returns noisy candidates, your query volume is high enough to justify the per-query cost and latency, and your eval harness shows a meaningful precision gain. If you cannot measure the improvement, the eval harness is what you should build — not the reranker. As of Q2 2026.

The feature still exists in my codebase. It is gated behind an enterprise tenant flag, fully tested, with circuit breaker protection and cost logging. When a client arrives with 10,000 documents and 5,000 queries a day, I will flip the flag and measure again. The harness will tell me whether the math changes at that scale.

Build the Harness, Not the Feature

The reranker took a day to build and sits unused. The eval harness took an afternoon and runs on every deployment. One was a guess that looked productive. The other is infrastructure.

Forty-two production queries — annotated with ground truth URLs and expected text snippets — were enough to build a permanent regression gate for a dental clinic chatbot's retrieval pipeline. The number matters less than the source: real user questions from real conversations, not synthetic data. Production queries catch failure modes that synthetic tests miss because they reflect how people actually search.

The harness already caught a regression when I refactored the search function's metadata handling — a silent bug that would have shipped without it.

Every hour spent on a retrieval optimization without a baseline is a guess dressed up as engineering. The reranker, the query expansion module, the hypothetical-document embedder — none of them tell you whether your system got better. The eval harness does.

The best optimization I made to my RAG pipeline was measuring it. The second best was not adding the thing I measured.

If you are building AI retrieval systems and want to know what your pipeline actually needs, get in touch.