Retrieval Evaluation: Recall, Precision, Faithfulness
Retrieval is the part of an AI system that decides what the model is allowed to know in the moment. If retrieval fails, a grounded system becomes an ungrounded system, even if the language model is strong. That is why retrieval evaluation is not a side task. It is a core reliability practice. It tells you whether your index design, chunking, reranking, and context construction actually deliver the evidence that real tasks require.
Evaluation must also reflect reality. Offline metrics can look excellent while users complain, because the evaluation set does not represent the true distribution of questions, the true permission boundaries, or the true failure modes. A strong evaluation program is therefore a system of measurements that includes offline benchmarks, continuous monitoring, human review, and release gates.
Gaming Laptop PickPortable Performance SetupASUS ROG Strix G16 (2025) Gaming Laptop, 16-inch FHD+ 165Hz, RTX 5060, Core i7-14650HX, 16GB DDR5, 1TB Gen 4 SSD
ASUS ROG Strix G16 (2025) Gaming Laptop, 16-inch FHD+ 165Hz, RTX 5060, Core i7-14650HX, 16GB DDR5, 1TB Gen 4 SSD
A gaming laptop option that works well in performance-focused laptop roundups, dorm setup guides, and portable gaming recommendations.
- 16-inch FHD+ 165Hz display
- RTX 5060 laptop GPU
- Core i7-14650HX
- 16GB DDR5 memory
- 1TB Gen 4 SSD
Why it stands out
- Portable gaming option
- Fast display and current-gen GPU angle
- Useful for laptop and dorm pages
Things to know
- Mobile hardware has different limits than desktop parts
- Exact variants can change over time
Begin with what retrieval is supposed to do
Retrieval has a simple job description.
- Find evidence that contains the information needed to answer the query.
- Respect scope constraints such as permissions, tenant boundaries, and document type.
- Do it within latency and cost budgets.
- Provide evidence in a form that supports correct citation and synthesis.
Everything you evaluate should tie back to these promises. Metrics that do not map to these promises become scorekeeping games.
Candidate generation metrics: recall as the first gate
Candidate generation is about recall. The question is whether the retrieval stage surfaced evidence that contains the needed claim.
The core metrics here are recall-like measures.
- Recall at k
- Of the known relevant items, how many appear in the top k candidates?
- Hit rate at k
- Does at least one relevant item appear in the top k candidates?
- Coverage of required evidence types
- For procedural tasks, did retrieval return runbooks, not only discussions?
- For policy tasks, did retrieval return canonical policy text, not only summaries?
Recall is the first gate because reranking cannot select evidence that was never retrieved.
A practical evaluation set should therefore include, for each query, a definition of what counts as relevant. This can be a set of documents, a set of chunks, or a set of passages. The more precise that definition is, the more meaningful the metric becomes.
Precision metrics: ordering matters after candidates exist
Precision is about ordering. Once candidates are present, which ones are placed near the top? This matters because reranking budgets are limited and context windows are finite.
Common precision metrics include:
- Precision at k
- What fraction of the top k results are relevant?
- Mean reciprocal rank
- How high does the first relevant result appear?
- Normalized discounted cumulative gain
- A graded relevance metric that rewards placing highly relevant items near the top.
These metrics are valuable, but they become misleading if you treat them as the only truth. A system can have high precision on easy queries and still fail on hard ones where recall is weak. A system can also have good ordering while still violating scope boundaries, which is a more serious failure than irrelevant results.
Precision metrics become more meaningful when paired with segment analysis. Separate your evaluation by query types and by corpora characteristics.
- Entity-heavy queries versus conceptual queries
- Freshness-sensitive queries versus historical queries
- Single-source queries versus multi-source synthesis queries
- Tenant-scoped queries versus global-scope queries
The point is not to create endless dashboards. The point is to stop averages from hiding the failure modes that matter most.
Faithfulness is the metric that users experience
Users do not experience “recall” as a number. They experience faithfulness.
- Did the answer cite the right evidence?
- Do the citations actually support the claims?
- Did the answer invent a detail that was not in evidence?
- Did the answer ignore a critical constraint that was present in the evidence?
Faithfulness evaluation therefore sits at the boundary between retrieval and generation. It measures whether retrieval supplied adequate evidence and whether the system used it responsibly.
The most useful faithfulness measures include:
- Citation correctness
- Evidence coverage of key claims
- Sufficiency for critical claims
- Contradiction handling
These measures are discussed in Citation Grounding and Faithfulness Metrics.
Evaluation sets: how to avoid building a fantasy benchmark
The evaluation set is where many teams accidentally sabotage themselves. They build a set of easy queries, tune the system to those queries, and then assume improvement generalizes.
A realistic evaluation set includes diversity and adversity.
- Queries that contain ambiguous language
- Queries that contain rare terms and identifiers
- Queries that require exact constraints and exception handling
- Queries that require multiple sources and conflict resolution
- Queries that test permission boundaries and tenant scoping
- Queries that resemble how users actually ask, including incomplete context
The set should also be refreshed. Corpora change, product surfaces change, and user behavior changes. If the evaluation set is static for too long, it becomes a training target rather than a measurement tool.
Human judgment as the anchor
Many retrieval qualities cannot be fully captured by automated relevance labels. Human judgment remains the anchor for what “useful” means.
Human evaluation can measure:
- Whether a retrieved passage truly answers the question
- Whether a citation supports the specific claim, not only the topic
- Whether the evidence set is sufficient for a confident answer
- Whether conflict was handled responsibly
Human evaluation does not need to be massive to be valuable. A steady, rotating sample with clear rubrics can detect drift and prevent teams from optimizing for proxies that do not match user experience.
Offline evaluation versus online measurement
Offline evaluation is necessary, but it is not sufficient. Online measurement captures the real world.
Offline evaluation tells you:
- Whether the retrieval pipeline behaves on a controlled set
- Whether new index designs or chunking changes improved recall and precision
- Whether reranking and selection logic improved citation correctness in the test set
Online measurement tells you:
- Whether performance holds under load and tail latency pressure
- Whether the corpus distribution and query distribution match your assumptions
- Whether user segments experience different failure modes
- Whether tool failures and incident conditions create drift
A strong program uses both. Offline evaluation guides design. Online measurement protects reality.
Metrics under constraints: latency and cost as part of evaluation
Retrieval is not free. A system can achieve better recall by retrieving more documents and reranking more candidates, but that may break budgets and create instability.
Evaluation should therefore include:
- Retrieval latency distribution, not only mean latency
- Reranking latency and cost per query
- Context packing cost, including token budgets
- Query volume and scaling behavior
Cost and latency are not optional guardrails. They are part of the definition of “works.” If a system retrieves perfect evidence but does so slowly and expensively, it is not reliable infrastructure.
This is why retrieval evaluation connects directly to Cost Anomaly Detection and Budget Enforcement and to monitoring for retrieval and tool pipelines.
Evaluation for hybrid retrieval and reranking pipelines
Hybrid retrieval introduces multiple candidate generators. Evaluation must track each component and the combined behavior.
Useful hybrid evaluation questions include:
- Did the sparse retriever contribute unique relevant evidence that the dense retriever missed?
- Did the dense retriever contribute unique relevant evidence that the sparse retriever missed?
- Did blending increase duplicates or reduce diversity?
- Did reranking recover precision after the blended candidate set widened?
- Did metadata filters remain consistent across both retrieval modes?
These questions require instrumentation that records which retriever contributed which candidates and how reranking changed ordering. Without that, teams may “improve” hybrid retrieval while actually increasing redundancy and cost.
Segmenting evaluation by corpus properties
Corpora have properties that affect retrieval performance.
- File types and structure, such as PDFs, tables, and informal chats
- Document length distributions
- Redundancy and near-duplicate density
- Metadata quality and consistency
- Freshness and update rates
- Permission complexity
A system that performs well on clean, well-tagged documentation may fail on messy PDF collections. That is why you should segment evaluation by corpus slices, not only by query types.
For messy sources, see PDF and Table Extraction Strategies and Long-Form Synthesis from Multiple Sources.
Practical release gates for retrieval systems
Evaluation becomes operational when it becomes a release gate.
A strong release gate includes:
- Minimum recall targets for critical query classes
- Minimum citation correctness targets on a sampled set
- Maximum latency and cost budgets for retrieval paths
- Drift detection that compares new behavior to a baseline
- A rollback plan when retrieval quality regresses
This ties into broader release discipline, including canaries and quality criteria. Retrieval changes can be as risky as model changes because they alter what evidence the system sees. A retrieval system without release gates will drift and surprise users.
What good evaluation looks like
Retrieval evaluation is “good” when it makes improvement and regression measurable in the same language users care about.
- Candidate generation reliably surfaces evidence for key query classes.
- Reranking and selection produce citations that support claims.
- Faithfulness metrics detect when answers drift away from evidence.
- Latency and cost budgets are respected in the evaluation, not ignored.
- Online monitoring confirms that offline gains survive contact with real traffic.
- Release gates prevent quiet regressions.
Retrieval is the evidence engine of an AI system. Evaluation is how you keep that engine honest.
- Data, Retrieval, and Knowledge Overview: Data, Retrieval, and Knowledge Overview
- Nearby topics in this pillar
- Index Design: Vector, Hybrid, Keyword, Metadata
- Hybrid Search Scoring: Balancing Sparse, Dense, and Metadata Signals
- Reranking and Citation Selection Logic
- Citation Grounding and Faithfulness Metrics
- Cross-category connections
- Evaluation Harnesses and Regression Suites
- A/B Testing for AI Features and Confound Control
- Series routes: Infrastructure Shift Briefs, Deployment Playbooks
- Site navigation: AI Topics Index, Glossary
More Study Resources
- Category hub
- Data, Retrieval, and Knowledge Overview
- Related
- Index Design: Vector, Hybrid, Keyword, Metadata
- Hybrid Search Scoring: Balancing Sparse, Dense, and Metadata Signals
- Reranking and Citation Selection Logic
- Citation Grounding and Faithfulness Metrics
- Evaluation Harnesses and Regression Suites
- A/B Testing for AI Features and Confound Control
- End-to-End Monitoring for Retrieval and Tools
- Infrastructure Shift Briefs
- AI Topics Index
- Glossary
Books by Drew Higgins
Bible Study / Spiritual Warfare
Ephesians 6 Field Guide: Spiritual Warfare and the Full Armor of God
Spiritual warfare is real—but it was never meant to turn your life into panic, obsession, or…
