Banking RAG retrieval accuracy testing checks whether a retrieval-augmented generation (RAG) system returns the correct, current and authorised banking source passages for a query. It measures retrieval separately from the language model's answer, so a fluent response cannot conceal missing evidence, stale documents or an access-control failure.

How to test banking RAG retrieval accuracy

Start with a versioned evaluation set of realistic questions and their expected source material. For each query, record the relevant document or passage identifiers, applicable product and customer context, document status or effective date, and any access restrictions. Have banking subject matter experts review the labels, especially where a policy, fee or product rule depends on a condition.

Include direct questions, paraphrases, banking terminology and abbreviations. Add ambiguous queries, questions with no supported answer, and queries that should return different material for different products or user roles. Keep personally identifiable and account data out of test queries unless the test specifically needs controlled synthetic data. Protect the dataset and results according to the bank's data-handling requirements.

Run each query through the production-equivalent retrieval path: query processing, metadata filters, access checks, search, reranking and top-k selection. Record the returned passage IDs, relevance judgements, ranks, scores, applied filters, corpus or index version, and execution outcome. A search against a hand-picked document subset will miss defects in filters and indexing.

Choose metrics that match the use case

Use more than one metric to understand different failure modes:

  • Recall@k measures the proportion of labelled relevant passages found in the first k results. Low recall means the system may lack evidence needed to answer.
  • Precision@k measures the proportion of the first k results judged relevant. Low precision can crowd useful evidence out of a limited model context.
  • Mean reciprocal rank (MRR) rewards placing the first relevant result near the top. Normalised discounted cumulative gain (nDCG) can account for graded relevance and result order.

Choose k to reflect the number of passages the application actually supplies to the model. A high average can hide a serious gap in a particular product, document type, language or access tier, so report results by risk-relevant slice as well as overall. Set release criteria from the use case and risk assessment; there is no universal acceptable score. Retrieval metrics do not establish that the generated answer is correct or that its citations support each claim. Evaluate those separately.

Example: retrieve a current transfer fee

Suppose the test query is: 'What fee applies to an international transfer from this current account?' The evaluation record should specify the account product and the applicable date, then identify the current fee schedule passages that answer the question. The expected results should exclude an expired schedule and fee information for another account type.

Run the query as a customer role and as a staff role if their permissions differ. Check that the customer receives only authorised material, that relevant current passages appear within the configured k, and that stale or unrelated results do not displace them. If two passages are relevant and only one appears in the top five, recall@5 is 0.5; if one of the five results is relevant, precision@5 is 0.2. Investigate the returned passages and filter trace to locate the cause, such as stale indexing, a wrong product filter or a poorly split document.

Keep the evaluation repeatable

Pin the corpus snapshot, parser and chunking configuration, embedding model, index settings, metadata filters and reranker version for each run. Otherwise, environment drift can make a score change difficult to explain. Include tests for document updates, withdrawn policies, duplicate or conflicting sources, and permission changes. Verify that an updated document becomes retrievable and that a withdrawn one no longer appears after the expected indexing process.

Run the labelled suite when retrieval configuration, source documents or models change, and retain the query, retrieved passages and configuration needed to reproduce failures. In continuous integration (CI), fail or flag a build when a risk-relevant metric drops below an agreed limit, or when a critical access-control or source-freshness case fails. Review regressions by query and source passage, then update labels only when the underlying banking policy or expected evidence has changed.

Frequently asked questions

Can an LLM judge replace expert relevance labels?

An LLM judge can help triage or expand relevance assessments, but it should not replace validated labels for banking policies and product rules. Review its decisions against subject matter expert judgements, particularly for conditional or conflicting source material.

Should retrieval testing include misspellings and abbreviations?

Yes, when customers or staff are likely to use them. Add realistic variants of product names, banking abbreviations and common typing errors, then label the same authoritative passages expected for the equivalent correctly phrased query.

How should retrieval accuracy be monitored after release?

Track retrieval outcomes using approved, privacy-controlled queries and sample results for review. Watch for changes by product, source type and user role, and investigate shifts alongside document ingestion, index or configuration changes.