How to evaluate document relevance in banking RAG systems

Banking RAG document relevance evaluation measures whether a retrieval-augmented generation (RAG) system finds the banking documents that are useful and authoritative for a given query. It tests the retrieval stage against labelled examples, including whether results match the product, jurisdiction, customer context and effective date. The aim is to detect retrieval errors before they produce unsupported or incomplete answers.

How to evaluate document relevance in banking RAG systems

Start with a representative evaluation set of questions and the documents that should answer them. Include routine requests, queries involving specific account or lending rules, and cases where the available material is incomplete or conflicting. Derive examples from approved requirements, service queries and business scenarios. Remove or mask customer information in line with the organisation's data-handling controls.

For each query, record its intended scope and the expected evidence. A query about an overdraft fee, for example, may depend on the account type, jurisdiction and policy version. Record the source document, relevant passage and effective period where these matter. Have banking subject matter experts review the labels, and resolve disagreements before using the set to compare system versions.

Evaluate retrieval independently from the generated answer. A relevant passage can still be summarised incorrectly, and a plausible answer can arise from a weak or incorrect retrieval result. Citation accuracy and answer groundedness therefore need their own checks.

Choose metrics that fit the retrieval risk

Use more than one metric, since each describes a different retrieval property:

• Precision at k (Precision@k) is the proportion of the first k retrieved results labelled relevant. It helps identify results that crowd out useful evidence.
• Recall at k (Recall@k) is the proportion of all known relevant results found in the first k. It requires a sufficiently complete set of relevant-document labels.
• Mean reciprocal rank (MRR) measures how highly the first relevant result appears across queries.
• Normalised discounted cumulative gain (nDCG) accounts for graded relevance and gives more weight to relevant results near the top of the ranking.

Set relevance labels and acceptance criteria before comparing runs. For example, a graded scale can distinguish an authoritative, current policy from a related but superseded document. Choose thresholds according to the use case and risk; a query used to explain a general process may have different retrieval requirements from one supporting a fee or eligibility decision.

Test banking-specific relevance conditions

Include examples that expose errors a general-purpose test set can miss. Check whether retrieval favours current documents over superseded versions, applies the correct jurisdiction and product scope, and distinguishes customer-facing guidance from internal procedure. Test near-duplicate documents and conflicting sources, and verify that the intended source hierarchy is reflected in the expected results.

Treat access control as a separate, mandatory property. A relevance score does not show whether the user was entitled to see a result. Test queries under distinct user roles and customer contexts, and assert that restricted documents never appear in results, citations or downstream logs accessible to unauthorised users. Keep these checks separate from aggregate relevance metrics so that a strong average cannot obscure a permission failure.

Run the same evaluation set after changes to document parsing, chunking, embeddings, search configuration or reranking. Record the index and configuration versions, query, retrieved identifiers, rank, labels and metric results. This evidence helps distinguish a retrieval regression from changes in the source corpus or environment.

Worked example

Suppose a customer asks, "What early repayment fee applies to my fixed-rate mortgage in Scotland?" The expected evidence is the current Scottish mortgage tariff and the applicable product terms. A result for a variable-rate mortgage, an England-specific tariff or an expired tariff should not receive the same relevance label as the current evidence, even if it contains the phrase "early repayment fee".

If two of the first three results are labelled relevant, Precision@3 is 2/3 for this query. Review the ranked results as well: if the current tariff is absent from the top results, inspect the query interpretation, document metadata, index and ranking configuration. Then check the generated response and citations separately to confirm that the system used the correct evidence. A retrieval score alone cannot establish that the final answer is accurate or safe to use.