How to test RAG systems in banking

RAG testing in banking evaluates whether a retrieval-augmented generation system finds authorised, relevant banking information and uses it to produce accurate, appropriately qualified answers. It covers the full path from document ingestion and search to generated text, citations, access controls and privacy. Testing should use controlled banking data and assess retrieval and generation separately as well as together.

How RAG works in a banking application

A retrieval-augmented generation (RAG) system searches a knowledge source for content relevant to a user’s question, then supplies selected passages to a large language model (LLM) to help it form an answer. A banking implementation might retrieve product terms, operational procedures or approved customer support content.

The result depends on several components: source documents, parsing and chunking, metadata, an embedding model, a vector or hybrid search index, access-control filters, prompts and the LLM. A failure in any component can produce an answer that is inaccurate, stale or disclosed to the wrong user. Testing therefore needs to cover the pipeline and its dependencies, rather than assessing generated wording alone.

Prepare a banking RAG test set

Build a controlled set of questions with expected evidence and expected behaviour. Include answerable questions, questions with no supporting source, ambiguous wording, conflicting documents and questions that require information from a particular effective date. Use approved, synthetic or properly de-identified material for test data. Avoid copying live customer records into an evaluation set without appropriate approval and controls.

For each case, record which user or role is asking, what information that role may access, the authoritative source, the expected answer points and whether the system should answer, qualify its response or decline. Keep the expected evidence separate from the model output so an evaluator can check whether the answer is supported.

Test retrieval, generation and permissions

Test retrieval first. Check whether the results contain the relevant passages, exclude obsolete or unrelated material, and respect metadata such as product, region, document status and effective date. Retrieval metrics such as Recall@k can help measure whether relevant passages appear within the first k results. Pair these metrics with manual review: a high retrieval score does not prove that the selected passage is authoritative or suitable for the question.

Then test generation against the retrieved evidence. Check factual accuracy, completeness, relevance, uncertainty handling and citation accuracy. A citation should identify a source that exists and supports the associated claim. Include cases where the supplied evidence is insufficient, where two approved sources conflict, and where retrieved text contains instructions that attempt to override system controls. The expected behaviour should follow the bank’s approved policy for escalation, qualification or refusal.

Treat access control as a retrieval requirement. Test that users with different roles receive only permitted passages, including when they ask questions designed to reveal another customer’s or business unit’s information. Check access filtering before content reaches the model, and inspect application logs and traces for unintended exposure of retrieved text or personal data.

Verify ingestion, freshness and change behaviour

Confirm that document ingestion preserves relevant text, tables, headings, metadata and version information. Test updates, removals and re-indexing so that superseded procedures or product terms do not continue to appear in answers. If source systems and the search index update on different schedules, verify how the application handles that delay and whether it indicates when its information may be out of date.

Run regression tests after changes to source documents, chunking, embeddings, retrieval settings, prompts or model versions. Record the configuration alongside results because a change to one dependency can alter retrieval and answer quality. Repeated runs may vary; use stable evaluation settings where possible, and review both aggregate measures and individual failures.

Practical example: a product terms question

Suppose a customer asks, "How much notice do I need to give to withdraw from this account?" Create a test case with the relevant account product, customer role and approved terms document, including its effective date. Verify that retrieval selects the current terms and excludes an expired version. Check that the answer states the notice period accurately and cites the supporting section.

Repeat the case with an account type the customer is not entitled to view. The system should not retrieve or disclose that account’s terms. Then remove the supporting source from the test index and ask the question again. The expected result is a safe response, such as asking the customer to consult an approved channel, rather than a guessed notice period. Capture the question, role, retrieved document identifiers, answer, citations and system configuration as test evidence.

Decide whether the system is ready for use

Set acceptance criteria around the risks and intended use. A customer-facing product-information assistant may require strict controls on unsupported claims and clear escalation paths. An internal procedure search tool may have different access rules and answer requirements. Define which failures block release, including cross-user data exposure, unsupported financial terms and incorrect handling of stale or conflicting sources.

RAG testing provides evidence about the tested data, configuration and scenarios. It does not establish that every possible answer will be correct. Pair pre-release evaluation with operational monitoring, incident handling and a process for updating or removing source content.