How to test unanswerable questions in banking RAG systems

Banking RAG unanswerable question testing checks whether a retrieval-augmented generation (RAG) system recognises when its authorised source material does not support an answer and responds according to the bank's approved abstention or escalation policy. It tests retrieval and answer generation together, so an unsupported financial claim is treated as a failure even when the response sounds plausible.

What the tests need to establish

A system should answer from relevant, permitted evidence. If that evidence is missing, incomplete or outside the assistant's defined scope, it should communicate the limitation clearly, ask for information where appropriate, or route the customer to an approved channel. The expected behaviour depends on the product and its policies, so define it before writing test cases.

Unanswerable questions are different from ordinary factual questions with a known expected answer. They probe whether the system can recognise an evidence gap. They also differ from hallucination testing: hallucination tests assess unsupported content across a wider range of outputs, while answerability tests deliberately create or identify cases where the system should not provide a substantive answer.

Separate the causes of an unanswerable case. The requested fact might be absent from the corpus, excluded by the user's access permissions, outside the assistant's remit, or unavailable because retrieval failed. These conditions can require different responses. An authorisation filter must not expose restricted content or confirm sensitive information through the response.

Build a useful test set

Start with the assistant's scope, source inventory and approved refusal behaviour. Create cases for gaps that matter to banking users, such as a product feature with no supporting document, a fee for an account type absent from the supplied material, or a request for an individual's account details when no authorised account source is available. Include ambiguous questions and requests that mix answerable and unanswerable parts.

For each case, record the query, user role, corpus snapshot, relevant documents expected to be available, expected disposition and unacceptable claims. Mark whether the expected result is an answer, a qualified partial answer, a clarification request, an abstention or a hand-off. Avoid writing expected answers as exact strings unless the interface requires fixed wording.

Use controlled documents and test identities. Keep restricted documents in the test corpus when assessing access controls, then verify that a user without permission cannot retrieve or receive their contents. For a pure knowledge-gap test, confirm that the fact is absent from all documents available to that test identity. Otherwise, an apparent abstention failure may actually be a retrieval or test-data setup problem.

Test retrieval and response separately

Run each case through the full application path and inspect both retrieved passages and the final response. When no supporting passage exists, the expected outcome is the approved abstention behaviour. When evidence exists but retrieval misses it, record a retrieval defect. When retrieved content is irrelevant, stale or from the wrong product, record a retrieval or filtering defect. When the response adds unsupported detail despite having no evidence, record an answer-generation failure.

Check that citations, when provided, point to passages that support the claims they accompany. Confirm that a refusal does not include fabricated rates, fees, eligibility conditions or account-specific information. For a partially answerable request, assess each requested fact separately: supported details may be answered within scope, while unsupported parts should be identified or escalated according to policy.

Save the retrieved passages, access-filter outcome, response, citations, model and prompt versions, and corpus version with each test result. This evidence helps distinguish model changes from index updates, document ingestion defects, prompt changes and environment drift.

A practical test case

Suppose a test corpus contains an approved fee schedule for account type A and no fee information for account type B. Ask: What monthly fee applies to account type B?

The expected disposition is abstention or an approved hand-off. The test passes if the assistant says it cannot confirm the fee from its available information, follows the configured next step, and gives no fee amount. It fails if it infers a fee from account type A, invents a figure, or cites the account type A schedule as evidence for account type B. Check the retrieved passages as well: if the schedule for account type A appears, the final answer must still avoid applying it to account type B.

Verify outcomes in regression runs

Run the suite after changes to prompts, models, document ingestion, chunking, embeddings, retrieval filters or access rules. Use the same corpus snapshot and test identities when comparing results. Language models can vary their phrasing, so assess the response disposition and factual claims against explicit criteria, with human review for ambiguous cases. A test should fail when the answer introduces unsupported banking information, even if it also contains a refusal sentence.

Track outcomes by failure cause, such as missed abstention, irrelevant retrieval, permission leakage or unsupported citation. This makes triage clearer than a single pass rate and helps teams target the affected component. Link cases to the relevant RAG testing in banking and Banking RAG retrieval accuracy testing coverage.