How to test citation accuracy in banking RAG systems

Banking RAG citation accuracy testing checks whether citations in a retrieval-augmented generation (RAG) answer resolve to the correct banking sources and support the claims they accompany. It also tests whether material claims have adequate citations, whether cited information is current and applicable, and whether the user is authorised to access it.

What citation accuracy means in a banking RAG system

A citation can link to a genuine document and still be inaccurate. The cited passage may not support the claim, may apply to a different account type or jurisdiction, or may come from an out-of-date policy. A correct citation should let a reviewer find evidence that supports the associated claim in the relevant context.

Assess citation accuracy at claim level as well as answer level. Check that each material factual claim has supporting evidence, that each citation supports the claim beside it, and that citations do not imply more than the source says. For numerical values, dates, eligibility conditions and exceptions, verify the exact wording and scope. A general statement about a fee, for example, may need a source that specifies the account type and effective date.

Citation accuracy overlaps with retrieval accuracy and groundedness, but each checks a different part of the system. Retrieval accuracy concerns whether the right source material was found. Groundedness concerns whether the answer follows that material. Citation accuracy checks whether the displayed references identify the evidence that supports the answer.

How to test citation accuracy

1. Build a representative question set. Include common customer and staff questions, account and product variations, different effective dates, and cases where the approved documents do not provide an answer. Use controlled test data and approved source documents. Avoid live customer information unless the test environment and handling controls explicitly permit it.
2. Create an evidence baseline. For each question, record the expected answer claims and the source passages that support them. Capture document identifiers, versions or effective dates, and page, section or chunk references where available. Have suitable subject matter reviewers resolve ambiguous policy wording before treating a passage as expected evidence.
3. Run the system and preserve its inputs. Record the question, answer, citations, retrieved passages, index or corpus version, and relevant prompt and model configuration. These details help distinguish a citation defect from a source-ingestion or retrieval change when a test fails. Use repeat runs where output variability could affect the result.
4. Validate each claim and citation. Confirm that the citation resolves, points to the expected document version, and supports the associated claim in context. Check for uncited material claims, irrelevant citations, omitted qualifications, and claims that overstate the source. Verify access permissions for the intended user role as part of the test.
5. Record failures in a useful way. Classify defects, for example, as missing citation, broken reference, wrong source, unsupported claim, incomplete coverage, stale source or access-control failure. Report results by question and claim, with the expected evidence and observed output, so the team can reproduce and investigate them.

A simple scorecard can report the proportion of tested claims with supporting citations, the proportion of citations that support their linked claims, and the proportion of references that resolve to the expected source. Keep these measures separate: a system can cite relevant documents while leaving important claims unsupported, or cite every claim with references that point to the wrong version. Agree acceptance criteria with product, risk and compliance owners for the system's intended use.

Worked example: testing a fee answer

Suppose an approved test document states that a named current account has a monthly fee of £5 from a specified effective date. Ask the RAG system, “What is the monthly fee for this account?” The expected result is a £5 answer with a citation to the applicable tariff passage.

The test passes only if the reference opens the correct document version and the cited passage supports the amount and account scope. It fails if the answer cites a different product tariff, omits a relevant qualification, gives an unsupported amount, or presents the figure as current when the cited tariff has expired. A separate question for a product with no fee information in the test corpus can check whether the system avoids presenting an unsupported answer.

Keeping citation tests reliable

Rerun the relevant cases when source documents, ingestion logic, retrieval configuration, prompts or models change. Include regression cases for source conflicts and superseded documents, and check that updates do not make restricted material visible to unauthorised roles. Keep the evidence baseline versioned with the test results; otherwise, a changed source can make a valid citation look like a system defect, or conceal a genuinely stale answer.

For high-impact banking information, automated checks can confirm citation structure, reference resolution and expected document identifiers. Human review remains useful for judging whether a passage supports a nuanced claim, especially where policy wording has conditions or exceptions. Link failures to the affected requirement or control so release reviewers can assess their impact.