How to test conflicting sources in banking RAG systems

Banking RAG conflicting source handling testing checks whether a retrieval-augmented generation (RAG) system detects when retrieved banking documents disagree and responds according to approved source authority, scope and effective date. Tests verify that answers use the controlling evidence, show accurate citations, explain material uncertainty or abstain when the conflict cannot be resolved safely.

Define what counts as a conflict

A conflict occurs when two applicable sources make incompatible claims about the same product, customer, transaction or rule. A difference in wording alone may not be a conflict. For example, a product summary may describe a fee as 'up to £10', while a current tariff lists the precise fee for a particular account tier. The test must establish whether the statements conflict for the customer’s circumstances.

Before testing, agree the source precedence rules with the relevant product, policy or compliance owners. Define which source controls, how effective dates and superseded documents are treated, and whether jurisdiction, customer segment or product variant changes the answer. The RAG pipeline also needs reliable document metadata and access controls. Without them, a test cannot distinguish a source-selection defect from missing or incorrect metadata.

Keep expected behaviour explicit. For a resolvable conflict, the system may answer from the controlling source and cite it. For an unresolved conflict, it may state that the sources disagree and direct the user to an approved support route. It should not silently combine incompatible terms or present an unsupported answer as certain.

Build conflict test cases

Create controlled document fixtures with known versions, dates, scope and authority. Include cases where:

• A current approved tariff conflicts with an older tariff.
• A product page conflicts with a policy document that has higher approved authority.
• Two current documents disagree and neither has precedence.
• Similar terms apply to different account types or jurisdictions.
• A superseded document is retrieved alongside its replacement.
• One source is inaccessible to the requesting user.

For each case, record the user question, expected controlling source or expected abstention, permitted answer content, required citations and prohibited claims. Include questions that expose the conflict, such as asking for a fee after a policy change, and questions that specify the account type or date. This checks whether retrieval uses the query’s context rather than matching a familiar phrase alone.

Verify the full RAG path

Run each case through the deployed retrieval and generation path, including document ingestion, indexing, metadata filters, retrieval, reranking if used, and answer generation. Inspect retrieved passages as well as the final response. If the controlling document never reaches the model, the likely defect lies in ingestion, retrieval or filtering. If it is retrieved and the answer still follows a lower-authority source, investigate source-precedence instructions and generation behaviour.

Assert outcomes at several levels. Check that the selected source is permitted and current, the answer matches the approved rule, and each citation supports the claim it accompanies. Verify that the response identifies unresolved disagreement or abstains where required. Also check that it does not expose restricted content, invent a reconciliation, or imply that a policy applies beyond its stated scope.

Generated wording can vary between runs. Use semantic checks for the business rule and fixed checks for critical invariants, such as the cited document identifier, effective date, required escalation wording or absence of a prohibited fee. Keep a human review path for ambiguous cases where an automated assertion cannot reliably determine whether the answer changes the meaning of a banking rule.

Worked example

Prepare three fictional test documents for a current account: an approved tariff effective 1 June lists a monthly fee of £5; an archived tariff lists £3; and a general product page says the fee is 'up to £5'. Ask, 'What monthly fee applies to this account from 1 June?'

The expected result is £5, supported by the current tariff and its effective date. The archived tariff must not control the answer. The general page may be retrieved, but it must not lead the system to describe the fee as variable if the tariff gives a precise amount. Then ask, 'What fee applied before 1 June?' The expected source and answer change with the date. Record the retrieved passages, answer, citations and test-data versions so a failure can be reproduced after an index refresh or model change.

Diagnose failures and keep tests current

Separate source-content defects from RAG defects. A missing effective date or contradictory approved documents requires content ownership and correction. Incorrect retrieval points to indexing, metadata or ranking. Unsupported wording with correct evidence points to generation or answer validation. This classification helps route defects to the team that can fix them.

Re-run the suite after source updates, embedding or retrieval changes, prompt changes, model upgrades and permission changes. Monitor for environment drift between test and production, especially differences in document versions, access filters and index refresh behaviour. Treat citation presence as insufficient evidence on its own: confirm that the cited passage supports the specific claim and applies to the customer context.