How to test response consistency in banking LLMs

Banking LLM response consistency testing checks whether a large language model (LLM) gives materially consistent answers to the same banking question, and to equivalent questions, under controlled conditions. It measures stability in facts, calculations, policy conditions and safety behaviour while allowing harmless wording differences. The aim is to detect changes that could mislead customers or staff, rather than require identical wording from every response.

What consistency means for a banking LLM

A response can vary in phrasing and remain consistent. A material inconsistency occurs when repeated answers give different fees, eligibility conditions, transaction states, dates or next steps for the same inputs and evidence. A change in whether the model discloses protected information or follows an approval boundary is also material.

Consistency and factual accuracy are separate quality dimensions. An LLM can repeat the same incorrect answer reliably, or give a correct answer in one run and an incorrect one in another. Evaluate both against approved requirements and source data. For background on broader evaluation, see [LLM evaluation in banking].

Exact-match comparison is useful for fixed output fields, such as a status code or JSON value. It is usually too strict for free-text answers. Semantic similarity tools can flag responses for review, but they may miss a changed number or condition hidden in otherwise similar wording. Define critical facts and prohibited claims as explicit assertions.

How to run banking LLM response consistency testing

1. Fix the test conditions. Record the model and deployment version, system and user prompts, decoding settings, retrieval corpus or index version, tool responses, and relevant conversation history. Use controlled test accounts and synthetic data. Mock external services where possible. If these dependencies change between runs, the test may measure environment drift rather than model variability.
2. Build a risk-based prompt set. Include repeated identical questions, paraphrases with the same meaning, questions with changed facts, and cases where the supplied information is insufficient. Cover high-impact areas such as fees, balances, payment status and eligibility. Keep the expected result traceable to an approved rule, policy document or API response.
3. Repeat each scenario. Run the same case more than once with identical settings and evidence. Then test equivalent phrasings separately. Compare material facts, conditions, calculations, citations where required, output structure and safety behaviour. Keep a record of each prompt, dependency version, response and assertion result.
4. Check boundaries and regressions. Test that changing a relevant input changes the answer appropriately, while irrelevant wording changes do not alter the underlying facts. Rerun the suite after prompt, model, retrieval, policy-content or tool changes. Investigate failures before updating expected results; a baseline update should reflect an approved change.

Set acceptance criteria before execution. For example, require a specified fee and its conditions to remain correct in every controlled run, while allowing different sentence structure. Any tolerance for numerical variation must reflect the product requirement. A tolerance that could change a customer decision is unsuitable.

Worked example: a transfer-fee answer

Use a test-only policy fixture stating: "Same-bank transfers cost £0. The fee applies to transfers submitted through the mobile app. Other transfer types are outside this fixture." Ask the assistant, "What does a same-bank transfer cost in the app?" Repeat the request unchanged, then use a paraphrase such as, "Is there a charge for moving money between accounts at this bank using the app?"

Assert that each answer states £0, preserves the mobile-app condition, and does not claim that other transfer types are free. Also check that the response does not invent an effective date or a policy exception. A reviewer can assess natural-language answers against these assertions; structured output can use field-level checks. The fixture is illustrative and must not be treated as a real bank policy.

If one run says £0 and another introduces a charge, inspect the captured prompt, retrieved passages, model settings and tool outputs. A changed retrieved document points to a retrieval or content dependency; identical inputs with differing answers point towards response variability. The distinction guides the next investigation.

Interpreting results and limitations

Track failures by scenario and assertion, not only by an overall similarity score. Review near-matches involving amounts, dates, eligibility or customer-specific data manually. A passing consistency suite does not establish that the source policy is correct, that every possible prompt is safe, or that the system behaves correctly in production. Pair it with factual-accuracy, privacy, refusal and integration checks appropriate to the use case.

Consistency also depends on the tested configuration. Results from one prompt, model version or retrieval snapshot do not establish behaviour after those components change. Preserve test evidence so failures can be reproduced and linked to the exact configuration.