How to test factual accuracy in banking LLMs

Banking LLM factual accuracy testing checks whether a large language model (LLM) gives correct, complete and appropriately qualified answers to banking questions, using authoritative evidence as the reference. It assesses individual claims as well as the overall response, including whether the model handles missing, outdated or conflicting information safely. The method applies to chatbots, staff assistants and retrieval-augmented generation (RAG) applications.

Define what counts as a factual error

Set the evaluation scope before writing tests. It may cover product terms, transaction status, fees, payment cut-off times, eligibility rules, or account-specific information. For every fact, identify the authoritative source and the conditions that affect its validity, such as product, customer segment, channel, currency and effective date.

Separate factual accuracy from related qualities. A response can cite a relevant document while misstating its contents; that is a factual error. A correct statement without required supporting evidence may pass a factual check but fail a groundedness check. Fluency and helpfulness scores do not establish factual correctness.

Define expected behaviour for unanswerable questions. If the supplied evidence does not establish an answer, the expected result may be a clear uncertainty statement, a request for clarification or a referral to an approved channel. A confident unsupported answer should count as a failure.

Build an evidence-backed test set

Create test cases from approved product documentation, business rules, API responses or controlled customer records. Record the source version, effective date, relevant conditions, expected claims and permitted qualifications. Keep test data synthetic or appropriately protected, and avoid using live customer information in an evaluation set without an approved basis and controls.

Include common questions and risk-focused edge cases. Vary dates, product variants, customer eligibility and wording. Test questions with incorrect assumptions, missing details and facts that have changed. For RAG systems, retain the retrieved passages and their source identifiers so the team can distinguish retrieval failures from answer-generation failures.

Review expected answers with banking subject matter experts. A reference answer should allow equivalent wording while making essential facts and restrictions explicit. Where a question has several valid answers depending on conditions, encode those conditions in the test instead of marking one sentence as universally correct.

Score claims and failure severity

Assess each material factual claim against the reference evidence. Record whether it is correct, incorrect, unsupported, incomplete or appropriately qualified. Track claim-level accuracy alongside question-level pass rates: one material error can make an otherwise polished response unsafe.

Weight failures by customer and operational impact. A wrong explanation of a general feature may have a different severity from an incorrect available balance, payment status or fee. Set release criteria for each risk class. Do not use one aggregate score to conceal severe errors or a high rate of unanswered questions.

Automated checks work well for stable, structured facts. Compare returned dates, amounts, status codes and product identifiers with an oracle such as a test API response or versioned fixture. Use expert review for nuanced explanations, conditional answers and whether a response implies more certainty than the evidence supports. An LLM-based evaluator can help triage results, but it should not be the sole authority for banking facts.

Run tests against the deployed answer path

Test the complete application path used by customers or staff. Include prompts, retrieval, permissions, tools and response formatting where they affect the answer. Verify that retrieved information belongs to the right product or customer context and that stale content does not override a current authoritative source. Environment drift, changed prompts, model updates and refreshed documents can all alter results.

Save the model and prompt versions, configuration, retrieved evidence, test data version, response and evaluation decision. Run the same cases after changes to these dependencies. Since responses can vary between runs, repeat selected high-risk cases and investigate inconsistent answers; a single passing response is weak evidence of reliable behaviour.

Worked example

Use a synthetic product fixture with an approved record stating that Easy Saver pays 2.8% annual equivalent rate (AER) from a specified effective date. Ask, "What rate applies to Easy Saver on [test date]?" The expected answer must give 2.8% AER only if the date falls within that record's validity. Check the rate and date against the fixture, then review whether the response adds an unsupported condition or presents the rate as applicable to other accounts.

Add a case where the effective date is outside the fixture's validity and no current rate is supplied. The expected behaviour is to state that the rate cannot be confirmed from the available information or direct the user to an approved source. This tests factual accuracy and uncertainty handling together.

For a release decision, examine failures by claim, severity, source version and application component. A failure in retrieval, source data or generation needs a different correction, and the affected tests should pass again after the change.