How to test numerical reasoning in banking LLMs

Banking LLM numerical reasoning testing evaluates whether a large language model (LLM) can interpret banking figures and rules, calculate or select the correct result, and present it accurately. Tests use known inputs and expected outcomes to check calculations, assumptions, precision and explanations across the model, its prompts and any connected calculation tools.

How to test numerical reasoning in banking LLMs

Start by defining the capability under test. A customer-facing assistant might explain a fee, while an internal assistant might summarise account data or call a calculation service. These tasks have different risks and should have separate test cases and acceptance criteria.

Build cases from approved product rules and representative user requests. Each case should record the inputs, applicable rule, expected numeric result, unit or currency, rounding method, expected explanation and any permitted tolerance. Include conditions that affect the calculation, such as transaction eligibility, rate tiers, dates or account status. If a rule is missing or ambiguous, specify whether the expected behaviour is to ask for clarification or decline to calculate.

Cover ordinary values alongside boundaries and failure conditions. For a fee calculation, vary the transaction amount around eligibility limits, test zero and negative values where relevant, and check the minimum or maximum fee. Add cases for unsupported currencies, missing inputs and conflicting values. For date-dependent calculations, control the date, time zone, calendar and day-count convention in the test data.

Separate arithmetic, interpretation and tool behaviour

A plausible final number can hide different defects. The model may extract an amount incorrectly, apply the wrong banking rule, calculate inaccurately, or format a correct result in the wrong currency. Record intermediate expected values where possible, so a failed response can be attributed to the relevant step.

When the LLM calls a calculator or banking API, test the full interaction as well as the calculation service in isolation. Check that the model sends the correct arguments, handles tool errors and uses the returned result without changing it. A calculator can make arithmetic dependable while leaving rule selection and input interpretation exposed to model error.

Use a deterministic reference implementation or independently checked expected results as the oracle. Compare numeric values after parsing, and apply a tolerance only when the business rule permits one. Monetary amounts normally require the specified currency precision and rounding rule; a generic floating-point tolerance can conceal a one-minor-unit error. Score the explanation separately from the answer, including whether it states the relevant assumptions and avoids unsupported claims.

Worked example: a transfer fee

Suppose a test policy says an eligible transfer incurs a 1.5% service charge, rounded half up to the nearest cent. For an eligible transfer of GBP 2,400.00, the expected fee is GBP 36.00 and the total debit is GBP 2,436.00.

A test case can ask: "What fee and total debit apply to an eligible GBP 2,400.00 transfer under this policy?" Assert that the response gives both values, uses GBP, applies the stated rounding rule and does not treat the fee as part of the transfer amount. Add a boundary case for an ineligible transfer and expect the assistant to apply the defined ineligibility behaviour, such as stating that the fee does not apply. Keep the policy in the test fixture so the expected result does not depend on unstated product knowledge.

Run and interpret the evaluation

Keep a versioned evaluation set with the model identifier, system and user prompts, tool configuration, test data, expected results and evaluation code. Run it against each relevant model, prompt or tool change. For non-deterministic responses, repeat selected cases and report exact-answer rate, rule-application accuracy, currency and formatting errors, and unsupported-answer rate. Review failures by category as well as aggregate score; a high overall score can hide a serious defect in a small, high-impact scenario.

Control environment drift by fixing rates, dates, account state and service responses. Use synthetic or suitably protected data, and ensure test prompts contain no real customer information. Store the raw response and tool inputs and outputs as evidence, with access controls appropriate to the data. A passing result applies only to the tested model configuration and cases, so changes to prompts, retrieval sources, tools or banking rules require impact-based regression testing.