How to test LLM prompt versions in banking

Banking LLM prompt version testing is the controlled comparison of a changed prompt with an approved baseline, using representative banking scenarios and recorded expected behaviours. It checks whether the new version improves the intended responses while preserving accuracy, policy handling, privacy controls and workflow behaviour. Results are meaningful only when the model, configuration, data and connected tools are controlled or recorded alongside the prompt.

What to hold constant

Treat a prompt change as a release change. Give each version an identifier and retain the exact system and user prompt templates, variable values, model identifier, model parameters, retrieval configuration, tool definitions and relevant application build. Record the test date and environment as well. A model provider may change behaviour behind a stable model name, so capture any available model revision or deployment identifier.

Use the same inputs for baseline and candidate runs. Where an application uses retrieval-augmented generation (RAG), pin or record the document set and retrieval settings. If the model calls APIs or other tools, use controlled test doubles where possible, or reset their state between runs. Otherwise, a response difference could come from changed source content, tool results or environment drift rather than the prompt.

Prepare test data that represents the intended use, including permitted synthetic or appropriately anonymised customer information. Keep production personal and financial data out of test prompts unless its use has been explicitly authorised and protected under the bank's data controls. Restrict access to prompt logs and test outputs, which may themselves contain sensitive information.

Build a prompt regression suite

Define observable acceptance criteria for each scenario before comparing versions. Assess the response against the task and applicable policy, rather than relying on a single similarity score. Useful checks include factual correctness against an approved source, required disclosures, refusal or escalation behaviour, output schema, tool-call arguments and whether the response invents account facts.

Include ordinary questions and difficult cases: ambiguous requests, missing information, conflicting source documents, out-of-scope requests, prompt injection in retrieved content, and attempts to obtain another customer's data. For banking workflows, test that an assistant distinguishes a payment's submitted status from confirmation that it has settled. If the model can initiate actions, assert that it follows the required confirmation and approval steps and does not repeat an action after a timeout.

Run the baseline and candidate against the same cases. Compare outputs with automated assertions for deterministic requirements, such as valid JSON fields or required status values. Use a documented human review rubric for qualities that need judgement, such as whether an explanation is clear and properly grounded. Repeat selected cases when output variability could affect the result, and assess the range of outcomes rather than treating one successful response as proof.

A practical comparison

Suppose a bank changes a prompt for an assistant that answers questions about transfers. The test asks, "Has my transfer arrived?" The controlled account data shows a transfer with status "processing", and the approved source explains that this status does not confirm settlement.

For both prompt versions, assert that the answer does not claim the funds have arrived, does not invent a completion time, and directs the customer to the available status information. Add a case where the account has no matching transfer and check that the assistant asks for clarification or follows the defined fallback. Record the prompt version, model configuration, input data, retrieved evidence, output, assertions and reviewer decision. This lets a failure be traced to a specific test run.

Decide whether a version passes

Set pass criteria before execution, with stricter treatment for high-impact failures. A candidate should meet its defined acceptance criteria and should not introduce a regression in safety or workflow controls. Track failures by scenario and severity; a small aggregate score can conceal a serious failure in a specific customer journey. Keep the approved baseline available for comparison and rollback.

A failed test can indicate a prompt defect, a changed model response, retrieval differences, a tool integration problem or a test environment issue. Reproduce the failure with recorded inputs and dependencies before assigning a cause. After release, monitor representative interactions under the bank's privacy and access controls, and rerun the suite when prompts, models, retrieval sources or connected tools change. Prompt version testing complements LLM evaluation in banking and should connect to the wider banking regression testing process.