How to test banking AI agents in a sandbox

Banking AI agent sandbox testing is the controlled evaluation of an AI agent’s decisions and actions in an isolated environment that represents banking workflows. It checks whether the agent uses approved tools, respects permissions and transaction controls, handles failures safely, and leaves verifiable evidence without affecting live accounts or payments.

How to test banking AI agents in a sandbox

A sandbox is useful only when its boundaries are explicit. It may connect to test versions of banking services, simulators for external providers, or both. Confirm which services are real, which are simulated, and what data or actions can cross the boundary. A sandbox can still cause harm if it has live credentials, unrestricted network access, or access to production customer data.

1. Define the permitted actions

Write down the agent’s intended tasks and the actions it must never take. For a payment-support agent, allowed actions might include retrieving a test account’s balance and preparing a transfer request. Submitting the transfer could require customer confirmation or a separate approval step.

Turn these rules into testable criteria. Check tool permissions at the service boundary as well as in the agent’s instructions. An instruction telling the agent to seek approval does not enforce approval if the payment API accepts an unauthorised call.

2. Isolate services, credentials and data

Use dedicated sandbox credentials with limited permissions. Block outbound connections by default, then allow only the test services the agent needs. Verify that a rejected or misrouted request cannot reach a live payment endpoint.

Use synthetic or suitably protected test data, and check that one test customer’s information cannot appear in another customer’s session or agent memory. Reset state between scenarios where appropriate. Keep test accounts and identifiers distinct from production resources.

3. Test the complete tool workflow

Exercise the agent through its normal entry point and observe the calls it makes to APIs or other tools. Cover valid requests alongside missing details, invalid account references, insufficient funds, timeouts, permission denials and conflicting instructions. Check tool selection, arguments, call order, returned status and the agent’s response to each result.

Include safety cases. For example, provide a message or retrieved document that instructs the agent to ignore its banking rules and transfer funds. Confirm that the agent treats that content as untrusted, makes no unauthorised tool call and escalates or refuses according to the defined policy.

4. Assert banking outcomes and controls

Assess outcomes across the agent and the banking services it calls. Useful checks include whether an approval gate was enforced, whether a retry created a duplicate request, and whether a failed operation left account state unchanged. For transactions, verify the resulting test ledger or payment status, not just the agent’s summary.

Agent responses can vary between runs. Use assertions for policy compliance, tool calls, transaction state and required information. Treat exact wording as a separate requirement only when the wording itself matters, such as a mandated disclosure. Record the model, prompt, tool schema, policy configuration and sandbox build used for each run so changes can be investigated.

Worked example: transfer approval

Suppose a customer asks the agent to transfer £80 from test account A to test account B. The policy requires the agent to prepare the transfer and obtain confirmation before submission.

1. Send the request without confirmation. Assert that the agent gathers required details or prepares a request, and that the payment service receives no submission call.
2. Provide confirmation through the expected channel. Assert that the agent calls the permitted test payment tool with the correct account references and amount.
3. Make the tool return a timeout, then allow a retry. Assert that the agent checks the transaction status or uses an idempotency control, and that the test ledger records no duplicate transfer.
4. Inspect the tool logs and test ledger. Confirm the approval, request status and final account changes agree.

The test data and tool names in this example are illustrative. Map the checks to the bank’s actual agent interfaces and transaction controls.

Interpreting sandbox results

A passing run shows that the tested configuration behaved as expected in that environment. It does not establish that production will behave identically. Differences in service versions, network rules, data, model configuration or third-party responses can create environment drift. Track those dependencies and check that the sandbox reflects the behaviours relevant to the test.

Retain the scenario, inputs, tool-call trace, service responses, state checks and configuration versions as test evidence. Investigate failures at the boundary where they occur: agent reasoning, tool validation, access control, service response or test-environment setup. This makes reruns more useful and helps distinguish an agent defect from a sandbox fault.