What is banking AI agent policy adherence testing?

Banking AI agent policy adherence testing checks whether an AI agent follows a bank’s defined rules and control boundaries while it interprets a request, plans work and uses connected tools. Tests verify both the agent’s responses and the actions it attempts, including whether it refuses, requests approval or escalates when a policy requires it.

What policy adherence testing covers

An agent can call APIs, retrieve account information or initiate banking workflows. Policy adherence testing examines whether its decisions and actions stay within rules such as access permissions, transaction limits, customer consent requirements and approval gates. The rules must be made specific enough to produce observable expected outcomes.

Testing should cover the full path from a user request to the resulting system state. A reassuring message does not demonstrate adherence if the agent also made an unauthorised tool call. Conversely, a safe refusal can be correct even when the user’s request is understandable, if the agent lacks permission or required information.

This work is distinct from factual accuracy testing, which checks whether an answer is correct, and task completion testing, which checks whether the agent completed an intended task. A response can be accurate and complete while still violating a policy. Policy adherence focuses on permitted behaviour and controlled outcomes.

How to make the tests executable

Start with an approved policy source and identify the rules that affect the agent’s scope. Convert each rule into conditions, an expected decision and assertions that can be checked. Include permitted requests, requests that exceed a limit, missing approvals, ambiguous instructions and attempts to bypass a control through supplied content.

Set preconditions explicitly. The test environment needs the relevant policy version, agent and tool configuration, user role, account state and test data. Use a sandbox or controlled tool doubles for actions that could move funds, change customer details or affect a real account. Confirm that test doubles represent the tool responses and errors the agent may encounter.

Observe more than the final text response. Capture the agent’s tool selection and arguments, authorisation checks, approval requests, downstream state changes and audit events. Keep traces linked to the test case, policy version and software configuration. This helps distinguish an agent decision error from a permissions defect or environment drift.

Agent outputs can vary between runs. Where exact wording is not part of the policy, assert stable properties such as whether a restricted tool was called, whether approval was obtained first and whether the final transaction state is allowed. Record model and prompt versions, and investigate unexpected variation rather than treating one successful run as proof of consistent behaviour.

Worked example: transfer requiring approval

Assume a bank policy requires an authorised customer confirmation before an agent submits a transfer, and requires an additional approval when a configured limit is exceeded. A test can set up a customer with a permitted account and a pending transfer above that limit.

The test should assert that the agent checks the relevant limit, does not submit the transfer before approval, requests the required approval and records the decision. After approval, verify that it submits the intended amount and beneficiary once, and that the resulting account and payment states match the expected outcome. Then repeat with approval denied or unavailable. The expected result is that no transfer is submitted and the agent explains or escalates the blocked request according to the policy.

A failure includes any prohibited tool call, a missing approval check, a duplicate submission after a retry, or a final state inconsistent with the approved request. A response that says the action was blocked does not pass if the payment system shows it was submitted.

Interpreting results

A policy test is only as useful as its rule coverage and evidence. Link cases to the policy clauses or control requirements they exercise, and review gaps where a rule has no observable assertion. Include negative testing for conflicting instructions, unavailable services and external content that tries to redirect the agent.

Treat a failure as a control issue to investigate, even when the immediate outcome was harmless. The cause may sit in agent planning, tool permissions, approval handling or downstream enforcement. Rerun the case after the fix and check relevant regression scenarios, including cases that should remain allowed. Record residual gaps where a policy cannot be tested reliably in the available environment.