How to test failover for a banking infrastructure provider

Banking infrastructure provider failover testing checks whether a banking service can continue or recover when an external provider or its connection becomes unavailable. It exercises the switch to an alternative provider, route or service instance, then verifies transaction integrity, customer and operational outcomes, and recovery against agreed service objectives.

What banking infrastructure provider failover testing covers

The provider may supply core banking services, payment processing, card authorisation, identity checks, cloud hosting or a network connection. The test should follow the dependency far enough to expose failures in the bank's own routing, application logic and operations. A provider's claim that its platform is redundant does not prove that the bank's integration can use the alternative path correctly.

Agree the scenario with the provider and service owners before execution. Record which component will fail, how the fault will be introduced, who can stop the test, and what signals trigger rollback. Use a controlled test environment where possible. A production exercise requires explicit approval, customer and transaction safeguards, operational cover, and a documented stop procedure.

Prepare the failover test

Map the transaction path, including the banking application, integration layer, provider endpoints, authentication, network routes, queues and downstream ledger or reconciliation services. Identify whether the design is active-active, active-passive or manually switched. Confirm which party owns each switch and whether the bank can initiate it.

Define the expected recovery time and acceptable data loss or duplication for the affected service. These should come from approved service objectives and business impact analysis, rather than assumptions made during the test. Check that the alternative route is configured, credentials and certificates are valid, capacity is adequate, and test data will exercise relevant rules. Establish a known starting state and capture transaction identifiers and balances before injecting the fault.

The scenario should include transactions already in flight. A timeout can leave the caller unsure whether the provider processed a request. Verify that retries use the required idempotency controls, or that the bank can identify and resolve an uncertain outcome without posting a duplicate debit or credit. Include pending, rejected and queued transactions where the service supports them.

Run the test and verify outcomes

Use an agreed fault, such as disabling the primary endpoint, blocking its route, or invoking a provider-supported test switch. Avoid uncoordinated failure injection against shared provider infrastructure. Observe the detection signal, routing decision, time to successful processing through the alternate path, and any manual action required.

For each transaction, compare the bank's request and response records with the provider's status and the resulting ledger entries. Check that accepted payments are recorded once, rejected payments remain rejected, and uncertain transactions enter a defined investigation or retry path. Confirm that balances, settlement files, fees and reconciliation reports remain consistent. An application returning a success response is insufficient evidence if downstream records disagree.

Also verify customer-facing status and operational alerts. A transaction should not appear completed to a customer while its outcome is unknown. Confirm that support and operations teams can identify the active route, see affected transactions and follow documented recovery procedures.

Worked example: payment processor outage

Assume a bank routes card payments through a primary processor with a secondary route available. In a controlled environment, submit a known set of test authorisations, then make the primary route unavailable using the agreed method. Send a request that times out during the switch and retry it with the same idempotency key.

Pass criteria should be agreed before the run. They could require the secondary route to process eligible new requests within the service objective, the timed-out request to resolve to one authoritative outcome, and the ledger and processor records to reconcile without duplicate posting. Restore the primary route only after the team confirms whether failback is automatic or controlled and verifies that queued work has drained safely.

Failback, evidence and failure analysis

Failback is a separate behaviour to test. Check how the system detects primary-provider recovery, whether operators approve the return, and how it handles traffic and queued transactions during the change. A successful switch to the secondary provider does not establish that switching back is safe.

Keep evidence of the injected fault, timestamps, route changes, alerts, transaction IDs, provider responses, reconciliation results and decisions made by operators. Investigate failures by layer: detection, routing, authentication, transaction processing, data consistency or operational response. Link defects to the relevant Banking third-party service outage testing and Banking disaster recovery testing practices where those topics apply. Re-test after changes to provider configuration, interfaces or recovery procedures.