How to test cross-customer data isolation in banking RAG systems

Banking RAG cross-customer data isolation testing checks that a retrieval-augmented generation (RAG) system only retrieves and uses customer records the authenticated user is authorised to access. It verifies the full path from identity and access policy through search, prompt assembly and response, including citations and cached results, so one customer's information cannot appear in another customer's session.

Define the isolation boundary

Start with the bank's access rules. A customer may have access to individual accounts, joint accounts or business entities through a mandate. Define which records each test identity is allowed to see, including any legitimate shared access. The expected boundary should reflect those entitlements, not simply assume that every record belongs to one person.

Map how identity and permissions reach the RAG components. Relevant points can include the application API, document store, embedding pipeline, vector index, metadata filters, reranker, conversation store and response cache. Check that the authenticated identity and its authorised scope are derived by trusted application services. A prompt instruction asking the language model to ignore other customers' documents cannot enforce access control.

Use synthetic, clearly distinguishable records in a controlled test environment. Include customer identifiers, account details and unique canary phrases in documents, along with metadata linking each document to its permitted customer or entity. Avoid production customer data. Confirm that fixtures have been indexed and that the test environment's access rules match the intended configuration before running the checks.

Build positive and negative tests

First verify legitimate access. Authenticate as Customer A and ask for a fact present in A's permitted documents. Check that the answer is grounded in the expected source and that its citations refer to authorised records. This confirms the test can detect retrieval as well as leakage.

Then test isolation in both directions. As Customer A, ask a question whose answer exists only in Customer B's documents. Inspect the retrieved document IDs and metadata, the assembled context where observable, the response, citations and relevant API fields. Repeat with Customer B querying for A's canary. A safe outcome should exclude unauthorised sources and avoid disclosing their contents; the system may return an appropriate refusal or say it cannot find the information.

Test variations that can expose weak filters: paraphrased questions, exact account references, customer names, aliases, misspellings and questions that combine authorised and unauthorised facts. Include metadata boundary cases, such as a joint account that A can access and an unrelated account owned by B. Assert the expected entitlement for each record.

Check every stateful and retrieval path

Isolation defects can sit outside the main vector search. Test document ingestion and updates to ensure customer ownership metadata is retained. Check keyword or fallback search, reranking and any secondary retrieval source. Verify that conversation history from one customer's session is unavailable to another, including after logout and a new login.

Exercise caches with the same query under different customer identities. A response or retrieval cache must be scoped so that a result produced for A cannot be served to B. Repeat requests in different orders and, where the architecture supports concurrent sessions, run them concurrently. Include session expiry, token refresh and identity changes, since stale identity context can cause requests to use the wrong scope.

Assert the boundary, not just the wording

A fluent answer that omits a customer's name may still disclose their balance or transaction details. Conversely, a model can produce an incorrect answer without retrieving another customer's data. Treat isolation as a data-flow property: assert that every retrieved source is authorised for the active identity, then check that responses and citations do not reveal protected canaries.

For example, let A and B have separate synthetic statement documents, each containing a unique phrase. After querying as A for B's phrase, a test could assert:

assert all(source.customer_id in authorised_ids_for_A for source in retrieved_sources)
assert B.canary not in response.text
assert all(citation.document_id in authorised_document_ids_for_A for citation in response.citations)

Adapt the assertions to the system's actual response and observability interfaces. If retrieval traces are unavailable to the test, use a test hook or controlled instrumentation; response-only checks can miss a leak in retrieved context that the model does not repeat in its answer.

Diagnose failures and preserve evidence

A failure may originate in identity propagation, a missing or incorrectly applied metadata filter, stale index metadata, cache-key design, conversation storage or citation generation. Capture the test identity, authorised scope, query, retrieved document identifiers, response and relevant correlation IDs in restricted test evidence. Keep canary values synthetic, and ensure logs do not create a new cross-customer disclosure path.

Run these checks after changes to access policies, ingestion, retrieval configuration, prompts, models or caching. Record the expected access rule and the test result so a failed assertion can be traced to the affected customer boundary and system component.