How to test knowledge freshness in banking RAG systems

Banking RAG knowledge freshness testing checks whether a retrieval-augmented generation (RAG) system uses the latest approved information that applies to a query. It verifies the path from a source change through ingestion, indexing, retrieval and response generation, including effective dates and source authority. The aim is to detect answers grounded in superseded, delayed or incorrectly prioritised banking content.

What freshness means in a banking RAG system

Freshness is a property of the evidence available to the model at query time. A document can be present in a vector index and still be stale if a newer version exists, its effective date has not been considered, or an update has not reached every retrieval component. A fluent answer does not prove that the system used current evidence.

Define freshness expectations by source and use case. A product terms document may change infrequently, while a service notice or operational procedure may need prompt propagation. Record the authoritative source, its version or update timestamp, any effective-from and expiry dates, and the expected update path. The acceptance criteria should state how soon an approved change must become retrievable and how the system behaves during that interval.

Separate freshness checks from retrieval relevance and answer accuracy checks. A retriever may find a relevant passage from an old document. A response may cite a current document while misrepresenting its contents. Test those properties independently, then include an end-to-end check that connects the retrieved evidence to the generated answer. See Banking RAG retrieval accuracy testing and Banking RAG citation accuracy testing for related checks.

How to test knowledge freshness

1. Set a baseline. Capture the source version, index state, query, retrieved passages, citations and answer for a controlled test question. Use synthetic or approved test content. Avoid putting real customer information into evaluation fixtures unless its use is authorised and protected.
2. Change the source. Publish a new version through the normal content process. Include a change that makes stale retrieval observable, such as a revised fee amount or updated support procedure. Preserve the previous version so the test can detect whether it remains eligible for retrieval.
3. Check propagation. Observe ingestion and indexing status, then query at defined points against the freshness acceptance criterion. Check the retrieval results as well as the final answer. This helps identify whether a failure sits in source synchronisation, parsing, chunking, indexing, caching or response generation.
4. Check applicability. Test documents with future effective dates, expired dates and overlapping validity periods. Confirm that the system selects the content applicable to the question's stated date, where date-sensitive answers are supported. Verify that a superseded version does not override the approved current version simply because it has a stronger semantic match.
5. Check removal and access changes. Remove or restrict a test document and verify that it no longer appears in retrieval after the required update process. Test relevant user roles separately: a fresh document must not become visible to a user who lacks permission to access it.
6. Repeat and retain evidence. Run the scenario after changes to ingestion code, embedding configuration, retrieval logic, prompts or source connectors. Store timestamps, document identifiers, versions, retrieved passages and expected results with the test run.

Worked example: an updated account fee

Create two approved synthetic policy versions. Version 1 states one monthly fee; version 2 changes the fee and has a defined effective date. Ask, “What monthly fee applies from [test date]?” before and after publishing version 2.

Before the effective date, assert that the answer reflects the version applicable to that date and cites its source. For a query on or after the effective date, assert that the updated amount and version are returned within the agreed freshness window. Inspect retrieved passages to confirm that the old version is not being treated as current. If the update has not propagated yet, the expected behaviour should be explicit, such as returning a controlled status or escalating the query, rather than stating an unsupported amount.

Common failure modes and trade-offs

A freshness test can pass at the document level while failing at the response level. For example, the new file may be indexed, but stale chunks may rank higher or a cache may serve an earlier answer. Check the complete query path and retain enough evidence to locate the failed stage.

Use controlled clocks or fixed test dates for effective-date cases. Otherwise, tests can become dependent on when they run. Define whether freshness is measured from source approval, publication or successful availability to users; those events may occur at different times.

Frequent polling can reveal propagation delays, while adding load to shared services. Run intensive checks in a representative test environment and use production monitoring that respects service limits and data controls. For risk-based coverage, prioritise sources used for customer-facing fees, eligibility or operational instructions. Link these scenarios to the wider Banking test strategy.