How to test AI-generated portfolio summaries in private banking

Private bank AI-generated portfolio summary testing is the validation of summaries produced from a client's portfolio data, market information and reporting rules. It checks that each statement is supported by the correct sources, calculations and client context, and that the wording stays within its approved purpose. Testing covers factual accuracy, omissions, unsuitable claims, data isolation and safe handling of generated text.

What the test must establish

Start with a defined use case: for example, a quarterly summary for a relationship manager to review before sharing with a client. Specify the intended audience, reporting period, permitted data sources, required content and prohibited content. Decide whether the summary can describe performance, explain movements or mention investment considerations. Those boundaries become testable acceptance criteria.

Record the input snapshot and generation configuration for each test. Inputs may include positions, cash balances, prices, foreign exchange rates, corporate actions, performance figures, benchmark data, the client's investment mandate and the reporting date. Capture the prompt, model version, retrieval sources and relevant settings as well. Without this context, a failed output can be difficult to reproduce or attribute to stale data, a calculation service or a change in generation behaviour.

Test data should include varied portfolio conditions: multiple currencies, concentrated holdings, cash-only portfolios, illiquid assets, missing prices and portfolios with restrictions. Use synthetic or appropriately protected data, and verify that one client's information cannot appear in another client's generated summary or logs.

Validate facts and meaning

Check numeric claims against an independent expected result or trusted reporting output. Validate values, currencies, signs, units, periods and rounding. A number can be correct in isolation and still be wrong in context, such as a percentage attributed to the wrong quarter or a return presented in the wrong currency.

For narrative claims, define assertions at fact level. Check that holdings, performance, benchmark comparisons and dates match the source snapshot. Identify required facts that must appear and claims the system must avoid, such as an invented cause for a market movement, an unsupported forecast or a recommendation outside the approved scope. Review whether uncertainty is expressed when source data is incomplete.

Exact sentence matching is usually a poor primary check for generated prose because valid wording can vary between runs. Use deterministic checks for figures, dates, currencies and required fields, then assess factual support and suitability with a documented human review rubric. Where automated language evaluation is used, validate it against reviewed examples and retain a route for escalation.

Worked test example

Given a test portfolio valued at GBP 1,000,000 at the start of a quarter and GBP 1,020,000 at the end, with no external cash flows, the trusted reporting calculation gives a 2.0% return. The benchmark return for the same period is 1.5%.

A test can assert that the generated summary identifies the correct quarter, reports the portfolio return as 2.0%, gives the benchmark return as 1.5% and does not reverse which result belongs to the portfolio. A reviewer then checks that any explanation of performance is supported by supplied data. If the model attributes the result to a specific holding when no attribution data was provided, the output fails even though both percentages are correct.

Test failure paths and release changes

Exercise missing, delayed and conflicting inputs. Confirm that the system follows the defined fallback, such as withholding the affected claim, showing a clear limitation or routing the draft for review. Check that a failed generation does not publish a partial or stale summary as current. For retrieval-based systems, test source relevance, access permissions and behaviour when retrieved material conflicts.

Re-run a fixed evaluation set after changes to prompts, source data, calculation services or model configuration. Include repeated runs for cases where wording can vary, and compare factual assertions and review outcomes instead of relying on identical text. Preserve the input snapshot, output, version details, checks and reviewer decision as test evidence. This supports investigation of regressions and helps reviewers distinguish a generation fault from upstream data or integration drift.