How to test entity extraction in banking chatbots

Banking chatbot entity extraction testing checks whether a chatbot identifies the correct values in a customer's message and assigns them to the right fields. It covers extraction, normalisation and safe handling of banking data such as payment amounts, dates, currencies and account references. Tests should verify both the extracted values and what the chatbot does when those values are missing, ambiguous or invalid.

What to test in banking chatbot entity extraction

An entity is a value mentioned in a message, such as an amount or date. The entity type describes its meaning: "£250" might be an amount, while "tomorrow" might be a payment date. Entity extraction is distinct from intent recognition, which identifies the customer's goal, such as making a transfer or checking a balance. A chatbot can recognise the intent correctly and still extract the wrong amount or account.

Start with the fields used by each supported banking workflow. For a transfer, these might include amount, currency, source account, beneficiary and execution date. Define accepted formats, permitted values and any normalisation rules. For example, decide whether "two hundred pounds" and "£200" map to the same amount, and how the service represents a date expressed as "next Friday".

Build a labelled test set from synthetic or appropriately protected messages. Include common wording, paraphrases, spelling errors, punctuation variations and realistic shorthand. Add negative cases where a value is absent, malformed, outside permitted bounds or could refer to more than one entity. Avoid production customer messages unless their use has been approved and privacy controls are in place.

A practical test approach

For each test message, record the expected intent, entity type, extracted value and normalised value. Include the conversational context when a message depends on an earlier turn. Then test the extraction service directly, where possible, and through the chatbot conversation. Direct tests help isolate model or parser behaviour; conversation tests check that the chatbot uses the result correctly.

For example, a synthetic test might use:

• Message: "Move 250 pounds from my bills account to Alex tomorrow."
• Expected intent: make a transfer.
• Expected entities: amount 250, currency GBP, source account bills, beneficiary Alex, execution date resolved according to the test's fixed date and locale.
• Expected behaviour: ask for confirmation or for any required detail that remains unresolved; do not submit a transfer based on an incorrect or ambiguous extraction.

Fix the test clock and locale when checking relative dates. Test similar phrases that change one field at a time, such as "Move 25 pounds" and "Move 250 pounds", to catch truncation and digit errors. Use account aliases and beneficiaries from controlled test data, and check that one customer's data cannot be selected from another customer's conversation.

Check accuracy, ambiguity and downstream behaviour

Compare predicted entities with labelled expected values. Exact-match accuracy is useful for fields where the full value must be correct, such as an account reference. Precision, recall and F1 can help assess entity detection across a labelled dataset, but aggregate scores can conceal errors in high-impact fields. Report results by entity type and inspect incorrect predictions, especially wrong beneficiary, amount or date values.

Test normalisation separately from detection. A system may identify a date correctly while converting it to the wrong day, or extract an amount while assigning the wrong currency. Include locale-specific number and date formats where the chatbot supports them, and verify that currency symbols and decimal separators are interpreted according to the stated rules.

Exercise ambiguity and missing information explicitly. If "move 500 to savings" could refer to more than one source account, the chatbot should ask a clarifying question. If a date cannot be resolved safely, it should request clarification rather than silently choosing one. Verify the resulting conversation state, including whether a correction replaces the earlier value and whether a confirmation displays the same values that would be sent to the banking service.

Keep results reliable in CI

Run labelled extraction cases as part of regression testing when prompts, models, parsers, entity schemas or upstream services change. Pin relevant configuration and model versions where practical so that a failed case can be reproduced. Separate extraction failures from integration failures by recording the chatbot's parsed fields and the downstream request in a protected test environment.

Set acceptance criteria for each field based on its use and risk. A mistaken payment amount or beneficiary needs stricter handling than an incorrectly extracted non-transactional preference. Verify that failed or uncertain extraction cannot trigger an unintended banking action, and retain test evidence without exposing real account details or credentials.