AI credit decision system testing evaluates whether a credit decision service produces accurate, consistent, fair and explainable outcomes under its approved model and policy, across relevant applicant data and operating conditions. It covers the model, data preparation, decision rules, integrations and human-review paths. Testing should confirm that decisions follow documented requirements and that defects or drift can be detected before they affect applicants.
Define the system and its expected behaviour
Map the full decision path: input data, validation and transformations, model output, policy rules, final decision, reason codes and any referral to a human reviewer. Record which component owns each step. A model can return a valid score while a faulty threshold, stale bureau response or incorrect rule produces the wrong outcome.
Agree expected behaviour with credit policy, model risk, compliance and engineering teams before writing tests. Document the model version, approved input features, decision thresholds, permitted overrides, missing-data handling and referral conditions. Requirements for fairness, explanations and adverse-action notices vary by jurisdiction, so have the relevant legal and compliance teams confirm applicable expectations.
Test data and model performance
Check data contracts and preprocessing first. Test required fields, types, ranges, category mappings, missing values and out-of-range values. Compare production transformations with those used during model development. Look for target leakage, where a feature contains information that would only be available after the decision, and confirm that validation data reflects the intended use of the model.
Assess predictive performance against an agreed validation dataset and appropriate measures for the credit outcome. Review results by relevant applicant segments as well as overall. A strong aggregate result can conceal weaker performance for a subgroup. Fairness measures can give different views of disparity, so select them with model risk and compliance stakeholders and document their limits. Use representative, privacy-controlled test data; synthetic records can help cover edge cases, though they may not reproduce real-world data patterns.
Verify decision logic and integrations
Test the model separately from the surrounding policy where the architecture allows it. Then test the complete decision path. Include scores just below, at and above each threshold, along with overrides, referral rules and conflicting inputs. Confirm that the final outcome, stored record, explanation and reason codes agree with the approved policy.
Exercise integration failures as well as successful responses. Examples include an unavailable credit bureau, a delayed response, duplicate requests, stale data or a malformed payload. The expected result might be a retry, a referral or a blocked decision, depending on the approved policy. Assert the specified behaviour rather than assuming that every failure should lead to approval or rejection.
Test fairness, explanations and change controls
Check that the system applies the same policy consistently to equivalent cases. Where authorised and appropriate, analyse outcomes across relevant groups and investigate material differences. Test whether an explanation refers to factors that actually influenced the decision, and whether the reason codes presented to downstream systems match the recorded outcome. An explanation that conflicts with the decision record can undermine review and customer communication.
Rerun affected tests when the model, feature pipeline, threshold, policy or upstream data source changes. Keep the model and rule versions with each test result so a failure can be traced to the deployed configuration. Production monitoring should track agreed indicators such as input-data shifts, score distributions, decision rates and subgroup outcomes. Define who investigates alerts and what action is permitted; monitoring alone does not correct a defective decision path.
Worked example: test a threshold and referral rule
Assume a hypothetical policy refers an application for manual review when a model score is below 0.60, and declines it when a separate policy rule flags a specified condition. The score and threshold here are illustrative.
Create controlled test records with scores just below, equal to and above 0.60, keeping other inputs constant. Assert that each record follows the documented boundary rule, then add a record that triggers the separate policy rule. Check the final decision, referral status, reason codes and persisted audit record through the service API. If the API returns the expected decision while the stored record shows another outcome, the end-to-end test should fail.
Frequently asked questions
Should fairness tests include protected attributes?
That depends on applicable law, privacy controls and the approved fairness assessment. Agree access and permitted use with legal, compliance and model risk teams, and keep any sensitive test data appropriately restricted.
What evidence should QA request from a third-party model provider?
Request documentation for intended use, input requirements, known limitations, validation methods, version changes and available performance or fairness evidence. If the provider cannot expose model internals, document the limits of black-box testing and assess whether the available evidence supports the institution's validation needs.
Should identical requests always return identical decisions?
For a fixed model and policy version with identical inputs and relevant context, define whether the service is expected to be deterministic. If time-sensitive data or an intentional randomised process can affect results, record those dependencies and test the permitted variation explicitly.