How the EU AI Act changes software testing for high-risk AI systems

When people talk about the EU AI Act, the conversation usually revolves around compliance, legal teams, and new regulations. But there's another team that will have a major role in making it work: software testing.

For companies building or using AI systems, especially high-risk AI systems, testing is no longer just about catching defects before a release. Teams also need to show that their systems behave as expected, are reliable, can be monitored, and include the right level of human oversight. In other words, testing becomes part of demonstrating compliance, not just product quality.

The good news is that most organisations won't face the strictest requirements. The Artificial Intelligence Act follows a risk-based approach, meaning the obligations depend on how an AI system is used. Some applications will have very few additional requirements, while others will need much more rigorous testing, documentation, and ongoing monitoring.

In this article, we'll look at what the AI Act means for QA teams, which AI systems are affected, and how software testing can help organisations meet the new requirements without adding unnecessary complexity.

TL;DR

• The EU AI Act is the world's first comprehensive law regulating artificial intelligence, using a risk-based approach rather than applying the same rules to every AI application.
• Most AI systems face minimal obligations, but high-risk AI systems require much stricter testing, documentation, monitoring, and human oversight.
• Software testing is no longer just about finding defects. It also helps organisations demonstrate that AI systems are reliable, secure, and compliant.
• QA teams should expand their testing beyond functionality to include accuracy, robustness, bias, security, transparency, traceability, and end-to-end business processes.
• Regression testing becomes even more important as AI models evolve. Every model update should be treated like any other software release.
• Compliance doesn't end after deployment. Continuous monitoring, incident management, and clear documentation are key parts of maintaining compliance over time.
• A risk-based testing approach helps organisations prioritise the workflows with the biggest business and compliance impact instead of trying to test everything equally.

What the EU AI Act means for software testing

The EU AI Act (officially the Artificial Intelligence Act) is the first law to introduce a common set of rules for artificial intelligence across the EU. It does not regulate every AI system in the same way. Instead, it classifies AI based on risk, with the strictest requirements applying to high-risk AI systems.

For banks and other regulated industries, this matters because AI is becoming part of more business-critical processes. It is already used for fraud detection, credit scoring, anti-money laundering checks, customer onboarding, and many other decisions that directly affect customers. Organisations now need to do more than build AI that works. They also need to show that it works reliably, safely, and consistently.

This is where software testing becomes much more important. Under the AI Act, testing is no longer only about finding defects before a release. QA teams help verify that AI systems behave as expected, produce consistent results, include appropriate human oversight, and can be monitored throughout their lifecycle. The evidence generated during testing also helps organisations demonstrate compliance with the regulation.

The legislation applies across the AI value chain, from companies developing AI solutions to organisations integrating or deploying them in their own products and services. For QA teams, understanding the EU AI Act is becoming just as important as understanding functional requirements. The sooner testing is considered, the easier it becomes to identify compliance risks before software reaches production.

Which AI systems are affected by the AI Act?

The EU AI Act does not apply the same rules to every AI system. Instead, it groups systems into four risk categories based on their potential impact on people's safety and fundamental rights. The higher the risk, the more testing, documentation, and oversight organisations are expected to provide.

For QA teams, understanding these categories is important because they determine how much evidence needs to be produced before an AI application can be deployed or placed on the EU market.

For many organisations, the biggest impact will come from high-risk AI systems. While most AI applications fall into the minimal-risk category, banks, insurers, healthcare providers, and public sector organisations are much more likely to develop or deploy systems that face stricter requirements. In those cases, software testing becomes an important part of demonstrating that the system is safe, reliable, and ready for production.

Why high-risk AI systems require a different testing strategy

Traditional software testing focuses on a simple question: does the application work as expected? For many high-risk AI systems, that is only part of the picture. Under the EU AI Act, organisations also need to demonstrate that their AI systems are reliable, transparent, secure, and can be monitored throughout their lifecycle.

This is especially important in regulated industries such as banking, where AI may be used for credit decisions, fraud detection, anti-money laundering checks, or customer onboarding. A system might pass every functional test and still create unacceptable business or compliance risks if it produces inconsistent results, cannot explain its outputs, or lacks proper oversight.

Here are some of the key areas QA teams should focus on.

Risk management

Testing should start with understanding where an AI system could fail and what the impact would be. Rather than treating every test case equally, teams should prioritise scenarios that could affect customers, financial decisions, or fundamental rights. A risk-based testing strategy helps ensure the most critical business processes receive the greatest attention.

Robustness

AI systems should continue to perform reliably even when they receive unexpected or incomplete inputs. Testing should include edge cases, unusual user behaviour, invalid data, and scenarios that may not appear during normal operation. The goal is to understand how the system behaves under pressure, not just when everything goes according to plan.

Accuracy

Unlike traditional software, AI models rarely produce the exact same output every time. Instead of checking for a single correct answer, QA teams need to validate whether results remain accurate, consistent, and within acceptable thresholds. Testing should also verify that updates to AI models do not unintentionally reduce performance in existing business processes.

Human oversight

The AI Act places a strong emphasis on human oversight, particularly for high-risk AI systems. If an AI system recommends rejecting a loan application or flags a payment as suspicious, authorised employees should be able to review that recommendation, understand the reasoning behind it where possible, and intervene when necessary. Testing should verify that these review and approval workflows function correctly.

Traceability

Good testing creates evidence. High-risk systems should generate logs that make it possible to understand what happened, when it happened, and which version of the system produced a particular outcome. Traceability helps organisations investigate incidents, support audits, and demonstrate compliance if requested by national competent authorities or a market surveillance authority.

Security

Because AI systems often process sensitive business or customer data, security testing remains essential. Alongside traditional vulnerability testing, organisations should consider how attackers might attempt to manipulate model inputs or influence AI behaviour. Strong cybersecurity helps protect both the application and the integrity of its decisions.

Documentation

Under the Artificial Intelligence Act, testing does not end with a successful test run. Organisations should document their testing strategy, the scenarios they executed, the results they obtained, and any limitations they identified. Well-maintained documentation makes it easier to demonstrate compliance, investigate issues, and support ongoing market surveillance activities throughout the software lifecycle.

For QA teams, the biggest change is that testing is no longer only about approving a release. It also becomes part of proving that an AI system is safe, reliable, and suitable for use in a regulated environment.

How to test AI systems for AI Act compliance

The EU AI Act doesn't tell organisations exactly how to test their AI systems, but it does make it clear what they need to demonstrate. For many high-risk AI systems, testing needs to go beyond verifying features and include evidence that the system is reliable, secure, and can be used responsibly. Here are the key areas QA teams should focus on.

Functional testing

Start with the basics. Verify that the AI application performs its intended function and integrates correctly with the rest of the software. Test complete business processes, not just the AI component. In banking, for example, that could mean validating the entire customer onboarding or payment approval workflow from start to finish.

Accuracy and reliability testing

AI systems should produce consistent results under normal operating conditions. Test with different datasets, edge cases, and incomplete information to understand how performance changes. If the underlying model is updated, regression testing is essential to ensure existing business processes still work as expected.

Bias and fairness testing

Some AI applications influence decisions that have a direct impact on people. For example, they may support credit scoring or fraud investigations. Testing should help identify unexpected inconsistencies and verify that similar situations are handled consistently. This isn't just a data science exercise. QA teams can help uncover issues by testing a wide range of realistic scenarios.

Human oversight testing

The AI Act requires many high-risk AI systems to include appropriate human oversight. Test whether users can review AI recommendations, override decisions when necessary, and continue working if the AI service is unavailable. Manual fallback procedures should be tested just like any other business process.

Security and robustness testing

AI applications should remain stable even when conditions are less than ideal. Test how the system responds to unexpected inputs, invalid data, integration failures, or heavy workloads. Security testing should also verify that the application protects sensitive data and cannot be easily manipulated.

Logging and audit trail validation

Testing should also confirm that the system records enough information to support audits and investigations. QA teams should verify that important events are logged, AI decisions can be traced back to the correct model version, and audit trails are complete. Good traceability not only helps with compliance but also makes it much easier to investigate production issues and improve future releases.

Testing generative AI systems under the EU AI Act

Not every general purpose AI model falls into the high-risk category, but that doesn't mean it can be deployed without testing. As more organisations integrate large language models into customer support, internal assistants, search tools, and business applications, QA teams need to validate how these systems behave in real-world scenarios.

The EU AI Act introduces specific requirements for general purpose AI models and general purpose AI systems, particularly around transparency and risk management. If a model is integrated into a high-risk AI system, the requirements become even stricter.

Verify transparency obligations

Users should know when they are interacting with AI. If an application generates text, summaries, or recommendations, QA teams should verify that the appropriate disclosures appear where required. Where applicable, AI-generated content should also be identifiable.

Test for hallucinations

Generative AI can produce answers that sound convincing but are factually incorrect. While hallucinations cannot always be eliminated, they should be tested using realistic prompts and business scenarios. In regulated industries like banking, an incorrect answer about a payment, loan, or compliance process could quickly become a business risk.

Include prompt injection testing

Prompt injection attacks attempt to manipulate a model into ignoring its original instructions or exposing information it shouldn't. QA teams should test how the application responds to malicious prompts, unexpected user inputs, and attempts to bypass built-in restrictions.

Validate guardrails

Most production AI applications include guardrails to limit what the model can generate or access. Testing should confirm that these controls work as intended, even when users try to push the model outside its expected behaviour. This includes validating restricted topics, content filters, permission checks, and escalation to a human when appropriate.

Regression test model updates

Unlike traditional software, AI models can change frequently. A newer model may improve one task while performing worse in another. Every model update should be treated like a software release, with regression testing covering the most important business processes and user journeys.

For many organisations, the biggest challenge is that generative AI is rarely used on its own. It is usually connected to internal systems, customer data, business rules, and external services. That makes end-to-end testing just as important as validating the model itself.

AI Act testing requirements throughout the software lifecycle

Testing for the EU AI Act doesn't begin a few weeks before a release, and it doesn't end once an application goes live. For many high-risk AI systems, compliance is an ongoing process that spans the entire software lifecycle. The earlier QA teams are involved, the easier it becomes to identify risks, collect evidence, and avoid costly changes later.

Requirements analysis

Before writing test cases, teams should understand how the AI application will be used and whether it falls under the AI Act. This includes identifying whether the solution qualifies as a high-risk AI system, what business processes it supports, and which regulatory requirements apply.

Risk assessment

Risk assessment should guide the testing strategy. Instead of treating every feature equally, focus on scenarios that could have the biggest business or customer impact. For example, an AI model used to prioritise support tickets requires a different level of testing than one supporting loan decisions or fraud detection.

Test design

Test cases should cover more than expected user behaviour. Include edge cases, unusual inputs, failed integrations, and scenarios where the AI produces unexpected results. Where possible, use realistic business data to reflect how the system will perform in production.

Validation

Validation combines functional testing with broader quality checks. Depending on the application, this may include accuracy, reliability, security, human oversight, and end-to-end business process testing. The goal is to confirm that the system performs consistently before it reaches production.

Deployment

Deployment should include more than a successful release. Organisations should verify that logging, monitoring, permissions, and fallback procedures are working correctly in the production environment. This is also a good time to establish a baseline for future regression testing.

Monitoring

AI systems continue to evolve after deployment. Model updates, changing business data, and new user behaviour can all affect performance over time. Continuous monitoring helps teams identify unexpected changes early and decide when additional testing is needed.

Handling serious incidents

For high-risk AI systems, organisations should have a clear process for identifying, investigating, and documenting serious incidents. QA teams can support this by ensuring logs are complete, test evidence is available, and issues can be reproduced quickly. Strong traceability makes it much easier to understand what happened and prevent similar problems in the future.

Continuous compliance

Compliance isn't something you achieve once. As applications, AI models, and regulations evolve, testing should evolve too. Regular regression testing, periodic reviews, and ongoing documentation help organisations maintain compliance while continuing to deliver new features. For QA teams, that means treating compliance as part of everyday software quality, not as a separate project before an audit.

What QA teams should document for AI Act compliance

Good testing is only part of AI compliance. Organisations also need to be able to demonstrate what they tested, why they tested it, and what the results showed. For many high-risk AI systems, documentation is just as important as the testing itself.

While the AI Act doesn't prescribe a single documentation template, it does expect organisations to maintain evidence that supports compliance throughout the software lifecycle. That evidence can also prove invaluable if national authorities or other public authorities request information as part of market surveillance activities.

Test results

Keep a clear record of the tests that were executed, their outcomes, and any defects that were identified. Test results should show that the system has been validated under realistic conditions and that identified issues have been resolved before release.

Dataset validation

Document how datasets were selected, reviewed, and validated before testing. For AI systems intended to support business-critical decisions, the quality of training and validation data can have a direct impact on system performance. Recording this process makes it easier to demonstrate that the application was tested using representative data.

Risk assessments

Maintain documentation showing how risks were identified, prioritised, and mitigated. Risk assessments should explain why certain scenarios received additional testing and how the organisation addressed potential impacts on customers, business operations, or fundamental rights.

Human oversight procedures

If the application requires human oversight, document how these processes work in practice. This should include approval workflows, manual review steps, escalation procedures, and fallback processes if the AI system becomes unavailable or produces an unexpected result.

Incident reports

Not every issue is a compliance issue, but organisations should have a process for recording and investigating significant failures. Well-documented incident reports make it easier to analyse root causes, improve future releases, and respond if regulators request additional information.

Traceability and audit logs

One of the most valuable forms of evidence is a complete audit trail. QA teams should verify that decisions can be traced back to the correct software version, AI models, datasets, and test execution. Comprehensive audit logs also help organisations demonstrate that appropriate controls remain in place after deployment.

As AI regulation continues to evolve under EU law, strong documentation will become an increasingly important part of building trustworthy AI. It not only supports compliance with the Artificial Intelligence Act, but also gives development teams, auditors, and business stakeholders greater confidence in the systems they deploy.

Common AI testing mistakes that can create compliance risks

Many organisations already have mature software testing processes, but AI introduces new risks that traditional QA doesn't always cover. The EU AI Act doesn't expect perfection, but it does expect organisations to identify risks, test appropriately, and maintain evidence that supports compliance. These are some of the most common mistakes to avoid.

Focusing only on model accuracy

A model can produce accurate results and still fail to meet the expectations of the AI Act. Testing should also cover reliability, security, traceability, human oversight, and how the system behaves in real business processes. Looking only at accuracy leaves important compliance gaps.

Ignoring edge cases

AI applications don't always fail under normal conditions. Unexpected inputs, incomplete data, or unusual user behaviour often reveal problems that standard test cases miss. Include edge cases and negative scenarios to understand how the system performs outside ideal conditions.

Overlooking transparency requirements

For certain AI systems, users must be informed when they are interacting with AI or consuming AI generated content. QA teams should verify that these transparency obligations are implemented consistently across different user journeys and continue working after software updates.

Skipping regression testing after model updates

Unlike traditional applications, AI models can change frequently. Even a small model update can affect business-critical workflows or produce different results for the same input. Every update should be treated like a software release, with regression testing covering the most important business processes.

Treating documentation as an afterthought

Without clear documentation, it becomes much harder to demonstrate compliance. Test results, risk assessments, dataset validation, and audit logs should be maintained throughout the software lifecycle, not recreated before an audit.

Forgetting production monitoring

Testing shouldn't stop after deployment. Organisations should monitor how AI systems perform in production, investigate unexpected behaviour, and review issues as they arise. Continuous monitoring helps identify problems early and supports ongoing compliance as applications and AI models evolve.

The organisations that adapt most successfully to the Artificial Intelligence Act won't necessarily be the ones using the newest AI technology. They'll be the ones with testing processes that continue to identify risks, validate changes, and provide clear evidence that their AI systems remain reliable over time.

Best practices for testing high-risk AI systems

Testing high-risk AI systems isn't about adding dozens of new test cases. It's about making sure your existing QA process covers the additional risks that come with AI. A few practical changes can make a big difference.

Start with risk classification

Before planning your test strategy, understand how the application is classified under the EU AI Act. If the AI supports a business-critical process such as fraud detection, credit scoring, or customer onboarding, expect a much higher level of scrutiny than you would for a simple chatbot or recommendation engine.

Combine functional and non-functional testing

Functional testing tells you whether the application works. Non-functional testing tells you whether it's ready for production. Alongside business logic, test performance, security, resilience, logging, permissions, and failover behaviour. AI doesn't replace these testing activities. It makes them even more important.

Test complete business processes

Don't validate the model in isolation. Test how it behaves as part of an end-to-end workflow.

For example, if an AI model flags a suspicious payment:

• Is the alert created correctly?
• Does it reach the right analyst?
• Can the analyst override the recommendation?
• Is every action logged?
• What happens if the AI service is unavailable?

Those are the scenarios that determine whether the application works in production.

Automate regression testing

AI models change frequently. Prompts change. Business rules change. APIs change. Every change introduces risk.

Automate regression tests for your most important business processes so you can quickly identify whether a new model version or software release has affected existing functionality.

Monitor production behaviour

Some issues only appear with real users and real data. Monitor key metrics after deployment, review failed transactions, investigate unexpected outputs, and compare production behaviour with previous releases. Monitoring should be part of your QA process, not something that only happens after an incident.

Keep testing evidence alongside your releases

Documentation shouldn't be a separate compliance exercise. Store test results, execution logs, screenshots, risk assessments, and approval records together with each release. If you need to explain why a feature was considered production-ready six months later, the evidence should already be available.

For most teams, these recommendations don't require a new testing methodology. They build on practices that already exist: end-to-end testing, regression testing, risk-based testing, and good release management. The difference is that, under the AI Act, those practices also help demonstrate that high-risk AI systems have been tested in a structured and repeatable way.

Frequently asked questions

Does the AI Act apply to every AI system?

No. The AI Act does not regulate every artificial intelligence application in the same way. Instead, it classifies AI systems based on risk. Most applications fall into the minimal-risk category, while certain AI systems are classified as high risk because they can affect fundamental rights, safety, or access to essential services.

The goal of the AI regulation is to regulate AI systems based on their potential impact while continuing to support innovation across the EU market.

What are prohibited AI practices under the AI Act?

The AI Act identifies several prohibited AI practices that create an unacceptable risk to people. These include harmful AI based exploitation of vulnerable individuals, certain uses of real time remote biometric identification and other remote biometric identification systems in publicly accessible spaces for law enforcement purposes, some emotion recognition systems, and untargeted collection of internet or CCTV material to build facial recognition databases.

These restrictions are designed to protect fundamental rights while limiting AI uses that could threaten national security or democratic processes.

How are general purpose AI models regulated?

The Artificial Intelligence Act introduces specific rules for general purpose AI models and general purpose AI systems. Most providers must meet transparency and documentation requirements, while models that pose systemic risks are subject to additional obligations.

The European AI Office oversees implementation at the EU level and is responsible for developing guidance, including the General Purpose AI Code. The AI Office's powers are supported by the European Artificial Intelligence Board, while national authorities are responsible for enforcement within individual Member States.

Who is responsible for AI Act compliance?

Compliance is a shared responsibility. AI providers, software developers, business teams, and QA all play a role in building trustworthy AI. The European Parliament introduced the AI law, while enforcement involves the European AI Office, the European Data Protection Supervisor, national authorities, and other public authorities.

For organisations, the most practical approach is to build compliance into the software lifecycle by combining good testing, documentation, human oversight, and regular monitoring from the start.

Why software testing will become central to AI governance

The AI Act raises the expectations for organisations using AI, especially when it's part of business-critical processes. For QA teams, that means testing is no longer just about finding defects. It's also about showing that systems have been validated, risks have been considered, and the right controls are in place.

A risk-based approach makes that much more manageable. Instead of trying to test everything, teams can focus on the business processes where failures would have the biggest impact.

Want to see what that looks like in practice? Try the TestResults QA Risk Agent and discover which workflows carry the highest risk in your application landscape. In just a few minutes, you'll get risk-based end-to-end test cases that help you focus your testing where it matters most.