If you're leading test automation and dealing with IBM AS/400 (now IBM i), you know it’s not built for traditional test automation (ID or UI-based). The system is old but still critical for many businesses, and automating it requires more than simply deploying a conventional test automation tool and initiating test execution.
ID-based automation struggles with AS/400’s terminal-based UI, unpredictable elements, and session IDs that don’t play nice with standard test automation frameworks. So, what’s the real solution?
This article breaks down why AS/400 automation is a challenge, why simply adding more scripts or engineers won’t fix it, and why a solid QA team is still worth every penny.
Why ID-based test automation struggles with AS/400
IBM AS/400, now known as IBM i, was never intended to support modern automation frameworks. Unlike today’s applications that rely on stable UI components and object IDs, AS/400 operates in a text-based terminal environment. This architecture makes it difficult for most test automation tools to function correctly, especially those that depend on identifying screen elements or maintaining consistent session states.
Traditional automation tools face limitations with AS/400 because they aren’t designed to handle dynamic screen content, inconsistent session IDs, and the lack of a true graphical interface. As a result, teams often struggle with reliability, script maintenance, and false positives during regression testing.
IBM does offer its own tool, IBM Rational Functional Tester (RFT), which is designed to integrate into the IBM ecosystem. However, RFT often requires heavy customization, significant infrastructure, and ongoing maintenance. Its cost and complexity make it a difficult choice for many teams trying to automate effectively.
Let’s take a closer look at what makes AS/400 automation so challenging and why a different approach is needed.
Session IDs and unpredictable elements
One of the biggest challenges in IBM i testing is dealing with session IDs and dynamic elements. Every time a user logs in, the system assigns new session identifiers, making it difficult for automation tools to track elements consistently. Most test automation frameworks need stable identifiers to function, but in the IBM i environment, these IDs frequently change.
Beyond tracking issues, this instability also affects regression and functional testing, as test automation tools struggle to match expected results with system behavior. Without a proper method to handle dynamic elements, testing teams often waste resources on maintaining scripts instead of improving test quality.
Structural limitations
Even when session identifiers remain stable, IBM i applications do not work like modern software. There are no standard buttons, dropdowns, or interactive elements—just blocks of text and command-line inputs. This makes automation unreliable and increases the risk of data errors.
Since IBM iSeries systems store sensitive business data in integrated file systems and databases, test accuracy isn’t just about functionality—it also affects security. If test automation tools can’t reliably evaluate test cases, teams risk overlooking errors that may impact both system performance and security, leading to gaps in validation coverage.
OCR-limitations
Because ID-based test automation tools do not work well with IBM i applications, teams often use screen coordinates or OCR (optical character recognition) to interact with elements. However, these methods create more issues:
- Small UI changes can break test cases, leading to unreliable results.
- OCR is prone to misreading text, reducing efficiency and performance.
- Job scheduling, system administration, and user management become difficult to automate, increasing maintenance costs.
The need for the right expertise
IBM i testing requires more than just automation. Testers need technical skills in RPG programming, CL programming, and database management. Understanding system administration, industry standards, and security protocols is necessary for maintaining high-quality updates.
Test teams must also evaluate testing work, test data, and system parameters to prevent data integrity issues. Without proper skill development, automation will not provide reliable results or seamless integration.
Finding a better approach
Rather than relying on fragile fixes like screen coordinates or OCR, testing teams need tools designed specifically for IBM i applications.
The right test automation approach should:
- Support system security, access control, and privacy regulations
- Reduce costs and errors while improving testing efficiency
- Help testers and developers focus on reliability and business knowledge rather than fixing broken scripts
Testing IBM i applications requires a structured method that respects the system’s limitations while maintaining industry standards and delivering high-quality results. With the right tooling, teams can improve performance without increasing headcount or maintenance overhead.
Why adding more engineers won’t solve the problem
Throwing more engineers at a problem might seem like the fastest way to fix it, but when it comes to software quality, it rarely works. More developers mean more code, more complexity, and more potential issues, without necessarily improving stability.
Instead of focusing on headcount, companies need to improve how they test. Without a structured approach to quality, new hires just add to the chaos.
A smarter way to automate AS/400
Automating IBM AS/400 (IBM i) is rarely straightforward. Its green-screen interfaces, cursor-based navigation, and legacy system design make traditional test automation tools unreliable. However, there are three main approaches teams often explore:
1. IBM Rational Functional Tester (RFT)
IBM Rational Functional Tester (RFT) is IBM’s own enterprise-grade test automation tool. It’s often the default choice for companies already operating within the IBM ecosystem.
RFT offers deep integration with IBM environments and supports a wide range of applications.
However, using RFT for AS/400 automation usually requires significant customization. The tool wasn’t originally designed for terminal-based interfaces, so it often struggles with dynamic screens, cursor positioning, and session variability.
Test scripts built with RFT can be fragile and require constant updates, making long-term maintenance costly and time-consuming for QA teams.
2. Java Toolbox (JTOpen)
Java Toolbox (JTOpen) offers a completely different strategy. Instead of trying to automate the UI, it bypasses the interface entirely by working through the system’s APIs. This open-source Java library gives testers direct programmatic access to database files, program calls, system values, and more.
As a result, it enables stable and fast test execution without dealing with the quirks of green-screen navigation. However, the API-based nature of this approach comes with trade-offs. It cannot validate the actual user interface or catch issues that impact the end-user experience.
Some AS/400 applications, especially those with compiled RPG or COBOL code, don’t expose all their business logic via accessible APIs, which can limit test coverage. Implementation can also be complex, with strict security requirements and setup overhead that may slow teams down.
3. TestResults
TestResults takes a different route. It bridges the gap by offering robust UI-driven automation specifically designed for AS/400 systems, without relying on brittle screen-scraping techniques.
Here’s what TestResults brings to the table:
- Accurate green-screen navigation – Handles cursor movement, input fields, and screen transitions without breaking
- Reliable object recognition – Dynamically detects elements, so tests don’t fail when text shifts or layouts change
- End-to-end automation – Connects AS/400 testing with web, desktop, and API validation for full process coverage
- No-code test creation – Set up and maintain tests without scripting, so more teams can contribute
- Resistant to UI changes – Works even when menu structures or screen layouts are updated
- CI/CD friendly – Easily integrates into DevOps pipelines to catch issues early
With no-code test creation and full CI/CD compatibility, TestResults empowers both technical and non-technical users to build and maintain stable automation quickly, without sacrificing visibility into the UI.
In short, while RFT and Java Toolbox provide partial solutions with either high maintenance or limited coverage, TestResults offers a modern, resilient alternative purpose-built for the challenges of AS/400 automation.
Rethinking automation for legacy systems
IBM AS/400 wasn’t built with test automation in mind, and forcing traditional testing tools onto it leads to more problems than solutions. Session IDs change, UI elements don’t behave like web applications, and workarounds like screen coordinates or OCR break too easily. More engineers or more scripts won’t fix the core issue. What’s needed is a smarter approach.
For many companies, legacy systems like IBM i remain mission-critical even as digital transformation efforts move forward. That’s why automation strategies must support both stability in legacy environments and flexibility for modernization, without forcing a complete overhaul all at once.
TestResults makes AS/400 automation stable, reliable, and easy to manage. No fragile workarounds, no constant maintenance. Just automation that works.
See it in action. Let’s talk.
