Agile Testing In A Controlled Environment – Case Study

Waterfall strikes back

Software developed in controlled/regulated environments requires a division between the development and testing team. The Verification and Validation teams are required to execute test cases against the specification and re-test bug-fixes only after the Software is officially released to them. That poses a big problem in terms of quality and SW delivery speed. You might imagine that following this waterfall approach can cause serious delays in the time to market as well as reintroduce the not so famous Tester-as-Quality gate approach.

How to make sure that you can increase the quality and have faster delivery even if you are bound to the regulated market restrictions? There are a couple of things you can do but the example below is based on an actual project I was in. The approach I took made a significant difference to the quality of the SW and allowed the product to be launched to market even though a lot of people expected that the project will die. Let me give you some background information first.

Quasi-Agile approach

Following the agile principles the project development (and by development I mean pure coding) was done in bi-weekly release cycles. Naturally, from the perspective of coding that was a great idea, but from a testing perspective it was a nightmare. Imagine that each two weeks at the day of the release (or shortly before) the specification has been updated. The V&V group had 2 weeks to write the test case, execute them, retest the bug-fixes and report the results back. Not to mention that a small change in a specs could affect several test cases. Since it was a regulated market and the tractability was quite obvious SSS-> TC1, TC2,….TC12, (Specs to test cases) you might imagine that with one single code change/specs change, multiple test cases had to be updated and executed. I assume you already know how this approach ended up. V&V was significantly behind in their testing activities and not keeping up-to date with the releases. Obviously, it was not their fault as they had to follow the strict process required by the regulatory bodies.

So what was the solution the problem of the testing team – more testing ?

Instead of putting more people to the verification team and following each projects managers favorite quote:

Nine women can deliver one baby in one month

new testers have been placed in the development teams. Sounds obvious doesn’t it? However, those testers could not do any formal verification as the official builds where not available until the release cycle was finished. So what did they do?

Pre-test the fixes before they are checked-in to the source code

You would probably say that this should be the task of the developer and you are probably right, but keep in mind that with an extreme pace there are cases where the developers simply don’t think about the possible effects of the fix. How did that work in practice is that once the software engineer developed a fix he either branched or shelved it. The module tester would then see the update on the Kanban board and immediately approach the engineer. The initial discussion usually ended up with addressing the following points:

• Developer: What has changed in comparison to the previous behavior
• Developer: What are the affected functionalities
• Tester: Did you think about….
  (here would be the list of the functionalities that the testers challenges)

As a result the developer either had to do some refactoring of the fix or the fix was ready for Module Testing. The Tester afterwards checked the fix against the newest source code on the target system (the test was not performed in isolation). The biggest advantage of that approach was that there way less bug reopens afterwards which naturally ended up with a better SW quality.

Help in the re-test steps

The Module Testers job didn’t end up there. As I have mentioned, the gap between the release and official verification start was at least two weeks. Given that, once the V&V team started to re-test the bug-fixes the development team was already far into the next sprint. Bummer! Extracting the information regarding the fix itself and possible scenarios to test was a challenge mainly because of the time which went by from the moment the fix was checked-in. Solution? Our Module Tester again. After the successful pre-test of the fix the module tester also added all of the scenarios which she/he has used for testing in the bug details. That gave the V&V group a head-up start and they could already be prepared with the test scenarios at the moment the official build became available. They still had to officially re-test but at least they knew exactly what has changed and what needs to be covered.

Regression Test Cases identification

How the process usually worked was that the official build was delivered, the V&V team looked at the release notes and determined what shall the Regression Test Cases for that build be. Again, that wasn’t really an effective approach – just imagine looking at 50 fixes and trying to figure out which of the 500 TCs in your repository are affected. How did we turn it around? The Module Tester had a great understanding of the whole software and therefore knew exactly what functionality is affected as well as which specification will be adapted. Therefore, he also provided the information about the possibly affected Test Cases that might need to be executed as a part of the Regression Tests. It was not a 100% bullet-proof approach, but it definitely reduced the time for the V&V team to make an assessment of what tests need to be executed.

Additionally if time allowed, the Module Tester did a dry-run of the test cases just to make sure that they will not fail during the official testing.

Conclusion

There is no way to circumvent the legal regulations in a controlled environment. You need to follow the process and that is it. You cannot bend the process but what you can do is make it less painful. The approach we have taken in that project saved us a lot of time and effort during the official verification runs and at the same time increased the quality of the delivered software. Keep in mind that the role of the Module Tester cannot be performed by a random person from the street. It needs to be someone skilled in testing who is not afraid to ask difficult questions and challenge the developers. At the same time this person needs to communicate with the V&V group to ensure that they have all of the information at hand once they start their part of the process.

My advice is, take your best test engineer from the V&V team and make him or her a module tester. He/she knows V&V and knows all the details of the software.

All of this works even better if the module tester just needs to click a button to run all his test automatically against the newest, just-built software version. Start today with test automation.