Traceability in Testing

Imagine your manager wants you to test this critical fix, that you and your team have just elaborated in a new version, on the old version. And not as you have some time left, but right now!  He wants to see the differences comparing the original results with the new.

How long does it take you to:

1. find the original results of the old version
2. setup the old test environments
3. install the fixed version
4. run the tests
5. and compare them to the original results? If it is more than 7 minutes, read on.

This post will explain:

If you do not know what traceability means, no problem.
Imagine that you can have a reference from every single executed step in a test case to the corresponding test plan, the corresponding software version, the corresponding test environment and how to set this environment up.

In general, the idea is that the available documentation is generated in a way that a new team could take over your project and could reproduce all your results.

Reproducing the results. This idea brings us to the main benefit of traceability. If your test automation system allows you to re-run tests the same way, they were originally executed you can easily do regression testing on old software versions.

It is a simple scenario: Version 1 and 2 of your software are in the field. R&D and testing is working extensively to get version 3 finished. As version 2 is the one that is current you still have everything ready to run regression tests on it. Even if it is for comparison only.
But here we go, a critical bug was found by an important customer who is still on version 1. If you have full traceability in your automation system you are up and running with your tests in a matter of minutes.

(old test plans & old environment) + (fixed version 1 & new test case for arisen bug) = Regression Testing

Select the old test plans, select the old environment, upload the fixed version 1 release, add a new testcase for the bug and re-run all tests as regression. Within less than one day you have a fully tested critical release. A happy manager, a happy customer and perhaps a cool gratification.

Having results, environments, Software (SW) versions, testcases & executions available for re-run saves time. Tracking everything for reproducibility can save you a lot of time. But what if the above scenario never happens. Perfect for you, no unexpected work! The additional tracing does not add any overhead to your testing as it is done completely transparent in the background.

The system should automatically save (with revisions):

If you have multiple system to track all this information today, think about how long it would take you to run a regression test on the oldest but still supported software version and create a set of documentation for your testing activities. Compare this to the 7 minutes it takes you in TestResults.io.

There is one more benefit. If you have full traceability you are always ready for audits. If you are working in any kind of regulated environment e.g., banks, medical, clinical, pharmaceutical, aerospace, you know how troublesome it can be to create all the cover documentation for your tests and your automated tests.

A complete test automation solution provides you all the required documentation together with your test results, for any version in any environment at any point in time.