Infinite Test Lab

Looking back at the first successes of test automation:

• RUNTIME
  Although your test automation is running and everyone was exited in the beginning, only few days have passed, and it seems no one is using the test-automation any more. Tests fails more and more or even no longer take place. What is worse, no one seems to care? What went so terrible wrong?

Your first success speaks for itself; you have reduced the test runtime form 14 working days executing test cases manually to being able to execute the same test cases with automation within only 61/2 hours.

• DEVOPS
  What is more, you even embedded the test automation within the DevOps pipeline. Continuous Integration (CI) is our keyword here.

The development team has up to this point only had unit tests, they are almost always green, but your testers as well as the end users still find errors in the software. A classic scenario. What`s new is the unit tests will be executed, followed by your automated end-to-end tests, integration tests, etc.

Everyone is excited because the errors have been found before the release. Quality has improved.

After the first excitement has passed it is back to daily business. You encounter the following: your tests have disappeared from the CI, or they are only being run once a week, and when they are being run, by this point no one is interested in the results, too many tests have errors or no longer run.

What happened? Your results are available too late! Even for a nightly build, at 6½ hours the runtime is too long.

• NEXT GOAL: To be accepted we need to bring the runtime of all the tests to a total of below 10 minutes.

A difficult task, but not completely impossible. All we need to do is run all the tests in parallel.

To start, there will be a few tests that are automated and that have a longer runtime. Respectively. we outsource those and run them only once per night, all the others can be run for each developer check-in.

To run all tests simultaneously, we need disproportionately more test systems, which are currently not available to us. We have 2 options: Extremely overdraw our project budget to purchase hardware that is no longer needed after project completion, or we switch our tests to dynamic execution.

Dynamic execution means that for the time of execution, we automatically start all the required systems with a cloud provider, execute the tests in a distributed manner, and then shut the systems down again.

As cloud providers fundamentally assume that the systems which they activate should perform a permanent service, we fall through the price cracks with our model of dynamic execution and receive nearly unlimited resources for an extremely small amount of money.

Let’s take Microsoft’s Azure Cloud, for example. Here we receive a so-called B4MS instance for $0.166/hour. This means that we have four virtual cores. Even more, the advantage of the MS variant is that we get 150% of the paid computing power for free for the first 30 minutes, i.e. we pay for the performance of 4 cores but receive 6. Therefore, if we assume that our tests run for about 10 minutes, then we pay $0.027 per test, so for 100 tests we pay $2.76 per test run. Instead of 6½h, however we have thus completed the entire test run in 10 minutes!

This is an investment of resources that is worth carrying out for every project, there is no requirement to have our own test infrastructure. i.e., this eliminates all of the internal IT costs. We pay about $55/month if we run all the tests every day.

This means that for $55/month you can run 100 tests a day simultaneously, with results after 10 minutes. Compare this to your current runtime and your current costs (including internal IT costs). Working past the actual cloud pricing model makes it possible.

The same is true of Amazon’s AWS or Google’s cloud.

This approach works only with some sort of repository. The repository is for your test cases. At any time, this is a central repository that allows the latest automations to be referred to during a test run.

It would be optimal if this repository also provided a direct overview of the available tests, the tests already automated, the successful tests, etc.

With this in mind, we would then have a central management dashboard for quality that reflects the current state of software quality at any point in time, and this not based on static analyses or unit tests, but on complete end-to-end tests which you have developed yourself.

Although the central control and administration is an important cornerstone. It is out of question, that the repository must still guarantee that all testers can work with it simultaneously. Preferably without additional tools and without directly requiring the application.

In the ideal case, each tester can create new test cases graphically via a web page based on the application. Directly stored in the repository are the test cases. And executed the next time the developer checks in. Every time.

In relation to test automation, it’s important to understand that we don’t automate for the sake of automating, we automate to increase quality. However, if no one is running the automation or paying attention to the results. Moreover, we have not done our job and quality levels will not be increased by our efforts.

It is therefore important to pay attention to the following during test automation :

I’ve given you some ideas to help you take your automation to the next level. I would like to share my ideas with you and learn from your experience. Contact me directly today by clicking on the button.