It might sound convenient to run your automated software tests on your machine, and it's how most people do it today. 

So why should this be a problem? Why should you even care?

The computer you work with today is a different computer tomorrow. In the background, your Windows or Unix machine is updating again and again: 

• Operating system libraries get exchanged
  ‍
• Runtime files get exchanged
  ‍
• Java virtual machines get exchanged
  ‍
• A new .NET version gets installed 

‍

This potentially influences the application under test. And that's a problem in software testing.

As it may interfere with the software under test, you need to have it under control with the so-called configuration management. This means you always know with which configuration you tested your software.

If you're not in control of your system, your test can't be reproducible.

You could even become the biggest problem in configurations

Let's assume the software you'd like to test doesn't start on your computer. You ask a developer what dependencies there are, and the developer says you need to install the runtime library XYZ. So you do that. The software to be tested now starts on your PC, and you can test without any findings. 

Sounds familiar?

But as the first customer installs the software and doesn't have this library either, he runs into the same problem: the software doesn't start.

You destroyed your configuration by installing the library, as you no longer have the system under configuration control.

Besides that, you have individual configurations on your machine. This may influence your tests as well.

As mentioned above, your tests can't be reproducible if you do not control your system. Your tests have to be reproducible. If they're not, you're left to chance.

And keep in mind that even the test software itself is already an interference that could have an interdependency with the software under test. 

Best practice of software test execution

There's an easy way to solve this: use a system where you have control over the configurations, like a virtual machine (VM). A VM is an image of the computer, which you can always reset and control the updates. It's not the idea to not update the system. It's about controlling it and updating all systems used for testing simultaneously.

Configuration control means there are no unknown changes in the system.

Benefits of NOT testing on your computer:

• After testing, you can archive the VM precisely as it was and reactivate it when later a customer reports a bug to reproduce it. It's not about finger-pointing why you haven't found the bug initially. It's about learning what process needs to be adapted to prevent this kind of bug from getting released. This way, you can reproduce if it is tested. If you found the bug but classified it as a minor defect, or you reported it, and no one cared, or it's not reproducible.
  ‍
• You don't block your computer during the execution, which could take a while. You're more efficient and flexible as you don't need to wait until night to run your tests. 
  ‍
• Providing VMs in AWS or Azure with their self-service is much faster than getting test systems from the IT department. 
  ‍
• Separation of test design and test execution, as this way, they are on a different system. If you need to install new software to test, there is no impact on the testing system. In other words, you're in control of every change.
  ‍
• Tests are reproducible anytime
  ‍
• You're in control of the configurations and updates
  ‍
• Unlimited scaling of your test execution, as you can run your tests simultaneously on multiple VMs

If you sign up for a free Azure account, you get a VM free of charge for 12 months. Create Your Azure Free Account Today | Microsoft Azure

‍

‍

‍

👉 With TestResults.io you get execution VMs out of the box. If you don't have an account yet, get a free membership now