Introducing automated testing is a topic that is mostly underestimated and not planned accordingly. In any successful development project, the development team, led by the software architect, evaluates the requirements and decides on the technologies that will be used to develop the product.
Limitations of the selected technologies are balanced against the simplicity of usage. All that contributes to the quality of the final product. And in all that, the testing is generally not being considered.
Developers are not testers and don’t like to be treated as such. What one can expect from a developer is that they’ve heard about TDD (Test Driven Development) and already gathered some experiences in that department. That’s where the expectations end and wishful thinking starts. There are some testing specialized dev’s but in general, it stops at writing unit tests or integration tests.
DevOps, the (new) way to develop and test software
At some point, the great minds of the century realized that such a scenario ends up in complications and additional costs and came up with a new job DevOps. It basically combines development and IT operations and is responsible for providing the tools required for seamless testing integration in the development process.
This post will focus on the tasks that are mostly covered by a DevOps engineer. If you’re a dev that was tasked with setting up an environment, then this is for you, my friend. If you’re a DevOps engineer that’s getting into the topic of unit testing and is being bombarded by requests, then hopefully also you will get something out of reading through that article.
The scope will focus on the infrastructure needed for unit testing and the decisions that must be made.
Our focus will be on these 4 topics:
- Technology choices
- Continuous Integration (CI), Continuous Testing (CT)
- Unit tests scope and execution
- Code quality
How to choose the right Technology?
To even start talking about the technologies we first need to understand the difference between the development stages, namely what’s commonly being called the greenfield and the brownfield projects.
Usually, it’s considered to be more advantageous to work on a greenfield project. Yet from our perspective, there are also advantages when picking the technologies for the brownfield projects.
Let’s look at the positive things when comparing those types of projects from a developer’s perspective.
Greenfield / brownfield comparison
What’s true for development, in general, is also true for us when focusing on unit testing and creating the environment for it. The technology question is also the point that has the biggest impact on our decision-making.
In the case of a greenfield project, a decision needs to be made on which environment to use for:
- continuous integration
- continuous deployment
- and monitoring
There is a multitude of tools available that cover that.
Unit test execution is covered in the “Test” part of the DevOps and that’s what we’ll be focusing on. Continuous integration is the definition and automatic execution of steps that must be performed to have a running software version with the latest changes integrated.
We’ll go with two different approaches, for the green field we’ll be selecting TeamCity as the example reference and for the brownfield, we’ll be taking the Azure DevOps Services, that are fully integrated into the Microsoft environment. The reason is quite simple, if you look at the strongest selling point of TeamCity, it’s highly customizable and easy to set up. The biggest advantage of Azure DevOps is that it’s fully integrated into the Microsoft landscape.
3 Steps to choose your CI for Greenfield Projects
Know your enemy! When picking technology for a greenfield project, knowing the limitations and capabilities is the most crucial thing. Not doing a complete analysis of the restrictions can result in very high costs to circumvent those restrictions down the road.
Most of this chapter’s content applies to any CI but I’ll take TeamCity as an example for some of the bullet points so it’s less theoretical.
Step 1: Getting clear on the technology’s limitations
Let’s look at some of the limitations that TeamCity imposes on a Unit Testing and Code Coverage CI. First, it supports only certain runners, namely: nUnit, MSTest, xUnit, MSPec, and Gallio.
Strictly speaking, Gallio and MSPec are not directly unit test runners but the reason I’ve wanted to mention them is that they might provide additional utilities or limitations. For example, Gallio in its turn supports additional testing frameworks like csUnit or MbUnit and provides on-the-fly test results back to TeamCity, which expands the limitations of the TeamCity itself.
Always keep an eye on the restrictions, those might change over time. With the introduction of .NET Core and .NET Standard a couple of years back, plus the new .NET 5 and soon to be .NET 6 (with longtime support) it’s always important to check whether the applications support the .NET version you plan on using.
TeamCity free version CI setup
Step 2: Keep an eye on the costs. This is always a limiting factor
The picture above is not there just because it’s pretty, it’s to showcase the cost limitation of using TeamCity. You get 3 build agents for free, after that there is a rather steep cost based on the number of agents. Part of that very limitation is also the number of the build configuration that one can set up. It’s limited to 100, which is rather generous considering that only 3 build agents are available.
That’s more than enough for fast builds, you won’t run into the bottleneck if your build only takes 10 minutes. Once it starts ranging on an hour, having only 3 build agents will become a bottleneck. The current price for the on-premises additional license is around 350 USD.
Step 3: Don’t Forget the Maintenance
Unlike the common assumption, the CI environment also requires maintenance. TeamCity offers on-premises and enterprise licenses. The difference is that in the first case you need to do everything yourself and are responsible for the updates and so on. In the second case, it’s done for you, but the price is considerably higher.
As I’m not promoting the product here and just trying to set you into a specific critical and thinking mindset, I won’t tell you which one to pick. That’s based on your specific setup. The only reason we’re talking about it is that we need a CI environment for our unit tests to be effective and useful. It could have been any other CI framework like Jenkins or even the Azure DevOps but I wanted to have Azure DevOps for the brownfield setup.
What’s important to understand is that in the case of greenfield projects, analysis and evaluation of the strengths and weaknesses are extremely important. If done properly it will save you a lot of time, money, and most importantly nerve cells.
Here are the questions that I deem important when selecting a CI environment for a greenfield project:
- Will my CI have to support only my project or also other projects in the company?
- How much will my CI environment cost?
- Where will it be hosted and what are the security standards that must be met to host it?
- Who will have to maintain the CI environment and if it’s you, how much effort will it be to maintain it? That one goes hand in hand with the previous one.
- What are the technological limitations of the CI environment? (Supported test frameworks, configuration management, and so on)
- How hard would it be to exchange the CI environment? Sometimes requirements change and there is no way to build around those new requirements in the old CI. That’s not something that happens in every project but the longer your project takes, the higher the chance that it might be needed. Being prepared on what steps will have to be taken will yet again save you money.
- I know TeamCity is not the best example for that, as it’s very easy to set up but in general, don’t be afraid of choosing a CI tool that’s rather complicated to set up. Maintenance is the more important factor. You only must set it up once, but you’ll have to maintain it as long as your project is ongoing.
Once you’re down to 2-3 choices, do a spike, like a day each, and try to set up a very basic CI environment. That will give you a feel for the CI environment.
How to choose a CI for Brownfield Projects
Everything that was said about the greenfield applies to the brownfield projects as well.
Here are some assumptions for our setup:
- It’s a .NET Core (or .NET 6) solution.
- The company (organization) is set up using the Azure AD
- The project uses GitHub repository to fetch the required data.
- There is a certain number of unit tests available.
- The build and running the tests takes less than 10 minutes
- The team uses TFS for planning and progress tracking.
- The team uses Teams for communication
Simply put, this is a solution built fully using Microsoft products for all layers of development. In such a scenario it only makes sense to use the azure pipelines for CI and reporting. There are a couple of prerequisites to be able to fully use the azure pipelines, like having the organization set up in Azure AD but we just assume that was already done.
Instead of reinventing the wheel on how to set up the azure pipelines by copying the already available information, let me point you in the direction of where to find all the good information.
In part two you learn about the tests scope and test execution, how to set it up, and code quality.
Besides unit testing, it’s always good to have End-to-End testing up and running automated. Not sure which tool to use? TestResults.io is always a good choice 😉 The only test automation tool that developers and testers ❤ love and QA Engineers, Test Analysts, Test Automation Engineers, Business Analysts, Management, …