Integration Tests with Medical Devices
Integration Tests with Medical Devices
If you are a tester for medical devices you have probably already hit the problem of Integration Tests. The scenario usually looks as follows:
- Each team (Software or Hardware) develops their piece of the puzzle
- Each team makes sure that their module works internally within the scope of their responsibilities
- All of the teams are checking-in to one central repository (branched or not doesn’t make a difference here)
- Sprint demo is taking place and BOOM! Nothing works. One Module doesn’t communicate with the other. Software doesn’t speak with Hardware.
How can that be? Each single team tested their own module and all was fine. That is indeed true, but nothing works in isolation. If you build a new engine for your car you would expect that the car will run with the gearbox, so that both modules work together. The same principal is applicable to testing medical devices. You cannot really test anything is isolation and assume it will just work out of the box on the target environment.
Integration tests to the rescue
This is stating the obvious but I cannot really stress enough how important integration tests are. This is often the biggest mistake that companies do when developing medical devices. They leave the integration tests till the very end of the sprint only to hit a wall and realizing that the entire system doesn’t work. The problem starts with the different teams responsibility attitude. Let me quote one of the best discussions I have heard to illustrate the problem.
Developer: I’ve checked-in the changes but someone should test it on the target machine
Team Lead: Does it work with the simulator?
Developer: Yes, it does.
Team Lead: If it works with the simulator then it’s good for us.
Developer: What if it doesn’t work on the real instrument?
Team Lead: That is not our problem.
As you can see this shows the big problem in a nutshell. People are always focused on their tasks but not the ultimate goal. In the end nobody will buy a medical device which “works great on the simulator”.
How to approach integration tests then? Based on my experience this needs to be a two-stage approach.
Stage 1: Integration Tests within the Team
The team responsible for each module should make sure that their module runs against the target system. What does it mean in practice? Instead of testing the new stories or bug-fixes in isolation, make sure to test them on the target device. Naturally, there will be cases where it’s not possible to do so (e.g. HW module missing). In such cases don’t be afraid to reach out to the other teams. Cooperate with them and don’t be afraid to raise you voice and say – hey it doesn’t work. In the end the whole system needs to work. Yes, in the beginning your teams velocity will probably drop and you will find some resistance in taking this new approach, however in the end what counts is the effect it will have on the end-product. There will be definitely some aspects you will not be able to test (e.g. due to the HW availability of the prototype units). Therefore, what you will need is a Dedicated Integration Tests unit.
In modern test automation environments a complete test lab control system is a must have. In an ideal world, most of the test environments would be available automatically in a scaleable way.
Every team could easily trigger their integration tests and they are executed automatically once a instrument becomes available. Or even better, testing would be staged. First, all tests are automatically executed against the simulator. And the simulator is in a VM that is automatically scaled in the cloud. And only tests that pass on the simulator are then queued to be executed on a real instrument.
Want to know more about modern test automation? Schedule a call today.
Stage 2: Dedicated Integration Tests Person/Team
Although, we would call the person/team integration tester what she/he would essentially do is perform exploratory tests and end-to-end tests. Basically, you need someone who would take each single daily/nightly build and ensure that it works as the teams claim it works. In other words, the tester performing the tests should act exactly like the target user would do. Test everything on the target platform, test around the stories that have been delivered, test as a human.
The Integration Tests should be a continuous process which allows you to reach a positive result and have a working demo at the end of the sprint. Keep in mind, you need to place yourself in the shoes of the customers. Not one customer but multiple customers. That means that you need to think outside of the box. I know it sound cliche, but you cannot just try to execute a sequence of actions as you would do it. One Happy Case scenario and that is it. There will be 100s, 1’000s or even 10’000s of users who will work with your device each day. It’s given that you will not be able to mimic each single behavior. What you can do though is to think of multiple scenarios for the same functionality, edge cases, unusual custom configuration etc.
By introducing these integration tests you will surly achieve a higher quality of your software release and in the end reach more satisfied customers.
Have fun in your integration tests. If you would like to share your thoughts about the integration tests within your company feel free to drop a comment below. You can also reach out to me by email.