Someone posted a question in a Ministry of Testing Slack channel about introducing mutation testing at work. My answer is below, as someone suggested I turn it into a blog post. I’ve edited it slightly to make it stand on its own rather than as a response to a particular person’s question.
It’s basically a small technical bit and the majority is change management. I don’t have proper experience of change management, but thanks to listening to Paul Boag’s podcast for long enough, I see it through the lens of user experience (UX).
Technical stuff with your CI pipeline
I’ve had a little experience of mutation testing (with Fettle, for C#). It can be an unpleasant surprise when you start mutation testing – so many mutants survive!
I suggest that you don’t set things up such that surviving mutants stop the CI pipeline to start with. There will be too many and it will take too long to sort them.
Instead of stopping the pipeline, have some kind of report that goes to the relevant people (something you’ll have to decide). Then I’d say it’s a people problem rather than a technical one.
Tackling the big problem – winning hearts and minds
How do you persuade managers to allocate time to fixing mutants? How do you help developers and managers move from “We have unit tests – we’re OK” to “But our unit tests give us a false sense of security because they’re only a very loose fit to our code, so we need to look at them again.”
I expect it will be the usual approach for improving testing – high risk or high value areas get addressed first, plus the standing order: if you’re already in a bit of code, make sure it has no surviving mutants / fewer when you leave than when you arrived.
I think a good technique for the persuasion would be to find a surviving mutant in a particularly valuable or risky part of the code. Then show what the mutant would mean in business terms – how could the business or customers suffer if the mutant went into production? For instance, we could allow products to have a negative price, so we’d lose money, or users could enter invalid data in step A of a process that would cause later step B to fall over.
Start with the people who will have to change
Think about the managers and developers – what do they already care about? Mutation testing could be seen as a way to impose even more work on them, so you need to show how it’s actually a way for them to get something they already want:
- More reliable and predictable releases?
- Reduced overall cost through less rework?
- Quicker good quality feedback on whether they’re done or not?
- Etc.
Another thing that might help is to write a manager’s guide and a developer’s guide and put them on e.g. an intranet. Make it easy for developers to know how to add mutation testing into their work routine. Make it easy for managers to understand this report that they’ve never heard of / only dimly remember (e.g. refer to the guide in the text of the report, ideally via a hyperlink).
If you have introduced code coverage metrics as a part of your pipeline, that might be a useful starting point (either as something to copy or to avoid). I imagine it would have had a similar path – sudden realisation that things are bad => plan to gradually make things better.
It’s important to phrase this as something positive – a way to get even better. It’s not about blaming anyone, or highlighting where e.g. developers are failing. The adoption of this new thing, that has net cost in the short term but net benefit in the longer term, will be hard enough without it being seen as a new stick to beat people with.
Summary
To cut a long story short: this is mostly a user experience (UX) issue, where the users are managers and developers. Apply UX type skills – make it easy for people to do the ‘right’ thing, and don’t try to persuade people about how good a new thing is – instead show them how this is a means to something they already value.
Random Tech Thoughts 