Mathieu Eveillard

Measuring Software Quality: Don’t Fight the Wrong Battle

Measuring Software Quality: Don’t Fight the Wrong Battle

Numbers are great: our gut feelings are one thing, but they can lead us astray. So, we set out to measure software quality. That’s a huge undertaking. How do we go about it? What metrics should we look for?

Let’s start by listing two or three things we’d be wise not to do…

These metrics do more harm than good

Let’s consider the unit test coverage rate: this metric overlooks the basics of [testing strategies], namely using different types of tests to detect different types of bugs. You can write as many unit tests as you want, but they will never replace an integration test…

So it’s off to a bad start, but that’s not all. The problem with this type of metric—which focuses on means—is that it dictates how the team should work. For very junior developers, this might be reassuring, but many senior developers will feel stripped of their responsibility—especially when the metric makes no sense, as it does here. What do you ultimately want? For there to be unit tests output, or for there to be no bugs outcome?

In other words, it would probably be better to hold the team to a higher standard by tracking a metric such as the number of bugs in production (something that matters to users and the business), and to empower them by letting them choose the means to achieve that goal (aiming for 0). Of course, automated tests will be needed, and of course, unit tests will be needed. But there isn’t just one right way to do things, and the development team remains best suited to make an informed choice in this regard (what balance between unit, integration, and end-to-end tests; which approach—test-after, test-first, or TDD?).

Let’s get back to the terminology. The output refers to what is directly produced: code, tests, documentation, deployments, etc. Outcome, on the other hand, refers to the actual changes or benefits generated by the output, from the perspective of users or the organization: a better user experience, acquisition of new customers, increased revenue, user feedback, reduced technical debt, etc.

An indicator related to output is particularly susceptible to Goodhart’s Law, which states that “When a measure becomes a target, it ceases to be a good measure.” If I decide to measure a team’s productivity by the number of lines of code it produces, everyone knows how absurd that is. It says nothing about the value of the features delivered to the user, nothing about code quality or technical debt—nothing at all. So, very quickly—whether intentionally or not—developers will start writing the same code by breaking it up into more lines, and you’ll have created a “watermelon” metric—green on the outside but red on the inside.

Finally, the outcome is far more interesting than the output because it holds the entire team—with its diverse roles—accountable. What good would it do to have excellent code—well-modularized, well-architected, well-tested, with little technical debt—if the features were poorly designed or, worse, if they didn’t address your users’ real problems? That would mean your developers are working for nothing, or even consciously doing absurd things, simply doing “exactly what they were told to do”. That’s clearly not the mindset you want. You want everyone to feel invested in the business objective and to lend a hand to their colleagues when needed, even outside their assigned responsibilities.

Some guidelines for useful metrics

Measuring software quality is a vast undertaking, which I’m only scratching the surface of in this article. Before measuring it, you first need to be able to define it—a topic that’s also worth spending some time on.

Regarding the metrics themselves, however, here are a few recommendations:

Here are a few examples of metrics that have a good chance of working:

  1. Time to Market: the time elapsed between the initial idea and the first release in production;

  2. The Feature Adoption Rate, which measures the speed at which users adopt a feature;

  3. The Production Bug Count, possibly weighted according to the severity of the bugs;

  4. The Mean Time to Recovery, the average time it takes to return to normal operations after an incident.

Conclusion

More generally, you shouldn’t be content with simply measuring software quality. What you likely want is to help your teams improve—by training them, fostering knowledge sharing across teams, and providing shared code libraries. Support and measurement are two sides of the same coin.

Finally, if you find bugs in production, think “Shift Left” (the latest buzzword): rather than chasing after bugs, ask yourself how and why they appeared. [Spoiler Alert] Functional design is often to blame.

← All posts