Mathieu Eveillard

100% Code Coverage: The Never-Ending Race

This metric isn’t very useful if you don’t know what lies behind it… It does tell you something, however, mainly when it’s low!

100% Code Coverage: The Never-Ending Race

100% code coverage is often touted as the goal to achieve within teams. Easy enough: all it takes is a few well-placed end-to-end tests to send this metric skyrocketing, without actually making the application any safer. By that I mean the ability to aggressively refactor the codebase without major risk of regression, because you have a real safety net.

The demand for 100% code coverage is a perfect illustration of Goodhart’s Law: “When a measure becomes a target, it ceases to be a good measure.” Ultimately, it’s sometimes better to have slightly lower code coverage with a good selection of tests and a measured level of risk. To do this, you need to know the application in detail, that is, its features, and, even before that, the business domain.

But first, let’s look at what 100% code coverage actually means.

An indicator to be taken with a grain of salt

Consider this example:

const positive = (n: number): number => Math.max(0, n);

describe("A single test achieves 100% code coverage…", () => {
  test("n > 0", () => {
    expect(positive(1)).toEqual(1);
  });
});

Here, the line-of-code coverage is 100%. But is the code actually well-tested? Clearly not, because the function Math.max hides a control structure, meaning two branches. The case of negative numbers is not covered. Since there is only one test for the two branches, the branch coverage is 50%.

But even this branch coverage isn’t sufficient. Consider the following example:

const twist = (n: number): number => {
  let result = n;

  if (n < 0) {
    result *= -1;
  }

  if (n < 1) {
    result *= -1;
  }

  return result;
};

describe("Two tests achieve 100% code coverage…", () => {
  test("n < 0", () => {
    expect(twist(-2)).toEqual(-2);
  });

  test("n > 1", () => {
    expect(twist(2)).toEqual(2);
  });
});

Here, the line-of-code coverage is 100%, as is the branch coverage: every branch of the two control structures is explored. However, the control structures are coupled through the variable result, which they modify jointly. We would therefore need to examine what happens between 0 and 1, in addition to specifically examining these boundary values.

All this to say that 100% code coverage is a necessary but not sufficient condition. 100% code coverage does not mean that your code is well-tested or that you have a real safety net against regressions.

Test-Driven Development to the rescue!

While it may not be the solution to all your problems, Test-Driven Development (TDD) nonetheless plays an important role in the range of available solutions. Indeed, it is the ideal tool for writing code that is exhaustively tested by design (and of course, there are counterexamples, which force me to oversimplify).

This follows from the first rule of TDD: “You are only allowed to write code if that code is used to pass a unit test that previously failed.” Thus, when you add a new execution path, it follows the writing of a test related to that same execution path.

Let’s state the obvious: TDD lends itself very well to writing the core business logic of an application, where business rules and, therefore, algorithmic logic are concentrated. But it’s unlikely that you’ll write more than 30% of the codebase this way. If that’s the case, you may already be suffering the pangs of the law of the instrument.

How should code coverage be used?

Should we throw out code coverage? Of course not: we need to view it for what it is, an indicator. Here’s what we can take away from it:

  • First, low coverage does tell you something: if it’s 10%, you need to write tests (start with unit tests).

  • Next, we can focus on its change rather than its value: it should always be increasing; if it decreases, there’s a problem.

  • Finally, as we’ve seen, 100% code coverage isn’t enough. It can’t be interpreted on its own; we need to go further and apply judgment, that is, adopt a real testing strategy. The ultimate test, of course, remains mutation testing, which is the test of all tests.

← All posts