50 shades of tests

• Best of

I’ve been wanting to write this essay on testing for a while now. Well, I say “essay,” but it’s really more of a survey. The value I hope to provide through this article is based on categorizing tests into three parts: purpose, granularity, and type of assertion.

Let’s start with an example: I’m often asked how acceptance tests fit into the testing pyramid. My answer is “almost everywhere,” because different types of tests can fulfill this role, from unit tests to end-to-end tests. This means that not all tests are comparable, nor are they mutually exclusive, so we must consider their various facets. We’ll call these facets “dimensions.”

These dimensions (purpose / granularity / type of assertion) are nearly independent:

• Within a single dimension, the alternatives are mutually exclusive: a test cannot be both deterministic and fall into the category of monkey testing (the “assertion type” dimension);

• On the other hand, you could very well consider a performance test (type of assertion) that is an API call test, and thus end-to-end (granularity), and that serves as an acceptance test (purpose).

Of course, it all comes down to the “almost,” since there are counterexamples. Nevertheless, I still find this classification useful.

Let’s get started, follow the guide!

Dimension: Purpose

This is, of course, where we must begin: not all tests have the same purpose. Some aim to validate the design, others the development, and still others to prevent regressions, etc. Let’s look at this in detail:

• User testing: This takes place early in the development process and is used to refine the functional design and the user interface (UI). It involves having a panel of users interact with wireframes or, after a few iterations, with high-fidelity, partially interactive prototypes. During these sessions, which are often recorded, the tester asks the user to perform a Job To Be Done (JTBD) and observes how the user performs, what difficulties they encounter, what they do not understand, and what suggestions they offer. I mention this type of test for the sake of completeness but will set it aside for now, because, although essential, these tests do not fall under the purview of development.

• Tests written as part of Test-Driven Development (TDD): these are unit tests, which serve to guide and facilitate development. A wonderful byproduct, these tests serve as documentation once development is complete, while also helping to prevent regressions. But their primary value, and this cannot be emphasized enough, is realized during the development phase. In fact, the name says it all! Test-Driven Development is therefore a development practice that cannot be compared to practices such as Test-First and Test-After, which are testing practices.

• Acceptance tests: These tests validate the code against the functional design. They are therefore necessarily part of the development of a new feature and must be defined at the design stage—obvious, you might say, but it’s better to state it explicitly. They thus constitute a form of contract between the Product Owner and the Developer. Example Mapping will help you with this practice.

• Non-regression tests: Any development carries a risk, the risk of causing regressions. In plain English: what used to work no longer does. Of course, good modularity minimizes this risk on a large scale by avoiding unnecessary couplings, but on a small scale, where cohesion is strong, tests are the only way to detect potential regressions. Tests act as a safety net: the finer the mesh, the less likely you are to slip through the net and fall. So where do non-regression tests come in? Most of the tests we discuss here serve this purpose, starting with acceptance tests, but also including unit tests, integration tests, and end-to-end tests.

• Golden Master tests: These are a special case of non-regression tests and are performed in the context of refactoring. It is impossible to lock down behavior using unit tests, since unit tests are coupled to the implementation, and the implementation will change. The only way to do this is to create a log trail expressed in purely functional terms (for example: “Paul’s bank account balance is €1,400 as of 10/31/2024”), known as the Golden Master. At every stage of the refactoring process, the logs generated by the application must match the master, because while the implementation may change, the business logic itself remains constant. Along the way, other types of tests must be written to lock down the behavior (unit tests, integration tests, etc.), and once the refactoring is complete, the master can be removed.

• Exploratory testing: These tests aim to “break” the application. Huh? Yes, it’s better for you to find a bug before a user does. This type of testing is the domain of QA engineers, whose mischievous minds seem designed to uncover all those twisted edge cases 🫶 I say this jokingly, because the value of this work is inestimable and actually requires a very detailed understanding of how the application works. Exploratory testing is anything but random. Exploratory testing involves standing at the crossroads of the specifications: are these two topics, which we’ve considered independent, really independent? Because QA professionals aren’t as “nose-to-the-grindstone” as the rest of the team, they bring a slightly external, cross-functional perspective to the application, which allows them to dig deeper in their search for anomalies. An important point: this exploration must be done manually. However, if a bug is found, fixing it will require a different type of test, one that should eventually be automated.

Dimension: granularity

This is the dimension most often discussed in connection with the concept of the test pyramid. This concept is essential because it encourages us to use the full spectrum of testing methods with the goal of testing efficiently. The underlying question is: “How can I achieve maximum reliability by writing as few tests as possible?”

• It all starts with unit tests: unit tests are the tests a developer performs to facilitate or validate the development of a basic building block of the software. This building block is akin to a technical task, a development effort that does not, in and of itself, add value to the user but is nonetheless necessary. These tests are therefore performed by the developer for their own purposes, as opposed to acceptance tests. They can be written using the Test-After or Test-First approach, or as part of a TDD practice. These tests operate on a small scale and in isolation, which sometimes involves using mocks, or more generally, test doubles. As a result, they are quick to write (a few minutes) and just as quick to run (on the order of 10 ms). Thus, they allow developers to explore and “freeze” the cyclomatic complexity of an algorithm in detail. You’ll need to write a lot of them, hundreds or even thousands, as soon as the business logic gets a little complex, which is why these tests form the base of the pyramid.

• Component tests: these fall under unit testing. The only distinction is that we are testing the components of a graphical user interface rather than the functions or methods of a class.

• Integration testing: Running unit tests is necessary, but it is not sufficient. The fact that two modules function properly as individual units, in isolation, does not guarantee that they are properly coordinated. For this, integration tests are required, and any test located above unit tests in the test pyramid necessarily plays this role. The question is how to strike the right balance between the two types of tests. Let’s consider an example: a module A with two execution paths, A1 and A2, which have been unit-tested; and a module B with three execution paths, B1, B2, and B3, which have been unit-tested. Assuming that A calls B, an integration test would, for example, cover the cases A1/B1 and A2/B2, because the way B is called will differ in the branches A1 and A2. However, all other cases in the combination (the cardinal product of the execution paths) will be intentionally omitted, namely A1/B2, A1/B3, A2/B1, and A2/B3.

• End-to-end (e2e) tests, on the other hand, sit at the top of the pyramid and are, in fact, a specific type of integration test: they simulate the behavior of a user interacting through the interface (graphical user interface or API), following a complete business scenario and traversing as many application layers as possible, from the graphical user interface all the way down to the database. These tests take a long time to write (sometimes several hours for a single test) and are also time-consuming to run (several seconds per test). You should therefore use them judiciously, that is, for a few representative happy paths or critical paths of the application. You should view e2e tests as smoke tests, in line with the proverb “there’s no smoke without fire”: if there’s smoke, it means there’s a fire somewhere in the application (necessary condition); but the absence of smoke does not imply that there is no fire (insufficient condition).

Dimension: type of assertion

The final dimension is the type of assertion. In other words, “what am I testing?” and therefore, “what level of assurance does this type of test provide?”

• Example-based testing (a.k.a., deterministic testing): given a certain initial state Given and a certain action When, I expect a certain output value or state Then. This is the type of test we’re all familiar with, but it offers very little assurance. Let’s take an example: if you’re implementing the affine function y=1+x/2, you might use the points (0,1) and (2,2) as test cases and consider that sufficient. This is reasonable because you know you’re constructing a line, but theoretically an infinite number of functions pass through these two points, starting with the parabola y=1+(x^2)/4. While these tests serve as a safety net, you have to admit that the net has holes!* On the other hand, these tests are particularly useful for guiding our development (TDD, once again).

• Property-based testing offers us a little more assurance, since it aims to verify that a business invariant is always satisfied for a large number of randomly selected input values (10, 100, 1,000?). In the case at hand (y=1+x/2), one invariant is that for any two x-coordinates, with the y-coordinates calculated based on your implementation, the slope of the line connecting the two resulting points is always 1/2. I suggest a detailed analysis of this example here. Again, this test does not offer an absolute guarantee; you still won’t have proof that your implementation is correct, but the statistics make an error less likely. As far as development is concerned, this is good enough for us!

• Snapshot testing: As in property-based testing, you do not specify that a particular value or state is expected as the test’s output. Instead, you simply run the test once and store the result (the “snapshot”), which you have manually verified to be correct (e.g., the virtual DOM of a React component). On subsequent runs, each time the test is executed, the result is compared to the snapshot. If the two are identical, the test passes; otherwise, the test fails. Use case: exclusively for non-regression testing.

• Mutation testing: It all starts with the observation that it’s easy to achieve very good code coverage without the associated tests providing a true safety net against regressions. In other words, line-of-code coverage is good, but not all cyclomatic complexity is explored. This is a significant issue, which we won’t delve into here. Mutation testing identifies these gaps by randomly introducing mutations into the code. For example, age > 18 becomes age > 10 or age <= 18, and if no tests fail, it means your code isn’t properly tested. It’s very powerful and, frankly, meta: it’s, in a way, a test of the tests.

• Monkey testing, on the other hand, addresses a very specific issue: the proper handling of error cases, particularly through the surface controls implemented in the graphical user interface. The idea is that your application should withstand completely random and therefore incoherent input, as if generated by a monkey. Your application must remain in a stable state under all circumstances and properly inform the user that their input is invalid. From experience, I can confirm that a cat works very well for this, and I’m told that a human baby also does a very decent job. In practice, this is the kind of thing we automate, so that the monkeys can go about their other business in peace.

• Performance testing: here, execution speed is what really matters. For the measurement to be meaningful, it must be based on a large number of “runs,” expressed as percentiles. For example, P50=100ms means that 50% of requests completed in 100 ms or less, and P95=400ms means that 95% of requests completed in 400 ms or less. Response time is a key non-functional requirement, particularly for APIs, because even a small tightening of requirements (a 150 ms reduction on the P95) may force you to completely overhaul your architecture.

• Smoke tests: We mentioned these earlier to note that end-to-end tests are smoke tests. More generally, smoke tests are performed, in particular, during the deployment of the application or the services that comprise it. All other types of tests have been run on the previous environments and have given you enough confidence to move to production; now that’s done, all you have to do is verify that the application is available and functioning “for the most part.” The ultimate smoke test is displaying the authentication screen when you want to verify that the HTTPS certificate is being properly recognized. Does that ring a bell?

Conclusion

One final note: all tests can and should be automated and run on the CI as much as possible, with the exception of user tests and exploratory tests, for which this would not make sense. There are two reasons for this:

• Regarding reproducibility (the “R” in the acronym F.I.R.S.T.): “manual testing” inevitably means “sooner or later, an error in test execution.” I didn’t use the correct values and the test passes (false positive), or I used the correct values but mistakenly reported that the test passed (also a false positive).

• For speed (the “F” in the acronym F.I.R.S.T.): no offense, but I doubt you’re any faster than a computer 😉

Next, knowing which tests to use requires a bit of experience and a lot of pragmatism. The key is to avoid falling into the “tool-centric” trap, that is, thinking, “I only know one type of test, so that’s all I do.” Your application won’t fare well if it relies solely on unit tests. Nor will it fare any better if it relies solely on end-to-end tests.

(*) Scott Wlaschin illustrates this very well with the following image: “Unit tests have been compared with shining a flashlight into a dark room in search of a monster. Shine the light into the room and then into all the scary corners. It doesn’t mean the room is monster-free—just that the monster isn’t standing where you’ve shined your flashlight.”