Mathieu Eveillard

Testing Randomness: A Textbook Example of Property-Based Testing

PBT verifies that a business invariant is always satisfied regardless of the parameters and the return value of the function being tested.

Testing Randomness: A Textbook Example of Property-Based Testing

Deterministic testing, based on examples such as f(0) = 1, is just one testing method among many. In some cases, such as functions that rely on random number generation, there are better approaches.

Problem Statement

Let’s imagine a random function that returns a real number between two bounds min and max, whose implementation would naturally rely on the Math.random function, which returns a real number between 0 and 1:

type Parameters = {
  min: number;
  max: number;
};

const random = ({ min, max }: Parameters): number =>
  min + (max - min) * Math.random();

How can we test this function?

This question actually requires us to determine what we want to test:

  • Whether the generated values are indeed within the range min to max?
  • Or the probability distribution between min and max?

In our case, we can assume that the distribution of the values from Math.random is indeed uniform, in accordance with its specification. What we want to test, then, is the value our function adds, namely, the fact that it “stretches” the randomness between min and max.

How should we proceed? Since (pseudo-)randomness has the good grace to be unpredictable, we cannot perform a deterministic test based on a single example. Indeed, testing that random({ min: 1, max: 10 }) === 5 would not only be false most of the time, but more importantly, it would be meaningless.

Approach 1: Dependency Inversion to “Control Randomness”

One initial solution is to inject the random number generator so that we can control its return value in a test. To do this, we use dependency inversion (DIP): Initially, the random function depended on Math.random; now, it defines for itself and in its own terms what it needs to function, namely RandomGenerator, and leaves it up to the “outside world,” that is, the caller, to inject a function that satisfies this contract (dependency injection).

This gives us the following code:

type RandomGenerator = () => number;

type Parameters = {
  min: number;
  max: number;
};

export const random =
  (generator: RandomGenerator) =>
  ({ min, max }: Parameters): number =>
    min + (max - min) * generator();

test("The lowest value possible should be 'min'", () => {
  const generator = () => 0;
  expect(random(generator)({ min: 1, max: 10 })).toEqual(1);
});

test("The highest value possible should be 'max'", () => {
  const generator = () => 1;
  expect(random(generator)({ min: 1, max: 10 })).toEqual(10);
});

This solution works, but you’ll agree that it’s still very “implementation-aware.” Put more plainly, this means that if the implementation changes, the test must change as well. We want to test a certain behavior, regardless of the implementation, period.

Fortunately, we can do better.

Approach 2: Property-Based Testing to Test a Business Invariant

In contrast to example-based testing (deterministic testing as we know it), property-based testing aims to verify that a business invariant is always satisfied for a large number of randomly selected input values (generally > 100, but you can configure the number of trials).

Let’s see how this works with the fast-check library:

import fc from "fast-check";

test("The generated value should fall within the (min, max) interval", () => {
  fc.assert(
    fc.property(
      fc.record({ x1: fc.integer(), x2: fc.integer({ min: 1 }) }),
      ({ x1, x2 }): boolean => {
        const min = x1;
        const max = min + x2;
        const actual = random({ min, max });
        return min <= actual && actual <= max;
      },
    ),
  );
});

Here, fast-check generates two random integers for us, x1 and x2, which we use to construct the interval [min, max]. Next, we call the random function under test and simply verify that the value it returns is indeed within the bounds of the interval. Note that at no point do we know the values of min, max, or random({ min, max }). That’s the beauty of it: we’re testing an invariant.

What Property-Based Testing Actually Guarantees

This example is one of the simplest I can think of to explain property-based testing, but it is typical, since randomness is a perfect use case for this type of testing.

How much assurance does this type of assertion provide?

This test proves nothing more than a standard test would: it is not a demonstration that your code works correctly, but simply a test. Even if you generate 1,000 values for each test, run the test 100 times, and the property holds true all 100,000 times, you cannot rule out the possibility of encountering a counterexample on the 100,001st try. But statistically, this is still quite unlikely, and for development purposes, we can be satisfied with that.

Just keep in mind that you increase your level of confidence by increasing the number of trials.

← All articles