Tests as sheep dogs for your code

Tests can act as constraints on your code, a bit like sheep dogs herd sheep into a pen.  One sheep dog stops the sheep from straying to the left, another stops them from straying to the right, and another moves them forwards towards the pen.  The sheep are free to roam where they wish, if they stay within the space between the sheep dogs.  The shepherd arranges the sheep dogs so that they constrain the sheep to just the space where the shepherd wants them, i.e. the pen.

A test author acts as the shepherd.  They must write tests such that they constrain the code under test appropriately.

A programmer is a bit like the sheep – they are free to produce whatever code they like that passes the tests i.e. if it is within the space defined by how the tests constrain the code under test.

Overlapping constraints

In a hand-waving kind of way, you can think of code as living in some abstract space.  This space contains all possible code.  To make this abstract thing even harder to imagine, I’ll represent it as a 2D rectangle, because you’re looking at this blog on a 2D screen:

The first test you write for that code will constrain it in some way.  For instance, the code will always return 2 if I pass it the arguments X=12 and Y=10.  As long as the code does that, it’s free to do whatever it wants.  You can think of that as the test defining a region of code space, which contains all code that returns 2 if it’s passed 12 and 10.

You could then write another test for the same code, such that it must return -3 if it’s passed X=-13 and Y=-10.  This will define a different region of code space from the first test, but the two regions will overlap.  Given that the two tests are applying to the same code, and the code must pass both tests, you’re now saying that the code must be in the overlap or intersection of the two regions.

You can add more tests and each one will define its own region of code space, such that the overlap of all regions gets smaller and smaller.

Picking the right arguments

If the test is implemented as code, the most fundamental constraint it places on the code under test is its signature i.e.

1. Its name
2. The number, order, names and types of arguments
3. The type of any value[s] returned

Most of the time, part A of the code under test will already place this kind of constraint on part B, because A will call B.  (There’s no point in having code that nothing ever calls.)  However, if you’re doing something like writing a controller for an MVC website or REST API, it’s highly likely that no part of the code under test will call the controller.  Instead, it will get called by the web server in response to an incoming HTTP request.

So, having a controller with no unit tests is even riskier than any other part of your code having no unit tests.

Good and bad behaviour

It’s up to the test author to make sure that that space defined by the tests:

• Contains no behaviour you don’t want;
• Leaves out no behaviour you do want.

In the example above, the tests are some of those that would fit a modulus function.  They are incomplete because they don’t include trying to do modulus 0 (which should return an error).  Even if there were tests for modulus 0, they would still be incomplete.

Remember, I said that the programmer is free to write whatever code they like that passes the tests.  I also said that tests should leave out no behaviour you want.  As there are infinitely many integers (i.e. even limiting the inputs to just integers is bad), then there should be infinitely many tests.

Without infinitely many tests, the programmer is free to implement the function as a look-up table from inputs to output, with an entry in the table for each test.  If I fail to test e.g. 17,234 mod 531= 242, then the programmer is free to miss that entry out of the table.

It’s obviously impossible to do that, which is where this model breaks down.

Risk and value

Instead of trying to write possibly infinitely many tests, the test author needs to be guided by value and risk:

• Where is the code going to deliver something of value to the user? Where is it going to deliver most value?
• What are the risks to this value? Which are the most likely, or will cause the most harm if they occur?

In the case of the modulus function, the user wants to calculate X mod Y for 2 integers X and Y.  It’s likely that the integers are positive (but not definite), so this is where the modulus function would be most valuable.

The most harm will come if the function can’t compute X mod Y for two positive integers X and Y (note that this breaks down into two sub-cases, where X is a multiple of Y and where X isn’t a multiple of Y).

However, due to human nature the more likely errors will be when

• Y = 0
• X = 0
• X < 0
• Y < 0

This is because the programmer is less likely to think of these cases than when X and Y are both positive.

So, it would be sensible to have a test for all the scenarios above, to protect the function’s value as well as possible.  After that, we could write 0 or more further tests.  At some point we must assume that all further tests will check the code in effectively the same way as previous tests (the details of the numbers will be different, but they will be similar to what’s already happened).

Conclusion

Tests constrain code, in terms of its signature and behaviour.  A programmer is free to alter the code if it still satisfies the constraints imposed on it by the test, for instance by refactoring it.

In many cases there could be infinitely many tests that could be written for a given bit of code.  Value and risk help you to identify which tests add further constraints on the code in ways that are most useful and cost-effective.