Taming Combinatorial Explosions in SpecFlow

The Problem

Imagine you are using SpecFlow to test a REST API.  In the common REST way, the URL is built up of 1+ units, where each unit is one of these:

• Give me a list of X;
• Give me the instance of X that has id Y.

You could easily have up to 3 different units strung together to form 1 URL:

/customers/1234/orders/5678/items

Actually, for testing, you might have extra types of unit:

• Give me the Nth X in its list, e.g. Give me the 1st customer;
• Give me an instance of X with a bad id, e.g. Give me a customer with a bad id.

The first will work even on production data, where you can’t rely on particular customer ids to be in the database.  The second will help test that things like /customers/999999/orders (where 999999 doesn’t match any customer ids) will return a 404 error rather than 200 plus an empty list.  Having “with a bad id” as a different test case has a couple of benefits:

1. It makes the intent of the test clear;
2. It hides unnecessary low-level detail (the particular value of the bad id).

So, you have up to 3 slots to fill, each with 4 different options.  This is 4^3 + 4^2 + 4 = 84 combinations.  Note that this is even though you use parameters to absorb variations in the particular id and nouns such as customer, order etc.  The 84 combinations are structurally different.  (Note that you probably can’t do e.g. all 64 combinations over 3 slots in practice, as the list of X option will probably come only at the end of the URL, but the total is still a large number.)

Spelling out all the possible combinations in your request steps file is a pain:

• There’s lots of copies of code doing almost the same job;
• Adding e.g. a 5th option and/or a 4th slot will make it even larger;
• It’s hard to read and so understand what’s going on.

I’ve not been able to get SpecFlow to match the regular expressions in the attributes attached to step code in the way I’d want – match on part of the string (so that many different patterns could match along the length of the string) but also give some idea of where on the string you matched (beginning, middle, end etc.)

Summary of the Solution

The answer is SpecFlow’s step argument transformations.  The basic approach is as follows:

• In the step definitions’ regular expressions you just worry about the high-level structure – how many units;
• You define a set of types that represent the different kinds of unit;
• In a custom step argument transformation you worry about just the different kinds of unit, and produce the appropriate type for the unit in question.

Step Argument Transformations

You define a small class hierarchy for the different kinds of unit, e.g.

• RequestElement
  • RequestElementList
  • RequestElementWithId
  • RequestElementBadId
  • RequestElementPosition

You then define a file for the transformations, which is a lot like a step file, except it has a different job.  Instead of matching the whole of a line and so working out which method to execute, it matches part of the line and returns an object that represents that part of the line (ready to be used as an argument passed to a method in a step file).

[Binding]
public class RequestElementTransform
{
    [StepArgumentTransformation(@"a list of (\w+)")]
    public RequestElement ListTransform(string listType)
    {
        return new RequestElementList { Noun = listType };
    }

    [StepArgumentTransformation(@"the (\w+) with id (\d+)")]
    public RequestElement SpecificIdTransform(string thingType, int id)
    {
        return new RequestElementWithId { Noun = thingType, Id = id };
    }

//...
}

This is missing out a lot of details for clarity, but I hope you get the idea.  Each method’s regular expression matches a different set of words in part of a test’s line, and produces the relevant child class as the parent class.

Simplified and fewer steps

With the step argument transformations taking care of the variation in the elements themselves, the steps can concentrate on how they are put together:

[When(@"^the client requests (.+) for (.+) for (.+)$")]
public void WhenTheClientRequestsXForYForZ(RequestElement x, RequestElement y, RequestElement z)
{
   PrepareRequest( new List<RequestElement> { z, y, x });
}

[When(@"^the client requests (.+) for (.+)$")]
public void WhenTheClientRequestsXForY(RequestElement x, RequestElement y)
{
    PrepareRequest( new List<RequestElement> { y, x });
}

//...

Again, this is leaving out details for clarity, but the point is to show how the URL is built up out of the elements of the test, in reverse order.  That is:

When the client requests the list of items for the order with id 456 for the customer with id 123

becomes the URL

/customers/123/orders/456/items

The different ways to handle the different kinds of element can be handled once, in a dedicated method that is called N times – the step with 2 elements calls this method twice, and the step with 1 element calls it once.

Summary

This approach might not work for all tests, but where it works it can be useful.  If X is the number of kinds of element, and there are Y of them per test, then instead of having X*Y or even X^Y different bits of code to handle all possibilities, there is X+Y which is less to understand and less to maintain.  Adding another type of element or increasing the number of elements allowed per test changes things from X to X+1 or from Y to Y+1, which is much nicer than going from X^Y to X^(Y+1).