How to mess up A/B tests

An excellent talk about A/B tests from someone who knows – Martin Goodson.

My favourite part is an A/B test that found a 2.5% improvement to (sales) conversions between the two versions of the software being tested.  Unfortunately there was a bug in the A/B testing framework, such that the old version was being tested against itself, rather than against the new version.  Oops.

It illustrates really clearly that A/B testing is actually doing experiments.  Maybe it would be better to call them A/B experiments?  Normal software tests are just kicked off, they run for as long as they take, and then give you a simple and clear-cut yes/no answer.  (There are issues about coverage etc, but I’ll gloss over those for now.)  With an experiment, you need to worry about three things:

1. Beforehand, you design the experiment to try to test your hypothesis (and not a different hypothesis) and to do so effectively;
2. During the experiment, you try to avoid clumsiness and other errors that would mess up your results (like telling test subjects that they have the real pills rather than the placebo);
3. After the experiment, you take care with interpreting your results – turning the numbers on your computer screen into some insight into how fruit flies jump or galaxies coalesce.

Doing an A/B test wrongly (at any of those 3 stages) could lead to mistakes that hurt your business.

The video has some statistics terms to help understand the background, and then some practical things to do and not do.