Metric grudge match: Which metrics you should target with AB experiments

I have a riddle for you…

How can an experiment fail but still be a success?

Easy! The goal of experimentation is to learn. As long as we’ve learnt something then the experiment must have been a success.

Nice try.

Another reason is that experiments can have multiple goals. While one goal might fail, another one may unsurprisingly succeed.

For example, an experiment may have these two goals:

• increase enquiry form submissions
• increase newsletter subscriptions

A change to a page layout may result in an increase in enquiries but a decrease in subscriptions. Whether the experiment as a whole was a success, or a failure may be a debatable topic, with answers differing depending on who you ask. Ultimately the purpose behind running the test will be in pursuit of the strategy and objectives of the product.

How do you measure the performance of a sales funnel? Which metric should you use?

Is it better to test a micro-goal within the funnel or instead to zoom out and measure all results against one key objective?

The components of a simple website sales funnel

An eCommerce website’s main goal is for people to buy a product. To purchase, a user would first need to visit the website, then generally add the product to their shopping cart and then finally go through the checkout process.

If we mapped this out as a simple sales funnel it might look like this:

1. Visit website
2. Add product to cart
3. Complete checkout (Sale)

The goal of the website is to encourage more people to buy products, the final step in our sales funnel.

Within this sales funnel there are 3 different ratios that we would want to monitor and optimise.

1. Website visits → Add products to cart (1)
2. Add products to cart → Checkout (2)
3. Website visits → Checkout (3)

Using the sales funnel to test and learn

Let’s imagine that we manage an eCommerce website and we’ve identified a potential optimisation that we think will encourage more people to add products to their shopping cart. We’re hoping that more people adding products to the cart will subsequently result in more purchases.

Therefore, we’ve decided that the metric we want to improve is the ‘Add products to cart’ ratio. There are three possible outcomes for the test:

1. Experiment wins: a higher proportion of people add products to their cart
2. Control wins: a lower proportion of people add products to their cart
3. No result: the number of people adding products to their cart stays pretty much the same

We would also need to keep an eye on the number of sales we receive (ratios 2 and 3).

Often experiments return some truly wonky results

Let’s say that our experiment concludes, and it loses. Not just a ‘no result’, but we can confidently say that our change has resulted in less people adding products to their shopping cart.

But now if we imagine that we were to then observe from our results that the checkout completion ratio increases in the experiment. In fact, the increase is so high that it counters the loss in ratio 1 and results in an overall gain to ratio 3. Our experiment has technically failed based on our key metric, but the most important metric (sales) has improved.

Should we have run this test using the funnel ratio as the key metric rather than just the add product to cart ratio? If sales are the most important outcome to us, shouldn’t we just measure all tests with the full funnel conversion ratio?

This is where good judgement is needed in determining which metric to use!

Running experiments across the whole sales funnel

One of the biggest challenges with running experiments over the whole sales funnel journey is that by the time we get to our goal, the conversion rate is too small to have a measurable impact.

Here’s an example to illustrate where using a full funnel metric wouldn’t work so well.

Suppose that on our website, the ratio of customers adding products to their cart is 5% and the ratio of those customers that then purchase is 20%. This means that the full funnel conversion rate is only 1% (5% x 20%). An experiment that is aiming for a statistical difference from say 1% to 1.1% would require quite a large sample size and therefore need much more time to run. To get a result in a timely manner, we would need either a lot of site traffic or we’d need to be aiming for large conversion improvement.

In this case, it would be better to try and impact individual steps within funnel, while making sure that there is no significant decrease to the overall funnel goal.

It’s all a matter of judgement

In most cases it generally makes more sense to run AB tests on specific stages in the sales funnel.

If your sales funnel is simple (e.g., one product, no cart) then perhaps targeting a sale as the key metric might be appropriate. This approach will depend, however, on the complexity of the user journey, the amount of traffic your site receives, and how much uplift you are expecting to see because of your change.

Whichever route you take, there often won’t be a ‘right’ answer. The hint is in the word ‘experimentation’, and you may just need to trial and error a few experiments to see what works best for you.

So it looks like the goal of experimentation is to learn after all.