The Difference Between an A/B Test and a Multivariate Test

Lately I’ve been writing a lot of posts answering questions that we get asked all the time. Well, here is another one:

What is the difference between an AB Test and a Multivariate Test?

Here is the best way to think about it. If you have a webpage and you are experimenting by replacing that page with a different page – then you are performing an A/B test. In the case of multivariate testing you are experimenting by replacing a number of elements within a page all at the same time. It’s like conducting numerous A/B tests on one page at the same time.

So now that you know the difference between an A/B test and a Multivariate test let me tell you about the pros and cons of each.

A/B Testing Pros

– You can vary whichever areas of the page you want

– You can test a more dramatic design or layout

– There are some awesome ways to evaluate each variation (e.g. click heatmap, phone call tracking, analytics integration, etc.)

– Experiments generally take less time than with MVT

– Many times the results are greater

A/B Testing Cons

– You may need multiple test rounds and these take planning

– Setup often must be done manually

Multivariate Pros

– Get more granular to better understand how small page elements effect conversion rate

– Figure out how combinations of small page elements have a compounding effect on one another

– See incremental conversion rate improvements – more conservative approach

Multi-Variate Cons

– You need a lot more variable combinations to be run than with an A/B test

– Generally only works well for large sites with a lot of traffic

– Dramatic design or layout changes are not possible

– Must make sure that all the variations that are being swapped out make sense together, which can hinder creativity

I hope this short post helped clear things up for you around the difference between an AB test and a Multivariate test.