A multivariate test is a test that simultaneously tests several combinations of several variables. The idea is to modify several elements simultaneously on the same page and then define which one, among all of the possible combinations, has the most impact on the indicators being tracked. Multivariate testing (MVT) helps test associations of variables, which is not the case of successive A/B (or A/B/C, etc.) tests.
During an A/B test, you may not modify more than one element at a time (for example, the wording of a button) in order to be able to measure the impact. If you modify both the button’s wording and colour (for example, a blue “Buy” button vs. a red “Make the most of it” button) and notice an improvement, how will you know if the wording or the colour really contributed to this performance? The impact of one change could be negligible or they each could have had an equal impact.
Multivariate testing looks to provide the solution. You can change a title and an image at the same time. With multivariate tests, you test a hypothesis for which several variables are modified and determine which combination from among all possible solutions performed the best. If you create 3 different versions of 2 specific variables, you then have nine combinations in total (number of variants of the first variable X number of variants of the second).
Testing 2 positions and 2 colours at the same time equals 4 different combinations
There are three benefits to MVT:
There are 2 main methods for performing multivariate tests:
The first limit concerns the number of visitors needed for your multivariate test’s results to be significant. By multiplying the number of variables and versions tested in your multivariate test, you will quickly reach a large number of combinations. The sample assigned to each combination will be reduced proportionally.
Where, for a traditional A/B test, you would assign 50% of your traffic to the original version in the tool and the rest to the variant, you will only assign 5, 10, or 15% of your traffic to each combination in a multivariate test. In practice, this often translates into longer tests and an inability to reach the statistical significance needed to make a decision. This is especially true if you test pages deep within your site with low traffic, which is often the case in order tunnels or landing pages for your traffic acquisition campaigns.
The second limit is linked to the way the multivariate test is defined. In some cases, it’s the result of an admission of weakness: the users don’t know exactly what to test and think that by testing several things at once in a multivariate test, they will eventually find a solution they can take advantage of. We then often find small changes at work in these multivariate tests. A/B testing, on the other hand, requires great rigour and helps better identify test hypotheses, which generally lead to more creative tests, backed up by data, with better results.
The third limit is related to complexity. Conducting an A/B test is often easier than a multivariate test, especially when analysing the results. You don’t have to do complex mental gymnastics to try to understand why a particular element interacts positively with another in one case but not in another. Keeping the process simple and quick to perform helps maintain confidence and rapidly reiterate on optimisation ideas.
