1,2,3 – It’s As Easy As A/B/C Testing

Author: Gab Goldenberg

A/B TestingI was emailing a prospect recently who mentioned that a competing firm had proposed doing A/B multivariate testing. If you’re familiar with the jargon of testing different ads/landing pages, you would know that A/B testing is different from multivariate testing. I can’t blame the prospect or my competition however, because ours is an industry enamoured with jargon and it sometimes gets me confused too! In any case, let’s see what A/B testing and multivariate testing really mean, and the implications for your test results.

(Bucket testing image by mil8.)

A/B testing is defined as testing different values for the same variable. For example, suppose you have three landing pages, called A, B, and C. You can carry out A/B testing by changing the titles on each landing page. The variable there would be the title, and each the A’s title, B’s title and C’s title would each be a value for that variable.

It isn’t about having only two versions and comparing one to the other. You can do A/B/C testing or even go super deep and do A/B/[…]/Z testing! A/B testing is also sometimes referred to as split testing or A/B split testing because you split your traffic (or your “sample,” if you’re testing something offline, like in-store displays)

Multivariate testing is defined as testing multiple values for multiple variables, simultaneously (hence the name “multivariate”). Let’s say we’re still working with pages A, B and C. Instead of just testing different titles, we’d also try different colour ‘buy now’ buttons, emphasizing different benefits in the copy and displaying different testimonials.

This is all done at the same time. If you tested each of these variables separately, you’d really just be carrying out A/B testing!

To give a human analogy, A/B testing is like having two brothers growing up side-by-side. They’re raised exactly the same way and are identical in every way, except that they play different sports. You’d be testing for the result of being enrolled in each of those sports and see who got a better job, or earned more money, for example.

Multivariate testing would take those brothers’ children, and have them being raised very differently. The variables there would be the brothers’ different wives, the ways they were being raised, their environments etc. You could still test to see who got better jobs and/or earned more money.

As this example shows, there’s a big difference in what you can learn from the results of these two types of testing.

With the two brothers, we know that since everything else was the same, any difference in job quality or salary is directly attributable to the sports they played. If the soccer-playing one got a better job, then you know that raising a kid to play soccer is better for the child than raising him on say, baseball. (Statistical significance is another issue for another day.)

With the various cousins, we can make educated guesses about what caused the differences in job quality and salary. Perhaps brother A’s wife worked in the recruitment industry and helped her kids more than her sister-in-law could help her kids. Perhaps the neighbourhood where brother B and his wife raised their kids didn’t have as good schools. You can make good guesses about why you got a particular result, but you can’t be certain.

Note: To my more advanced readers, I’m vaguely aware that depending on your sample size and how you play with the numbers, you can get a very high degree of certainty (if not 100%) with multivariate testing. My point above is made at a basic level, and is the main distinction between A/B testing and multivariate testing.

Update 1: Sphinn button is broken, but you can [should? ;)] sphinn the story here. Also, I forgot to point it out, but if analytics are your bag, check out these 3 new metrics for measuring social media and this post on messing with competitors’ analytics.

Update 2: Tim from Convert Offline gets our first dofollow of the day :). He’s a local hound apparently. Me too! Miriam’s going to interview him too, so keep your eyes peeled! (No, that doesn’t mean that you should put a sharp metal implement near your cornea. :P)

Update 3: More link loveƂ abounds! The very succesful Brian Chappell gets his and Hannah Smith gets some for the Gravy Train and her post on rogue Adwords due to personalization.

Please let me know in the comments whether this helped clear up to you what A/B testing is vs multivariate testing. Also, please do comment about how you’re using A/B testing and/or multivariate testing and any case studies/experience you might have in the matter! Testing is a fascinating topic and while I know a little bit about it, I’m sure many of you could teach me more! Good comments get dofollow links, as usual.
Liked the post? Get my rss feed.

Tags: ,

Sidebar Story


  1. I was just reading a "For Dummies Book" about this stuff and this post was clearer. Thank you I got it. I had it backwards before.

    Comment by Local Hound - February 11, 2008 @ 10:11pm
  2. My pleasure Tim! That's a big compliment too, so thanks a bunch!

    Comment by Gabriel Goldenberg - February 11, 2008 @ 10:51pm
  3. From my understanding, you want to use a/b testing when your site does not get much traffic, and vice versa for multivariate testing. I found a great multivariate analysis excel tool to help you calculate when you have enough statistical evidence to equate one sample is the clear winner. Email me if you want me to send it to you. Don't think I am allowed to post the link.

    Comment by Brian Chappell - February 12, 2008 @ 11:55pm
  4. Nice post - I think it's really important to explain the jargon - particularly for clients. Did you know that A /B testing orginated in good old fashioned print advertising? Half of the print run of publication contains with one variant of the ad, and the other half of the print run contains the other variant. I'm in danger of showing my age, no? :)

    Comment by Hannah - February 13, 2008 @ 11:53am
  5. Brian, thanks for sharing that. I honestly had no idea that that was the case. I guess the idea is that you want to leverage your greater traffic to run more tests in a shorter period of time? I'd love to see that tool anyways. Hannah, that's a fascinating point. I thought it was always run with a single version of the ad! I guess it would offer different coupons, for example, to test the response? And no, no worries about showing your age. In any case, both have you earned yourselves a couple of links; I'll update the post a little later today when I get a minute.

    Comment by Gabriel Goldenberg - February 13, 2008 @ 12:13pm
  6. You around Gab? I got a question or two for ya about *the project*, and you havent been around lately...

    Comment by SlightlyShadySEO - February 16, 2008 @ 10:27am
  7. Hmm, seems a lot easier to track A/B testing with paid search than SEO because the new versions of the page/category templates need to get reindexed before the search engines decide on providing you with additional natural traffic or turning off the faucet on you.

    Comment by Billy - November 24, 2008 @ 5:37pm

Leave a Reply