Google Website Optimizer is a very capable testing platform allowing you to setup complex multivariate experiments using a simple, intuitive interface. So what’s the problem?
AVERAGES LIE
Simplicity, while being Website Optimizer’s greatest strength, is also a tremendous weakness when it comes to data segmentation. Google Website Optimizer doesn’t offer the data granularity that Google Analytics does.
In Website Optimizer all of your traffic, no matter how different the traffic source, is lumped together into a single bin. You might be sending traffic from paid search, organic, and content to the same page. When you view data for an experiment on the page, you’ll see aggregate data across all traffic sources…
TEST RESULTS WILL DIFFER ACROSS TRAFFIC SOURCES
When we run a Website Optimizer test results at ROI, we manage our experiments just as you would expect – running a test until we have a winner within the Website Optimizer interface. However, when the test is completed, that’s when things get interesting.
WHAT DOES A WIN REALLY LOOK LIKE?
We use Google Analytics to dive deep into our Website Optimizer test results, breaking up the test data by traffic source, campaign, ad group, and any other significant way we can segment traffic. When you view the results on campaign by campaign basis, things start to look a little different.
Here’s an A/B test we recently ran for one of our clients testing a revamped headline on their paid search landing page:
Sweet – a 60% win overall! But our job isn’t over yet…
When we break the test down campaign by campaign in Google Analytics, we can see that the new headline actually led to as much as a 40% decrease in conversion rate in some of our campaigns (Seeing A/B Experiments in Google Analytics). So what should you do about it?
ELIMINATE THE LOSSES
Well, the short answer is to segment your traffic. Typically, when people have a win in an A/B test they send all traffic to the winning page. When you do this, you understate your win.
Take the following example. We run a Website Optimizer test and find after a week of testing that Page B gets 100 more conversions then Page A. Accordingly, we send all traffic to Page B for a 6.45% improvement.
However, if you look closer at the data from our test, you’ll see that Page A actually performed better than Page B in Campaign 2. If we optimize where we send our traffic, we can get an extra 50 conversions bumping our actual improvement up to 9.68%!
I GET IT, HOW SHOULD I SEGMENT MY TRAFFIC?
There are a number of different ways to send visitors from different traffic sources to different pages. By far the easiest method is simply update your destination URLs to point to an optimized page for each traffic source.
Really, it doesn’t what method you choose to split your traffic to the appropriate page. Just make sure whatever system you go with is robust/manageable enough to handle ongoing testing.
