Google Website Optimizer Case Study: Daily Burn, 20%+ Improvement

120 Comments

This post will show exactly how one start-up improved their homepage conversion rate (visitor to sign-up flow) more than 20%, then 16% again, with a few simple changes and Google Website Optimizer.

Once reading this, you will know more about split-testing than 90%+ of the consultants who get paid to do it…

There are a few advanced concepts, but don’t be intimidated; just use what you can and ignore the rest.

Along with Founders Fund (Dave McClure), Garrett Camp (CEO, StumbleUpon), and others, I am an investor in Daily Burn, one of the premier diet and exercise tracking sites.

Following investing, first priorities included introducing them to Jamie Siminoff, who taught them how to purchase the domain name for DailyBurn (Jamie’s method is described here), and look at their conversion rates for the homepage and sign-up process (sign-up flow to completion of sign-up). This post will look at the former, since the latter cannot happen without the former.

The first step was simple: remove paradox of choice issues.

Below is the homepage prior to tweaking. The bottom of the screen–the “fold”–was right around the second user under the running calorie counter.


Click here for larger version.

Offering two options instead of six, for example, can increase sales 300% or more, as seen in the print advertising example of Joe Sugarman from The 4-Hour Workweek. Joe was, at one time, the highest-paid copywriter in the world, and one of his tenets was: fewer options for the consumer.

DailyBurn (DB) was two founders at that point in our conversation, so instead of suggesting time-consuming redesigns, I proposed a few cuts of HTML, temporarily eliminating as much as possible that distracted from the most valuable click: the sign-up button.

Here is the homepage after reducing from 25 above-the-fold options to 5 options and raising the media credibility indicators. Note the removal of a horizontal navigation bar. The “fold” now ends just under the “Featured On”:

The results?

Test 1 Conversion Rates: Original (24.4%), Simplified (29.6%), Observed Improvement (21.1%)
Test 2 Conversion Rates: Original (18.9%), Simplified (22.7%), Observed Improvement (19.8%)

Conclusion: Simplified design improved conversion by an average of 20.45%.

To further optimize the homepage, I then introduced them to Trevor Claiborne on the Google Website Optimizer (GWO) team, as I felt DB would make a compelling before-and-after example for the product. Trevor then introduced DB and me to David Booth at one of GWO’s top integration and testing firms, WebShare Design.

Why not just use Google Analytics?

David will address this in some detail at the end of this post, but here are the three benefits that Google Website Optimizer (GWO) offers over Google Analytics (GA):

– GWO offers integrated statistics – is new version B better by chance or better because it’s better?
– GWO splits traffic – half traffic runs to A, half of traffic runs to B (if A/B test); it also ensures, using cookies, that a returning visitor will see same the same variation
– GWO really tracks visitors – GA works on idea of a session (a person bounces around on the site for a bit and leaves, which is considered a “session”); if they return, that is generally a new session); GWO uses unique visitors (no matter how many visits, they’re counted as one visitor, assuming they don’t delete cookies). On a fundamental level, it’s the difference between visits and visitors. This is critically important for determining if your result in statistically valid, as ten people and ten visits by one person are not the same.

GA can do a lot of what GWO does, but you need to do a lot of custom work and intricate number crunching to make it work.

Enter Google Website Optimizer

The following is a report of the WebShare / Gyminee Website Optimizer landing page test, and includes a description of the test that was run as well as analysis of the test results. This report was authored by David Booth, to whom, and to whose team, DB and I owe a debt of gratitude. I’ve included my (Tim’s) notes in brackets [ ]. Don’t be concerned if some of the graphics are hard to read, as the text explains the findings.

1. Test Description

The landing page identified for this test was identified as:
http://www.gyminee.com

This A/B/C test included three distinct page versions, including the original (control) homepage as well as two variations designed with conversion marketing best practices in mind:

Original (control)

[same as simplified version above]

Variation B

Variation C

2. Test Results and Analysis

During the first run of the experiment the test saw ~7500 unique visitors and just under 2,000 conversions over the course of about 2 weeks. When the experiment was concluded, both variations B and C had outperformed the original version, and specifically Version B left little statistical doubt that it had substantially increased the likelihood that a visitor would convert, or sign up for the Gyminee service.


Larger version here.

We can see from the analysis of the data that Variation B had a large and significant effect on improving conversion rate. The winning version outperformed by the original by 12.7%, with a statistical confidence level of better than 98%. [This means there is less than a 2% likelihood that you would duplicate these results by chance, which can also be called a p-value of <0.02]

Interesting to note is that the B version, which does not have a “take a tour” button, nor horizontal navigation bar, performed a few percentage points better than their current, more polished design which does offer both.

A follow up experiment was then launched in order to provide more data and ensure that these results were repeatable. The follow up experiment was conducted as an A/B experiment between the original and Variation B, and ran for approximately 1 week, over which time almost 6,000 unique visitors and ~1,400 conversions were recorded.

The results of this follow up experiment showed that Variation B outperformed the original by 16.2%, with a statistical confidence level of better than 99%.

Further analysis concludes the following:

* The absolute difference in conversion rates between Variation B and the original during the test was 3.7%.
* During the test, Variation B’s conversion rate was 16.17% greater than that of the Original design.
* The p-value used in these calculations was <0.01, corresponding to a confidence level of >99%.

The Bottom Line: The results of this experiment were extremely successful.

Putting these test results into plain terms in another way, there is a 98% chance that the true difference between the conversion rates of these versions is between 7.8% (1.8% raw) and 24.5% (5.6% raw).

3. Supporting Analysis (A/B/C Test Only)



A Pearson Chi Square test answers the question: “Out of all the combinations, is any one combination better than another?”

The values here tell us that with >95% confidence, at least one variation was statistically better than another. This further validates the conclusions drawn by Google Website Optimizer.

Was Version C statistically better than the Original?

At an acceptable level of statistical confidence, it was not. However, had we continued to run this test for a longer time period, it is very likely that we would have eventually proven that it was indeed better than the original with >95% statistical confidence. The estimated sample size needed to prove this would have been an additional ~21,000 unique visitors (~7,000 for each variation).

The table below shows you the various sample sizes you would need at different confidence levels to show different relative improvements [Tim: this is my favorite table in this analysis]:

Was Version B statistically better than Version C?

We can be approximately 94.1% certain that Version B is also better than Version C. After applying a Bonferroni correction for the test set, we would still be >90% confident that Version B is better than Version C. The p-value for these calculations is 0.059.

Recommendations:

As Version C did test well, and we believe would have eventually proven itself better than the Original, it is very likely that certain elements of Version C resonated well with visitors to the Gyminee website.

To continue down this path of testing, we would recommend using the winning Version B as a test page for a multivariate experiment. In this experiment, we would suggest testing certain page elements from Version C in the framework of Version B.

Additionally, as testing only covered the homepage, we would highly suggest performing testing on the form found at:
https://www.dailyburn.com/signup

Many concepts such as calls to action, layout, design, contrast, point of action assurances, forms & error handling, and more could be used to increase the likelihood that a user enters information and submits the form.

Lastly, it may be beneficial to begin running tests where the conversion is measured as the paid upgrade. As this conversion rate is much lower than the free sign-up, it should be understood that all other things held equal these tests could take significantly longer to run to completion.

Google Website Optimizer vs. Google Analytics – Parting Thoughts

From David Booth, whose team performed and compiled the above:

1) GA doesn’t have any capability of doing statistical analysis to compare two groups (and it’s not meant to), but it can collect all the data you would need with the best of them. GWO records data very differently and is not meant as (and should never be used as) an analytics package. It runs the stats for you and tells you when you have a statistically significant difference between variations/combinations, but is limited to a single goal or test.

2) The real beauty is to integrate GWO with GA – this gives you the best of both worlds by letting each tool do what they were built to do. You can use GWO to create the test, split traffic, and crunch the numbers for your primary goal, and you can then pull the data out of GA on anything you have configured and run the numbers in a stats package like JMP or Minitab. A very useful case for this is an ecommerce purchase: GWO can tell you if one version / combination was more likely to get an ecommerce purchase (binary – they either purchase or they don’t), while GA data can record things like revenue, and running a different statistical analysis can tell you if one version was more likely to make you more money.

###

Related and Recommended:

Daily Burn 90-Day Fitness Challenge – Starting August 17th! Lose fat and gain muscle with better data and accountability.
How to Tim Ferriss Your Love Life

Posted on: August 12, 2009.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Comment Rules: Remember what Fonzie was like? Cool. That’s how we’re gonna be — cool. Critical is fine, but if you’re rude, we’ll delete your stuff. Please do not put your URL in the comment text and please use your PERSONAL name or initials and not your business name, as the latter comes off like spam. Have fun and thanks for adding to the conversation! (Thanks to Brian Oberkirch for the inspiration)

120 comments on “Google Website Optimizer Case Study: Daily Burn, 20%+ Improvement

  1. Great post with detail on the art of testing for real results, rather than just for testings sake.

    Did you go on to use mixpanel or crazyegg for testing this site as well? results?

    thanks.

    Like

  2. Thank you for the post. Will have al look at GWO and Daily Burn as well. My next project will be tested a little bit more I think.

    Every time here is an inspiration.

    Like

  3. GWO is certainly a valuable tool. Unfortunately it’s only useful for testing static HTML which I’ve often found limiting

    To test dynamic content that is generated by code, I still usually have to code up my own solution. Here is an example I did recently on a Ruby on Rails app:

    http://www.startbreakingfree.com/1003/results-of-universitytutor-com-price-testing/

    If anyone knows a better way to test dynamic content in GWO I’d be curious to hear more.

    Thanks,
    Brian

    Like

  4. This was excellent, Tim and David. We use both GA and GWO at AwayFind but weren’t aware of the depth of statistical analysis that can be drawn from the results. Additionally, that table that you liked will be stapled to our wall during our next launch.

    Much appreciated!

    Like

  5. Thanks for posting such an in depth example of GWO. I checked it out after your video about getting web traffic to your blog without killing yourself. I’ve been wanting to use it, but haven’t had any changes to make to the site worth measuring yet.

    Like

  6. That is fascinating, it can be very difficult for any business, offline or online, to test how changing the product and design affects the consumer. This is a long way from the days that we had to do polls or just go by our gut. I anticipate being able to use this on my own site someday.

    Like

  7. Ohh fantastic information, I didn’t even know how the confidence levels worked before now :p

    I personally use the Conversion Chicken (www.conversionchicken.com) for my testing as it does full proper multivariate testing (where it tests how well variables interact with each other to get much more accurate results) while GWO does not.

    But Conversion Chicken costs a little bit of money (only like $40 a month so still much cheaper than hiring consultants), so I guess GWO is fantastic for being free (and if you only want to do normal a/b split testing).

    I can’t believe how many people don’t test, it’s like 20 – 50% more sales / signups in a week and takes an hour or so to set up.

    To get 20 – 50% more traffic to your site could take anywhere from 3 – 6 months yet people focus on more traffic instead! really odd.

    Like

  8. Hello, Tim and David.

    A little advanced compared to what we see in the current blogs. Thank you for this excellent article and keep staying ahead of the pack!

    Could you give us other case studies, such as a website with a call to action: (pick up the phone?).

    Like

  9. Thanks for this post Tim. P values and Chi Squares! I thought I was the only one that was this anal about analytics! You simply talked about the stats here, but I’m wondering, simply based on a visual ‘hunch’ did you know option B for instance was going to convert higher? If so what do you think made the difference between them.

    I ask this as I had a massive jump in conversions (from around 1.5-4.0) by just changing my buttons from (Order Now) to (Next Step). I of course used multivariate tracking but I already knew I was going to get a massive jump.

    Like

  10. Awesome, I was waiting for this post ….. I recall you had hinted about this in previous posts. This is a great morning post to read with some coffee. Thanks for the sweet tips.

    Jose

    Like

  11. Awesome breakdown. Thanks Tim, this is where you shine. You’re focus on the details, but method in conveying them has been honed for clear communication. I’ll be “evernoting” this for when I know that next big site I’m working on needs this type of breakdown.

    Since you’re close to the matter…personally I wish I could fly to California (assuming the heads of Gyminee are there) and sit down with the CEO of DailyBurn and let them know why I’ve tried to use the site earnestly 3 different times now (once because of a igoogle app, twice on your recommendations on it), and each time go back to dailymile http://www.dailymile.com …. for 6 months consecutive I’ve been able to login into dailymile, log my workout and get out in under 2 minutes. I just can’t do that with DailyBurn. The excel export, tracking, and facebook/twitter connect are nice too. Thoughts?

    Like

  12. Ah man, Great stuff! I love posts like this!

    I really like GA over the analytical tool provided by my hosting company. I have been looking for more in depth analysis lately because I’ve been playing with the layout of my site. I usually track visitors and not visits.

    I can’t wait to try it out!!!

    Like

  13. Multivariate testing is very cool, thanks for sharing your experience with this Tim :) If clients were only patient and understanding enough to allow us to do this work for them. For the time being, we try to second guess what the best layout is for the best conversion – we found that adding any form of video and testimonials is a big help, not to mention reducing form fields.

    Like

  14. Excellent post! Most people get stuck on “traffic”. But traffic is secondary to conversion. The only way to increase conversion is through testing, testing and then testing again.

    It doesn’t matter if it’s UGLY as long as it converts.

    Like

  15. geek nitpicking: you might get better click-through rates for geeks if your page works with javascript disabled (ie. using noscript in firefox). I see a big blank box in the middle of the dailyburn front page.

    Like

  16. This is great. We’re in the process of moving our site from one hosting service to another and we wanted to remove all from the website that wasn’t working and enhance our conversion rate. We just weren’t sure how besides staring at GA.

    Thanks so much! This is invaluable!

    Like