Measuring Social Media ROI: A Case Study (Plus: Tweet to Beat Winners)

51 Comments

Total Read Time: 10-15 Minutes

Have you really seen ROI from Social Media?

Social media marketing! Twitter consultancies! Conversational communications! Oh, these are exciting times.

It seems like everyone and their grandma are getting into social media. On a whole, I think this is a good thing, but here’s the problem: whenever technology becomes fashion, return-on-investment (ROI) tends to get lost in the excitement of the latest .com catwalk.

It’s going to help “the brand”? Show me data. It’s going to drive more “awareness”? Define it, isolate it, and translate it into a sales increase.

In this post, we’ll look at some real numbers (total capital, conversions, redemptions, etc.) from my latest educational non-profit campaign, the Twitter-based Tweet to Beat, which was a follow-up to the blog and leaderboard-based LitLiberation campaign, which outraised Stephen Colbert 3-to-1 with no staff and no material hard costs…

My hope is that more non-profits and hybrids (like the impressive Tom’s Shoes) will share their best practices and experimental findings so that everyone can benefit.

Here was the basic pitch from the original post:

The gist: To benefit U.S. public school students, I will bribe the entire world to follow me on Twitter for $3 each.

For every new Twitter follower in the next two weeks, I will donate $1 to DonorsChoose.org, and an anonymous supporter will match $2, for a total of $3 to U.S. public school classrooms per follower. For now, the matching limit is tentatively capped at 50,000 new followers, though I’m open to increasing it later. 50,000 new followers would mean $150,000 to U.S. public school education, and I hope to double or triple this total with a few twists.

The goal is directly helping 25,000 U.S. public school students in low-income and high-need areas in two weeks. This timeline is half the time dedicated to LitLiberation. My current follower count is, at the time of this writing, 22,782, so we’ll round down and begin the count at 22,500.

How many followers did I get (hint: about 17/hour), and how much was raised? We’ll get there, but let’s start with the fundamentals.

First, Fundamentals and Hypotheses

I run experiments for one of two reasons: 1) to produce results I feel comfortable predicting, or 2) to gather data and findings I can then (hopefully) incorporate into follow-up experiments. Let’s call the latter an “investigative campaign”.

Tweet to Beat was an investigative campaign, as I was less comfortable with large-scale predictions about Twitter user behavior. I have mountains of data and replicable outcomes from the blog, but I did not have either for any micro-blogging platforms.

That said, it is important to begin with a hypothesis or hypotheses (predictions) that you will test. Why? Because backwards correlation is bad science.

If you gather enough data and measure enough variables, you will inevitably find chance correlations that you — in lieu of predefined hypotheses — will want to label as causal (“A causes B”, when these are coincidental). For more in-depth discussions of proper study design and fascinating phenomena like negative publication bias, I highly recommended Bad Science by Ben Goldacre.

Decide what you’re testing upfront (e.g., “If A, then B”) so you don’t succumb to wishful thinking and forced causation. This is also true for online analytics.

For Tweet to Beat, there were just a few assumptions I wanted to test:

1) If I redistribute monies raised, it will increase the per-person donation amount. In other words, if each person helps raise an average of $3 and I return that $3 to each person in the form of a coupon, some percentage of those donors will give more than $3 when the coupons are redeemed.

2) This can be done at close to zero cost. Tweet to Beat was done in partnership with DonorsChoose.org (DC), just as LitLiberation was done in partnership with DC and Room to Read. Since their labor costs aren’t zero, the value of the campaign (whether in amount raised or actionable data) cannot be less than the ROI from investing time in other fundraising or activities.

3) People will participate more if just a single click is required. LitLiberation required fundraising or donations, whereas Tweet to Beat would require nothing more than clicking on “follow” in Twitter.

The Findings

Ultimately, the Twitter follower count increased from the 22,500 starting mark to 31,739, for a total follower increase of 9,237 in two weeks. The total donation (x $3 per new follower) is then $27,717. Therefore, the third hypothesis about more engagement with less required action does not appear valid; at least, it is not independently sufficient to overcome other variables that might cause resistance.

So what about increased yield from distributing donations to donors themselves? The announcement of the coupon redistribution was saved for a second-wind PR effort, as I’ve found phased announcements to increase yield at least 10% in the past with third parties.

In post titled “Upping the Ante”, the campaign was also extended by one week and bonuses totaling $168 in value were offered to all followers (DropBox, RescueTime, and PhoneTag).

The coupons were distributed simply:

1) Twitter followers were notified of a link via a protected update.

2) Interested followers clicked on the shortened URL, which took them to a Google Form asking for their e-mail address, to which the coupon code would be sent. The Google Form took 60 seconds to set up and is free.


Google Forms: Simple and Free

3) DonorsChoose automatically generated as many codes as needed and emailed out welcome letters to each donor, including their coupon code and a link to a Project Page where they could apply it to the classroom project of their choice.

4) Donors were shown via screenshots how to use Facebook Connect to share their donation with friends via Facebook status update.

There were two batches of e-mail coupons sent out. The first was a link to a $3 coupon, available to approximately 30,000 followers, while the second was a link to a $12 coupon, available to just the first 1,000 takers.

Batch 1 (Wed. 3/25, 4pm EST, $3 Giving Codes)

To 30,000 Followers, Open to All

Total Sign-ups (coupons sent via e-mail): 1,108 (3.69% conversion)
Total Redeemed: 666 (~60% redemption)
Total Donated: $1998
4 day expiration (March 25 – 29th; Wed. – Sun.)

Batch 2 (Fri. 3/27, 6pm EST, $12 Giving Codes)

To 30,000+ Followers, First 1,000 Respondents
Total Sign-ups (coupons sent via e-mail): 583 (1.94% conversion)
Total Redeemed: 352 (~60% redemption)
Total Donated: $4224
4 day expiration (March 27 – 30th ; Fri. – Mon.)

The below block quotes, and all block quotes from this point forward in this post, are from DonorsChoose with some edits for space:

Total Dollars Allocated by Tim Ferriss’ Twitter Followers in the Tweet to Beat Challenge: $6222 (FYI: this is higher than the number on the project page — $5821 — because some of the participants may not have redeemed directly via this Giving Page).

But how much more did TweetToBeat redeemers spend in addition to their GivingCards? Total to date: $1868.34

Thus…

Grand Total Dollars Allocated by Tim Ferriss’ Twitter Followers in the Tweet to Beat Challenge: $8090.34

The below spreadsheet offers a bit more granularity on how many people decided to add onto their donations and the average size of donations by GivingCard denomination.


Click here for a larger version.

This is the eureka data set I was looking for. Things to note:

-The % of $3 donors who gave more than coupon value was higher than those in the $12 donor group.

-The average donation amount for those who added was greater in the $3 group than in the $12 group: $37.65 vs. $36.72. Keep in mind that the additional differential from the $3 group ($12-3 = +$8) makes the average amount $34.65 above par vs. $24.72 for the $12 group.

-In sum, based on this limited sample size, we were able to increase total donations 30% ($6,221.66 –> $8,090.34) simply by distributing the amount raised back to supporters vs. writing a check to DonorsChoose directly. What this means: enable your supports to “recycle” what you raise and give directly, and you could add 30 cents to every dollar you’ve already raised.

There are no doubt problems with the data, like the limited sample size, different days of the week for mailings, and our inability to isolate outliers or measure overlap between the $3 and $12 groups, but this preliminary data is both counterintuitive and testable. If you want to throw caution to the wind, as the downside is next to nothing if you automate coupons, it’s actionable.

The best part is that this ROI is based on one campaign and not Lifetime Value of each new registered donor. If we factor in the below from DonorsChoose, we see that getting $1.30 for each $1 distributed could well be a minimum assumption, with $2 – 2.60 ROI for each dollar distributed on the high end.

Doing these calculations is as important for for-profits as it is for non-profits, of course. Determining your “contained” LV over 3 or 6 months, for example, helps determine what you can spend to acquire each customer while still remaining profitable.

On Lifetime Value (LV or LTV):

Unfortunately, we don’t have a ton of historic data around LTV since we’ve just recently begun capturing these figures. That said, a recent analysis which sought to look at class of donors to determine how much the group would donate in subsequent years, suggests an appropriate multiplier that we could use for this case. This conservative multiplier would be 2. Obviously, there are some outliers like donors who give large sums of money, but our data suggests this is pretty close.

Ex. If an average class of donors gave $100K in year 1, we can expect them to give $200K in ALL their subsequent years (~5-10 yrs) as a donor. While this doesn’t necessarily equate to what the average ‘individual’ donor would give, it does give us a general benchmark with which we can guesstimate LTV. Also note that the time frame that we’re using 5-10 yrs is based on where the data is trending towards, but not 100% exact since we’ve been in existence for less than 10 years.

In Summary, from DonorsChoose:

In terms of takeaways, I think this was a huge success for a number of reasons in addition to funds raised:

1) We acquired 300+ followers (~42% of our total followers) over the course of the campaign (from when it first launched to when the last batch of codes were mailed), lending mounds of exposure to our very nascent Twitter account

2) Even two weeks after the campaigns conclusion, your fans are still talking about it. [Note from Tim: Thank you to Jonathan Hinson for setting up this tracking page for T2B — as another data point for building Twitter followings, it shows my current follower add rate to be 17 per hour and my current Qwitter rate as 7 per hour, for a net gain of 10 new followers per hour.


From Jonathan Hinson’s Tweet to Beat Mash-up

Areas for improvement:

1) We should brainstorm further internally about how we can allocate deliverables quickly for campaigns over social networks. These are a lot more fast-paced than the Blogger Challenge and there was a sense of immediacy that we were not 100% prepared for.

2) We should have implemented an official hashtag from the outset (like #tweet2beat) for this campaign so as to widely broadcast it in the Twitter community (i.e., trending topic).

Bottom line: I think we were all really pleased by the results and are anxious to do something similar on Twitter in the near future. We’ve already been talking about using Twitter as part of our social media strategy for our upcoming “Give-Back” Birthday Campaign [From Tim: read more about this here]. Trust that we’ll be keeping an eye on participants who redeemed the GivingCards and reveal themselves as ongoing donors.

Tim’s Conclusions

$29,585.34 ($27,717 + $1,868.34 additional donations) over three weeks is a non-trivial amount, but it was dramatically less than expected, and the LitLiberation response dwarfed it (more than 10x this to date). There are several reasons, subject to further testing, why I believe this was the case.

1) Everyone who participated in LitLiberation was eligible to win large prizes, and their progress up or down was publicly visible. Tweet to Beat did not make everyone eligible for prizes, only those who chose to compete as fundraisers, and there was no public accountability or recognition.

2) Using Twitter follower count seemed self-serving. Though follower count is nice, Twitter was chosen intentionally for several other reasons: one-to-many broadcasts via protected updates, a public counter for transparency, and the expected backlash from purists, which would drive additional links and participants. Alas, the public backlash never came en masse, and the net-effect seemed to be less engagement due to convolution of the campaign goals and self-interest. Powerful partners also couldn’t commit publicly due to this. Much of this could have been avoided with a stand-alone site (vs. a post on my blog) like LitLiberation.

3) It was too complicated. More specifically, there were to many incentives for different groups. Once again, simple is better.

So where to from here? Simple: don’t repeat the above mistakes, combine LitLib-like leaderboards with the redistribution force multiplier, and test, test, test. The next campaign will be bigger than both of these combined, a melding of best practices from the two, and it will be — in part — thanks to a “failed” experiment that yielded extremely valuable data.

Remember: the best vehicle for testing (gathering actionable data) is often not the best vehicle for roll out.

Isolate your variables, test predefined predictions, and — please — share your results. Don’t expect the p-values to be ideal, but, in a world of surprisingly little measurement, good experimental design can make the difference between pure guesswork and educated sniper shots.

###

Tweet to Beat winners!

My sincerest apologies for the delay with this reporting. 452 comments and submissions kept assistants under water for a bit.

* Best results: Jonathan Kramer — His bit.ly link had more than 33K clicks! John wins the grand prize, a roundtrip anywhere in the world. John, keep an eye on your inbox.

o First comment posted on March 9th, the first day it was possible to do so.
o “Tweeted about it http://twitter.com/kramertron/statuses/1301594140 … Already got retweeted before I got a chance to finish this comment!”
+ [Note: This is a good example of reaping the benefits of posting the first working bit.ly link — other people used it as their own, and the clicks added up very quickly.]

* Best effort: Miltownkid — Created a few microsites and videos to promote the campaign. He also gave away his Xbox 360 as an incentive to get more people following me on Twitter. Miltownkid, keep an eye on your inbox for your runner-up prize info, a fully-loaded 15″ MacBook Pro.

o First comment (of six) posted on March 9th
o “What I did is run a contest giving away my Xbox 360: http://mydeadxbox.com/
“The goal is to get the most followers for you BUT the people are suppose to be working together as a team and the person voted most helpful to the cause will be the winner. What I did is take things out another notch and made custom pages for all the early contestants. Just like my page, but using their name. Like this one: http://mydeadxbox.com/contest/LaughingLeaf.html
+ [Note: His video on how to participate in Tweet to Beat was seen approximately 1,800 times. Only 250 clicks through bit.ly]

* Most potential exposure: Nicolette. The crux word here is “potential”, as there were no links to this feature, and it could not be found on the CurrentTV website. Nicolette, all the same and for the attempt, please look for an email with a $217 gift coupon to DonorsChoose.

o Comment posted on March 12th
o “I just got the word out to 59 MILLION homes! I’m a producer at Current TV which reaches viewers in the US, England, Whales, Ireland and Italy. After pleading, I was able to get a mention on our headlines show “Current Tonight” which airs in almost 59 million households world wide. It will air tonight at 11PM EST, 8PM PST. Look out for headline #12: Twittering for a Better World.”

Other Random Goodies and Happenings:

Tim Ferriss on Weebly – this is a new site I whipped up in about 1 hour to show how fast it can be.
Tim Ferriss interviewed by Leo Babauta of Zen Habits – I just found this in-depth interview I’d forgotten all about. Some fun stuff discussed, including:

– What started me on the road to 4-Hour Workweek
– My biggest inspirations
– My daily routine (in detail)
– How I set up my environment for writing
– Why it’s better to control environment than behavior
– How I figured out my most productive time of day
– How technology tools can make it easier to be distracted
– How I motivate myself when I don’t feel disciplined

Posted on: May 11, 2008.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Comment Rules: Remember what Fonzie was like? Cool. That’s how we’re gonna be — cool. Critical is fine, but if you’re rude, we’ll delete your stuff. Please do not put your URL in the comment text and please use your PERSONAL name or initials and not your business name, as the latter comes off like spam. Have fun and thanks for adding to the conversation! (Thanks to Brian Oberkirch for the inspiration)

51 comments on “Measuring Social Media ROI: A Case Study (Plus: Tweet to Beat Winners)

  1. Heady material, Tim. You know how to run a campaign!

    I will do a re-read in consideration of a non-profit I encountered last week in Oaxaca, that could really use some assist. May need to streamline what you have; thanks for speaking from experience which assists.

    Congrats on the results!

    Now I wanna know where Jonathan goes on his roundtripper.

    Lani Muelrath

    Like

  2. ….Oh, i almost forgot to add.

    Current Economy v.s. Economy(when LitLiberation took place), could this have been a cause, and if so, assuming one could test that measure, what would the metrics be?

    Jose

    Like

  3. This is a really solid breakdown of your two campaigns into a case study that I think any person/organization can benefit from entering the land of social media.

    Particular congratulations to Jonathon Kramer, Nicolette, and Miltownkid. Excellent work to help spread it.

    Also, I really appreciate your efforts with charities, social change work these days Tim. It’s inspirational.

    Like

  4. Tim,
    1. People need to be incentivized in a way that gut punches them personally. Most like large sums of money.

    2. People like short term ROI. Promising heaven upon their passing is not nearly as appealing as a 1 in a billion chance to win the lottery this Thursday.

    Like

  5. This is very interesting data.

    On a personal note, I just want to thank you. One of my projects was funding almost completely from “Tweet 2 Beat”.

    If Nicolette needs a project to support, I hope she’ll let me know!!

    Like

  6. Something that is not addressed here, or in the comments sections of “Tweet to Beat” or “Let’s Up The Ante” is the distribution of the RescueTime, DropBox, and PhoneTag coupons. As of now, I have yet to hear anything about them and I know a few other people have mentioned it as well in comments. While I certainly feel great about donating and would have helped the cause regardless, I don’t understand why I never got anything.

    I may have overlooked some critical info somewhere, a lot of the times you or commenters talk about receiving a “coupon” and that could mean the coupon to donate or the coupon for the discounts, so maybe I overlooked it, but maybe you could clear this up for us. To clarify, I DID get the Donation coupon and donated $3 in the campaign first round, I DID NOT hear anything about the Rescuetime, PhoneTag, and DropBox coupons. Thanks!

    Like

  7. Way to make a difference! Thank you for this valuable (and free!) case study. Metrics are a challenge for me (I’m more of a language guy), but I am determined to understand them better so I can get my message out more effectively.

    I have had people retweet my links, going to thousands of people, and yet when I checked my analytics, it showed only the tiniest spike.

    I love branding as much as the next guy, but it can’t always be about branding, right? Sometimes, we want to actually draw people to our site. I’m glad thousands of people read my tweets, and I’m even gladder when they click on a link and come for a visit!

    ~Dane

    Like

  8. Tim-

    Fascinating results, I specifically enjoyed how you thoroughly explain the process you go through when setting up an experiment like this. You’re obviously very analytical and intentional with the process and I have a lot to learn from that.

    My question for you is, why? What is the intended purpose of an experiment like this? What do you get out of it? Personal interest? Perhaps building a case for advisory with another not-for-profit or just excitement over the data/gathering methods? Winning a bet? I’m very curious of your overall goal of this experiment.

    Cheers,
    Ryan

    Like

  9. Hey Tim,

    You have taught me to Micro test everything (and I do). Why assume anything “define it, isolate it and translate it into sales”. I love it! Simply jumping on the next social media band wagon without empirical data is a waste of time. Nice work!

    Best,
    Rob

    Like

  10. Tim, great post and very interesting analysis. Will certainly refer back to this often…but I gotta imagine this took you more than 4 hours to write this article this week! ;)

    Like

    • @Zaw,

      LOL… this post did grow into a bit of a beast! I started out with “this should take about 25 minutes” and then I got on a roll. Happens to us all sometimes :)

      Best,

      Tim

      Like

  11. 3) It was too complicated.

    I think thats what did you in. I took the plunge and signed up for twitter because of your campaign but going through the remaining set of steps was too complicated/time consuming. I’d stick with the KISS principle.

    Like

  12. Hi Tim,

    Utterly fascinating experiment…I took part in this and received & used the $3 coupon as well as the $12 coupon…I’d like to add what I feel I learned from this study, if I may, as follows:

    1. One of the reasons Tweet to Beat may have raised less than LitLiberation, is the overwhelming choice of donation recipients once at the DonorsChoose website; as per the business marketing principle that ‘too much choice creates indecision & delays or even prevent the sale’. I know I was certainly surprised and overwhelmed enough to delay donating, because I couldn’t choose from the huge number of equally worthy recipients.

    2. In your conclusions, you mention LitLiberation raised more than 10 times to date. This also raises the point of longer term involvement, and along with the counter intuitive results regarding your 3rd hypothesis, makes me wonder if requiring more from your participants leads to more long term engagement, and therefore more results over a longer period of time (more sustainable return)?

    3. Alternatively (or also), did asking more of your participants, combined with the added motivation provided by the extra incentives & accountability you mention, pre-qualify the recipients (as per the marketing principle of focusing on ‘qualified prospects’ to increase ROI)? I haven’t looked at your data for LitLiberation, but was there more money raised from a smaller number of people over the initial 3 weeks?

    Of course, my comments above, are not based on microtesting, or data – just thoughts (hopefully insights) based on your article that may lead to new hypotheses to test.

    Like

  13. @patrick I received everything promised…

    Maybe I am too tired for this article… but I didn’t see where a real ROI (in a for-profit sense) was addressed?
    Thank you for the time, energy, and thought you put into your work.

    Like

  14. Hi All,

    To one question about ROI — this post looks at rationale design for testing and the evaluation of multiple levels of conversion, cost-per-acquisition, and lifetime value, all of which can be applied to ROI testing in for-profits. In this case, the key metric is $1 distributed bring back a 30%+ ROI at no additional cost.

    @Patrick, the DropBox and other coupons (links to free sign-ups) were provided via Twitter, which is the only way I would be able to communicate with followers. The links were up for a few days and then taken down to prevent problems with link forwarding.

    Hope that helps :)

    Tim

    Like

  15. This is a timely post for me Tim. We’ve been watching everyone jump on the SM train for the last few years and experts coming out of the woodwork left-right-and centre. But I worry that many are just taking advantage of the hype and excitement around the rush to social media without providing any meaningful ROI.

    At your Sydney party I asked you your opinion regarding how much social media had contributed to the success of 4HWW. I was interested because I know you like to analyze the numbers.

    Once again with this exercise you are using the tools well and delivering (and showing) an ROI.

    If only all marketers could do that…..

    Like

  16. This question is unrelated to the post, but I couldn’t find a recent blog post that it could relate to and I really wanted your opinion about this! I’m sorry for the off topic question.

    I spend about 10-12 hours a day in front of the monitor. I’ve recently read about negative side effects that this can cause, fatigue, headaches, sore and red eye, etc and I’ve been suffering from each of these symptoms. I’m not sure to what extent it may be caused by my excessive PC usage but I’m sure that to some degree it is.

    On another forum post that was discussing this someone said that they wear some type of necklace that helps to eliminate the EMF (electro magnetic fields) around the body. I went to the website and the description essentially hypes it as the answer to eternal life. I’m not convinced, but it also made me wonder if there is anything (aside from going out more lol) that can negate the negative health effects of monitor/PC use and perhaps combat sleep/headache and fatigue problems that come along with it (and cause huge impairment for me)

    Is there anything you or anyone else can recommend that helps with this? I’m sure many of us are guilty of PC overuse and could use some suggestions to protect ourselves from the harmful psychological/physical effects of this.

    BTW, this is my first post here but definitely not my first time on the blog. This is easily the best and most diverse blog out there and a great companion site to an already complete book.

    Like

    • @KattleKod,

      Thanks very much for the kind words and extended comment. The quick answer: I hope to do an entire post on this in the next 4-6 weeks. Please be patient and remind me if you don’t see something by end of June, but this is on the schedule.

      All the best,

      Tim

      Like