Measuring Offline Advertising Effectiveness (Advertising Elasticity)
Intro
Disclaimer - the statistical techniques in this article can be a bit tricky. If you have never studied stats before you may want to consult the help of somebody who has in order to get the most out of the techniques described herein.
This article explains how I helped a friend of mine measure the effectiveness of offline advertising for his online business. In technical terms, what we are measuring is the advertising elasticity.
Most e-commerce businesses have built their customer base using mainly online methods and most people in the e-commerce business understand that the return from online marketing methods usually exceeds those from offline marketing. However, not every potential customer is online and even ecommerce businesses can benefit from some offline marketing.
In this case, we were experimenting with advertising on the radio. Radio is often the easiest place to start because the ads are easy to produce and usually can be produced by the radio station, who keep talent on staff for reading ad spots. All you the advertiser have to do is write the copy, choose the channels and times that you want and, of course, measure the results to ensure you are getting a return.
We ran the ads over 8 months on 11 channels, spending about $160,000. We knew the ads were having an impact because he noticed an increase in orders on the days when the ads played, but he didn’t know exactly how big the impact was and he didn’t know if the advertising was helping him financially. We began by gathering data.
Gathering Data
We expected to see the impact of the ads on daily sales, so we exported each days sales for the past 1.5 years from google analytics (you have to have ecommerce tracking turned on in google analytics to collect such data). We also asked the radio company we were advertising with for the full history of ads that we had played. They were able to provide that data in a spreadsheet, which we were able to combine with the .csv file exported by google analytics, so we had our data all in the same place. The data, which has been scrubbed to protect the privacy of my friend, is available in the attached excel spreadsheet.
Multiple Regression
Multiple regression is a statistical technique that can be used to determine how several variables, called independent variables (ex. Number of ads played on that day, channel the ads were played on, etc.), impact one variable, called the dependent variable (ex. Daily sales). The details of multiple regression are complicated, but in practice it is a pretty easy tool to use, can be done within Excel, and it is commonly taught in high school and college statistics classes.
Have Enough Variability
Multiple regression works by observing the correlation of a change in one variable with changes in another variable. If the independent variable that you are trying to measure doesn’t change very often (i.e. if this business advertised the same amount every day), then you will not get reliable results. Fortunately, in this case, there was lots of variability in the advertising. Sometimes this business ran 10 ads / day and sometimes they would go a full week without advertising, creating a useful natural experiment (Steve Levitt on why businesses should run experiments).
Building the Model
For a multiple regression to be truly correct, we should try to make sure every variable that impacts the regression is included. The reason for that is to avoid confounding variables. For example, in this case, the business tended to advertise more around the holidays. Sales also tend to rise naturally around the holidays. So, if proximity to the holiday is not included in the model, but the amount of advertising is included in the model, we risk incorrectly giving the advertising credit for a rise in sales that would have happened anyway. It is particularly important to include price in your equation if prices changed over the time period (they did not for this business).
I began by brainstorming with my friend what variables other than advertising tend to impact daily sales. We came up with the following:
- Closeness to Christmas - sales are higher directly before and after Christmas for this business
- Day of the week - sales on Monday are higher than sales on Sunday for this business
- Days the company has already been in business - for a variety of reasons (such as increased brand recognition and repeat customers), this business has grown over time
- Sales yesterday - just like the weather - the best predictor of today’s weather is yesterday’s weather
I did some data transforms to make the data more suitable for regression analysis, using the natural log of the Closeness to Christmas variable and creating a dummy variable for each day of the week.We ran that regression in Excel (the .xls file is available for download above) and got the following results:
The R-Square (in red) of 79% means that our model is able to explain 79% of the variability of each day’s sales. The other 21% is due to factors not included in the model. An R-Square of 79% is considered high and is a really good starting place.
The Coefficients (in blue) can be interpreted as a mathematical formula that predicts the sales for today. Our best guess as to what todays sales will be is: $1,472 + YesterdaysSales * 0.71 + [3527 if it is Monday or 768 if it is Tuesday, etc, etc] + 3.96 * the number of days since the business was started + 348 * ln (days away from Christmas).
Adding in the Advertising Variable
This is what we are here to measure! After some experimentation we added in three variables: Advertising spend today, advertising spend yesterday (to see if the effect from advertising lasts more than one day) and cumulative advertising spend (to see if there is a residual benefit to the advertising) and this is what we got:
Interpreting the results is reasonably easy, we just look at the coefficients. The coefficient for Todays Advertising Spend is 0.81, meaning that, for every dollar we spent on advertising, we got an $0.81 increase in sales that day.
The coefficient for Yesterdays Advertising Spend is (-0.57) which is somewhat surprising. Why is it that advertising yesterday would *hurt* our sales today? It doesn’t really. We already included yesterday’s sales in the model. When the business is not doing any advertising, we can calculate a large part of tomorrow’s sales by including 0.70 of yesterday’s sales, in other words, the growth is highly consistent. However, if yesterday we spent $10,000 on advertising, this regressions is telling us that we cannot count on getting a $10,000 x 0.81 = $8,100 impact today as well as a $8,100 x 0.70 = $5,689 impact tomorrow. Rather, we need to add -0.57 x $10,000 = -5,700, for a total 2nd-day impact of about zero. In other words, if we had truly organic growth yesterday, we can count on a high amount of that growth translating into today. But if we inflated yesterday’s sales numbers with a massive ad blitz, we cannot count on really any 2nd-day impact from that ad blitz.
It is important that we have this Yesterday’s Spend variable in the regression (remember what I said earlier about confounding variables?) because if we don’t include it the regression underestimates the impact that advertising has on today’s sales, because this business tended to advertise in streaks, with a high correlation of advertising dollars between one day and the next.
Finally, the coefficient for Cumulative Advertising Spend is 0.005, meaning that if, over all past history we have spent $100,000 on ads, we expect the residual impact from those ads to be $500 / day.
Did the Ads Pay Off?
The business owner says that his gross margin is about 30%. If the immediate impact from each dollar of ad spending is $0.81 additional sales today and $0.00 additional sales tomorrow, then the total impact $0.81 in immediate additional sales from each $1 of advertising, or 30% * $0.81= 0.24 of additional gross margin for each $1 of advertising. Unfortunately, the ads do not pay off in the short term. They may pay off in the long term, however. Given that the short term loss on each ad dollar spend is $1.00 - $0.24 = $0.76 and each add dollar generates an additional $0.005 in ongoing residual sales, the ad pays for itself after 151 days. We should take this number with a grain of salt, however, because the t-stat was for that variable was less than 2. We also have to question whether it is logical that an ad would still be producing a residual value after 151 days.
Caveats
As I alluded to above, multiple regression models have a number of shortfalls:
- If a variable is missing from the model it could skew your results.
- The coefficients are not exact. In fact, each of them has a confidence interval, which is the range in which we are reasonably certain the coefficient falls.
- Regressions are based on assumptions which rarely are 100% true, when the assumptions are close to true (as I believe they are in this case) the results are ok, but as the assumptions become less true the results become less reliable.
How Do the Results Compare?
An article entitled How Advertising Affects Sales: Meta-Analysis of Econometric Results has long been considered the authority on this topic. The full- text is not available online but if you want to email me I could send you a copy (the license prohibits online distribution) - or you can just go to the library. In that article, the authors aggregated the results from a few dozen published studies and found that in those studies the average short term impact from $1 of advertising was a $0.22 increase in sales and the long term impact was $0.46. The average R2 in those studies was 0.78, in line with ours. That study was written in the 80s, however, and recently at the empirical generatlizations in advertising conference it was reported that short term advertising elasticity has dropped to $0.10 for each dollar spent and the long-term impact is still about 2x the short term impact.
References
For those who want to learn more about how regression can be used to solve business problems, two good books are Freakonomics and Super Crunchers. For those who want to brush up on their statistics, check out Statistics by Freedman, which is the most useful statistics book I have read. There have also been lots of academic studies on advertising elasticities in various contexts - most can be found simply by searching for “advertising elasticity”.
