Sean’s blog

I co-founded an online retailer a few years ago - we have a pretty large customer list (approaching 100,000) and we like to send them emails from time-to-time (maybe 6 times / year) because to tell them about new products.

There are several services that help manage such mailings. The three that I am most familiar wth are Mailchimp, IContact and Constant Contact. My company chose IContact because, at the time, we found it easier to use than Constant Contact and had not heard of Mailchimp. The advantages of using such a service rather than just sending out the emails yourself are: 1. deliverability - their mailservers are on whitelists to ensure that more of your emails get through 2. tracking - they track clicks and opens 3. managing the usubscribes so that you don’t make your customers angry by sending them stuff they don’t want 4. templates for helping to create attractive mailings.

With a list as large as ours, this service is a considerable expense for us - $699 / month - almost as much as we are paying for our dedicated server, bandwith, etc. which seemed a bit out of whack to me. I wanted to see if we could create a system with comparable functionality ourselves and spend the savings on something else.

I felt pretty sure that we could replicate #2 (tracking) and #3 (managing unsubscribes), and even improve a bit on the tracking by seeing what sales result from the mailings (in addition to clicks and opens, which all of the software packages described here track). We don’t really need #4 (templates and formatting) since we already have a template that we like and we prefer using Dreamweaver over the javascript html editors that the services use.

However, #1 (deliverability) was a big question mark. If the emails don’t get through when I send them from our own servers then it could be worth the extra expense for one of the services. I decided to run a test, but I got a bit carried away. While I was at it, why not test the deliverability of all the services relative to each other, as well as relative to our own solution.

It cost me a few hundred bucks and some time to do this, so I am posting it in hopes that somebody else can benefit from it as well.

For each of the 4 treatments (homemade and the 3 vendors), I chose about 3000 customers randomly from our mailing list. It wasn’t exactly 3000 because of the way that I randomized the list. Then I asked my friend Simone, an economics PhD student at MIT how I should run the test. She replied:

An easy way would be to run a regression with response=”yes” on the left hand side and dummy variables equal to one to indicate whether than customer got treatment X (so if you had 4 different treatments, you’ll need 3 dummies). The coefficients on dummies will tell you if the relevant treatment is significantly different from the response rate for the omitted treatment.

Ok, that’s easy enough. I sent the same message to all four lists at the same (rough) time, midnight on Tuesday 1/22. Then I waited until today (3.5 days later), downloaded the results from each of the services and aggregated them manually in Excel. Here were the results:

These results show that the difference in deliverability for the four treatments was small, but it is not clear whether the difference is statistically significant, so I ran the regression that Simone recommended. These were the results:

Coefficients - the first bolded column, shows that each of the services achieved deliverability slightly worse than my own solution. P-value - the second bolded column shows us how certain we are of the coefficients. For example, the 0.40 p-value for constant contact shows that we are 60% certain that it actually performed worse than the control (my solution). In other words, the negative result for constant contact was not statistically significant. Actually, none of them were. IContact came the closest - we are almost 90% sure that it actually performed worse than my home-made solution.

Conclusion: I am going to start sending out my email myself and save the $700 / month that I was giving icontact.

Another wrinkle - I told my hosting company about this plan and they were not big fans of it:

8:48:25 AM Sean: can you imagine any issue with me using php tosend 75000 emails from my box?
8:48:44 AM gino: you bet your ass volume alone will get yout blocked
8:48:55 AM Sean: by whom?
8:49:13 AM gino: unless you have a great personal approved relationship with alot of the major isps, they will block you based on volume -
8:49:48 AM gino: they may let 500 in wihtin x time, but the rest will get blocked and eventually they’ll see it as abusive and blacklist your IP and the range even - whcich puts us all in jeapardy
8:50:07 AM gino: can you prove these 75k are clients or have subscribed or doe busienss with you?
8:50:14 AM Sean: of course.
8:50:17 AM gino: and is the email being sent can-spam compliant?
9:01:09 AM gino: they have an email relationship specialist on staff - and several load-balanced clusters each with its own distinct IP range and they slowly dribble out the emails over time -
9:06:50 AM Sean: but if i think about it differently and instead of sending out 75,000 at a time, but instead send out
something to each customer 1 month after they have shopped with us, that could work
9:07:03 AM Sean : because i would be only sending like 200 / day
9:08:16 AM gino: yeah much more likely to work

So the new plan is to re-arrange my mailing strategy, so that we no longer send out clumps of emails, but rather gradually tell customers about new products. In some cases, that won’t work (when something really hot and cutting-edge comes out). In that case I will use Mailchimp, which has per-message pricing. I think I can cut these clumpy mailings to 2 / year, at their pricing of $0.005 / email that’s only $1000 / year, must less than the ($699 * 12 = $8,388 / year that I was paying previously).

I am working on putting some analytical rigor around many aspects of this ecommerce business. If anyone else has done testing of this sort and is willing to share, I would really like to speak with you, maybe we could collaborate and save ourselves some work.