Announcing S3CDNFilter, possibly a cute little idea

We recently had a little bit of success with StyleFeeder which resulted in a truckload of extra traffic coming from Mary-Kate and Ashley Olsen’s website ever since they started using StyleFeeder (yes, really… yes, pretty amazing). The sudden burst of extra traffic made me wonder what my next step would be in terms of capacity enhancements.

Traditionally, you’d call up Akamai, get dragged through their OMG-this-is-painful sales process before someone would give you a quote for way more than basic content delivery should cost. But Akamai has great products, so it’s usually worthwhile. But it begs the question: why is the lowest price that I’ve ever seen coming out of Akamai around USD $1500 for 1Mbit worth of content? Why is there not a simple pay-as-you-go service available for anyone with a credit card who is willing to pay $50/month?

I took a look around and the closest thing that I saw was CacheFly. I tried to sign up with them at 8:30PM on a Sunday. Signup failed. I called their tech support number. I was surprised that someone answered, but less surprised that this person knew as much about content delivery networks, DNS and caching as I expect my mother does. It turns out that their service basically gives you an FTP login. You upload content and they push it out on their network, which just seemed really clunky.

Akamai’s basic service involves is simple if you serve your static content from a dedicated hostname like static.whatever.com. If you do that, it’s just a simple DNS change to make static.whatever.com a CNAME for something.g.akamai.com. You tell Akamai where the real location of the content is and the rest is transparent to you. Pretty cool, pretty easy to set up, pretty awful that you are forced to talk to a slow salesperson and pretty annoying that you can’t sign up by yourself and turn it on at 8:30PM on a Sunday.

Step back a few days to a short conversation that I had with Matt MacDonald of PRX about how they’re starting to use Amazon’s S3 (Simple Storage Service) to serve up their 800Gb of audio content to the Web. This prompted me to look into S3, at which point I was happy to discover a rather simple and well designed service with some basic support for HTTP 1.1 virtual hosting built right in. So this got me thinking: it’s so close to what I need in a content delivery system.

It’s a good point to establish some differences between S3 and Akamai and other CDNs. Amazon S3 is housed on presumably some kind of high-availability distributed filesystem (which has become unavailable at least once since launch) across several Amazon datacenters. Akamai, by contrast, has thousands of servers located at key points on the Web. They play some fancy tricks with DNS and intelligent routing to give you content from the servers that are closest to your network location. S3 gives you no such guarantee. However, for many needs, serving from S3 will give you the ability to burst pretty well up to levels that most sites can’t. It’s so close to a CDN.

So then I wrote some code and now the process of using Amazon S3 as a source for serving your static content is a lot easier. I’ve posted a java servlet filter called S3CDNFilter that automatically uploads static files to S3 and gives you a application a hook for rewriting your local URLs on a per-asset basis to indicate their new home on S3. It seems to work fine, although it’s in an early stage and certainly will have buglets to work out.

Here’s the code, available for download and fun. It’s licensed with an Apache license.

I have two objectives with this little project: the first is to get Amazon to give me a reverse caching proxy server that hosts content on S3. The second is to get Akamai to give me a pay-as-you-go option for $50/month that I can sign up for myself. May the best company win.