Reducing Your Website's Bandwidth Usage

Don’t forget your ISP. I use roadrunner and there is a 25MB limit for ‘users home page’, which I store only images for my web page.

Feedburner has worked very nicely for me. I have over 10,000 subscribers to my feed at macmegasite, so it’s made a huge difference in my traffic. The ads in my feed also pay very nicely. I like your mod_rewrite code better than what I’m doing now - specifying a private feed for feedburner and redirecting the public feeds to feedburner. I’ll probably have it use the user_agent instead.

nice article, has giving me something to think about myself.

Hi Jeff,

I had some initial fears in using FeedBurner as well. I went with their “my brand” feature which lets me use my own URL via a CName record for about $3/month. This way if they ever go belly-up or get sold/etc I can simply remove the CName and handle the feed on my own again.

-Scott

Many hosting firms offer 1tb/month transfer nowadays, there would probabaly be problems if all the sites hosted on each server actually uses this amount, but i’m sure theyre using at least t3 its probabaly best getting a dedicated server hosted somewhere if you want your own server unless you either manage the transfer really well or are not expecting too much traffic. Your own T3 or OC3 would probably be fairly pricey!

How about “none of the above” and ditch the T-1 and get 10/100/1000 fiber? We went from a T-1 ($600/month) to 10 megabit Fiber, unlimited transfer, for $700. Easily paid for itself. Linux ISO downloads in 5 minutes!

I’ve used Django (the python web framework) for a while, and they have a simple template tag called {% spaceless %} which automatically removes all unneeded whitespace and line breaks from your html without having to un-uglify your code every time you want to edit it. It’s the bees knees!

I would love for one of the Amazon adopters to contact me at michael@sophio.com to help with integration of our software (paying gig).

Three naive questions from a desktop application developer:

  1. Is it possible to exclude inefficient RSS feeds?
  2. Is it possible to throttle the polling frequency of the feeds, e.g., set a cap of one poll per minute?

Not really. The problem is not the feeds(which are just XML files on a server). The problem is inefficient feed readers(programs that pull the feeds and possibly way too often). There’s probably no reliable way of identifying them. If they set the user agent header(part of the http message that identifies the program making the request), you can detect which one is making it, but a poorly coded reader probably didn’t bother to set this optional setting and even if you could, you probably don’t want to tell a user which feed reader they’re allowed to use.

You could track the frequency of traffic by IP and not serve up the feed more than once a minute to any given IP, but I’d imagine proxies would mess up that method since they are requesting on behalf of many different users. You could use an expires header that tells the proxy not to re-request for some amount of time, but that only works if they bother to honor it. And blocking that re-request may hurt the end user experience because even a well written feed reader might be blocked because of a poorly written one on the same proxy.

  1. Is it possible to have a “no images” version of your posts, and restrict RSS access to only those versions?

RSS readers only access an XML file that lists a number of articles and some text content. They shouldn’t be requesting any images in the content, at least not until someone tries to actually view the content, in which case you would want them to be able to load the images anyway.

This particular feed could probably shed 90% of its size if the whole article weren’t present in it though. Most feeds I’ve seen only have a short description. I’m sure including the whole article was a conscious choice though because the owner didn’t want users to have to navigate to his site to read the post.

Jeff, glad you looked into my comment on the oop vs poo comment, hope it didn’t come off as overly sarcastic.

Flickr’s Terms of Use indeed say, “professional or corporate uses of Flickr are prohibited”, but Yahoo!'s don’t. Flickr was acquired by Yahoo and accounts now fall under the Yahoo terms: http://www.flickr.com/terms.gne

Nice tutorial. Usefull if bandwith really is a big problem.

I think S3 is a pretty good solution. I just completed a test for http://www.famundo.com using it for all static content and the results are great. We are going to deploy this into production in a few days.

You can even serve compressed jc/css directly from S3, but there are some limitations with it. You can see my post about doing it here: http://devblog.famundo.com/articles/2007/03/02/serving-compressed-content-from-amazons-s3

It has code sampled on how to upload compressed assets from S3.

The big lure of S3 is the price. Even if you transfer 30GB a month, we’re talking a few dollars only. Nothing else even comes close. And it is a pretty stable netwrok.

This is good stuff, Jeff! Thanks for the tips. The HTTP compression stuff was news to me – I didn’t know that was built into Apache and IIS. My bandwidth is creeping up there so I might have to look into that, along with hosting my images elsewhere (it’s kind of a chore, though). Thanks!

However nice external image hosters can be, I did notice that all of the images on the post I read yesterday were ugly “this user has exceeded bandwith” images. It makes the article rather hard to track.

I’ve always been curious as to how much bandwidth savings you get for doing HTML, CSS and JS compression in addition to HTTP Compression. I’m no compression expert, but it seems that HTTP Compression would be doing roughly the same thing.

LiveJournal.com also has very innexpensive image hosting, $25 per year. (I don’t think there is a bandwith limit, at least none that I’ve hit. lol)

I’ve never load tested my own little web server, so I guess I won’t say anything but that I get 400 GB in transfer from Network Solutions for 15 or 20 bucks a month. And I’m just buying as a consumer, so I always thought I was grossly overpaying in the name of having a name which would care if I griped loud enough.

I might guess that if you called up your hosting company and said, “Um, yeah… I run a blog that’s mostly text and I went through 9 GB of transfer yesterday. It turns out that my userbase is heavily biased towards highly skilled programmers, and I’m wondering if you would like to put a link to my hosting company on the lower left hand side of my page’s sidebar in exchange for some leniency next time I go overboard on the hosting bill?” your bandwidth issues might not reappear.

I’m reading Founders at Work; generally it appears that on the Internet popularity is a problem which solves itself.

@thereign

“pages will inevitably drop key bytes”

I think you misunderstand how TCP sockets work. If data is dropped either the connection is lost - or the data is retransmitted - this happens regardless of whether the packet represents compressed data or not

Of course with uncompressed data there is a higher probability of loss (more packets transmitted). Moreover compressed data transmits less packets overall, increasing data-transfer rate by spending much less time in acknowledgement cycles. (there to prevent data loss in the 1st place)

Thanks a lot for these tips, I’ve been using photobucket for my images …