Of herds and hosting…
About a month ago, Kevin Kelly wrote a short post called “Prepare for flash crowds” in which he talked about the herding behaviour of people online and the poor readiness of the spaces they herd to. I’ve been thinking about this for a while as well, ever since a couple of years ago Steven Fry turned into whatever the opposite of King Midas is, by inadvertently overloading websites with traffic merely by referring to them on twitter (or did he actually turn into King Midas, as everything he touched became detrimentally inanimate? I don’t know - simile fail…)
It also occured to me that hosting readiness is really about the ‘commercials’ rather than infrastructure - i.e. most hosts can cope with the traffic, it’s just that when you sign the service level agreement you don’t agree to pay for the bandwidth necessary to serve pages during a traffic spike. Indeed most traditional service level agreements simply authorise the host to shut down the site when it goes over it’s monthly limit.
But what if the agreement was commercially flexible? What if the host could detect a sudden social media-driven traffic spike and automatically insert high-value advertising into the page to offset the bandwidth costs? Wouldn’t that be brilliant?!
I thought so.
So about six months ago, before I switched my blog over to Tumblr, I asked my then hosting provider whether they’d ever thought about this. This is how the conversation went:
(Disclaimer 1: I’m not going to name and shame the host, because a) I only spoke to a couple of their service reps and b) I don’t think they are any different to any other hosting company so it would be wrong to single them out).
(Disclaimer 2: I think I sound like a bit of a dick. My only excuse is that I thought anyone involved in hosting would surely also be thinking about this. And sometimes I’m a bit of a dick. That’s not two excuses - the second one isn’t an excuse.)
Anyway…
Me:
Hi there,
I’m an existing customer of yours and while I’ve never been anywhere close to the 30Gb per month bandwidth limit, it strikes me that the Internet doesn’t work with predictable traffic flows at all and that if it’s all about the ‘economy of attention’ then past performance is no guide to the future.
So, what would happen in this scenario:
say I’m hosting a blog and suddenly a post on that blog gets referred to on twitter by someone with a lot of followers - how does the server scale to cope with the sudden load and (more importantly to me) how does the service plan scale to cope with my pocket? Obviously I don’t want to be simply cut off - that’s a) a terrible customer experience for my readers and b) just not what this whole Internet thing is about, right? So is there a sensible way to keep the service going, but use Google Adsense to offset the additional bandwidth charges (whatever they are - I can’t find any mention of penalties in your FAQs…)? So, in effect, can I use Adsense to insure against a sudden spike - do you have any experience with this? Do the numbers work?
Any advice appreciated!
Best Regards,
Chris Dymond
Sheffield, UK
Them:
Hello Chris,
The Bandwidth transfer can be viewed in cpanel of your account. Currently it is showing 431 MB. You can also have a detailed view of bandwidth at cpanel->logs->Awstats.
The server your account is on can handle large amount of traffic and it is a Shared environment in which the server resources are shared across other customers in the server. This will not dampen your site performance and if we find that your site is using most of the resource we will suggest you a VPS in which you will be in an isolated environment. This significantly improves the site performance since you will have your own web server, database server, email server, etc. This will help insure the best performance for your application.
—
A Level II System Administrator
Me:
Thanks very much for your reply, but I’m not sure it fully answers my question: what happens if a page on my blog gets retweeted by someone with 100,000+ followers and a huge amount of traffic gets driven to the page for just a few hours before tailing off?
I’m concerned that as it stands you will either block the page or charge me for exceeding the bandwidth limit, or is there another alternative?
Them:
Hi Chris,
Generally, yes, if you exceed your bandwidth limit then your site will be blocked. You will then have to contact us to increase your bandwidth and then pay accordingly for the extra bandwidth. However, you have a monthly bandwidth of 30GB and I don’t think it should be an issue. You can continue your current plan and see if it suits you. If your bandwidth does begin to spike, you can contact us before it runs out and have it increased.
If you do indeed expect heavy bandwidth usage on your site, then you can either upgrade to the Developer Level 2 plan or migrate to a VPS plan. The Dev Level 2 plan allows 200GB monthly bandwidth while the starting VPS plan allows 100GB.
Do let us know if you have any further concerns.
Me:
Thanks for your reply.
But now that the Web is real-time and attention-driven and there are many very powerful nodes on the network that can fling 50,000 page views at a site with a simple tweet, things are: a) not predictable and b) happen in a matter of minutes not days.
So my regular usage and the 30Gb limit don’t mean anything should such an event occur, and neither will I have enough time to observe the bandwidth increasing and contact you to have it increased - how much should I increase it to? How long for? How might I offset the cost? What if it happens while I’m asleep or not paying attention? How do we both gain from the event?
Essentially you are saying that you have no services or policies designed for this scenario, so the only way to mitigate the risk seems to be to no longer ‘self-host’ but to sign up with squarespace or wordpress.com (which I’d rather not do…)
Them:
Hi Chris,
I can understand your concern. However, since we don’t have an idea of the exact bandwidth consumption your site will expect, we’re unable to suggest the limit for your purposes. I would suggest that you continue to check your bandwidth usage on a regular basis. You should then be able to estimate the approximate bandwidth limit you might need.
Regarding upgrading the plans, you can take a look at them here:
http://hostingcompany.com/plans-products/developer-hosting/
Do let us know if you have any other concerns.
And that was it - no point carrying on the conversation. I got the classic “empathise, reiterate and upsell”…
And so I’ve moved to Tumblr - and while protection against (highly) unlikely traffic surges is not the main reason I moved, it did play a role. And the wider issue still remains: does *anyone* provide a guaranteed service during a large spike while offering some form of commercial insurance against it? Sure cloud hosting providers provide metered access where you pay for what you use without fear of being disconnected, but that still leaves you with a bill for becoming momentarily popular instead of a profit…
Kevin Kelly says: “…the nature of spaces is that in order to accommodate a flash crowd when they do come, you have to be ready, tooled up.” - yes, he’s probably right, but what exactly does that mean…?!
