Some of you might have noticed that access to my blog has been less than stellar… well, that’s an understatement. The blog has been down for almost a week.
This has been tracked down by DreamHost to a faulty RAID card. What DreamHost does on these occasions is to run everything from the backup server, mounted over NFS, while they bring in new hardware, and then copy everything over. This has the advantage, of course, that everything continues to work.
Sort of…
Unfortunately, NFS mounts are perhaps 1000x slower than direct access to hard disks, specially disks on RAID on a local server. What this means is that the poor CPUs are endlessly waiting for packets to arrive. This floods its networking buffers until they cannot handle the load no more, and everything screeches to a halt. Ironically, the server does not crash, because the CPUs are actually not doing much. They have no data to do much with. So they just idly wait until some packet arrives… and wait… and wait. When they finally can process it — in plain English: when they have a page to feed back to the browser — the network connection has long ago timed out. In the mean time, a few thousands of new requests have just arrived in… and the CPUs have to go back on waiting and waiting.
This particular server has reached an overall load of 27.200% — or, since it has 8 CPUs, 4600% load per CPU. Think about it. That means that each CPU was, on average, forty-six times as slower as a regular machine at full capacity. What usually takes seconds, started taking minutes. Since network connections are usually measured in seconds these days, this means that, for all purposes, nobody ever got any page served from the server. Very, very occasionally, a page would finish and get delivered in time.
Now, I’m writing this in the past tense, but things are still not operational — not by far. DreamHost claims that 3/4 of all the users on this server have been already migrated to the new hardware. I’m not one of them. The last time I’ve checked, I accounted for perhaps 10% of the overall load on the server, and I guess DreamHost is placing my websites at the bottom of the queue.
What I did was to restore the scripts and images on another provider, and continue to run WordPress from DreamHost’s excellent MySQL cluster — I just needed someplace else to place the scripts. It wasn’t a painless move — since everything related to “disk access” (and copying files from a backup is disk access) is about 1000x slower than usual, it meant that it took 48 hours just to get my backup off the server. I know, I know, I should have made a backup on a different server. But even though nobody believes me, I’m actually a very poor girl — statistically, I’m accounted as being part of the 20% of Portugal’s citizens who live under the poverty line. It sounds far more dramatic than it actually is: the “poverty line” is measured differently between different countries and takes into account the cost of living depending on the cultural environment — I’d be reckoned as filthy rich for the standards in most of India or sub-saharian Africa. Unfortunately, I live in Europe.
Closing parenthesis… I really cannot afford to pay for extra space on a different server. I can keep some backups on “free” services like Dropbox, Google Drive, Microsoft Drive and so forth, and that’s what I do in some cases, but not all. DreamHost is very cheap for all the features they give me, and because I took a risk with them as a beta tester long time ago, I get unlimited everything service from them — unlimited disk space, unlimited users, unlimited databases, unlimited bandwidth, unlimited domains. I even get unlimited mailboxes with unlimited space of them, if I used their services (which I don’t). Think of any metric you care to mention, and I get it “unlimited” 🙂 So obviously I host everything with them.
They’re also cool guys. For a week of “downtime”, they refunded me 2 months of service. Now that’s actually quite nice of them, because, technically, the service is not down. It’s just insanely slow to the point that it seems to be down. A less ethical hosting provider — and I’ve been on the “dark side” as well, knowing all sorts of excuses that providers use to avoid refunding their customers — would be able to claim that their service is running, so, by sticking to the letter of the law (or, in this case, the Terms of Service) they wouldn’t refund me anything. And DreamHost does not have a Service Level Agreement. They’re so cheap that they prefer to offer Best Service instead — and, when they feel they have hurt their clients (which is what they feel in my case), they refund them whatever they think is acceptable. Getting two months free for one week of “downtime”(or rather, slowdown time) is rather nice of them.
With that money I was able to afford to pay for a few months on the competition. It’s been some time since I’ve shopped around for hosting service, and, of course, I wanted to go as cheap as possible, since this would be just a temporary measure anyway. My first choice was BlueHost, because I have worked with them before, and a few friends of mine speak quite well of their service — but unfortunately they required me to pay a whole year in advance. Oh well. So the next choice was the popular 1&1 hosting service, which is even cheaper than BlueHost, and allows for paying just 3 months. In fact, these days, they’re even cheaper than DreamHost, so I was curious to see how good their service is.
Well, of course it meant getting a very limited service, but I just needed to worry about a single blog (the rest of my 100 or so websites can wait 🙂 ). Soon enough I figured out that I have been spoilt by DreamHost just by looking at 1&1’s idea of a back panel. I’m familiar with quite a lot, from the popular cPanel (which I hate) to Parallel’s Plesk (which I also hate, but is marginally better than cPanel). I’ve even survived using GoDaddy’s backoffice… which I hate with a vengeance 😛 With so much hate I thought there couldn’t be any backoffice left that I could waste any more hate on — but I was wrong. 1&1’s backoffice earns my award of Worst Hosting Backoffice Ever. cPanel looks like something designed by Apple when looking at 1&1’s backoffice. I could even survive a few years on GoDaddy’s backoffice after taking a look at 1&1 😛
What’s so bad with it? Well, I guess that anyone familiar with it would come to appreciate it — I mean, I do know people that actually like cPanel and Plesk, and think that GoDaddy did a reasonable job. These people are insane, but, then again, I suppose one can get used to anything, if you never tried anything else. After all, people use Windows all the time. Millions swear by Linden Lab’s old 1.X viewer interface and think it’s the best thing since sliced bread (when compared with the 3.3 viewers).
My personal peeve with 1&1 is how insanely hard everything is. I mean, there are just a handful of options, right? It shouldn’t be too hard to click on two or three buttons to set up a disk area that is connected to a certain domain name — right? Well, no. It is hard. How logical it is to be forced to register for a domain name, then cancel the registration by telling 1&1 that you already have a domain name, and, when it starts asking for codes for the domain transfer, abort everything, and refresh the page? With luck, if you do it just right, 1&1 will point a domain to an area of the disk. You cannot change anything else, but at least it will be pointing to the right place. Gah! It took me hours to figure out the right sequence of when exactly to abort the procedure to make sure I got things pointed to the right place…
A lot of people always hated the DreamHost backoffice, too. It predates commercial (or open source) backoffices — DreamHost is so long in business that they never reinvented the wheel. They created something that people later on called “wheel”. So their backoffice doesn’t look nowhere near to what you usually think of a backoffice — specially if you’re used to cPanel or Plesk (which most of you will be, since they’re the two most popular options). DreamHost are a bunch of geeks, and they figured out that people want to tweak everything from a webpage. I mean, everything. There is little you cannot do on DreamHost from the backoffice — assuming it doesn’t require root (superuser) ability — and for the little you cannot, you still get shell access…
And, of course, they continue to have unbeatable technical support. If you complain because “my site is slow”, the usual reply is a two-page report on website optimisation. Personalised — not really a “form letter”. They also pro-actively run probes on your sites to see if they have been hacked, solve whatever is obvious, and send a detailed report back to you for further investigation.
Aye, there are downsides. Living in a shared environment means that often your “friendly neighbours” are not that friendly and starve the server for resources. DreamHost usually tries to do their best to give everybody a fair share of the resources, but of course they cannot work miracles. But for the little you pay, it’s more than you’d expect.
So, enough ranting — I’m not really writing a long praise of DreamHost. After all, as said, all my sites (except this one and another one I have on GoDaddy — a semi-abandoned project) are down, and I think they will continue to be down for another few days. Obviously this means lots of angry people. I should be angry too — but how can I, when I get reports every 3-4 hours to let me know how well the restore procedure is going? I cannot even complain that “I want my money back!!” because I already did get a refund (and didn’t ask for it!) — for far longer than the sites were down. So it’s hard to get angry at DreamHost. I guess that’s why they keep faithful customers. And, of course, they have hilarious newsletters. It’s probably the only company in the world that writes “promotional monthly newsletters” that I’m always eager to read — because they’re simply hilarious. They switched CEOs, and I thought, “oh, now we’re getting the serious stuff from a serious senior executive who thinks that this is a serious company and should stop pretending they’re clowns”. Not so. The new CEO is as hilarious as the previous one — fitting right into the corporate culture. Those guys really know how to take it easy 🙂 But the techs writing back on the support channel treat you seriously. They can be fun in their spare time, but they won’t joke with you when you’re having problems. That’s the kind of company I like!
For other DH users, of course, they can be a pain to bear 🙂 For example, many of my in-world devices use web services running from DH’s servers. All these might be down these days — I haven’t checked on all of them. This means that by now I must have lots of angry customers, waiting for refunds because their devices stopped working — or are so slow as to become useless. I might have to prepare myself to issue refunds to everybody who asks for them. So if your business depends on DreamHost, well, it might mean thinking twice about their services — because while this is the most serious issue I have encountered with DreamHost so far since May 2006, it was not the only one. For many people, 99.9999% uptime is the least they expect — and DreamHost is not the kind of company to rely upon that. But if you’re happy with 99% of uptime, then they’re unbeatable — in price, services, and fun! — and that’s why I will stick with them for a while longer.
Even though there are millions of alternatives, some with more features, many with lower prices. But sometimes you become a fan of a company for mostly irrational reasons. And I’m known to be irrational about Apple, Linden Lab, and lots of similar companies 🙂
