Ouch!
Well that sucked. Late yesterday afternoon all, and I mean all, of my Dreamhost-hosted sites took a nose dive due to the Los Angeles power outage. No mail, no web, no FTP. It was off for hours and didn't come back online until sometime around midnight.
For kicks I loaded my entire "Blogs" directory of bookmarks to see how many other people were down, and nearly 10 were also afflicted. Media Temple appeared to be down for a while as well. All that because some guy at the power company accidentally cut a line.
Sure would have been handy if they had some backup generators to keep the ship afloat, but oh well. Glad it all came back in one piece.
Comments
This is not a critic to Dreamhost or any other hosting company... but maybe you should consider a hosting provider that offers a SLA (SERVICE LEVEL AGREEMENT).
Just my opinion... cia'
Posted by: Denis at September 13, 2005 9:42 AM
yeah I noticed this as well heck I noticed alot of places that appeared to be affected by the outage, you could actually feel the slowdown online.
Posted by: drew at September 13, 2005 9:44 AM
Todd -
According to their blog, Dreamhost did have backup generators in place.
However, their building was evacuated and all three of their network providers went down. Eventually, their generators failed, too. Seems like everything was kind of FUBAR.
My sites were down because of it, too, which really sucks on a Monday. Email was down until about 11PM Mountain. Bummer of a day. To Dreamhost's credit, this is the first time I've ever had an outage longer than an hour since I've been a customer (1998).
Who knew I would be so affected by a power outage in LA up here in Montana?
Posted by: Dave Simon at September 13, 2005 9:49 AM
Who knew I would be so affected by a power outage in LA down here in Sydney, Australia!?
Posted by: UniversalHead at September 13, 2005 9:51 AM
Yeah all my DH and MeditTemple sites were down, although it was interesting how much faster MediaTemple got everything back online. But that could just be down to the order in which they brought machines back online. Or maybe they didn't have evactuate their data center...either way maybe Montana or Australia would be better places to host...
Posted by: Scott M. at September 13, 2005 10:13 AM
How can "nearly 10" be affected? Is that 7? 9? And am I the only one wondering?
Posted by: Erik J. Barzeski
at September 13, 2005 10:26 AM
(mt), as most customers are probably aware now, did have to evacuate their data center. Apparently, and this is not based on first hand knowldege, there was an outage due to some human error upgrading the lines which put the power out at about noon pacific time for 2 hours.
After everyone was getting their power back there was another outage later in the evening but the redundant systems, at least at (mt), were effective this time. (mt) seemed to come back online around 8pm Mountain time for me.
Posted by: Mike S. at September 13, 2005 10:26 AM
I host 5 sites on Dreamhost. I'm lettin' them slide on this, though. This wasn't just a small power outage. New York had one of these a few years back that even went up to Canada. I had a few sites go down for a few hours then, too.
Posted by: M.e. at September 13, 2005 10:31 AM
My company's sites are hosted by Media Temple. Here is a link to their response. Seems their engineers tried to 'help' out the backup power and ended up cutting it off entirely.
http://www.mediatemple.net/support/incidents/power_issue_09-12-2005/
Posted by: treevis
at September 13, 2005 11:37 AM
FYI: In some cases (like mine), Dreamhost was up a lot faster than Media Temple. I think it depends what server you were on on either provider.
I too am willing to let this slide as an "unavoidable circumstance" because the only way to have stopped it would be a redundant failover somewhere else across the country. There's a lot of work and money which goes into setting up and maintaining full redundancy and for how rarely you need it, I'd rather not pay for it. It would be nice, perhaps, for ISPs to offer this as a special add-on service though. Then you could decide whether the extra price was worth it.
Posted by: Mike D. at September 13, 2005 11:54 AM
I only noticed the outage last night when I couldn't get email and didn't think too mcuh of it. This is the first time I've ever had an outage with DH and there is no way that it could have been avoided.
Posted by: Kevin Tamura at September 13, 2005 12:12 PM
I'm actually a Media Temple Engineer (and regular reader) -- a few quick clarifications. We're actually hosted in the same building as Dreamhost, and we had the exact same set of problems. It wasn't either of our engineers that managed to kill the power to the building, it was the building itself's staff who are no doubt in a world of hurt at this moment.
As far as when your sites came back up, it probably was the luck of the draw -- because computers eat so much more power when booting than when normally running, we all have to use a "phased turn on" approach to get everything turned on safely so it doesn't blow the power again! Anyway, hope this helps, what a day.
Posted by: Joshua Barratt at September 13, 2005 12:23 PM
This was waaaaaaaaaaaaay out of the hands of either Dreamhost or (mt). Level 3 and Global Crossing, the folks who supply those beautiful OC3 lines (that's like a hundred T1 connections or something technical) to the hosts were completely out of commission.
No Level 3, no internet machine.
By the way, the http://status.dreamhost.com/ was a nice feature to have and kept me informed on the dozens of sites I have hosted there... I then found it amusing that the (mt) notice page asked themselves the question "Have you considered hosting a blog or status site outside your network?"
Posted by: FAA at September 13, 2005 5:40 PM
Oh so that could have been why my website was acting really weird yesterday. Glad its back up and running okay now.
Posted by: Yannick at September 13, 2005 6:37 PM
We had just launched boy sets fire's new website at boysetsfire.org and another band's site at thelookmachine.com and we were afraid that our streaming files knocked some of their servers offline!
Ha! Then we checked some other sites we knew were using Dreamhost. . . like this one, and we knew it was much bigger then just us.
Curiously, all of our sites came up by 8 PM ET (not the streaming servers though), while I noticed a number of other dreamhost sites were still down. Lucky us I guess.
Posted by: Jason Latshaw at September 13, 2005 11:55 PM
I hear you! I was in the same boat: "what's wrong with my email"... "why can't I get to the control panel"... (visits status.dreamhost.com) Oh.
According to The Dreamhost Blog, there were five generators, but two failed and the other three weren't enough for the load. So much for backups and redundancy.
Apparently, this is the first time status.dreamhost.com has ever been used! Now that's impressive.
At least they didn't delete ALL my email (in IMAP), like my other host!
Posted by: Calrion at September 15, 2005 1:12 AM
It is not the first blog where I've seen such comments on this matter and I'm always a bit confused about that. I just wonder why people bother that much. Good post anyway...
Posted by: Popeye at September 16, 2005 8:57 AM
It's interesting how there are downtimes with the best hosting companies on the web. Its kind of strange that there is a downtime with MT and DH, usually you would think that there wouldn't be any.
There is another hosting company on the net, which I have recently found. It's called indexcore.com ( I guess we can call it iC? ) seems that they aren’t based in CA where power is ridden, because of some really smart power line engineer( he he ). I saw them like a couple of days ago, when I was looking for a host that was someone new, rather then DH and MT. I guess there will now be 3 hosting companies? MT, DH, iC? Just thought it might be worth a look if your looking for some stability vs the power outages.
Posted by: Jeff at October 10, 2005 11:43 PM
It is not the first blog where I've seen such comments on this matter and I'm always a bit confused about that. I just wonder why people bother that much. Good post anyway...
Posted by: Fox Trot at October 15, 2005 9:47 PM
You have to go a long way with a hosting company before you really know them (and can trust them).
Posted by: Jack at November 8, 2005 2:59 PM
