This has been sitting in my Draft queue for like six months. My colleagues at ServerBeach are having some power issues in their Virginia data center, so this seemed like an opportune time to dust it off…
If Mother Nature has anything to say, there simply are some places where you shouldn’t place a data center. Do you really want to have your servers where there is a high risk of earthquakes, tornadoes or hurricanes? We didn’t think so…
It seems that the U.S. is “blessed” not only with large areas where the likelihood of earthquakes is very high, but also with some very harsh weather conditions in the form of hurricanes (no one is going to forget Katrina) and tornadoes, the most violent weather phenomenon in nature. Power outages, breakdown of infrastructure and loss of lives follow in the tracks of these hazardous monstrosities of nature.
— “Royal Pingdom” blog, Where NOT to keep your servers according to Mother Nature
And they created this lovely graphic:
If you have servers in a data center somewhere, odds are that they’re in a Red or Yellow zone. Truth is, nature hasn’t proven itself to be a substantial threat to most data centers and a shining example of his is how Zipa survived Katrina in downtown New Orleans.
What is the real threat to your data center? Power loss. Not the kind caused by hurricanes, tornadoes, earthquakes, or blizzards. No, if your data center goes dark it will probably be due to failures in UPS, PDUs, generators, or straight-up human error on a perfectly ordinary day.
LiveJournal has been hit many times by someone accidentally pressing the emergency power-off button, disgruntled employees sometimes hit the EPO deliberately, DreamHost was plagued by power outages because their facility had insufficient backup power. An nylug poster recalls several data center power failures over the past few years. It’s hard to get good information about the causes of DC power failures because most operators don’t want to discuss them, but I remember a bunch of outages in ‘04 where enough was said that my colleagues were sure that their problems were the same we’d experienced the previous year with a certain vendor’s redundant switching controllers failing to fail-over.
My advice for choosing a data center would be this:
- Make sure that all EPO buttons are covered.
- Make sure that there’s enough backup power.
- Make sure that the backup power is regularly tested.
- Make sure that there is sufficient fuel storage on site.
- Make sure that there are provisions for fuel delivery in the event of an extended outage or regional disaster.
- Make sure that the company you’re paying actually controls the backup power.
- If you are co-locating, protect yourself with enough UPS in your rack(s) to gracefully shut down.
Even with all of those bases covered, Murphy’s Law still applies. You can accept the risk, or mitigate it by utilizing multiple data centers in different regions, but placing all of your servers in BFE, Montana is not the answer.
