In October 2012 Hurricane Sandy churned ashore, and a small datacenter in the heart of New York’s Financial District struggled to stay online. As Manhattan went dark and the floodwaters rose, an overwhelmed and battered team of Peer 1 Hosting employees and customers came together to “wage a 72-hour battle against exhaustion, time and Mother Nature.” And the story of the “Bucket Brigade” was born.
A new documentary that was recently released by Peer 1 Hosting tells their story about how they “persevered — one second, one step and one bucket of fuel at a time” to keep the datacenter up and running for their customers.
As the Peer 1 employees explain in the video, the company’s data center on the second floor of their building. When the flood waters from Hurricane Sandy started pouring into the basement of the building, Peer 1 employee Jeff Burns explains that what he thought was lightning was the sound of transformers exploding. Burns says that after the power in the building went out, the datacenter was being powered by a generator located on the roof, 18 stories from the street.
“Within 48 hours of Sandy hitting, basically every generator on the eastern seaboard was gone,” said Ryan Murphey, VP of Datacenter Operations for Peer 1.
The time for a decision came: Shut the data center down by letting the generators run out of fuel and go offline or make sure there’s enough fuel to keep it running.
When there wasn’t a good solution for getting fuel to the roof, someone jokingly suggested they carry the diesel fuel in 5-gallon buckets up the 18 flights of stairs. And that became the plan. To keep servers up, the datacenter tenants from Fog Creek Software and Squarespace carried fuel for 72 hours straight. They took turns sleeping on the concrete floor in a closet in the building.
“It’s kind of a miracle that everything ended up working out and we never went down,” said Burns.
After 72 hours, the building management was finally able to run a fuel hose from the street to the tank on the 18th floor, and the “Bucket Brigade” was retired. All in all, the documentary explains, it took the building 23 days to return to the power grid after Hurricane Sandy hit. But even months later, portions of the financial district were still dealing with the aftermath.
“When a datacenter goes down, it’s not about your servers that go offline. It’s people’s businesses that go out of business.” said Gary Sherlock, Chief Financial Officer of Peer 1. “It’s their customers who are impacted.”
Why This Should Never Happen to You
Businesses and hosted datacenter providers should never need to scramble to prepare for disaster. There should be no reason to experience network downtime, even in critical situations — and especially no reason to carry 5-gallon buckets of fuel up 18 flights of stairs for 72 hours to keep your business up and running.
Though hosted services are virtual, the cloud is tied to a physical location along with the information stored there. Businesses can avoid a breakdown in business continuity by simply choosing a provider that has a geographically dispersed network infrastructure with switches in more than one location. Geo-redundancy insulates businesses from having to experience the effects of a core network crash and prevents disruption in business communications. In the event of a switch outage, another location can serve as backup and preserve uptime.
Momentum’s core network infrastructure is hosted from two separate offices across the country — in Las Vegas and Atlanta. These markets were selected because of their stability, situated in areas that are not prone to major disasters like hurricanes or earthquakes.
The geographically diverse offices make Momentum’s network geo-redundant, meaning that in the event of a failure that takes one office’s switch down, the other location will back it up, maintaining 100% network uptime. As we saw from hurricane Sandy, flooding that knocked out switches in New York and New Jersey left thousands of business lines across the country down. Geo-redundancy prevents network outages and disruption in business communications.
Organizations that aren’t prepared for disasters often lose communication capabilities which in turn risks sales opportunities, revenue and valuable client contact. When a business’ lifeline is its communications, every enterprise should take the time to ensure that it has a disaster recovery plan in place, and that its provider is prepared to keep communications up and running.