UPDATE #7 (11/3/12 7:24pm PDT): Great news to report this evening. Our Secaucus, NJ data centers (NY2, NY4 and NY5) are now back on utility power. That means all of our sites in the New York metro area are now operating normally on utility power.
Please send us any questions you may have in the comments below, or tweet us with your questions to @equinix.
UPDATE #6 (11/3/12 12:15pm PDT): Good news to report this morning. Four of our New York metro area data centers (NY1, NY7, NY8, and NY9) are now back on utility power and operating normally. Our Secaucus, New Jersey sites (NY2, NY4 and NY5) remain on generator power. These data centers have ample fuel onsite, and we are receiving daily fuel deliveries. Daily fuel deliveries will continue until utility power is restored. We are in regular contact with the utility company (PSE&G), who has indicated that utility power could be restored as early as today at our New Jersey sites.
UPDATE #5 (11/2/12 2:45PM PDT): Thus far, no Equinix IBX fuel supply has reached a level of lower than 50% and we continue to successfully manage fuel deliveries to all of our sites. Fuel deliveries were received overnight and today. Additional deliveries to the New York Metro area are scheduled with fuel vendors across the Northeast and Mid-Atlantic regions throughout the day and night as needed. We are communicating with the utilities (ConEd and PSE&G) regularly and discussions with both companies have indicated that they expect to have utility restored to our New York and New Jersey sites over the weekend.
UPDATE #4 (11/2/12 12:30pm PDT): UPDATE #4 (11/2/12 11:30am PDT): Customers are receiving service as usual at all of the sites that have been impacted by the storm. Five of our eight New York/New Jersey sites remain on generator power (NY2, NY4, NY5, NY8 and NY9). These sites are operating normally. We are receiving fuel deliveries daily and daily deliveries will continue until utility power has been restored. Our NY9 site is now stable and running on power from redundant generators. We are in regular contact with the utility companies (ConEd and PSE&G), who have indicated that utility power could be restored over the weekend at our New York and New Jersey sites. Our Reston, Washington D.C. and Philadelphia sites are running on utility power and operating normally.
UPDATE #3 (11/1/12 12:20pm PDT): All of our New York and New Jersey sites are accessible to customers. Three of those sites (NY1, NY7, and NY10) are back on utility power and are operating normally. The other five sites (NY2, NY4, NY5, NY8, and NY9) remain on generator power. All of these sites have ample fuel onsite, and we have successfully received fuel deliveries over the past two days with regular scheduled deliveries to each site until utility power is restored.
This morning our NY8 site experienced a failed generator that impacted service to several customers. The generator was restarted and service has been restored. Over the past two days our NY9 site, which is housed in a multi-tenant building with shared generator infrastructure and fuel service, has experienced three outages due to generator failures that impacted service to several customers. Repairs have been made and service was restored. The site remains on generator power.
And finally, our sites in Philadelphia and Washington, D.C. are operating as usual on utility power.
UPDATE #2 (10/31/12 1:45pm PDT): Three of our sites in New York and New Jersey (NY1, NY7 and NY10) are back on utility power. The other five sites in the region remain on generator power. All of these sites have ample fuel onsite, and we have successfully received fuel deliveries today. We are scheduling regular deliveries to each site until utility power is restored. Early this morning, our NY9 site experienced a failed generator that impacted service to several customers. Repairs were made and service was restored within one hour and 15 minutes. Our sites in Philadelphia and Washington, D.C. are operating as usual on utility power.
UPDATE (10/30/12 5:30pm PDT): Four of our sites have returned to utility power (NY1, NY7, DC7, PH2) while another six remain on generator power. We have ample fuel onsite and fuel vendors available to deliver more as needed to the remaining sites. All reported water leaks have been contained. There is no impact to customers at this time.
The East Coast of the U.S. has been inundated with severe weather due to Superstorm Sandy. The storm buffeted one of the most highly populated corridors in the country – an area that also happens to be home to many Equinix data centers. Here is how our IBX data centers and our staff are faring through the storm.
Several days ago when it became clear that a major storm was coming, we took additional precautionary measures at our data centers in Boston, Buffalo, New York/New Jersey, Philadelphia and Washington, D.C. to guard against interruption of customer services. We tested generators, filled fuel tanks and made arrangements for backup fuel. In addition, we arranged to have full time staff onsite at each facility to maintain operations. That included securing hotel rooms near the sites, and bringing in cots, MREs, and other emergency supplies, so that our staff could comfortably support customers even in extreme conditions.
As the storm progressed, we faced a number of incidents:
- All sites in New York/New Jersey, and several in Philadelphia and Washington, D.C., experienced power outages and customer loads were transferred to generator power. All of these sites have at least 48 hours of fuel onsite with fuel vendors on standby to deliver more as needed. As of this morning, some of these sites are back on utility power, while some remain on generators.
- NY9 experienced a failed generator that impacted service to several customers. We made repairs and service was returned this morning. The site remains on generator power.
- Sites in New York and Washington, D.C. experienced water leaks. While most were minor and quickly contained, at least one leak at a New York site impacted a customer. We are currently working onsite with the customer to contain the issue.
The Equinix Operations Center staff has been sending regular communications to keep our customers apprised of the situation as it unfolds and have been onsite troubleshooting 24X7.
Our Site Operations personnel have worked tirelessly to ensure that our data centers in Boston, Buffalo, New York/New Jersey, Philadelphia and Washington D.C. continue to operate during the worst of the storm. We appreciate the commitment and dedication from our staff, many of whom have left their families and homes to keep our data centers safe and operational.
As the storm progresses, we’ll continue to monitor the situation closely and take additional steps as needed to ensure the highest level of customer service.
Watch this space for more updates…