13 Jun 2013

Facebook: there’s a cloud in the data centre

Posted by iwgcr

On June 8, 2013,  The Register published a quote from Jay Parikah, Vice President of Infrastructure Engineering at Facebook, about an event happened in 2011 at the data center in Prineville, Oregon :

“I got a call, ‘Jay, there’s a cloud in the data center’,” Parikh says. “‘What do you mean, outside?’. ‘No, inside’.”
There was panic.
“It was raining in the datacenter”

Basically, due to a problem in the environmental management systems.

This resulted in cold aisle supply temperature exceeding 80°F and relative humidity exceeding 95%. The Open Compute servers that are deployed within the data center reacted to these extreme changes. Numerous servers were rebooted and few were automatically shut down due to power supply unit failure.

Facebook learned from the mistakes, and now designs its servers with a seal around their power supply, or as Parikh calls it, “a rubber raincoat” and also improved its building-management system to make sure that the error couldn’t happen again.