Google's backup generators failed to start during a London outage

Google has explained why a London data center went dark on October 20. When the power failed, the backup generators were supposed to start. They never got the signal. Customers of Lloyds, HSBC, Royal Mail and Revolut had service interruptions as a result.

What went wrong

Google posted its explanation on November 4. The affected site is one zone in its London cloud region, called europe-west-2-b. It normally gets power from two separate utility feeds, called A and B. A cable carrying feed A underground, outside Google's site, broke.

That alone should not have caused trouble. But feed B was already switched off. Google says it opened the feed B breaker so repairs could be made after problems turned up in a generator test on October 14. So one feed was carrying the whole load, and then it failed.

Why the backups did not help

The site has batteries, known as UPS units, built to bridge a power loss. Google says they can run things for five minutes. They worked as planned. The generators were meant to take over before the batteries ran out.

They did not. Google blames a voltage-sensing design flaw and a misconfigured PLC. A PLC is a small industrial computer that controls equipment. This one wrongly concluded a healthy utility source was available, even though both power feeds were dead. So it never told the generators to start. The batteries drained and the site lost power.

What Google is changing

Once the utility connection was restored, Google reprogrammed the PLC. It says it has made many changes to the site's electrical setup and tested them, and that they work. It also named two follow-up tasks:

• Build an automated alert system so customers hear faster about problems at a specific location.
• Expand platform-wide testing for infrastructure-level events.

What is not known

The material does not say how long the outage lasted or how many customers were affected beyond the companies named. It also does not say when the follow-up tasks will be done.

What to do

Even the biggest cloud companies can fail. If your business relies on an online service for payments, email or orders, work out how you would cope if it went down for a few hours. A simple plan beats finding out in the middle of an outage.