Google says routine upgrades to blame for Gmail outage

Google is blaming load miscalculations during routing server upgrades for the widespread failure of its email service.

Google is blaming load miscalculations during routing server upgrades for the widespread failure of its email service Gmail for almost two hours last night.

Millions of users of the free Gmail and paid for business version of the service were faced with an "Unable to reach Gmail" error message as their computers failed to connect to the service.

In February, Google blamed a similar failure of its Gmail service on routine maintenance in a European datacentre.

Ben Treynor, Google's engineering vice-president, apologised to users for the latest outage in a blog post, saying the outage was a "big deal" and Google was treating it as such.

According to Treynor, Google took a small number of Gmail servers offline to perform routing upgrades, but underestimated the load that it put on routers that direct web queries to the appropriate Gmail servers.

"At about 12:30pm Pacific a few of the request routers became overloaded and in effect told the rest of the system 'stop sending us traffic, we're too slow!'," he wrote.

The load was transferred to the remaining request routers, causing a few more of them to also become overloaded, and within minutes nearly all of the request routers were overloaded.

"As a result, people couldn't access Gmail via the web interface because their requests couldn't be routed to a Gmail server," said Treynor.

However, he said IMAP/POP access and mail processing continued to work normally because these requests do not use the same routers.

According to Treynor, the Gmail engineering team was alerted to the failure within seconds and brought additional request routers online to restore the Gmail service.

"We've turned our full attention to helping ensure this kind of event doesn't happen again," said Treynor.

Google has already increased request router capacity beyond peak demand and plans to implement further reliability improvements in the coming weeks.

"Gmail remains more than 99.9% available to all users, and we're committed to keeping events like today's notable for their rarity," said Treynor.

Read more on Voice networking and VoIP