Google Explains Why You Didn’t Have Gmail

http://www.wired.com/epicenter/2009/09/google-explains-why-you-didnt-have-gmail/



Epicenter The Business of Tech
Google Explains Why You Didn’t Have Gmail

* By Ryan Singel Email Author
* September 2, 2009 | * 1:04 pm |

Want to know why you couldn’t use your Gmail account Tuesday?

Blame the outage on overworked routers that decided to go on strike. At least that’s what Google engineer (and VP) Ben Treynor wrote in a blog post Tuesday night, explaining in relatively clear engineer-ese why Gmail went down for about 100 minutes earlier that afternoon.

At about 12:30 p.m. Pacific a few of the request routers became overloaded and in effect told the rest of the system, “stop sending us traffic, we’re too slow!” This transferred the load onto the remaining request routers, causing a few more of them to also become overloaded, and within minutes nearly all of the request routers were overloaded. As a result, people couldn’t access Gmail via the web interface because their requests couldn’t be routed to a Gmail server. IMAP/POP access and mail processing continued to work normally because these requests don’t use the same routers. That’s good to know. Not because I know anything about configuring request routers to keep them from acting like French factory workers, but because Google is letting me know clearly what happened, what they did and how they will take steps to keep it from happening in the future.

snip

Update: In response to a reader’s question, Wired.com asked Google if a hack or an attack was involved. According to a Google spokesman, “There was no attack or hack. They got overloaded because we had slightly underestimated the the load which some recent updates placed on them.” And if you are one of the Enterprise or Education customers who pays Google for their app service, you just got three free days of service added to your account, on account of Gmail failing to deliver its promised 99.9 percent uptime every month. At least that’s what a Google spokesperson told Wired.com.

(For those of you doing math at home, Google’s 100 minute outage means that even if Gmail is perfect for the rest of September, the best performance it can get is 99.8 percent. That falls below the Service Level Agreement Google promises to its top-level customers.) Gmail’s last major outage was in May, and it had a number of outages in 2008. But Google says its service is still far more reliable than corporate networks, many of which run Microsoft Exchange servers.

sufficiently to handle a minor spike in traffic. Which means that we've been running them too close to capacity for far too long. In addition, our fall back policy for router overload is asinine."

That doesn't say much!
We have to reboot our exchange servers at LEAST once a day!
1000+ users ~1,000,000 messages/day.

I assume that wasn't your decision. You have my sympathy.

We had a RISC CPU that was doing nothing. Loaded with IBM's flavor of UNIX. We would have had to spend ZILCH on an e-mail system. But NOOOOOOOO, management wanted the Micro$oft product. It has been nothing but headaches since.

If M$ had TRIED to design Exchange to be more hacker friendly they couldn't have done a better job. "Here, install something and have it PHONE HOME!"

The exchange server is behind a well protected firewall. Yeah, I know that that will not stop them, and many times people have downloaded garbage that phones home. You don't want to know the number of malware calls we get in a week. Management just shrugs its shoulders, and the support staff keep re-imaging the PCs, and nothing happens to the users. I advocated for a training, where users are told what the threats are, and how they could guard against them, but management nixed it, citing the cost.
Oh well, what do they call it, job security? I think that one of the managers actually told me that my job was secure, and one of the reasons was because MS is so insecure. Such is life.

It reminds me of the Fred Flintstone cartoon where he got fired and is going through the help wanted section:

"Stuffing cotton in little bottles"

later:

"Taking cotton out of little bottles"

I've worked on a lot of servers where that is exactly the routine. I think managers should be forbidden from taking ANY industry trade rag into the bathroom. That causes more problems than probably anything else. Anytime you hear the words "the entire industry is...", you know you're in for a lot of late nights and long weekends because of some boneheaded decision to purchase a clusterfuck.