Microsoft outage: Months-old bug, malfunctioning internal connections, and server capacity led to blackout

Each of these factors may have easily been resolved on their own, but, when taken together, they led to a major outage that was harder to mitigate, Microsoft executives said
Microsoft issued an initial report on Monday that identified three factors that led to the 24-hour blackout of its services: A months-old bug, malfunctioning internal connections , and server capacity. 

Microsoft issued an initial report on Monday that identified three factors that led to the 24-hour blackout of its services: A months-old bug, malfunctioning internal connections , and server capacity. 

Microsoft has identified the root causes of the global outage that crippled Outlook, Exchange, and a host of other vital office software last week.

The office software giant issued an initial report on Monday that identified three factors that led to the 24-hour blackout of its services: A months-old bug, malfunctioning internal connections , and server capacity. 

Each of these factors may have easily been resolved on their own, but, when taken together, they led to a major outage that was harder to mitigate, Microsoft executives said.

The disruption began on Thursday with users reporting interrupted access to their Outlook email accounts. As it continued into the next day, millions of people worldwide said they could not access their email in any capacity as well as other Office 365 services. 

Jails could not release prisoners, hospitals turned to paper records, schools lost contact with parents, and police departments could not receive crime reports. 

Microsoft said it had resolved the issue and restored service by Saturday, also issuing an apology. The company did not attribute the blackout to a cybersecurity breach.

In a post on X on Monday, Charles Lamanna, the executive vice-president of Copilot, Agents, and Platform at Microsoft, said there was a latent bug or defect from July that caused Microsoft’s servers to “become overwhelmed and unresponsive”.

Lamanna wrote that the company typically has two available networking gateways, or entry and exit points that connect separate systems and allow data to pass between them. 

However, one of them was not working for Microsoft Teams which overwhelmed the primary gateway. Subsequently, over 14% of the company’s server capacity was also not working due to an unrelated issue from the previous day.

“Taken together, the increased load and the reduced server capacity, caused the network gateway to encounter the latent bug from July,” Lamanna wrote. 

“This caused requests to build up until servers ultimately stopped responding. This occurred during the peak window of traffic in the United States.” 

The company also had trouble mitigating the bug because it originated months before, Lamanna said, so it would not be easy as rolling back a recent change to undo the defect.

“Recovery took longer than usual because as servers returned to service, they were immediately overwhelmed by waiting traffic and became unresponsive again,” he wrote. 

“Our normally automated recovery processes were not designed to restore service from this broad unresponsive state. 

"Additionally, since the main code defect was a latent bug from several months previous, our normal rollback, flighting, and feature flags were unable to quickly revert the change.” The company plans to publish a final report which will include more information, he said.

Analysts say Microsoft’s global outage highlights the need for companies to examine how much they rely on a single operating system.

“This is a prime example of the importance of not only cyber resilience, but business resilience,” said Matt Hull, vice-president of cyber intelligence and response at NCC Group. 

“Reports that hospitals, schools, and other essential services are affected demonstrate the wide impact that a system outage can have, regardless of the cause.”

Guardian

More in this section

Cookie Policy Privacy Policy Brand Safety FAQ Help Contact Us Terms and Conditions

© Examiner Echo Group Limited