top of page

Microsoft traces 24-hour Microsoft 365 outage to a July gateway server bug

1 day ago
6 min read
Microsoft 365 outage traced to a July gateway server bug, Data Studios original cover

Microsoft has published a preliminary post-incident report on the global Microsoft 365 outage of November 12 and 13, and it places the origin of the failure in a software change made in July 2026 to the network gateway servers that accept and route connections for Exchange Online. According to the report, which Microsoft shared with customers in the Microsoft 365 admin center on November 15, the defect sat in production for more than three months without causing problems before a combination of extra traffic and reduced capacity exposed it.


The Exchange Online incident, tracked in the admin center as EX1550543, began at 16:20 UTC (11:20 a.m. Eastern) on Thursday, November 12, and Microsoft confirmed full restoration at 16:10 UTC on Friday, November 13, which puts customer impact at just under 24 hours. During that window users in organizations worldwide could not reach their mailboxes through Outlook, Outlook on the web, mobile apps and other connection methods, and mail delivery was heavily delayed.


The report says the defect spread across Microsoft's gateway fleet in less than two hours and that, at the worst point, more than 80% of the servers were unresponsive. Microsoft says its final post-incident report is due on November 20, 2026.


··········


⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣

MICROSOFT 365 OUTAGE ROOT CAUSE: LOCK CONTENTION IN A WINDOWS PROCESS.


Microsoft describes the underlying fault as lock contention inside an internal Windows process running on the Microsoft 365 network gateway servers. Under certain traffic conditions the contention consumed the machines' CPU and memory until they stopped responding. The change that introduced the behavior shipped in July, passed Microsoft's load testing and was deployed across the fleet without any visible effect.


The report explains why testing did not catch it. In Microsoft's words, "Our load testing tended to focus on weaker hardware with fewer cores," while "the contention issue is worsened on machines with more cores and newer hardware." The newer, higher-core production servers were therefore the machines most exposed to the defect, and the lab environment never reproduced the conditions that triggered it.


Two operational factors pushed the gateways over the edge on November 12. At 01:07 UTC, about fifteen hours before customers felt any impact, Microsoft Teams automatically failed over to the primary gateway that also serves Exchange Online, adding load to that infrastructure. At the same time almost 14% of the networking gateway capacity was out of service because of an unrelated issue. Microsoft states that the service still had sufficient capacity headroom, so the defect is what turned the extra load into a fleet-wide failure.


........


Contributing factor

What Microsoft's preliminary report says

Software defect

July 2026 change caused lock contention in a Windows process on gateway servers

Testing gap

Load tests ran on weaker hardware with fewer cores than production

Extra traffic

Teams failed over to the Exchange Online gateway at 01:07 UTC on Nov 12

Reduced capacity

Almost 14% of gateway capacity was already offline for an unrelated reason

Spread

Fleet affected within two hours; more than 80% of servers unresponsive at the low point


........


··········


WHY EXCHANGE ONLINE STAYED DOWN WHILE TEAMS AND SHAREPOINT RECOVERED.


The gateway layer is shared by several Microsoft 365 workloads, and the report draws a clear line between the services that could escape the faulty infrastructure and the one that could not. Teams, OneDrive, SharePoint and the Office apps were able to fail out of the affected gateways onto other infrastructure and recovered sooner. Exchange Online could not do the same, which Microsoft attributes to the scale of the mail service, so mailbox access remained broken for most of the incident.


Recovery itself proved slow because the usual tools did not work. Microsoft writes that its "normally automated recovery processes were not designed to restore service from this broad unresponsive state." Restarting gateway servers brought them back only briefly, since the backlog of client connections waiting to reconnect quickly overloaded them again. Microsoft also says that the regional isolation safeguards designed to keep a failure inside one geography did not contain this cascade, which is why customers in every region were affected.


A conventional rollback was also impossible. Because the faulty change was months old, it could not simply be reverted, so engineers wrote a new update that returned the affected Windows code path to its previous known good state and applied it to groups of servers in stages.


........


Time (UTC)

Event in the Microsoft 365 outage

Nov 12, 01:07

Teams fails over to the gateway that supports Exchange Online

Nov 12, 16:03

Connectivity anomaly detection flags rising connection failures

Nov 12, 16:20

Customer impact begins for Exchange Online (EX1550543)

Nov 12, 16:33

First customer support case received

Nov 13, 00:30

Connection negotiation identified as the main cause of access failures

Nov 13, 07:20

Mitigation starts: update restoring the previous Windows code path

Nov 13, 16:10

Microsoft confirms the service is healthy

Nov 14, 04:15

Final customer update posted in the admin center

Nov 20

Final post-incident report due


........


··········


WHAT MICROSOFT SAID DURING THE OUTAGE AND WHAT IT HAS CHANGED.


While the incident was still running, Microsoft said there was no indication of malicious or cyber activity and described the problem as a technical issue unrelated to cyberattacks or AI. Its first explanation was that the fault had been introduced during an earlier network capacity upgrade and surfaced after a recent network change. The preliminary report replaces that summary with the detailed account of the July defect.


Charles Lamanna, Microsoft's executive vice president for Copilot, Agents and Platform, apologized on his official X account, saying the company is "deeply sorry" for the impact on customers and their business operations, and pointed administrators to the admin center for the root-cause details.


As immediate measures, Microsoft says it paused ongoing and planned deployments and configuration changes on the affected infrastructure, added capacity, rerouted traffic, improved monitoring, and added safeguards meant to stop a traffic redirection from setting off the same kind of cascading failure. The repair items it commits to for the longer term are expected in the final report on November 20.


··········


[[ADPLUS_300x250_1]]

WHAT THE MICROSOFT 365 OUTAGE MEANS FOR COPILOT USERS AND ADMINS.


Microsoft's service health messages during the incident said that features interacting with Exchange Online and email were affected, while parts of Microsoft 365 that do not depend on Exchange stayed available. The boundary is relevant to Microsoft 365 Copilot, whose work in Outlook, such as summarizing threads, drafting replies or answering questions grounded in a user's mailbox, depends on Exchange Online data. Microsoft has not published a Copilot-specific list of affected features in the material reported so far, so the precise effect on Copilot Chat and the Copilot experiences in Outlook remains undocumented. Readers who rely on those features can find how Copilot is licensed and where it runs in our guide to Microsoft 365 Copilot in 2026.


For administrators there are a few concrete steps. The preliminary report is available under Health, Service health, Issue history in the Microsoft 365 admin center, filed under EX1550543 for Exchange Online, with a companion incident, MO1550821, covering knock-on effects on other Microsoft 365 services. Organizations that want to document their own impact should record the start and end times seen in their tenant, the users and locations affected, and any support ticket numbers, since that evidence is what a service credit request under Microsoft's Online Services SLA will need. Under that SLA, credits are tied to each service's monthly uptime and are not applied automatically, so a tenant has to request them through the admin center or its Cloud Solution Provider.


The final report on November 20 is the document to wait for before revising continuity plans, because it should list the committed repair items and explain why the regional safeguards failed. The reported failure sat in the layer that accepts and routes client connections, and the report as described does not mention damage to mailbox databases. If Copilot buttons or panes still look wrong in Outlook after the recovery, the cause is more likely licensing or client configuration, and our troubleshooting guide for Copilot missing in Outlook covers those cases. The same week also saw a separate multi-hour incident on another major AI service, documented in our report on the Claude outages of November 13.


··········


⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁤

MICROSOFT 365 GATEWAY FAILURE: CONFIRMED FACTS AND OPEN QUESTIONS.


Microsoft has confirmed the chain of events in its preliminary report: a July change to gateway servers introduced lock contention in a Windows process, testing on smaller hardware missed it, a Teams failover and a 14% capacity shortfall triggered it, more than 80% of the servers became unresponsive at the low point, and an update restoring the old code path ended an Exchange Online outage that lasted just under 24 hours.


Several questions remain open until the final report. Microsoft has not yet explained in detail why the regional isolation safeguards failed, how it will change load testing so that production-class hardware is covered, which longer-term repair items it commits to, and whether the companion incident affecting other services will receive its own analysis. The answers will determine how much confidence enterprise customers can place in the gateway layer that every Outlook connection, and every Copilot request grounded in mailbox data, passes through.


··········


FOLLOW US FOR MORE.


·····


DATA STUDIOS


·····


datastudios.org


bottom of page