Top
Best
New

Posted by Bender 11 hours ago

Microsoft blames monster email outage on software defect introduced in July fix(www.theregister.com)
199 points | 136 comments
mike_hearn 8 hours ago|
No big surprises here. Big outages at big firms seem to always look like this.

The electricity industry has this notion they call a black start, where they bring the grid online from zero. It's a very difficult operation because you can't start a generator straight into an offline grid otherwise it gets hit with full demand and can't keep frequency in the right range, which can damage equipment. So they have to start by disconnecting all the loads and then bringing load blocks back in sequence, trying to roughly balance available generation with the size of the next block. In practice black starts are extremely rare, but they can happen after a cascading failure in which something causes some generating stations to trip, redirecting load onto the remaining stations which can't handle it causing them to trip in turn. The entire grid can collapse in just 60 seconds or so once that kind of runaway failure starts.

This kind of outage is very analogous to a black start. A few servers get into trouble - maybe they're in a crash loop or maybe their caches are cold - the load balancers shift traffic onto the remainder which then can't handle the load either and so the failure cascades. Once a service is in this state sometimes the only remedy is to disconnect all load, bring the servers back to stability, then bring load blocks back online in small chunks, making sure to never hit anything with the full load immediately to ensure any caches or JIT compilers have time to warm up.

It's the sort of thing that requires a bit of preparation because naive ways of doing it like just randomly dropping 90% of requests will yield a broken service for everyone. Ideally your LB can drop traffic by modulus of the session cookie or similar, so users are let back in atomically.

Black starting a service is obviously quite hard to practice, but is probably something that sufficiently well resourced SRE teams should try once a quarter or so on a test cluster. I used to be an SRE at Google and although we regularly started up new clusters and moved traffic around between them, actually starting an isolated service instance entirely from scratch was never something we practiced. I remember in a few cases teams discovered devs had introduced circular dependencies by accident, i.e. starting service X required service X to already exist, or it depended on service Y that in turn depended on X. Sometimes you can only find these things by practicing.

nilptr 1 hour ago||
There's literally no such thing as "black start" at Microsoft that I am aware of. The way services are brought online initially is through a build out process coordinated through hundreds of Teams. It's an organic, messy, and painful way of doing things. As such, build outs are a human driven process with bits of automation spanning sometimes 20+ years of engineering touches, troubleshooting guide, and some additional tribal knowledge thrown in passed down through the years. There is little consistency across teams and sometimes even within a team as every Azure Cloud is effectively slightly "different".
CobaltFire 2 hours ago|||
I never worked in civilian power generation, but I can vouch for this on Nuclear Ships.

We had to practice load pickup on a normal start at the pier often, but we likely practiced a full SCRAM to back online (including a FULL load shutdown) more.

Having predefined lists of what gets brought up at various points helps a ton, but it's still not an easy or simple operation.

TomatoCo 50 minutes ago||
I imagine the predefined list is easier when it's all consumers that are under your control. I figure with civilian power some random percent of houses'll kick their AC on after power is restored.
Rygian 2 hours ago|||
Back in 2025, Spain did a very-dark-brown-almost-black start (with help from neighbours), starting with one hydropower plant and increasing from there.

It went remarkably well.

toast0 20 minutes ago|||
> Black starting a service is obviously quite hard to practice, but is probably something that sufficiently well resourced SRE teams should try once a quarter or so on a test cluster.

I mean... you can try, but unless you spend a lot of resources on it, I don't expect test load to be close enough to really practice a black start.

From experience, the best way to prepare for restoration from nasty outages is to have them frequently! The team I was on was pretty good at outage restoration, until we started regularly hitting about a year between outages. After a few years of one or maybe two outages per year, we had lost the muscle memory for protecting services that get overloaded during or after an outage and setting outage messages (which both informs customers and protects customer service queues).

Regularly starting up an isolated cluster at least lets you get ahead of sequencing issues though; hopefully it's enough if you can get the whole cluster started and then modulate inbound traffic (most complex services will need something like you mentioned so that you limit the number of sessions, but sessions that start are likely to proceed successfully).

rawgabbit 3 hours ago|||
It's more testing theater on their end; it's just perfunctory rubber stamping. There was and is no real performance testing under simulated near-real world conditions.
solyin 1 hour ago||
Did you do anything to try and fix that?
rawgabbit 48 minutes ago||
Yes. As another commenter said, you can partition the clients into pods-clusters. And use your test suite software against a realistic pod-cluster. Azure has such a test suite: https://learn.microsoft.com/en-us/azure/app-testing/load-tes...
tonfa 3 hours ago|||
Somewhat related: https://company.sbb.ch/en/railway-development/services-rus/e...

SBB just finished their project to be able to restart the grid in 5min. (There was a famous multi hour outage a few decade ago that I think motivated this work)

nickdothutton 2 hours ago||
3-5 days in the UK according to "regulations". Some talk of it potentially taking 7-14 days under certain bad circumstances.
brunoborges 2 hours ago|||
The biggest challenge of bringing a resilient system back online after an outage, is the resiliency of said system fighting back.
scheme271 1 hour ago|||
Texas was on the verge of having to do a black start a few years ago when winter weather started causing failures. I think people were quoting potentially weeks to get everything back online. As it was, I think people didn't have power for days during sub-freezing temps due to load shedding.
2muchcoffeeman 1 hour ago|||
Good to know even Google introduces circular dependencies!
jvidalv 3 hours ago|||
Happened to Spain last year. The black start.
EGreg 7 hours ago|||
In software servers this is called the “thundering herd” problem. It is why clients are coded do not just exponential backoff but random offsets too.
chinathrow 6 hours ago|||
It's also a well known problem, so no excuse for that.
dijit 2 hours ago|||
a well known problem with more than a few known fixes, one of which is load shedding your upstream based on protocol aware load balancrs.

I've had similar issues where backends couldn't come up due to load killing them as they came up: my solutions was to turn off the load balancers until the backends all came up before opening the door again...

I agree: no excuse.

VirusNewbie 1 hour ago|||
Not only did they not handle thundering herd, it happened across all regions.
nialv7 2 hours ago|||
that's not the problem with grid black start at all? black start is difficult because many types of generators themselves need power to start. it's a circular dependency problem, not a load management one.
avianlyric 1 hour ago|||
It’s both, but every grid has a sub-set of power stations with black-start capability (a crap ton of diesel generators), that exists to break the circular dependency problem and bootstrap the rest of the grid.

But load management while bootstrapping the rest of the grid is still very hard. Often the reason for really long blackout, ones measured in days rather than hours, is repeated cascading failures as operators try to reconnect parts of a the grid, and accidentally overload recovered parts of the grid, causing generators to trip off again.

bragh 1 hour ago|||
That was a very enlightening moment in Factorio when I had already converted everything to power and the whole grid blacked out and I needed to get it going somehow again.
xunil2ycom 3 hours ago||
Well stated.
mindwok 2 hours ago||
The biggest surprise here is that they're using Windows for critical network infrastructure. I'm surprised a Windows process has any role here outside the Exchange servers themselves.
devy 1 hour ago||
Not a surprise. It's Microsoft, Windows is their bread & butter for a long time. And Windows Kernel team is actually full of amazing engineers, reputable computer scientists. Disclaimer, my former boss worked at Windows Kernel team, she's the most talented engineers I have ever worked with, she then went on lead core platform engineer at Shopify many years later.

Before Linux took over the server market in 2000, Windows server was the dominant player in mid-market (enterprise market has IBM mainframes too). But you would have be old enough to remember that decades ago.

mindwok 1 hour ago|||
I have heard this repeated a few times now and it surprises me. Since going deep on Linux I've resented any time I had to use Windows, but folks I really respect often speak highly of the kernel. It'd be cool if they open source it one day and we get a good deep dive into the internals.
makeitdouble 47 minutes ago||
Another angle on this: most ATMs were running windows a decade or two ago (I'd assume most still do, but not sure).

You'd see occasional crash screens, but given the sheer number of machines out in the wild, having only a few crash once in a blue moon speaks amply to the reliability of the thing.

freecodeio 1 hour ago|||
windows kernel might be the only thing high in quality in entirety of microsoft, although that's just my personal opinion
anonymous908213 1 hour ago||
C#, TypeScript, and VsCode are all extremely high quality.
spartanatreyu 29 minutes ago|||
C# was great, Typescript was great (especially considering it also needed to support JS' historical baggage), VSCode was great.

Typescript paused new feature development to port to a new implementation language. It's too early to see if it's going to be better or worse for TS devs, will probably need to wait a few years (at least after the easiest things to pick are gone) to see if it was worth it for anyone who isn't building vscode sized projects.

VSCode was great, until LLMs hit the scene. Now there's a new update every 2/3 days either shoving more LLM-ware to the point that the seams are breaking, or it's a security bugfix for something that should never have been added in the first place.

anonymous908213 17 minutes ago||
I use a pre-LLM VsCodium fork. Works fine, I think VsCode was already feature complete years ago and you can easily extend it yourself if you need something it doesn't provide out of the box.
freecodeio 1 hour ago|||
vscode has become trash

I use it since day 1

syngrog66 1 hour ago|||
engineering unwise. but might be wise purely from biz POV: dogfooding.
RachelF 2 hours ago|||
Someone has to do the testing, and Microsoft tests in production.

In 2014 Microsoft cancelled the dedicated SDET (Software Development Engineer in Test) role, triggering significant layoffs.

mosura 2 hours ago||
Just be grateful they no longer put the configuration in SharePoint. (In an Excel document).
nashashmi 9 hours ago||
Original source information: https://x.com/clamanna/status/2122209697226948729

Better summary: https://www.neowin.net/news/microsoft-finally-explains-why-o...

ncr100 9 hours ago||
They talk about their relatively low-end testing environment PASSing the July code, and being different from their high-end production environment machines, sadly hiding the concurrency bug during the test phase.

Is this commonly acceptable? I'd imagine it's a costly trade-off?

bayindirh 9 hours ago||
Depends on what low/high end are. If low end is 4 core and deployment is 128 cores, this is not acceptable.

But if low is 64 and high is a whole cluster, now that's a different kind of maths.

Edit: The article states the following:

> "lock contention inside an internal Windows process which caused the machine's resources (CPU & memory) to be exhausted."

So, it's more the former. The testing machine is probably a 8 year old server on an office shelf, and prod machines are 64-128 core cutting edge Xeons/EPYCs.

Lock contention. It was MUTEX spins in Windows 95, it's lock contention in 2026. Why this is still a thing, Microsoft?

(Man, I felt old while writing this.)

aspbee555 4 minutes ago|||
I have found that AI really loves using locking/Mutex when working with rust and I have to specifically tell it to not use any locking/use atomics/swaps instead
lokar 9 hours ago||||
I always this was standard to have a test setup in the production environment on the latest hardware (that’s how you qualified that hardware in the first place) running one “unit” of the system (whatever that is) at full scale and load (ideally higher).
jakewins 2 hours ago||
We had this conundrum early on when we were building the Neo4j DBaaS - there was zero chance to afford to run test environments as big as production.

We solved that by sharding customers, so about every 1000 database clusters got their own k8s cluster; production then could run hundreds of those “orchestras”, but we could afford to load test one of them and be relatively confident the test was valid for production.

This relied on the various cloud vendors running the front-line load balancers that took traffic and moved it into the shards though. I was about to say that seemed to be the case for Microsoft here as well, but actually it seems they load tested an individual server with different specs than prod.

Tricky problem though - it seems they reasoned if a lower specced server passed the load test, a much beefier prod server would handle the load too. Wrong, but you can see why someone made that assumption.

awesome_dude 58 minutes ago|||
When i saw the lock contention - I immediately thought distributed lock component
notyourwork 8 hours ago|||
Testing at production scale is extremely difficult. It’s also financially expensive. In my experience, you balance the two where big changes are tested with some simulation setup to gain confidence. Though, the ci/cd pipeline never has close to full scale automated testing.
Choad_20261104 7 hours ago|||
...and many times you test to experience, and experience cannot possibly build interferences that simulate the instances you have not experienced nor what has been imagined based on that experience. some ways of precipitating the demons that are in the ontology of not knowing what you don't know are monte carlo, or perhaps noise injection, but those are time consuming, quite wasteful, and truly hated by people used to working with deterministic machines. that makes finding corner cases very difficult

plus, mistakes are the essence of learning and sometimes mistakes are necessary.

likewise, we are a very efficient world and efficiency hates redundancy, and hates nonconformity...of which those are two of the features that make exploits widely successful and failures massively impactful

pwarner 2 hours ago|||
I think MS could afford a bit of a test lab?
marcta 2 hours ago||
Yes but something that works fine in a test lab won't necessarily cope when confronted with hundreds of millions of users.
munchbunny 1 hour ago|||
At Microsoft's scale, it's extremely difficult to simulate production scale because the sheer size of the production system is overwhelming, and many issues emerge from the combination of sheer volume and the complexity that reality tends to introduce.

Microsoft tests its stuff on their own employees (old citation, but still true today: https://devblogs.microsoft.com/oldnewthing/20110802-00/?p=10...), and that includes Exchange online. However, even though Microsoft is a pretty big company, it's still teeny compared to its customer base.

awesome_dude 1 hour ago|||
I read that part of their statement as "QA were set a priority to ensure existing hardware wasn't affected, which reduced their (QAs) capacity to test on new hardware which was assumed to be safe"

They find QA dirty in their statement, because they don't state who set that priority, nor why the new hardware was felt to be less likely to be affected (what did their risk matrix look like for their hardware profiles i wonder)

flycensed 2 hours ago||
[flagged]
hmokiguess 9 hours ago||
> The engineers who worked feverishly to restore services deserve commendation and will hopefully never have to buy their own beverages in Redmond again (or at least escape the next round of firings).

What does this mean? I didn't understand

giancarlostoro 8 hours ago||
People should buy them drink, as for the firings side note Microsoft recently had a "voluntary exit" offer mailed to a fourth of their workforce:

https://www.geekwire.com/2026/microsoft-makes-new-voluntary-...

How long until Microsoft fires key people to the point that outages become more common? All in the name of AI, which I don't believe for a second, they're just trying to maintain stock market profitability math by firing enough, they'll just hire more people when they make more.

Choad_20261104 8 hours ago|||
The opportunity/command to exit is not always negative. In many of my career moves, it turned out that the next job was better and came far sooner than the severance calculation had allowed for. Some within weeks. Hardly had time to relax. It seems like for talented individuals opportunity always pops up.
giancarlostoro 7 hours ago||
I'm saying its negative for those "left behind" not so much those who took it.
thewebguyd 8 hours ago|||
> How long until Microsoft fires key people to the point that outages become more common?

Already happening. Granted, not a global outage like this one, but there's not a single day without a service degradation or advisory on the admin portal for 365. The "It's now called Microsoft 360" memes are well earned, and their support is atrocious. I get the feeling that everyone who knows how it works is gone.

abrichr 9 hours ago|||
I think it means that people will hopefully buy them beers as thanks
nashashmi 9 hours ago|||
An artistic way of saying the engineers should get praised and rewarded.
AdamN 8 hours ago||
You mean like the Covid nurses? With stickers that say 'You Rock!!' or 'Hero' and a $5 gift certificate to Dairy Queen?
deathanatos 2 hours ago|||
Nurses reported many hospitals paid significant bonuses and hazard pay during COVID, including (already mandated) time-and-a-half, as nurses were in short supply. Many nurses were being paid at hourly rates above that of senior software engineers.

That's not to say their job was or is easy: nursing is far more hazardous than SWE, and significantly more physically and mentally demanding, too. But I do think many got more compensation than your $5 gift certificate implies.

But as a software engineer, if I work overtime on an outage like the above, I'll get precisely $0 of additional pay for it.

BoiledCabbage 6 hours ago|||
> and a $5 gift certificate to Dairy Queen?

Valid only upon a qualifying transaction with a minimum purchase amount of fifty dollars ($50.00). Subject to additional terms, conditions, and restrictions. The display, wear, or possession of any "Hero" sticker shall not be construed, interpreted, or implied as an official or unofficial representation, certification, or endorsement that the wearer possesses the status, title, or attributes of a "hero".

bayindirh 9 hours ago|||
In sysadmin speak this is hoping that Microsoft gives them free beverage credits (or give them some kind of bonus because of their efforts) and don't fire them as soon as the fix is deployed to scapegoat them.

In a rare occurrence, The Register sides with the Microsoft developers. Not that they're anti-developers, but they are anti Microsoft because of its well-earned reputation for EEE and Halloween Documents.

voidfunc 8 hours ago|||
It means they deserve "Thanks" (buying a beer/coffee is a token Thank You) and hopefully they are removed from the pool of candidates for the next layoff.

Surprised so many people are struggling with this...

Having been on incidents like this before though, yea, that aint happening.

bayindirh 8 hours ago|||
> Having been on incidents like this before though, yea, that aint happening.

Been there, through that, and sadly you're right. The bitter part is I didn't want extra compensation or anything. Just a thank you and a day off is more than enough for me, so to recharge and ride the wave of what we (not I, we) have accomplished, so I can work with extra motivation for some time.

Instead, got a 5 minute coffee break and dived to the next non-urgent ticket. Yeah, great way to thank and motivate the people who spent 24 hours in an underground office cutoff from the world, trying to save a system from melting down.

Glad I'm not there anymore.

27183 8 hours ago|||
> Having been on incidents like this before though, yea, that aint happening.

Yeah, instead what will happen is their manager will ask them why they didn't get as many story points done this sprint as they did last sprint and their performance will be marked "degraded".

Molitor5901 9 hours ago||
It kind of feels like they are saying, look, we know we caused the problem, but we fixed it. So we're good, right?
k0morebi 2 hours ago||
Not a network engineer so genuinely curious how these things are handled at the epic scale of 365, azure, aws and so on.

Extent of my knowledge is nginx or traefik load balancing which at this scale is virtually equal to no knowledge. But seems like there would be systems in place to simply drop a sliding percentage of excessive traffic or block some regions once critical thresholds are reached rather than compounding the problem with forwarding the growing number of requests downstream to next victim server.

colonCapitalDee 8 hours ago||
Seems like a pretty standard big company severe outage to me. These things happen!
p4ul 43 minutes ago|
Indeed! A "normal accident" as they say.

And it's also possible that these normal accidents might become more frequent with all the layoffs, AI code, and AI-generated exploits. I guess we'll see.

syngrog66 1 hour ago||
broadstrokes: bitten by concurrency, contention, thundering herd, and a proximate cause which was not in fact the root cause and the root cause had been lurking latent for several months so hard to undo surgically fast. combined with insufficient load/release testing and insufficient mitigation in prod.
toss1 9 hours ago||
>> "the processes that led to an update with a latent defect reaching production deserve a long, hard look. "

Indeed they do deserve a long hard look.

Toss1's conjecture:

We are now seeing the beginning of an era of latent technical debt brought about by too rapidly firing developers and switching to AI code generation (nevermind vibe-coding). Even well-prompted and well-monitored by top expert coders and architects, there is latent technical debt, and it is more diffuse, harder to find, and harder to fix, and will start showing up in outages like this.

(It first seemed the ATC outage was another example, but it now seems it was merely the usual management incompetence if not an outright manipulative ploy to 'justify' more spending for a single-bid contract [0])

[0] https://ajsignalnotnoise.substack.com/p/faa-air-traffic-cont...

nialse 4 hours ago||
Suggested reformulation of the conjecture:

"Complex systems are kept stable by organisational knowledge of their weaknesses and how to compensate for them. Organisational change erodes this knowledge while the underlying weaknesses remain. A previously stable system thus becomes increasingly vulnerable to failure."

When it is said that software is a solved problem, what is ment by that is really that programming, typing up the code, is a solved problem. Software engineering, systems architecture and the complexities in distributed systems are not, while leads to confusion when not being mindful of the difference.

toss1 49 minutes ago||
And so we now have additional mysterious banking system issues today... Hmmm
wduquette 8 hours ago|
So maybe we shouldn’t be putting all our eggs in one basket?
falserum 1 hour ago||
Why? Outage is not so bad if 1) everybody is suffering and 2) it is not your fault.

Cynic in me does not see any need to change anything.

650REDHAIR 15 minutes ago|||
I went through this thought experiment.

The amount of time, energy, money it would take to have an objectively bad fallback option is crazy.

And then what? Most of the rest of the world is still down...

jkahrs595 7 minutes ago|||
exactly my thoughts, but won't somebody think of the shareholders!?
dweinus 2 hours ago|||
Multiple eggs AND multiple baskets? In this economy?
dgellow 8 hours ago|||
Hmm, the best I can do is offer you a larger basket
marcta 2 hours ago||
Subscribe now to my egg-as-a-service product
mxuribe 8 hours ago||
No, no, i think we can and should still load all the eggs into a single basket...we simply have to do so at random frequencies and speeds in order to avoid cracking the eggs, plus when contention inevitably happens, we ask that the egg loaders simply hold off a little...to avoid such traffic, and until egg loading can resume! /s
More comments...