Top
Best
New

Posted by Bender 11 hours ago

Microsoft blames monster email outage on software defect introduced in July fix(www.theregister.com)
199 points | 136 commentspage 2
nelaggy 7 hours ago|
what role did ai play in all of this?

can we blame it for the bug? can we credit it for the fix?

might be the most important part of the full report

devy 2 hours ago||
Interesting that it's their EVP of Copilot, Agents, and Platform at Microsoft, building the next generation of AI-powered tools for work who spoke up for he post-mortem instead of their CTO.

Does that mean Microsoft's internal code are 100% outsourced to agents / AI and human review just won't be able to catch latent bugs like this?

feverzsj 9 hours ago||
Poor Microsoft can't afford more than one low-end testing environment.
preommr 9 hours ago||
This is what happens when you burn billions for no reason.

Github copilot had INSANE usage. Cursor switched away from per request plans in 2025, Copilot did it this year with all the reasoning and more expensive models. It was insane, you could write out a whole page of features, and have it run on one request for hours depending on the loopholes you exploited (some models were discounted for some time, basically free in some cases, and shenanigans around switching in betwen agents)

We talk about opus et. al being heavily subsidized at like 1-2k for $30-$100. I remember seeing figures in the 10s of thousands in subsidizations for GHC. There's no way they weren't losing just massive piles of money.

dgrin91 9 hours ago|||
Doesn't the article state the opposite? The low end was not the problem. It was high end that was the problem.

I like ragging on Microsoft as much as the next guy, but honestly this problem sounds complex & hard to predict. AI or not, this feels like the type of bug that would have slipped by me as well.

The response clearly could have been stronger though.

thesuitonym 9 hours ago|||
The point they were making is that had Microsoft used high-end equipment in testing they may have found this issue before it hit production.
lkjdsklf 2 hours ago|||
Considering the bug was latent for several months in production before they ran into it, unlikely they would have caught it in a contrived test
deeringc 2 hours ago||
Seems like the issue happens readily at high load on a system with many cores. Once the conditions were right it seemed to impact a large proportion of their fleet. They just didnt have that combination in their test/validation pipeline. They likely have some sort of load test in their pipeline, but it's running on machine that's much smaller than their production machines... so it didn't trigger the lock contention bug.

The opposite is true running in production for the last several months. They were runnign on big production machines with enough cores, but never quite had enough load to trigger the lock contention. But once both conditions were true, big machines and high load, due to some unrelated failover elsewhere suddently bringing more load, we then had the magic combination.

It's really not unreasonable to expect an enormous hyperscaler like Microsoft to do load testing on hardware that exactly matches their production systems.

bethekidyouwant 8 hours ago|||
Not unless it was also under high end load
zdragnar 9 hours ago|||
I read "more than one low end" as "more testing environments than one low end" rather than "more low end environments than one". It's a snarky way of saying they should have been testing high end environments for no more reason than they had the money to do it, rather than saying they should have been testing the low end more.
elcritch 9 hours ago||
Think of the poor starving shareholders and CEO pay packages.
rs999gti 9 hours ago||
> poor starving shareholders

You mean you and me? You realize pensions, institutions, and individual investors are shareholders as well?

triceratops 8 hours ago|||
The top 1% wealthiest own 50% of all public stock. https://www.fool.com/research/how-many-americans-own-stock/ So it's mostly not you and me at all.
rs999gti 6 hours ago||
False. A lot of working people, probably even you, are tangentally invested into these stocks through indexes, pensions, endowments, and retirement accounts.
triceratops 5 hours ago||
What's false about it? I didn't say working people don't own stocks. They just don't own most of them. Half of the benefit of any increase in stock prices goes to 1% of the people.

Workers benefit much more from higher wages than higher stock prices. Simply because a significantly higher proportion of their income comes from wages.

Whenever someone says "Think about the retirees and other ordinary shareholders" in response to a suggestion to pay workers more than absolutely necessary, they're full of shit.

qwerpy 3 hours ago||
I don't care who owns most of them, I care that a big part of my retirement savings are in stocks. It may be an inconvenient fact for some people, but my interests are aligned with "the rich" for this particular thing. If some rich guys get richer when MSFT does well, but I also get to have a more comfortable retirement as well, then that's fine by me.

This "rich people will benefit so it's automatically bad" mentality is counterproductive and borne out of jealousy.

rs999gti 2 hours ago|||
This person gets it. A+++
triceratops 3 hours ago|||
Your interests are only fully aligned when you're actually retired and 100% of your income comes from stock investments.

If part of your retirement income comes from rental property, workers being paid more is better for you because they can afford higher rents.

Ordinary workers would better off if they were simply paid more and could put more money into their retirement savings. There's a crossover point where investment gains outpace contributions but it takes most workers a very long time to hit it. Front-loading retirement saving and letting compound interest do its magic over a long time is the way to go. And front-loading is only possible when workers get paid good money early in their careers.

This "whatever's good for rich people is equally good for me" is a mentality born out of being bad at math, and assuming that everyone who disagrees is a raging communist (sorry).

Capitalism is great, moar stock gains are great - I enjoy both. But it's important to look at real numbers.

rs999gti 2 hours ago|||
This is HN, 99% are white collar workers like me invested into pensions, investment accounts, funds, etc. that are mostly stock backed due to working at companies with these programs. Even government and non-profits are the same.

We all need to side with whoever makes the charts go up and to right to ensure our post-retirement lives are well funded.

Myself and many like me don't care what side that is and we don't care much for policies, groups, or programs that are counter to that.

triceratops 2 hours ago||
My guy what's going to make your retirement account go up faster? Contributions or gains?

For at least the first 11 years of a career, contributions outweigh gains. And only if you continuously contribute the same amount every month.[1] But in fact most people's contributions increase because their salary increases, so the actual crossover point is much later, such as year 20.

Being paid more, especially early career, is just more useful to the 99% white collar workers on HN. It lets you save more money for retirement. Compounding more money for longer gives you a better retirement. This is simple math. It isn't a class conflict argument.

[1] https://finance.yahoo.com/news/crossover-point-compounding-m...

qwerpy 2 hours ago|||
Did I miss something? I don't see where someone in this comment chain argued for workers to not be paid more. I very much enjoyed being paid well so that I could retire early! It would be great if others get to be similarly fortunate.

All me and the other guy are saying is that it's possible to not be filthy rich and still want stocks to do well. "Think of the poor starving shareholders" implies that we commoners shouldn't care or want that. But that's false. I'm no longer working, and have no rental income, so I need my investments to do well.

triceratops 1 hour ago||
> I very much enjoyed being paid well so that I could retire early! It would be great if others get to be similarly fortunate.

That necessarily means lower profits to shareholders. The two goals are in tension with each other.

> I'm no longer working, and have no rental income, so I need my investments to do well.

Then you're still rich. Not a 1%-er but at least a 10%-er and very possibly a 5%-er. I want my investments to do well too. I don't pretend it's in the common good.

elcritch 8 hours ago||||
It's a rhetorical jab of course.

The mantra of "shareholder primacy" sounds great in theory but just becomes an excuse for short term profit boosts for a small class of shareholders usually at the expensive of the long term health and growth of the company and long term value seeking shareholders.

Sort of similar to how most countries with "people's" or "democratic" in the government's name tend to be democratic only for the ruling party.

In this case ensuring adequate engineering resources for possibly the most widely used email system in the world doesn't seem crazy. How many long term customers will they loose because of this incident? Perhaps they did have enough resources, but given the recent layoffs to chase AI I sort of doubt it.

rs999gti 6 hours ago||
> It's a rhetorical jab of course.

Not for me and probably a lot of people and institutions too. We have retirement accounts, endowments, and pensions tied to index funds and the index funds have these stocks in them.

We need these companies to do well to ensure we retire and grow well.

elcritch 34 minutes ago||
In my opinion, the trick to ensure they do well is to _not_ shortchange their core engineering to boost quarterly profits.
panarky 7 hours ago|||
Only about 9.5% of Microsoft shares are held directly or indirectly by people with a net worth of $1 million or less.
rs999gti 2 hours ago||
Who cares? For most working people like myself invested into these companies through funds, pensions, investment accounts, savings, etc., all we care about is up and to the right forever or at least when we come into retirement.
dogscatstrees 1 hour ago||
> "lock contention inside an internal Windows process which caused the machine's resources (CPU & memory) to be exhausted."

Looks at my Windows Task Manager...

vessenes 9 hours ago||
Pretty interesting that testing was aimed the worry: low spec boxes, and missed a different problem: concurrency contention for high spec boxes. Gotta be covered by someone's rule of testing, I imagine. Also, woof.
taspeotis 3 hours ago|
They've done this before! https://randomascii.wordpress.com/2017/07/09/24-core-cpu-and...
pluc 3 hours ago||
Nothing more cuts and more AI won't fix!
ZiiS 9 hours ago||
- we added a bug (yes, everyone does).

- our testing didn't cover it (again, this is expected occasionally).

- this cascaded to take down more than a billion people's service (WTF; this is not how you design systems).

- it took 15 hours to deploy the fix (not good, but at this scale maybe).

Feels like they are completely missing the point of where the problem is and how to fix it?

topaz0 56 minutes ago||
Antitrust
VirusNewbie 1 hour ago|||
>- it took 15 hours to deploy the fix (not good, but at this scale maybe).

I've definitely been in similar scale war rooms and 15 hours is not acceptable.

lokar 9 hours ago||
In terms of cascading failure and inability to rollback or fix quickly, MS has always been terrible.
bayindirh 8 hours ago||
Also, don't forget that e-mail is a distributed system and servers retrying always keep a pressure on you, despite their best intentions, and spaced retries.

So, even when you fix the issue, it takes a bit to clear up, and the load gauge stays at the red zone for a while.

lokar 8 hours ago||
Anytime you run a proxy system for a large number of clients you don’t control you need a way to cold start. This means being able to allow traffic through (cheaply!) on a client basis, rejecting the rest.

This lets you start at a very small amount of load and gradually ramp up. But you can’t reject randomly or goodput stays at zero and the backlog never decreases.

bayindirh 8 hours ago||
Sounds reasonable.

I believe SDN systems can do that with ease, but what would be the equivalent method on older systems? Selectively allowing IPs to go through the trickle load into the system?

toast0 18 minutes ago|||
Selectively allowing IPs is a reasonable way to do it. If you think about it, it's terrible, but it will get you going, and it's pretty easy do something like start with allow nothing, then allow all ips under 2.0.0.0, then 3.0.0.0, etc.

Distribution of users by /8 isn't very even and the order that customers will get online wouldn't be an order anyone would pick, but it should be reasonable to implement and synchronize (if needed) and it lets you make progress.

You could do similar in IPv6, start allowing IPs under 2001::, and bump it up little by little... probably each increment would be the 2nd byte, but you do need to increase the first byte too ... or you could be a terrible person and drop ipv6 while you modulate ipv4 and only let ipv6 in when ipv4 is at 100%.

lokar 8 hours ago|||
IP is often hard on the proxy/load balancer, you get a really large list, which becomes expensive.
bayindirh 8 hours ago||
This is also what I thought, so why I asked, actually.
spprashant 9 hours ago||
These kinds of incidents should inject healthy dose of techno-skepticism in the larger conversation. The world is teetering on the edge of turning over all our processes to LLMs and retiring human labor for good.

But at the end of the day these are servers strung together sending packets over copper wires. They are extremely reliant till they are not.

When crisis strikes you can only hope there is a squad of well trained humans who can step in to untangle these messes. But as more of the labor force is stripped out of responsibilities, the longer these outages will take to resolve. Imagine being jolted out of your year-long UBI-funded vacation to login and resolve a multi-continent server outage that you only have a vague memory of.

dgellow 8 hours ago|
> The world is teetering on the edge of turning over all our processes to LLMs and retiring human labor for good.

That’s Silicon Valley fantasy, it’s not even close to reality

greggoB 2 hours ago||
From what I've observed, SV is doing anything and everything it can to make it reality. It's scary to see how successful their campaign has been so far.
tenderfault 3 hours ago||
This happens when you are trying to do weird things with emails rather than just deliver them to destination.

Also this is karma because Microsoft blocked in ignorance entire datacenter networks making SMBs unable to deliver messages to their networks.

Ta-ta!

cyberax 3 hours ago|
So apparently this is the report: https://www.365fanatic.com/wp-content/uploads/2026/11/PostIn... I propose a "latent bug" drinking game.

So to recap:

1. They did not have proper cell-based architecture to partition their system into failure domains ("In this case, due to a separate issue, the normal bulkheads between these different partitions did not stop the cascading failure.").

2. They did not have proper monitoring to see the encroaching collapse.

3. Stuff just keeps falling off: "Third, at the same time, nearly 14% of server capacity for the networking gateway was out of service due to an unrelated issue from the previous day"

More comments...