Top
Best
New

Posted by dkpk 5 hours ago

Service degradation on Microsoft 365(status.cloud.microsoft)
https://www.abc.net.au/news/2026-11-13/microsoft-outage-affe...

https://www.reddit.com/r/sysadmin/comments/1xuje83/microsoft...

https://downdetector.com/status/microsoft-365/

412 points | 402 commentspage 2
matt3210 5 hours ago|
Software / software service quality has been declining for a few years IMO.
TacticalCoder 3 hours ago||
> ... has been declining for a few years IMO

Interesting: there's a huge correlation with everybody beginning to use LLMs.

And we dismiss it as being non-causal by saying it all started going downhill before the advent of LLMs?

I don't buy it. I think we're seeing the results of what happens when you hand the keys of the kingdom to LLMs.

perching_aix 2 hours ago|||
I really don't see the correlation, actually. On the contrary, I'm finding nasty defects with it that have been dormant for long and handwaved away, with repros to reveal them.

It's been full speed downhill for many years, and the failure modes are pretty distinct.

CamperBob2 3 hours ago||||
Using LLMs properly leads to better software, not just more of it.
10111001 3 hours ago||
> Using LLMs properly leads to better software, not just more of it.

Current industry best practice is to use LLMs improperly.

hackerbrother 2 hours ago|||
How do you use LLMs properly?
NichoPaolucci 2 hours ago||
With engineering rigor. They are fantastic tools for quickly navigating large codebases, ingesting large amounts of data, refactoring legacy code, compiling multiple formats of data, following defined patterns, and many other proper uses for the tools.

That doesn’t mean the code/result doesn’t still need to meet our engineering standards.

CamperBob2 2 hours ago|||
Hopefully the people who feel that way are already close to retirement, so the choice won't have to be made for them.
10111001 26 minutes ago||
> Hopefully the people who feel that way are already close to retirement, so the choice won't have to be made for them.

One of the best ways to deal with doing something the wrong way is to get rid of all the people who know how to do it correctly. That way no one knows any better, so no one has to confront the fact they're actually doing it wrong.

vconnor 2 hours ago||||
It's the same with layoffs: there's always a top comment trying very hard to convince everybody that AI usage has played no part.
skeaker 1 hour ago|||
Well no, Windows has been more or less all downhill since Vista.
ocdtrekkie 5 hours ago|||
This usually isn't a software quality issue, it's overcomplicated infrastructure failures cascading through a complex environment causing additional systems to fall over.
worik 5 hours ago|||
> This usually isn't a software quality issue... overcomplicated infrastructure failures cascading through a complex environment

That is a software issue.

cheesemonk66 4 hours ago|||
Basically the number one issue these days
unethical_ban 4 hours ago|||
I don't consider fat-fingering a BGP route or a DNS entry to be a "software" issue directly.
elxr 3 hours ago||||
Infra is >90% software nowadays
iAMkenough 5 hours ago|||
AI agents should be able to resolve this quicker than employees, right?

Service quality is in better hands after the layoffs, at least that’s what investors are led to believe.

SoftTalker 4 hours ago|||
Assuming AI agents are not the cause of the incident.
bitteralmond 4 hours ago||||
AI is good at incident diagnosis, but AI slop code may have actually caused this: designing to avoid cascading failures is something AI doesn't do so well
davidee 4 hours ago|||
The place where I consistently see AI/LLMS/Agents/language-widgets-with-tools fail at making decent code (and more often configuration) is infrastructure.

The number of times it has spit out a python program where a much simpler bash script would do, is too many to count.

I have a pet theory that if you don't know what you really want, and aren't skilled enough to understand what you got, infra code is THE WORST.

Anecdote: I wanted a simple change to an ansible playbook today. GPT-6-Astra/codex came up with the most convoluted, brittle solution possible the first time, even though it was extending and already clear, idiomatic playbook. It took some amount of offensive and critical prompting to get it to use ansible's built in modules and tooling to do the same job - and do so with nearly 80% less configuration and like 1000% more human readability.

Some part of me harbours a sneaking suspicion that the "throw everything at it" ethos is by design, not as a defensive measure against crappy code, but as a way to ensure there's so much code and it's so convoluted orgs and operators (or developers) are stuck/addicted.

It's still a force multiplier - but it's by no means driverless, especially for existing/already-running infra.

MiguelPablo 3 hours ago||
I work as a (depressed lol) sysadmin apprentice in a small MSP company (I live in Europe) and boy do I got a story about Ansible playbooks: We made an offer to one of our clients to automate the deployment of software to their computers (this is a client with around 150-200 employees, depending on whether you count only the "core" employees or also security and maintenance staff). Before we made and implemented this offer using Ansible, they did not have any sort of Software Deployment tool whatsoever, everything got installed fully manually.

(they also used to have local admin rights on all of their client computers until one employee got phished and literally gave away their personal ID card via a webcam)

In any case, guess who got to implement all the relevant playbooks for installing software and what tool they used for it? Our company literally handed the job to an intern who had no IT background whatsoever. I say this not to be mean, he was legitimately just at our company because a government program required him to do an internship.

He used Deepseek (our company does not believe in purchasing LLM subscriptions, at least not for us normal employees) to write literally all of the playbooks for any kind of software that mattered to our clients.

Now, as I said before, I am only an apprentice, and I am certainly not a good programmer by any means, but I read through parts of the Ansible documentation and had worked with it before, and I understood enough to at least be able to roughly judge whether a playbook made sense or was just way too complicated and convoluted.

I had to help this intern out multiple times, because he hadn't yet understood that, in order to write a functioning playbook, you first had to somewhat understand how to deploy a piece of software on a single test machine using CMD or Powershell syntax with an according .exe or .msi.

He simply prompted, pasted the playbook and then reprompted with the error codes from the failed playbook until it somehow worked or until I had to help him out.

This took place in December or November of last year and the playbooks are mostly still pretty shitty if I am being honest.

I did take away an important lesson from this "episode", though: LLMs can be very usefuly for writing code, or in this case, Ansible playbooks, but only if you have a somewhat decent and basic grasp of how software installation works via CMD/powershell and if you took the time to read through and learn about at least the basic Ansible concepts.

Since I am only an apprentice (on paper and in terms of my knowledge and how I get paid, not in terms of how much I am expected to earn for the company), I have painfully experienced how important it is to possess a certain level of understanding about the infra-related code and scripts that you would like LLMs to write for you, because otherwise it very quickly spirals out of control and starts throwing everything at the problem, just as you described.

worik 5 hours ago|||
Coincidentally with massive programmer layoffs?
combobyte 4 hours ago||
I blame bad vibes
RajT88 3 hours ago||
ThousandEyes while it was loading seemed to have centralized around it mostly being a Microsoft outage.

https://status.cloud.microsoft/m365

Azure status does not document anything:

https://azure.status.microsoft/en-us/status

All that said... Stuff on my home PC started getting various DNS issues around the time my email started acting weird. Hard to say how widespread it is. It may be bigger than Microsoft even.

jwnin 1 hour ago||
This is an M365 outage, not an Azure outage. M365 has a separate status page with information: https://status.cloud.microsoft/m365
fock 2 hours ago||
everything non-microsoft works fine here!
vermontdevil 20 minutes ago||
Looks like a new vector attack for the future.
testplzignore 5 hours ago||
https://x.com/MSFT365Status/status/2120938282204688468

> We're investigating reports of users experiencing issues accessing Exchange Online mailboxes through multiple connection methods in the Americas. More details can be found in the Microsoft 365 admin center under EX1550543.

jwnin 1 hour ago||
It's insane that Microsoft continues to have global outages. Tech debt is real, service delivery on a planetary level means change is measured and deliberate, but...it's been years. YEARS.
infn8loop 3 hours ago||
I suspect some possible relation to SignDrop or attempts to patch it.

SignDrop is an unpatched Windows zero-day local privilege escalation (LPE) exploit that allows a standard user to gain NT AUTHORITY\SYSTEM privileges by abusing the Common Log File System (CLFS) kernel driver (clfs.sys). Security researcher Abdelhamid Naceri (also known as MSNightmare or Nightmare Eclipse) published a detailed technical write-up and functional proof-of-concept (PoC) code on GitHub on November 10, 2026.

antx 2 hours ago||
Not impossible, but very unlikely. CLFS isn’t remotely reachable, it's a kernel driver reached through local API calls and has no network listener. AFAIK the historical CLFS CVEs are all rated as local.
stackghost 3 hours ago||
Are you finding these vulns with Mythos/Daybreak or some other tool? Chinese models perhaps?
JPascal_ 3 hours ago||
In my language many would say “descansito” :)
rossdavidh 2 hours ago||
I need to start working on adding this word to English...
tech_ken 3 hours ago||
Oh estoy haciendolo
xtiansimon 2 hours ago||
WOO-HOO! I'm going home.
andrewinardeer 4 hours ago|
GitHub the other day went down because of an influx of anonymous users. Microsoft has hinted that surging connections caused the outage. I wonder if these two are linked?
More comments...