Posted by dkpk 1 day ago
https://www.reddit.com/r/sysadmin/comments/1xuje83/microsoft...
[1] https://github.com/customer-terms/github-online-services-sla
7:52 PM UTC: We've identified that affected servers are failing automated health checks, which is causing them to become unavailable, and this may be related to a recent change to a component involved in processing those checks. We're testing a configuration change on a subset of affected infrastructure to address this and, if successful, will apply it more broadly. In parallel, we've initiated a restart of all affected servers and are validating whether they're successfully returning to service.
7:30 PM UTC: We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've determined that the elevated load isn't driven by an overall increase in traffic, but by a large number of servers being unexpectedly unavailable, which is shifting load onto the remaining infrastructure. We're investigating what's causing these servers to become unavailable while we work to safely return them to service.
7:16 PM UTC: We're aggressively pursuing all pathways to reduce impact. As a precaution, we've paused all deployments and configuration changes while we review recent changes for any correlation to this event. We've also moved traffic for some services off the affected infrastructure to further reduce load, with additional traffic moves in progress.
6:59 PM UTC: We've completed redistributing connection traffic across our worldwide infrastructure, and we're moving additional traffic off the affected infrastructure to further reduce load. We're also restarting affected servers to return them to service. In parallel, we're investigating whether the elevated traffic reflects a genuine increase in requests or a shift in how traffic is being routed across our infrastructure.
When your outsourced system to Microsoft goes down every few months your boss can’t yell at you
Outsourcing is about shifting blame elsewhere
> I dislike Microsoft as much as the next guy, but many businesses impacted by these outages are completely innocent and don't deserve these consequences.
At no point is "keeping people alive" mentioned. Businesses use email for everything. Businesses usually don't care about where their email server is hosted. They just want to send emails. They shouldn't have to care either. I don't put blame on a business that happened to choose Outlook, one of the largest email providers, to host their email.
421 4.3.2 The maximum number of concurrent connections per resource forest has exceeded a limit, closing transmission channel. xyzxyzxyz.outlook.com. PRX8 [Name=InboundProxyTrackingAgent][AGT=IPT][MxId=xyzxyzxyz] [xyzxyzxyz.ausprd01.prod.outlook.com 2026-11-13T01:08:15.373Z xyzxyzxyz]451 4.3.2 Temporary server error. Please try again later ATTR2 [MI3PEPF00008486.eurprd02.prod.outlook.com 2026-11-12Txxxxxxxxxxxxxx]
I want to believe this is the cause. It'd be especially funny because businesses were already anticipating the productivity loss when GTA6 comes out, but now we have actual productivity loss because of this outage (and it's affecting people who were never going to play the game anyway, lol)
Sure GTA6 may be a slight uptick in the amount of data being downloaded like any new game being released, but I would wager that the amount of data they are normally pushing out for downloads just on a regular basis (wether new purchases, redownloading games, game pass, etc) is already quite high.
If they roll it out evenly the week prior, then today (1/7th) is just 75 PB. To get that done in a single day would require a sustained 6.9 Tbps.
I'm sure this is all handled by a CDN at some level... but again it's not nothing.
6.9 Tbps to get it done in a single day is even 3x less than the minimum on Steam in the last 48 hours https://store.steampowered.com/stats/content/ and they don't even have GTA VI to distribute.
Maybe if people are spinning up simple VMs and VPCs they can manage multicloud through a hub, but if you're using the big three for what they're meant for, you're using cloud DBs, API gateways, lambdas, and whatnot. With how much tuning, billing monitoring and specific configuration is needed, it seems like that'd be a hard thing to abstract. Even using an abstraction like Terraform to try to keep out of the weeds takes a lot of work, from what I remember five years ago.