Top
Best
New

Posted by MattIPv4 6 hours ago

GitHub is once again down(www.githubstatus.com)
345 points | 179 comments
guywithabike 6 hours ago|
The worst part of all this is that GitHub's CTO and VP of Engineering sent out the usual "here's what we'll do to fix things" letter to their larger customers and, without exaggeration, it boiled down to: 1) "Here's a bunch of stuff we already did!" which... clearly isn't working, and 2) "We're continuing our Azure migration." also clearly not working.

So needless to say, if you depend on GitHub for critical business operations, you need to start thinking about what a world without GitHub looks like for your business and start working your way toward that. I know my confidence in GitHub's engineering leadership is at rock bottom.

Eji1700 6 hours ago||
I could sorta see a situation where the reality is "we're in the middle of a miserable transition and it'll clean up when we're done" but I don't think anyone has confidence that's all it is at this point.
everforward 5 hours ago|||
Even that doesn’t really make sense to me, unless they’ve done it in a way where everything has to move at once.

Everywhere I’ve worked, if a migration is causing this much downtime then you kill the migration or slow it down. If every change has a 10% chance of bringing the site down, you only do a change every week or two until you can work out the kinks.

shrikant 4 hours ago|||
...or you keep fighting forward with the migration, because if it's seen as a failure then some pretty big heads will have to roll...
acedTrex 5 hours ago|||
I mean, they are seemingly breaking every week or two so that might be what they are doing.
suriya-ganesh 5 hours ago|||
also it should be noted that LinkedIn had a 5 year plan of migrating everything to azure but abandoned it after a year.
sysworld 5 hours ago|||
ooooh, they're migrating to Azure, now everything makes sense.
spondyl 5 hours ago|||
Here are some relevant excerpts from an October 2025 article[1]:

> In a message to GitHub’s staff, CTO Vladimir Fedorov notes that GitHub is constrained on capacity in its Virginia data center. “It’s existential for us to keep up with the demands of AI and Copilot, which are changing how people use GitHub,” he writes.

> The plan, he writes, is for GitHub to completely move out of its own data centers in 24 months. “This means we have 18 months to execute (with a 6 month buffer),” Fedorov’s memo says. He acknowledges that since any migration of this scope will have to run in parallel on both the new and old infrastructure for at least six months, the team realistically needs to get this work done in the next 12 months.

If you consider that six month parallel window to have started from the time of the October memo (written presumably at the start of October), then that puts us currently or past the point where they would have cut off their old DC and defaulted to Azure only.

Whether plans or timelines changed, I have no idea of course but the above does make for a convenient timeline that would explain the recent instability. Of course, it could also just be symptomatic of increased AI usage generally and the same problems might have surfaced at a software level regardless of whether they were in a DC or on Azure.

Putting that nuance aside, personally I like the idea that Azure is simply a giant pile of shit operated by a corporation with no taste.

[1]: https://thenewstack.io/github-will-prioritize-migrating-to-a...

Barrin92 5 hours ago||
>It’s existential for us to keep up with the demands of AI and Copilot

if by chance the CTO reads this, as a user of GitHub I would find it really existential if GitHub continues functioning as a reliable hub for git workflows (hence the name), and I have the strong suspicion nobody except for the shareholders gives a lick about copilot or 'AI' if it makes the core service the site was designed for unusable

ncruces 4 hours ago|||
AI and Copilot increase the load on git workflows.
jwoq9118 4 hours ago|||
For GitHub to remain profitable they have to appease those shareholders you mentioned.
denkmoon 4 hours ago|||
Why? What is the correlation between profit and shareholder sentiment (besides the fact that shareholders want said profits)? They don't really influence the operation of the business meaningfully.
kevin_thibedeau 2 hours ago||
Growth chart gotta go up. Only chumps run a business that makes a steady return.
denkmoon 40 minutes ago||
Sure, but I think it's the wrong way around. Appeasing shareholders doesn't make you profitable, being profitable appeases shareholders. I think there is a wealth of evidence that appeasing shareholders actually impedes profits overall.
conception 4 hours ago|||
Incorrect. They need to appease/trick/threaten/etc those that are paying for their services. Shareholders just demand they do so at the greatest (often short term) rate.
comice 5 hours ago||||
yeah currently working with Azure. what a PITA.

I wonder if the extended downtime is just due to the on-call engineers waiting for their azure auth tokens to refresh within azure's own damn network.

pm90 5 hours ago||||
i heard that they asked LinkedIn to do this too and they either refused or their systems were too complex so they refused to. Maybe that explains why LI availability seems ok
kleene_op 5 hours ago||||
Azure, the color of BSOD
cyanydeez 5 hours ago|||
they're not just migrating to Azure, they're vibrating to Azure!
AustinDev 2 hours ago|||
Vibe coding and it's consequences.

"The evidence is clear: Either you embrace AI, or get out of this career." -Github CEO

"Sooner than later, 80% of the code is going to be written by Copilot. And that doesn’t mean the developer is going to be replaced." -Github CEO

justinko 2 hours ago|||
There’s plenty of alternatives, but people continue to stay. Therefore, it’s not as bad as you think it is.
packetlost 6 hours ago|||
I second this. I'm done.
ryukoposting 5 hours ago||
Is "migrating to Azure" the new "migrating to SAP?"
trvz 5 hours ago||
That’s not for to … SAP.
proc0 6 hours ago||
It's starting to really look like the AI effect. It might be coincidence but I've noticed a lot more downtime and bad software lately. The last Nvidia drivers gave me a blue screen (last week or so), and speaking about Windows, I froze updates last year because it was clear they were introducing a bunch of issues with every update (not to mention unwanted features).

I like AI but actually not for coding because code quality is correlated to how well you understand the underlying systems you're building on, and AI is not really reasoning on this level at all. It's clearly synthesizing training data and it's useful in limited ways.

davebranton 3 hours ago||
Interesting how many people "Like AI" because it's good at all the jobs other than the one they happen to make a living doing.

Did you hear about the screenwriters school in which the professors said to avoid AI for writing, but it's great for storyboards. And the storyboard school where the professors said the opposite?

The reality is that AI isn't actually "good" at anything. It produces passable ersatz facsimiles of work that can fool those not skilled in the art. The second reality of AI is that everyone is busy cramming it into their products at the expense of what their products are actually useful for.

Once people realise (1), and stop doing (2), the tech industry has a chance of recovering.

proc0 3 hours ago|||
Yeah, I think I heard about that. Within certain domains it is certainly a useful tool. I would say things like online search are much nicer now (in that asking an AI is equivalent to searching online but it summarizes it for you). Online search fits the strengths of LLMs nicely, but right now it's being sold as a silver bullet, which it's not.
danillonunes 2 hours ago|||
It's a version of the Gell-Mann Amnesia effect.
someperson 5 hours ago|||
GitHub has been unreliable since before AI. Though it's definitely gotten far worse.

Seemingly the decline started with the Microsoft acquisition in 2018, and subsequent "unlimited private repository" change in 2019 (to match Gitlab's popular offer)

hparadiz 4 hours ago||
One example is the search being broken for CI logs. It takes over your browser's search hotkey too. What happens is every stage of the log is collapsed so the search doesn't work until you trigger the expansion but if you attempt to search before expanding the search will never work after it's been initialized. It's pretty infuriating when you're trying to find something in a giant build log.
thewhitetulip 1 hour ago|||
We had more and more outages on EVERYTHING since AI. Not sure what they're doing.
ivanjermakov 4 hours ago|||
I bet on rushed Azure migration. A lot can go wrong it devops.
newbish 5 hours ago|||
I think maybe it's not that GitHub is using AI, but that the amount of AI slop going into GitHub may be more than they expected.
qbane 5 hours ago||
Productivity is finite. If you pivot entirely to the AI stack, you're going to lose bandwidth for everything else. It's an opportunity cost problem.
boxingdog 5 hours ago||
[dead]
paxys 6 hours ago||
Took a full 8 years for a Microsoft acquisition to go to shit, which is probably a record. Kudos to the Github team for holding out this long.
AndroTux 5 hours ago||
I’m still baffled that Minecraft is doing so well, despite the whole Bedrock thing. At this point I think Microsoft just forgot that they bought Mojang.
PaulKeeble 5 hours ago|||
Its had its fair share of outages and outrageous changes that overreach the bounds as well. Its more stable than github is but its had at least 2 sessions of downtime this year that I recall and they were both quite long (day length).
spauldo 5 hours ago||||
They'd lose a whole lot of users if they killed Java edition, since the modded community is so large. They'd quickly find one of the Minecraft clones reaching feature parity. And there's no good reason for it - it's not like Java is a threat anymore.
AndroTux 5 hours ago||
Exactly. So why isn't Microsoft doing just that? Isn't that how Microsoft usually handles things? Just look at Xbox. They essentially screwed up everything they could and then some.
joshribakoff 2 hours ago||||
They don’t enforce or even default to 2fa to change the account email. In addition, they have no process to get a human to reverse account takeovers. Just a web form that tells you to call a number that redirects you back to a web form
cedws 4 hours ago||||
If I remember correctly UK players can no longer chat at all until they verify their ID.
7777332215 5 hours ago|||
Minecraft is a trick up their sleeve yet to be used. Manipulate and indoctrinate the youth.
Biganon 5 hours ago||
Indoctrinated by cubic cows
merlindru 5 hours ago|||
GitHub has always been incredibly outage riddled no? This is not a MSFT thing
theamk 29 minutes ago|||
It was not nearly as bad... I remember our company migrating to github.com, and believe it or not, it was significant performance/uptime benefit over our self-hosted instance.

(And the first thing to go was occasional 500's on github-hosted files.. the core service itself - git, PR, actions - were pretty stable until recently)

nine_k 5 hours ago||||
I don't remember that happening so much (if ever) in, say, 2016. But the frequency of noticeable incidents seemingly has been rising steadily since around 2023. The Azure migration apparently only exacerbated it.
georgel 5 hours ago|||
I remember it going down semi-regularly in the 2013+ era, and seeing HN posts about it. Especially if you were using a package manager reliant on GitHub like Cocoapods. It seems to me it is more "impactful" on the dev community now that they have gone past just being a centralized Git server for the team, to being the thing that does deploys and all sorts of other things.
0x457 3 hours ago||||
I remember seeing unicorn daily and "webhook delivery delayed" weekly. I think it got better, but also they got more traffic, now millions of agents read files separately over and over again.

IMO it's much better now.

gbear605 5 hours ago|||
Circa 2019, my office had a bell that we would ring whenever GitHub had an outage, and it was rung several times per week.
bandrami 2 hours ago|||
I think having a single 9 of uptime is a relatively new thing even for GH
xeonmc 6 hours ago|||
How fast was Skype?
cm2187 5 hours ago||
MSFT acquired Skype in 2011, so I would say only a few months:

https://trends.google.com/trends/explore?date=all&geo=GB&q=s...

htrp 6 hours ago||
To be fair a bunch of this is because the CEO after Nat Friedman (Thomas Dohmke) was pushed out in August 25.
carlosft 5 hours ago|||
And a ton of the top end ruby staff have left. Many of them ended up at shopify. There is a growing about of non ruby/rails code at github, but most of the system that people think of when they think github are ruby/rails.
ambicapter 3 hours ago||
Shopify is on the AI-everything train as well, we'll see how that goes.
workfromspace 5 hours ago|||
Who was also the last CEO, right? Is this a coincidence?
inaros 6 hours ago||
Every day more Microsofty...they should rename to "Your Repository Needs To Restart To Apply Updates"
Waterluvian 6 hours ago||
It's now safe to turn off your expectations.
kenhwang 6 hours ago|||
I wonder what the average career tenure of the userbase here is now, because Github was slow and flaky well before Microsoft got involved.

Maybe it wasn't as noticeable when Github had less features, but our CI runners and other automation using the API a decade ago always had weekly issues caused by Github being down/degraded.

jmtulloss 4 hours ago||
The best stretch Github ever had was post-acquisition when Nat Friedman as CEO.
bartread 6 hours ago|||
"It looks like you're trying to develop some software.

Would you like help?

- Get help with developing the software

- Just develop the software without help

[ ] Don't show me this tip again"

morkalork 5 hours ago|||
Would you like to setups repository backups with OneDrive?
amarant 6 hours ago||
Lol, someone should make a pre -commit hook that reboots your computer with a message like this!
corvad 6 hours ago||
Just wait until github comes up with an outage tuesday.
mememememememo 6 hours ago||
Down? No sir we are not down. There are elevated error rates and degraded performance.
karim79 6 hours ago||
The update to .NET framework went badly and we need to reinstall Windows.
xtracto 5 hours ago||
An isolated group of customers are experiencing elevated error rates and degraded performance.

FTFY. (I've read AWS word it like that)

mememememememo 4 hours ago||
Technucally a set is a subset of itself.
stevepotter 6 hours ago||
I'm just going to stand by until Microsoft is back in everyone's good graces again by releasing some oss software that we all swoon over
kace91 5 hours ago||
Is that a cyclical thing? I went through the VS code+typescript good graces era but I didn't know there were previous cycles.
RevEng 5 hours ago||
Not that I remember. VS Code was a surprising turnabout for a company that was both adamantly closed and threw FUD around like monkeys in a zoo.
philipallstar 5 hours ago||
You've been here before!
hirako2000 6 hours ago||
I'm glad I moved over to forgejo. Being selfhosted, the UI loads faster. Most importantly, the thing is always responsive.
mfenniak 5 hours ago|
As a developer working on Forgejo -- glad you like it!
hirako2000 5 hours ago||
It hosts all the repositories backing applycreatures, we ran dozens of git projects on the same instance, have teams, you guys did a phenomenal work. I would say it's even easy to customise.

https://foja.applycreatures.com

Edit: it has a wonderful API so I posted the link it may tempt some to ditch MS/Azure hub.

sathish316 2 hours ago||
How much of this is due to Microsoft Culture of not innovating and buying leading companies with their revenue from windows/office and slowly destroying the aspects of companies that made them great in the first place?

Is all the recent GitHub downtime entirely attributable to GitHub AI Copilot related development? How hard can it be to reduce the blast radius of new AI features to not affect the core parts of hosting repositories? Because of Copilot everywhere, The UX has become bad and I had to click all over the place and on my profile to find repositories.

odiroot 6 hours ago||
They invented the perfect solution to stop supply chain attacks.
anematode 4 hours ago|
Credential stealers hate this one trick!
dsm4ck 6 hours ago|
Microslop at it again
kraemahz 5 hours ago||
Oh, they've gone beyond micro. It's Macroslop now!
workfromspace 5 hours ago||
Please don't use that term; it makes them look bad :p /s
More comments...