Top
Best
New

Posted by salkahfi 2 hours ago

An Update on GitHub Availability(github.blog)
123 points | 121 comments
embedding-shape 1 hour ago|
Hah, love that now they say "Our priorities are clear: availability first, then capacity, then new features" when 6 months ago, it was seemingly exactly the same except Azure supposedly was gonna save them:

> GitHub Will Prioritize Migrating to Azure Over Feature Development - GitHub is working on migrating all of its infrastructure to Azure, even though this means it'll have to delay some feature development.

> In a message to GitHub’s staff, CTO Vladimir Fedorov notes that GitHub is constrained on capacity in its Virginia data center. “It’s existential for us to keep up with the demands of AI and Copilot, which are changing how people use GitHub,” he writes.

https://thenewstack.io/github-will-prioritize-migrating-to-a...

So the currently delayed feature development is now gonna be further delayed, yet almost every week we see new features and changes, just the other day the single issues view was changed, as just one example. And it was "existential" 6 months ago yet they keep stumbling on the exact same issue today?

Even if they're focused exclusively on reliability and uptime, we get the experience that we have today, kind of incredible how a company with the resources of Microsoft seemingly are unable to stop continuously shot themselves in the foot. It's kind of impressive actually. As icing on the cake, they've decided to buy up all popular developer services then migrate them all to the same platform, great idea too.

madeofpalk 2 minutes ago||
This seems uncharitable. Priorities aren't exclusive, especially at scale across large engineering orgs like GitHub. It could be that these are the top level priorities, but teams or individuals who aren't able to contribute to these priorities will work on other things like new features.
ncruces 3 minutes ago||
> So the currently delayed feature development is now gonna be further delayed, yet almost every week we see new features and changes, just the other day the single issues view was changed, as just one example.

They did that as a panic mode hack to mitigate performance: https://news.ycombinator.com/item?id=47912521

maccard 1 hour ago||
It's kind of hard to read this with a straight face.

The unlabelled graph with big numbers on top, the priorities that don't match with what we're experiencing, and a list of things that they're doing without a real acknowledgement of the _dire_ uptime over the last 12 months....

georgyo 1 hour ago||
These are not the worst graphs in the world... Sure the bottom left axis is not labeled, but it still conveys the point correctly. The growth between 2023->2024->2025->2026 is growing quickly. And that in the end/beginning of 2026 they say more growth than the three years before, combined!

You don't need to know the bottom left axis number. We do have to assume the graph is linear, and not some kind of negative exponent log graph. But given the rest of the content, I think that is safe to assume.

Any company that experiences significantly more growth than they were planning for will have capacity issues.

The priorities are most inline with that. The are way beyond the point that they can just add more hardware. They need to make the backend more efficient, and all the stated goals are about helping there.

johndough 43 minutes ago|||
> You don't need to know the bottom left axis number.

We very much do. The graph suggests an insane growth in PRs from almost zero to 90M. Now compare this misleading graph with this much clearer one, which shows that the growth over the last three years has been less than 80%: https://github.blog/wp-content/uploads/2025/10/octoverse-202...

SkiFire13 34 minutes ago||
That link shows the number of PRs created to be less than 10M though.
johndough 22 minutes ago||
Yes, to be honest, that graph could use some improvements as well. I should probably just link to the full blog post with actual numbers: https://github.blog/news-insights/octoverse/octoverse-a-new-...
maccard 47 minutes ago|||
> These are not the worst graphs in the world... Sure the bottom left axis is not labeled, but it still conveys the point correctly.

No, they're completely useless. Using the "New repos per month" as an example, if the bottom left is 1m, then that's a 20x increase in 2 years which is a lot. If the bottom left is 19m, it's a 5% increase in 2 years which is nothing.

The massive surge on their labelled X axis starts in 2026, and these issues have been going on for a lot longer than that. GHA has been borderline unusable for a year at this point, if not longer.

> But given the rest of the content, I think that is safe to assume.

The rest of the content is "we're working on it", and "here's two outages in the last 14 days, one of which caused actual data loss"

ncruces 1 hour ago|||
More numbers: https://x.com/kdaigle/status/2040164759836778878

What's the question here, you don't believe growth is currently exponential, or do you think it shouldn't be hard to scale, when 10x YoY is not enough?

OtherShrezzing 1 hour ago|||
As a business user, our costs have gone up while service has gone down dramatically. Meanwhile our marginal cost to GitHub has hardly changed. Where our costs to them have increased, they mostly charge us per cpu minute, so obviously aren’t making any kind of loss on our account.

I’m sure they’re experiencing scaling issues across the platform, but it’s unacceptable for that to have a negative impact on us when we're sending them $250/dev/yr for (what is in all honesty) hosting a bunch of static text files.

ncruces 41 minutes ago|||
I understand that, and maybe GitHub became a bad deal because of that.

But if anything, their post and your reply are precisely an endorsement of usage based billing.

The bit that's growing 13x YoY (and which they expect will easily blow past that) is unmetered - commits. The bit that is metered (for some, not all folks) - action minutes, grew only 2x YoY.

GitHub was not built to limit the number of commits, checkouts, forks, issues, PRs, etc - nor do we want them to - but that's what's growing ridiculously as people unleash hordes of busy beaver agents on GitHub, because their either free or unlimited.

Where there are limits - or usage based billing - people add guardrails and find optimizations.

Because for all the talk, agents don't bring a 10x value increase; otherwise, they'd justify a 10x cost increase.

Besides, other forges are having issues too. Even running your own. We have Anubis everywhere protecting them for a reason.

rdevilla 53 minutes ago||||
> we're sending them $250/dev/yr for (what is in all honesty) hosting a bunch of static text files.

You know, you can just host your own code forge. Or you can just drop gitolite on a server. Or pull directly from each others' dev machines on a LAN.

GitHub is not git.

dist-epoch 52 minutes ago|||
> we're sending them $250/dev/yr for (what is in all honesty) hosting a bunch of static text files.

so start a GitHub competitor which bills $50/dev/yr for solving this easy problem and make a lot of money?

maccard 44 minutes ago|||
These numbers should have been in the blog post, not the graphs that are present.

> What's the question here, you don't believe growth is currently exponential, or do you think it shouldn't be hard to scale

I think you're putting words in my mouth here; I didn't say either of those things. I'm saying that this blog post is a meaningless platitude when the github stability issues predate this, and that all this post says is "we hear you're having issues".

ncruces 29 minutes ago||
Sorry if I misread your intent.

I just think their charts, taken at face value, show substantially the same thing (for PRs, commits, new repos).

Either those charts are a bald-faced lie (the tweet could be as well) or there is no way for that chart to be something else.

The only way to fake exponential growth like that would be to use an inverse log scale (which would be a bald-faced lie).

It doesn't even really matter what's the y-axis baseline, unless we really think growth was huge in 2020, then cratered to zero by 2023, now back to the previous normal.

As for the rest of the post, I do think it's panic mode platitudes. But I honestly don't know what I'd write instead that's better.

You can already see people complaining loudly where they instead of "we'll do better" decided to limit usage.

ramon156 1 hour ago|||
"We hear you" in ~300 words, basically.
ferguess_k 1 hour ago||
You can do the same with so many clients.
mijoharas 1 hour ago||
> we started working on path to multi cloud.

Is this microsoft stating that they aren't able to get acceptable reliability from Azure? (I mean, I think a lot of us have heard that, but it's interesting to hear it from microsoft themselves).

derwiki 1 hour ago||
It’s pretty damning. But as someone who has used Azure, I buy it.
everfrustrated 35 minutes ago||
Pretty damming that two Microsoft subsidiaries - GitHub and LinkedIn - either shelved their forced migration to Azure or are looking at non-Azure options.
cbg0 1 hour ago|||
I think this is more tailored towards enterprise clients that lose money when Github is down, that would probably help with retention.
bombcar 56 minutes ago|||
You’d think they could have had the existing GitHub on whatever continue as is (maybe for paying customers) while all the AI new inrush goes to the Azure setup.
jofzar 50 minutes ago|||
Yeah that's a top tier enterprise plan feature if I have ever seen ut
jasoncartwright 1 hour ago|||
Seems pretty sensible to not rely on a single provider for their large complex system?
embedding-shape 49 minutes ago|||
Man, you should have been there 6 months ago when they decided to start tearing down GitHub's own data centers and move everything exclusively to Azure. Seems they themselves realized this after they started moving, but imagine if you could have helped them realize this before they even started :)
nextaccountic 9 minutes ago|||
Made me think. Why not convert Github datacenters into Azure datacenters that have Github as their sole customer?

Then it's up to Azure how they will manage this

benterix 25 minutes ago|||
> Seems they themselves realized this after they started moving

I guess most people at Github knew exactly it makes no sense but they didn't really have a choice. Maybe some voiced their statement, got "we hear you" in response and were told to proceed anyway.

embedding-shape 21 minutes ago||
Yeah, I don't know how it went down, but I also know exactly how it went down:

Microsoft Execs: Everyone needs to move to Azure!

GitHub developers: But Azure is not gonna be able to handle our load, we literally have our own data centers!

Microsoft Execs: Sure, but you're Microsoft now, please publish blog post about how in half a year you'll be 100% on Azure.

Few months later...

GitHub Developer: We've tried our best, users are leaving in droves and Azure can't keep up!

Microsoft Execs: Ok fine, you can use something else too, but only if you mainly use Azure and continue publishing blog posts about how great Azure is.

mijoharas 1 hour ago||||
I mean, amazon (shopping, along with prime video e.t.c.) runs on AWS.
ksimukka 45 minutes ago|||
When I was at AWS, retail was not yet running on AWS. Has that changed?

Prime video does use some AWS services, but live and on-demand are two entirely different beasts.

mijoharas 17 minutes ago||
Really? I thought retail was. It's been almost a decade since I worked at prime video but I think everything was running on AWS. (Some things didn't use brazil etc, but I think all the servers etc. were on AWS)
jasoncartwright 1 hour ago|||
Prime video uses a non-AWS CDN when I watch football on it here in the UK
farfatched 52 minutes ago||
The BBC were unable to find a single CDN that could serve the UK during its peak football matches. https://www.bbc.co.uk/webarchive/https%3A%2F%2Fwww.bbc.co.uk...
cyanydeez 1 hour ago|||
This isn't a mom and pop shop. They have locations all over the world: https://datacenters.microsoft.com/

There's no intrinsic reason they should be vulnerable to themselves.

farfatched 56 minutes ago|||
+1. Multi-cloud is typically done for vendor independence.

But Github don't have that rationale.

jasoncartwright 1 hour ago|||
That website (for me) uses Cloudflare via WPEngine, which also isn't Azure
jansan 44 minutes ago||
The entire concept of multi cloud is amusing if you think what cloud originally was supposed to be. They could call them meta clouds (might infringe trademarks), and with the current growth trajectory of AI generated code eventually multi-meta-clouds, renamed to beyond-clouds, and then multi-beyond-clounds. I see no limits.
GS_Projects 3 minutes ago||
The bit nobody covers in these write-ups: small teams without dual-cloud failover budget. Last big GitHub outage cost me a deploy day. Not catastrophic but the kind of thing you don't budget for when GitHub is your single source of truth.

Status page is also still doing that thing where every component is green but in practice clone is hanging, push is timing out, actions are stuck. Per-service uptime is a managed number. The user-experience number is the one that matters and it's not in the post-mortem.

darkwater 1 hour ago||
Glad that they released some data about new repo/issues/commits over the last years. It confirms what everyone else already believed from the outside: agents are putting a lot of extra, sudden pressure on GitHub. It's like a startup that is growing exponentially, with the difference that they already have a large user base to serve - and that keeps them in the bullseye - and probably a not-so-fast-moving organization when it comes down to changes. On the other side of the coin, they also have a lot of talent, infra and money a startup might not have yet.
maccard 1 hour ago|
What data is that? There's an unlabelled graph and a number at the current peak.
ncruces 1 hour ago|||
Some previous numbers: https://x.com/kdaigle/status/2040164759836778878
maccard 52 minutes ago||
This is the data that should be in the blog post. Thanks for sharing.
torben-friis 51 minutes ago||
Not enough attention is being put in the production/delivery mismatch.

GitHub is claiming they require 30x scale due to the giant increase in repository creation, PRs, commits, etc.

I have not seen a single product increase in features or quality as an end user, nor new significant products have come out in this period (other than the LLMs themselves).

Where is all this code going?

jmbwell 38 seconds ago||
I understood it to mean, GitHub is being crushed by LLM/AI/Agentic code review and submission, not GitHub’s code itself

What I’m not seeing here but I am seeing with the Linux kernel is, most of the automatically submitted code is irrelevant or not useful

whstl 47 minutes ago||
I for one believe Microsoft when they say this code is going to Github... to die.

Half of my friends is vibe-coding something but they can barely get the rest of the group chat to use it once.

In companies, I see people vibe-coding "miracle apps" that fall under the smallest amount of scrutiny.

Basically people are doing the same developers do when they say "I can do this in a weekend", which is getting a prototype sort of running and then immediately losing energy (or in this case lacking ability) to push it forward.

jansan 41 minutes ago||
> Half of my friends is vibe-coding something but they can barely get the rest of the group chat to use it once.

Some people I know can't even explain what they are trying to create.

frangonf 1 hour ago||
What are we doing?

Stop subsidizing tokens now that we extracted enough training data from you and we have enough agentic junkies business to keep the flywheel going up and cut on the loss leaders. [0]

[0] https://news.ycombinator.com/item?id=47923357

LiamPowell 49 minutes ago||
I can not figure out what on Earth they've done with these graphs, it almost seems like these are an artists impression of a graph.

Looking at the commit graph: Why do commits have big steps followed by slow rolloffs? Why do the steps not happen at uniform points Why do larger steps sometimes have less of a slope than smaller steps but not all the time?

Then looking at the other graphs there's completely different effects going on.

icy 1 hour ago|
I'm biased (founder of tangled.org), but the future really should be federated forges. Host repositories on sovereign infra with global identity + federated "metadata" (issues, pulls, etc.).

Global indices for this should be trivial to spin up so availability is never a concern (we're working towards this!).

ljm 21 minutes ago||
I would love if it coding agents didn't default to GitHub for their deep VCS integration.

If I could get the same bells and whistles by wiring up another forge, so long as it offered a decent API and/or sent events over a webhook, I'd have everything self-hosted.

The agents would need to expose an interface on their own end but as long as you implemented it with a plugin, it'd take the dependency of GitHub and you could use MCP or skills for the rest of it.

icy 16 minutes ago||
The neat thing about Tangled is it's built on an open protocol (https://atproto.com)—this allows us to effectively build an API-free system since all data on Tangled can effectively be ingested via the AT Protocol firehose.

Which is to say, this is perfect for agents given they don't need any bespoke SDK from us: simply write Tangled records for issues, pulls, whatever to your PDS and it'll show up on Tangled. We plan to start working on some exemplar agents first-party that would 1. enhance Tangled itself, 2. showcase cool things you can do with an open data firehose.

ArcHound 1 hour ago|||
But, there are? I can host a repo on GitHub, Codeberg and self host it too. Then I need to watch over main to keep it consistent between those. After that's established, I can do updates from wherever. Link'em in the README.
nibbleyou 1 hour ago|||
There's also a tool to automatically push it to multiple repos: https://github.com/prashantsengar/GitEcho

Disclaimer: the author is a colleague of mine

Though to be fair, what the parent meant by federated forges is different than this approach.

embedding-shape 58 minutes ago|||
There are distributed forges? Yes, git is distributed, but often everything around it isn't. The case parent is trying to make, is that the rest ("federated forges") should also be distributed, not just git.
ArcHound 44 minutes ago||
Ok, gotcha. So there's a demand for the additional features that are not bundled within git to be federated somehow.

I'd say we have emails, mailing lists and bug trackers. Or maybe: what is the missing killer feature that needs federation?

embedding-shape 43 minutes ago||
> what is the missing killer feature that needs federation?

Issues, pull requests, collaboration/permissions/access, "staring"/"favoriting", etc.

I think ultimately the goal is that people can run their own forges, yet still collaborate on repositories hosted in other forges, leveraging your existing authentication so you no longer need to sign up individually for each forge.

sikozu 48 minutes ago|||
I've never heard of this before, going to sign up and check it out!
icy 40 minutes ago||
Thanks! If you need anything, email me anirudh@!
ramon156 1 hour ago|||
Love the idea, would replace the LLM generated content ony our site, though.

I recently migrated to codeberg because I'm okay with self-hosting big runners, while using codeberg's available runners for smaller cron-based things (they even have lazy runners for this).

icy 55 minutes ago||
It’s… all hand written? We just sound “professional”.
beernet 1 hour ago||
What is "sovereign infra" exactly?
mathgeek 58 minutes ago|||
I know it's just marketing speak, but the term made me think of the scenes in the Matrix where what's left of humanity (ignoring all the cyclical lore that was added on top of it) has to make sure the machines can't remote in to any of their tech.
tfrancisl 1 hour ago|||
No less than self hosted, imo. If youre on some cloud it doesnt really matter that you pay them absurd amounts of money, you arent sovereign.
embedding-shape 57 minutes ago||
So literally a computer at home/in the office, as with anything else you don't really "own" the infrastructure? Or is this just about "cloud"?
icy 43 minutes ago||
Yeah sorry it's marketing BS speak for self-hosted or just infra that you control. It could be a VPS, it could be a Raspberry Pi at home. Your repos live on your servers. (And we support this on Tangled today!)
embedding-shape 42 minutes ago||
> just infra that you control

But a VPS isn't actually infrastructure you control, you essentially have as much control over it as "cloud", so I don't think that'd be counted as "sovereign", would it?

icy 34 minutes ago||
Perhaps, but it's still better than nothing!
More comments...