An Update on GitHub Availability

Posted by salkahfi 6 hours ago

An Update on GitHub Availability(github.blog)

215 points | 175 commentspage 4

guidoiaquinti 6 hours ago|

> While we were already in progress of migrating out of our smaller custom data centers into public cloud, we started working on path to multi cloud. This longer-term measure is necessary to achieve the level of resilience, low latency, and flexibility that will be needed in the future.

Wild

throwatdem12311 5 hours ago||

> The main driver is a rapid change in how software is being built.

Leopard, meet face.

Too little too late, yesterday was the straw that broke the camel’s back for us and we’ve started a migration to a self-hosted GitLab.

steve1977 5 hours ago||

I know that I'm simplifying (probably too much), but it seems like things were fine when GitHub was still a Ruby on Rails monolith and all the rigmarole with microservices etc. only made things worse.

remus 5 hours ago||

Unless everything else stays the same (underlying traffic etc.) then you can't really compare. Could be that you hit some fundamental scaling limit with the old design and it completely falls over after a certain scale.

steve1977 5 hours ago||

Oh as said I'm pretty sure things are more complex. It's just funny in a way that all these technologies that are usually being sold as "enablers for scale" don't seem to do their job very well.

tankenmate 5 hours ago||

This sounds more like a belief, based on little more than "correlation is causation", than analysis that controls for macro-trends backed by evidence.

sgarland 4 hours ago|||

It is, but everyone is entitled to beliefs. Anecdotally, I feel the same way. Everywhere I’ve been, there has always been a legacy monolith that was stable as a rock, with dozens of new microservices scattered around it in an attempt to exit the monolith. The microservices have never once been stable. People fail to take the most basic things into consideration, like “you can’t have Consistency and Availability when everything is a network call.”

I’m sure survivor bias is at play here, but when I look through the older code bases - especially the data model - it’s an entirely different world than the newer stuff, and it’s clear which of the two was written by people who understand systems.

embedding-shape 5 hours ago|||

GitHub been oscillating between long phases of "Never any new features but rock-solid and no downtime" and "New features every week but also unicorns (used to be the "service unavailable page") every week" for as long as I can remember. Seems they're on some interval switching between the two.

perbu 3 hours ago||

fwiw, I've had good luck scaling git, specially doing clones, in the HTTP layer, using Varnish. this was CI bringing Github Enterprise to it's knees.

nraynaud 6 hours ago||

So I gather that nobody is working on a search that stays on the current branch?

mendyberger 4 hours ago||

I wonder if this mess has anything to do with talent loss resulting from layoffs after the pandemic

pointlessone 4 hours ago|

I’d guess it has much more to do with the extra load agentic ai generates. If we take the charts in the OP at face value, do you think gh suddenly exploded in popularity? At this point I think almost everyone who has any use for gh already has an account and use it as much as they ever would. But all the charts go to the moon. Gh obviously didn’t take into account that ai agents can generate a lot of activity they don’t have capacity for.

dzonga 4 hours ago||

blame MySQL. Blame Ruby.

on another note - is the exponential growth from 'agentic' workflows actually resulting in productive software in the wild. Or it is just noise. On my end I haven't seen the software I use getting better.

fontain 6 hours ago||

Personally, I’m sympathetic. We know that GitHub did a huge amount of work over the last decade to make Git scale, which has benefited us all. These new scaling challenges are real challenges, 30x growth would be a nightmare for any system that was already pushing the limits of what was possible, I think we are being far too hard on GitHub, they deserve a little grace.

remus 5 hours ago||

For all the negatives about github I agree. They offer a lot of free stuff, and LLMs seem likely to put massively increase their costs with no guarantee they'll be making money off it. I can't think of many (any?) large businesses which could scale up to meet so much new demand without some significant growing pains along the way.

someone_eu 5 hours ago||

GitHub's scaling issues are caused by their own vendor-lock approach and monopoly. Yes, of course _their_ goal is to be even bigger and even more all-consuming, so _they_ have to deal with the scale. Why a user would be sympathetic to that?

The user (and not a big tech monopoly) answer to scaling issues is almost always to stop scaling and start federating and interoperating.

rootnod3 5 hours ago||

> Our priorities are clear: availability first

That's a delayed April fool's right?

embedding-shape 5 hours ago|

No, just a 6 month old memo that was first opened today, as they said literally the same 6 months ago.

jameskilton 4 hours ago|

Nice, they have availability numbers now on their status page, but they aren't aggregating.

If you multiply all current numbers together (as of Apr 28), you find out that GitHub has a 97.26% uptime.

One ... single ... 9.

They can do better.

embedding-shape 4 hours ago|

Kind of unfair though, do the same for any platform with multiple services and you'd probably get <99% for most of them.

> you find out that GitHub has a 97.26% uptime

Calculating that to "Downtime per day" you get ~40 minutes of downtime per day, almost a week per year. Crazy stuff for something essential like this.

More comments...