Top
Best
New

Posted by 0xedb 2 hours ago

The August 17 outage, and the work ahead(github.blog)
43 points | 62 commentspage 2
kvemkon 1 hour ago|
I fear to ask, how archive.org keeps up to catch all those events for archiving...
ChrisArchitect 58 minutes ago||
Related recently:

GitHub has alternatives, but no replacement

https://news.ycombinator.com/item?id=49135365

Why developers are ditching GitHub for Codeberg and self-hosting alternatives

https://news.ycombinator.com/item?id=48842611

and new entry:

Cursor Origin Code Hosting

https://news.ycombinator.com/item?id=49334209

annoyingnoob 1 hour ago||
Github down, no hard drives available, no memory available, thanks AI!

Seems like we are headed for Tech Gridlock.

jdm2212 1 hour ago||
This stuff is good! This is what a booming economy looks like. There are people out there competing with you for resources because they have cool ideas they want to implement.
a2ff6eeb0 1 hour ago||
Or at least they asked the AI to come up with cool ideas, which is even more interesting. It's exciting watching the world transition away from humanity being in the driver's seat!
yipinwong 1 hour ago||
What they can implement is to slowdown the commit rate, rate limt or just queue-up messages not to overburden their downstream service.

I don't think GH has any of those, but just keep scaling, but that scaling failed.

Just bad architectural decisions from the postmortem.

--

It will only get worse due to AIs spawning massive commits, and they don't have unlimited cloud resource.

They can scale but not scalable in terms of effort, resources, and $

jdm2212 1 hour ago||
How would any of what you're saying help with this?

> The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path and causing widespread authentication latency and failures. The problem was worsened by optimistic retry logic which overloaded internal load balancers. Pausing HAProxy on those nodes simultaneously produced immediate broad recovery.

yipinwong 18 minutes ago||
load balancer failure? rate limit woudl address concurrency limits? throttle or queue up messages. auto-scale failed cause was misconfiguration policy, which i admit cannot be handled by my suggestions. The cascade? it's downstream service degradation, which I mentione should have had been prevented with queues. One of the jobs that queues/kafka solve is to prevent these downstream outages.
jdm2212 13 minutes ago||
If your LB is down, you're just kind of screwed. You can't enqueue things if requests aren't getting through at all. Same deal with authn/authz issues, which they also had. If you can't answer the question "is this message allowed to be added to the queue" you can't enqueue stuff.

GitHub does use queueing for all kinds of stuff internally, though, because they're not morons.

lenerdenator 1 hour ago||
We need to have a package of FLOSsoftware that you could run on the cloud of your choice that offers most of what GitHub does (niceties on top of Git) without the centralization.

GitLab was close last I remember but there was some sort of enterprise tier when I tried hosting stuff on a local server years ago. I want true FLOSS, not another SaaS equivalent of the coke dealer giving clients the good uncut stuff when they're just starting out only to sell crap when they're addicted.

verdverm 50 seconds ago||
I'm particularly fond of the concepts within git-appraise, gits back to the federated foundations

https://github.com/google/git-appraise (unmaintained)

cschep 1 hour ago|||
https://forgejo.org/ promises to be this, have only lightly used it on https://codeberg.org/ but it seems nice?
0x457 1 hour ago|||
Forgejo is pretty neat. Their CI story is sad because it's based on act and it lacks features like GH Apps so service accounts are not possible (using a user account as service is barf).

I self-host it and its pretty easy to have uptime higher than github when you have 3 users total.

Absolutely do not recommend using GitLab.

0xblinq 1 hour ago|||
"Forgejo is a self-hosted lightweight software forge"

That says absolutely nothing. The "What is Forgejo?" question is unanswered and instead you get a lot of words about their values, their inclusivity, etc. And the next thing in the docs is how to install it. It's ridiculous.

I still don't know what it is or what it does.

denzen 1 hour ago||
Have you looked into Forgejo?
amazingamazing 1 hour ago||
Exponential growth. No company could handle that without some issues. Good luck to them. And for those who cannot tolerate this, there are many self hosted options.
rvz 1 hour ago||
And another outage. [0] Looking forward to the subsequent post-mortem on that one.

You might want to not go all in on GitHub anymore since it is very unstable to use. A self-hosted instance would have a far better uptime than GitHub over the years.

6 years ahead [1] on not going all in an centralizing everything on GitHub.

[0] https://www.githubstatus.com/incidents/bhbcjn4n3jzp

[1] https://news.ycombinator.com/item?id=22867803

djha-skin 59 minutes ago||
[dead]
nycpig 1 hour ago||
Almost 8 hours of downtime across all core workflows, and the word "sorry" or "apologize" appears nowhere in this post.

"If you were trying to ship software that day, we let you down" is classic corporate non-apology speak.

I’m done.

bibimsz 36 minutes ago||
thats what i liked about it. its fact and action oriented. what does a "sorry" buy you that the "we let you down" doesn't.
sajithdilshan 1 hour ago||
Bye Felicia
codegeek 1 hour ago|
[dead]