Posted by Footkerchief 16 hours ago
Also, doesn't even have RAG offering.
Seems like the only reliable way to run GHA jobs is to not use their runners. Hope they at least didn’t break self-hosted runners operations
> Customers using self-hosted runners may see errors or rate limiting when runners register.
Maybe they're hosting in us-east-1 though :)
Self-hosting a service like GitHub that operates at GitHub scale is difficult.
Self-hosting a service like GitHub that operates at the typical small/medium company's scale is trivial.
A single machine (with separate runners for CI) will cover many companies' needs. It being a single machine eliminates a lot of the complexity and failure modes associated with a distributed system and makes backups/restores/maintenance easy.
Even self-hosted runners are impacted.... How can that be?
The cost of this globally has got to be in the hundreds of millions to companies that use CI/CD through GitHub Actions. What if prod is broken and GitHub actions is stalling the deployment of your hotfix? What if this makes your organization miss and SLA and diminish user trust? What if this makes you miss a release that you were contractually obligated to meet? This is happening during peak dev hours on a Thursday (not that it would be acceptable at any other time).
I don't understand how a service this critical to the global technical infrastructure can fail like this at all, let alone for more than a few hours. Like where's the backup generator for crises like these? You can't even use self-hosted runners? WTF? Like how can you not bring your own backup in a crisis event like this?
Not that Microsoft has a good reputation, but holy moly, you'd think they would prepare from something inevitable like this.
The fix is merged, but won't deploy... it's been hours
Thankfully it's a batch job, and isn't interrupting production ATM
There's always the escape hatch of running you GHA workflows locally, but unfortunately, despite the existence of packages like `act`, there is no way to fully recreate the GHA runtime locally. Tons of the special YAML syntax just can't (more accurately, "just doesn't") get interpreted by those local actions runners.
We never went this route, but at my old org, I always advocated for considering GHA to be wrapper around a single bash script (or whatever script you want to run), as a means of completely breaking out of the GHA hellscape that is programming in YAML, who's turing-completeness is pretty dubious.
Unless you have things set up this way, you (the client of GitHub) would have to completely redesign your CI on the fly, run it locally, and then figure out how to get the D compliment of the I to work in a way that is auditable. Fat chance for most teams I bet.
Thank god you're dealing with a batch scenario. Silver lining for sure. Still, embrace the anger.
What makes my blood boil is that there's millions of DEVs literally crying at the moment worrying about how GitHub's failure to be responsible will put their jobs in jeopardy.
And fingers crossed for you my friend. We're at 5+ hours at the time of this writing.... You're batch job may still have a chance!!!
It's hard to draw a direct analogy there, but I feel like it echoes the same sentiment.
A soapbox I have is that GHA workflows are scripts that could run on your machine without any of the YAML stuff. Who gives a flying about the DAG or the logs? Which, by the way, if you're willing to walk to the milk store to buy your milk, could be recreated in a much more testable and maintainable way without any of the YAML bs that GHA prescribes....
But DAGs are pretty, and logstreams showing up in a browser application instill trust (for reasons that fly far above the head of yours truly). So people go for that. Pretty DAG, nice logstream; therefore, deliver my milk. All of a sudden.... The CI/CD platform is having its merry way with your SLAs, contract abidements, and hotfix deployments.
What a time to be alive.
Not to rail on the South Park thing, but the blast radius of this issue also reminds me of the episode where the internet dried up.
If this bs with GitHub continues, Parker/Stone will have to make a GitHub episode. How seen would we all feel if that happened?
latest server i set up is simply a bare repo + hooks to make a local deploy after running tests and shit
super easy to set up having ai do it, zero dependencies, deploy is still 'push it to the main'
i have several remotes for backups and stuff