Posted by MattIPv4 6 hours ago
So needless to say, if you depend on GitHub for critical business operations, you need to start thinking about what a world without GitHub looks like for your business and start working your way toward that. I know my confidence in GitHub's engineering leadership is at rock bottom.
Everywhere I’ve worked, if a migration is causing this much downtime then you kill the migration or slow it down. If every change has a 10% chance of bringing the site down, you only do a change every week or two until you can work out the kinks.
> In a message to GitHub’s staff, CTO Vladimir Fedorov notes that GitHub is constrained on capacity in its Virginia data center. “It’s existential for us to keep up with the demands of AI and Copilot, which are changing how people use GitHub,” he writes.
> The plan, he writes, is for GitHub to completely move out of its own data centers in 24 months. “This means we have 18 months to execute (with a 6 month buffer),” Fedorov’s memo says. He acknowledges that since any migration of this scope will have to run in parallel on both the new and old infrastructure for at least six months, the team realistically needs to get this work done in the next 12 months.
If you consider that six month parallel window to have started from the time of the October memo (written presumably at the start of October), then that puts us currently or past the point where they would have cut off their old DC and defaulted to Azure only.
Whether plans or timelines changed, I have no idea of course but the above does make for a convenient timeline that would explain the recent instability. Of course, it could also just be symptomatic of increased AI usage generally and the same problems might have surfaced at a software level regardless of whether they were in a DC or on Azure.
Putting that nuance aside, personally I like the idea that Azure is simply a giant pile of shit operated by a corporation with no taste.
[1]: https://thenewstack.io/github-will-prioritize-migrating-to-a...
if by chance the CTO reads this, as a user of GitHub I would find it really existential if GitHub continues functioning as a reliable hub for git workflows (hence the name), and I have the strong suspicion nobody except for the shareholders gives a lick about copilot or 'AI' if it makes the core service the site was designed for unusable
I wonder if the extended downtime is just due to the on-call engineers waiting for their azure auth tokens to refresh within azure's own damn network.
"The evidence is clear: Either you embrace AI, or get out of this career." -Github CEO
"Sooner than later, 80% of the code is going to be written by Copilot. And that doesn’t mean the developer is going to be replaced." -Github CEO
I like AI but actually not for coding because code quality is correlated to how well you understand the underlying systems you're building on, and AI is not really reasoning on this level at all. It's clearly synthesizing training data and it's useful in limited ways.
Did you hear about the screenwriters school in which the professors said to avoid AI for writing, but it's great for storyboards. And the storyboard school where the professors said the opposite?
The reality is that AI isn't actually "good" at anything. It produces passable ersatz facsimiles of work that can fool those not skilled in the art. The second reality of AI is that everyone is busy cramming it into their products at the expense of what their products are actually useful for.
Once people realise (1), and stop doing (2), the tech industry has a chance of recovering.
Seemingly the decline started with the Microsoft acquisition in 2018, and subsequent "unlimited private repository" change in 2019 (to match Gitlab's popular offer)
(And the first thing to go was occasional 500's on github-hosted files.. the core service itself - git, PR, actions - were pretty stable until recently)
IMO it's much better now.
https://trends.google.com/trends/explore?date=all&geo=GB&q=s...
Maybe it wasn't as noticeable when Github had less features, but our CI runners and other automation using the API a decade ago always had weekly issues caused by Github being down/degraded.
Would you like help?
- Get help with developing the software
- Just develop the software without help
[ ] Don't show me this tip again"
FTFY. (I've read AWS word it like that)
https://foja.applycreatures.com
Edit: it has a wonderful API so I posted the link it may tempt some to ditch MS/Azure hub.
Is all the recent GitHub downtime entirely attributable to GitHub AI Copilot related development? How hard can it be to reduce the blast radius of new AI features to not affect the core parts of hosting repositories? Because of Copilot everywhere, The UX has become bad and I had to click all over the place and on my profile to find repositories.