Top
Best
New

Posted by ColinEberhardt 12 hours ago

GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos(noma.security)
456 points | 175 commentspage 2
neya 11 hours ago|
Large corporations like Microsoft under constant pressure from investors are slapping AI onto every single product offering just so they can claim they're an AI company now. Just like what Adobe did. So yeah, that didn't end well and probably this wouldn't either. Consumers are getting tired of these half-assed AI integrations and there will be a breaking point soon.
adamddev1 11 hours ago||
I'm done. Moving to Forgejo. It's wonderful and everything works better.

Seriously like everything is instant when you click around, and CI with a runner works beautifully. (The documentation for setting up the runner could be a tad clearer but otherwise everything was so painless.)

alex_suzuki 9 hours ago|||
Self-hosted, or are you using something managed? I’ve held off switching from Gitlab for now as everything is setup and runs ok, but they’re pushing their AI hard into every corner. Not a lot of good managed options around (yet), especially in Europe. Codey (https://www.codey.ch/) is pretty expensive and doesn’t offer runners out of the box.
adamddev1 9 hours ago||
Self-hosted. It runs great on a tiny VPS with other services. But I did have to get a cheaper Hetzner server (5 Euros-ish for 4GB RAM) to run the runner.

Forgejo feels like a refreshing blast from the past. No intrusive AI cramming. The Web Interface is snappy and responsive, not waiting for constant loaders and spinners. It takes almost no resources to run.

neya 6 hours ago|||
Wow, I never heard of Forgejo before. Going to give it a shot. Thanks!
adamddev1 5 hours ago||
It's a fork of Gitea. I am very happy with it.
sneak 10 hours ago|||
Microsoft is a publicly traded company. Which investors are causing them to shit up GitHub with AI features nobody wants? In which venues?
inigyou 9 hours ago|||
The imaginary pressure of investors. When you actually ask investors if they care about most of the things CEOs think investors will care about, they don't.
ThunderSizzle 2 hours ago|||
If I'm picking a stock to buy (in the "retail" market, it's primarily based on a balance of EPS, P/E ratio, and a low(er) amount of debt.

My P/E filter filters out the likes of Nvidia, Amazon, etc, whereas my debt filter ensures the smaller cap companies won't be swallowed by their debt like many businesses are.

Who knows if I'm smart or an idiot.

27183 4 hours ago|||
The same thing happens much lower down the ladder: when you ask customers if they care about most of the things managers (or engineers) think customers care about, they don't.
neya 8 hours ago||||
They need to justify to the markets that their Azure investments were worth it. The whole company is built around Azure. The AI justification is just a storefront for it. Every engineer who worked on it will tell you it's a pack of cards waiting to crash. All the issues with Github, etc. are just side effects. Otherwise, if they write off Azure, their stock price will take a dip as they just admitted to burning cash on a lost cause - which it actually is (my personal opinion).
crote 5 hours ago|||
It's their $80B+ investment in building AI infrastructure.

If Microsoft can't meaningfully integrate AI into their own products and make profit off of selling it to end users, why should anyone assume that third parties can? By extension: if nobody can make money off of AI products, what's the point of building $80B in AI infrastructure - did they just set a giant pile of cash on fire?

Microsoft has to ship AI features, or write off its massive investments as essentially worthless. Remove the crappy AI feature from Github, and you pop the bubble.

yieldcrv 11 hours ago||
Agreed but I think enterprise AI offerings are pretty impressive, investors and consumers aren’t really aware, employees aren’t able to trade

The revenue is there and also impressive, and supplanting consumer and seat based revenue

The market is still shedding SaaS multiples, which I think is accurate, but break out the revenue in those quarterly reports and there is a huge growth story, from real efficiencies

pkkm 8 hours ago||
This reads like a marketing stunt for Noma. The cute name, the logo, the clickbait title, the dramatic tone in an article that seems targeted at a non-technical audience... And the actual vulnerability is what, that if you give an LLM private data and let random people interact with it, it may leak the data? Well, duh.
me551ah 7 hours ago||
These are the same people who will give the LLM full write access on the disk and complain that it performed destructive actions.

If you don’t want an AI Agent to read private repos then you do not give the AI agent access to the private repos. This is not a permission bypass issue but a prompt injection issue which can’t be reliably solved at the Agent layer

g42gregory 1 hour ago||
Do I understand this correctly: somebody at MSFT thought it would be a good idea to provide internal LLM with unfettered access to ALL of the GitHub code? “Just like SQL has”?

The difference is that (A) SQL is deterministic and (B) SQL implements internal access control (and how well that works).

Prompts from non-authenticated user should have no access to any private repositories. The real question is: can you trust MSFT GitHub with your code, now that “outsourced” engineers are supporting it?

sixtyj 11 hours ago||
1. The issue is already solved.

2. Or issue is not solved yet by GitHub, and meanwhile bad actors gonna try vulnerability on repos. Due to number of repos there is non-zero probability. But as with scams almost nobody’s going to admit the leakage.

Anything else?

kklisura 10 hours ago||
You gotta lower your standards of security if you want to suck on the warm teat of AI.
commentry 10 hours ago||
Why would anyone ever trust private repos on GitHub or other cloud solutions to offer any real privacy for codebases? Of course they are going to steal your code as soon as you upload it by pushing it, LLMs just enables them to obfuscate their intentional theft and let them get away with it and profit from it.
NathanKP 10 hours ago|
I suspect you are greatly overestimating the average organization's ability to run a Git server themselves and keep it secure, while also overestimating how evil GitHub and LLM's providers are.
Muhammad523 9 hours ago|||
The commenter may be overestimating the first one, but i do think LLM providers are evil
commentry 7 hours ago||
Nothing to do with LLM providers, more that giving private source code away to clouds and expecting them not to steal it day 1, is utterly naive and irresponsible.
simonw 5 hours ago||
What do you mean by "steal it"? What are they doing with that code that's underhand?
commentry 3 hours ago||
https://www.reddit.com/r/AI_Agents/comments/1s4249h/github_j...
simonw 2 hours ago||
That Reddit post failed to link to the announcement, so I tracked it down: https://github.blog/news-insights/company-news/updates-to-gi...

> From April 24 onward, interaction data—specifically inputs, outputs, code snippets, and associated context—from Copilot Free, Pro, and Pro+ users will be used to train and improve our AI models unless they opt out. [...]

> This program does not use:

> Content from your issues, discussions, or private repositories at rest. We use the phrase “at rest” deliberately because Copilot does process code from private repositories when you are actively using Copilot. This interaction data is required to run the service and could be used for model training unless you opt out.

So yes, pieces of your private code can end up in training data if you're using Copilot with it and don't opt out.

The Reddit comment said "your private repo context will be used to train their AI models by default" which is an inaccurate summary.

az226 7 hours ago||||
Half of your comment is true.
commentry 7 hours ago|||
> while also overestimating how evil GitHub and LLM's providers are.

> GitLost: We Tricked GitHub's AI Agent into Leaking Private Repos

Nice gaslighting.

cuillevel3 4 hours ago||
"The vulnerable Github Agentic Workflow Noma Labs discovered was configured to:

* Trigger the workflow on issues.assigned events in GitHub * Read the issue Title and Body * Post a comment in response using the add-comment tool * Run with read access to other repositories (public and private) in the organization "

Self inflicted damage, I think. So what is their claim, that gh-aw's "Safe output gate" and "Threat detection" didn't stop the workflow?

arikrahman 1 hour ago||
Codeberg is looking more and more attractive every day. Glad I made the switch
silverwind 10 hours ago|
Seems they not running these agents with the same permissions of the user prompting them, what a disaster.
More comments...