New Vulnerability in GitHub Copilot, Cursor: Hackers Can Weaponize Code Agents

Posted by pseudolus 4/14/2025

New Vulnerability in GitHub Copilot, Cursor: Hackers Can Weaponize Code Agents(www.pillar.security)

233 points | 135 commentspage 2

t_believ-er873 4/17/2025|

Recently, I've seen a lot of information on the internet on how attackers use AI to spread malware, like jailbreak vulnerabilities that allow attackers to modify the tool's behavior. Here is the good article also on the topic: https://gitprotect.io/blog/how-attackers-use-ai-to-spread-ma...

AutoAPI 4/14/2025||

Recent discussion: Smuggling arbitrary data through an emoji https://news.ycombinator.com/item?id=43023508

Oras 4/14/2025||

This is a vulnerability in the same sense as someone committing a secret key in the front end.

And for enterprise, they have many tools to scan vulnerability and malicious code before going to production.

throwaway290 4/14/2025||

Next thing, LLMs that review code! Next next thing, poisoning LLMs that review code!

Galaxy brain: just put all the effort from developing those LLMs into writing better code

GenshoTikamura 4/14/2025|

Man I wish I could upvote you more. Most humans are never able to tell the wrong turn in real time until it's too late

mock-possum 4/14/2025||

Sorry, but isn’t this a bit ridiculous? Who just allows the AI to add code without reviewing it? And who just allows that code to be merged into a main branch without reviewing the PR?

They start out talking about how scary and pernicious this is, and then it turns out to be… adding a script tag to an html file? Come on, as if you wouldn’t spot that immediately?

What I’m actually curious about now is - if I saw that, and I asked the LLM why it added the JavaScript file, what would it tell me? Would I be able to deduce the hidden instructions in the rules file?

Etheryte 4/14/2025||

There are people who do both all the time, commit blind and merge blind. Reasonable organizations have safeguards that try and block this, but it still happens. If something like this gets buried in a large diff and the reviewer doesn't have time, care, or etc, I can easily see it getting through.

simiones 4/14/2025|||

The script tag is just a PoC of the capability. The attack vector could obviously be used to "convince" the LLM to do something much more subtle to undermine security, such as recommending code that's vulnerable to SQL injections or that uses weaker cryptographic primitives etc.

moontear 4/14/2025||

Of course, but this doesn’t undermined the OPs point of „who allows the AI to do stuff without reviewing it“. Even WITHOUT the „vulnerability“ )if we call it that), AI may always create code that may be vulnerable in some way. The vulnerability certainly increases the risk a lot and hence is a risk and also should be addressed in text files showing all characters, but AI code always needs to be reviewed - just as human code.

tsimionescu 4/14/2025|||

The point is this: vulnerable code often makes it to production, despite the best intentions of virtually all people writing and reviewing the code. If you add a malicious actor standing on the shoulder of the developers suggesting code to them, it is virtually certain that you will increase the amount of vulnerable and/or malicious code that makes it into production, statistically speaking. Sure, you have methods to catch much of these. But as long as your filters aren't 100% effective (and no one's filters are 100% effective), then the more garbage you push through them, the more garbage you'll get out.

bryanrasmussen 4/14/2025|||

the OPs point about who allows the AI to do stuff without reviewing it is undermined by reality in multiple ways

1. a dev may be using AI and nobody knows, and they are trusted more than AI, thus their code does not get as good a review as AI code would.

2. People review code all the time and subtle bugs creep in. It is not a defense against bugs creeping in that people review code. If it were there would be no bugs in organizations that review code.

3. people may not review or look only for a second based on it's a small ticket. They just changed dependencies!

more examples left up to reader's imagination.

Shorel 4/14/2025|||

Way too many "coders" now do that. I put the quotes because I automatically lose respect over any vibe coder.

This is a dystopian nightmare in the making.

At some point only a very few select people will actually understand enough programming, and they will be prosecuted by the powers that be.

ohgr 4/14/2025||

Oh man don’t even go there. It does happen.

AI generated code will get to production if you don’t pay people to give a fuck about it or hire people who don’t give a fuck.

rvnx 4/14/2025||

It will also go in production because this is the most efficient way to produce code today

cdblades 4/14/2025|||

Only if you don't examine that proposition at all.

You still have to review AI generated code, and with a higher level of attention than you do most code reviews for your peer developers. That requires someone who understands programming, software design, etc.

You still have to test the code. Even if AI generates perfect code, you still need some kind of QA shop.

Basically you're paying for the same people to do similar work to what they do now, but now you also paying for an enterprise license to your LLM provider of choice.

GenshoTikamura 4/14/2025||||

The most efficient way per whom, AI stakeholders and top managers?

bigstrat2003 4/14/2025||||

Sure, if you don't care about quality you can put out code really fast with LLMs. But if you do care about quality, they slow you down rather than speed you up.

ohgr 4/14/2025|||

It depends somewhat on how tolerant your customers are of shite.

Literally all I’ve seen is stuff that I wouldn’t ship in a million years because of the potential reputational damage to our business.

And I get told a lot by people who really have no idea what they are doing clearly that it’s actually good.

TZubiri 4/14/2025||

May god forgive me, but I'm rooting for the hackers on this one.

Job security you know?

GenshoTikamura 4/14/2025||

There is an equal unit of trouble per each unit of "progress"

gregwebs 4/14/2025||

Is there a proactive way to defend against invisible Unicode attacks?

Tepix 4/14/2025|

Filtering them?

handfuloflight 4/14/2025||

The cunning aspect of human ingenuity will never cease to amaze me.

ekzy 4/14/2025||

Not saying that you are, but reading this as if a AI bot wrote that comment gives me the chills

almery 4/14/2025||

Yes, but are you sure a human invented this attack?

handfuloflight 4/14/2025|||

Maybe, maybe not, but right now humans are the ones who are exploiting it.

bryanrasmussen 4/14/2025|||

Is there any LLM that includes in its training sets a large number of previous hacks? I bet there probably is but we don't know about it, and damn, now I suddenly got another moneymaking idea I don't have time to work on!

jdthedisciple 4/14/2025|

simple solution:

preprocess any input to agents by restricting them to a set of visible characters / filtering out suspicious ones

stevenwliao 4/14/2025||

Not sure about internationalization but at least for English, constraining to ASCII characters seems like a simple solution.

cess11 4/14/2025||

Nasty characters should be rather common in your test cases.

More comments...