Posted by speckx 17 hours ago
The catalyst is the shift towards software transparency: both the radically increased adoption of open source and source-available software, and the radically improved capabilities of reversing and decompilation tools. It has been over a decade since any ordinary off-the-shelf closed-source software was meaningfully obscured from serious adversaries.
This has been playing out in slow motion ever since BinDiff: you can't patch software without disclosing vulnerabilities. We've been operating in a state of denial about this, because there was some domain expertise involved in becoming a practitioner for whom patches were transparently vulnerability disclosures. But AIs have vaporized the pretense.
It is now the case that any time something gets merged into mainline Linux, several different organizations are feeding the diffs through LLM prompts aggressively evaluating whether they fix a vulnerability and generating exploit guidance. That will be the case for most major open source projects (nginx, OpenSSL, Postgres, &c) sooner rather than later.
The norms of coordinated disclosure are not calibrated for this environment. They really haven't been for the last decade.
I'm weirdly comfortable with this, because I think coordinated disclosure norms have always been blinkered, based on the unquestioned premise that delaying disclosure for the operational convenience of system administrators is a good thing. There are reasons to question that premise! The delay also keeps information out of the hands of system operators who have options other than applying patches.
Probably goes without saying but the last line of defense is not deploying your software publicly and instead relying on server-client architectures to do anything. Maybe this will be more common as vulnerabilities are more easily detected and exploited. Of course its not always feasible.
It has been annoying seeing my (proguard obfuscated) game client binaries decompiled and published on github many times over the last 11 years. Only the undeployed server code has remained private.
Interestingly I didn't have a problem with adversaries reverse engineering my network protocols until I was updating them less frequently than weekly. LLM assisted adversaries could probably keep up with that now too.
How easy to do you this is for LLM to build decent emulator of the server in question by just observing what you send and what you get as response?
Care to mention these reasons?
With "convenience of system administrators", I'm guessing you mean that there's a patch available that sysadmins can install, ideally before the vulnerability is disclosed? What else are sysadmins supposed to do, in your opinion? Fix the vulnerability themselves? Or simply shutdown the servers?
With the various copyfails of recent, it at least was possible to block the affected modules. If that were not the case, what would you have done, as a sysadmin?
That’s why Microsoft has been obfuscating its binary builds for at least the last two decades so that even the two builds from the same source would produce very different blobs.
If you saying there is a whole step just scrambling blobs, i will be very surprised.
Current coordinated disclosure practices have a dependency on patching and disclosure being separate, but the gap between them seems to be asymptomatically approaching zero.
The actual policy responses to it, I couldn't say! I've always believed, even when there was a meaningful gap between patching and disclosing, that coordinated disclosure norms were a bad default.
What would the best solution be? And where do you believe the industry is headed (which may very well be something other than the best solution) ?
I can’t think about anything other than improving operations, but given the state of the industry, this seems like a pipe dream.
Day -X + 1: Engineer at Alibaba finds the vuln and tells Apache. Patch is pushed to git while new release is coordinated.
Day -X: A black hat sees commits fixing the bug. Attacks start happening.
Day 0: Memes start circulating in Minecraft communities of people crashing servers. Some logs are shared on Twitter, especially in China, of people getting pwned.
Day 0 + ~4 hours: My friend DMs me a meme on Twitter. I look up to find the CVE. Doesn't exist. My friend and I reproduce the exploit and write up a blog post about it. (We name it Log4Shell to differentiate it from a different, older log4j RCE vuln)
Day ~1: Media starts picking it up. Apache is forced to release patches faster in response. CVE is actually published to properly allow security scanners to identify it.
Today: AI makes this happen faster and more consistently. Patches probably should be kept private until a coordinated disclosure happens post-testing and CVE being published?
Hard to say what the right move is, but this is gonna be happening a lot over the next 1-3 years. Lots of companies are going to be getting cooked until AI helps us patch faster than attackers can exploit these fresh 0-days.
So prevention will be more important than ai-assisted rapid containment or patching, though both of those capabilities will be necessary as part of defense in depth.
And some sort of AI-enabled security analysis across the organization’s architecture that is done as part of testing ahead of new software entering production to ID potential vulnerabilities caused by configuration changes or upgrades that modify how systems interact with each other.
I’ve been trying to guess the timeframe for seeing improved secure development, but I’m hoping it’s a bit closer to 6 months - 1 year given the speed of AI adoption and AI progression. May be closer to 3 years as you stated.
In the meantime, is there more to be done than this (not in order)?
- Patch COTS software
- re-evaluate the scoring for previous vulnerabilities
- set up up containment measures capabilities for systems that can’t be patched / high risk vendors
- use frontier model vuln scanning and patching for home grown systems that may have more 0-days than COTS depending on the organization’s capability
- limit the number of vendors / simplifying the tech stack.
I’d be happy to hear how others are thinking about this.
I think there's opportunity to reinvent the pipeline with AI powered tools to assist but the onus is still on the person to ensure they are deploying something that has been tested.
people were already diffing kernel commits and figuring out which ones were security fixes long before llms. if a patch lands publicly, the race has basically already started.
also not sure shorter embargoes really help. the orgs that can patch in hours are already fine. everyone else still takes days or weeks.
if anything, cheaper exploit generation probably makes coordinated disclosure more important, not less.
With skill, and usually not consistently and systematically. With AI, anyone can do this to any software.
> not sure shorter embargoes really help
Why 90 days versus 2 years? The author is arguing the factors that set that balance have shifted, given the frequency of simultaneous discovery. The embargo window isn’t an actual window, just an illusion, if the exploit is going to be found by several people outside the embargo anyway.
> cheaper exploit generation probably makes coordinated disclosure more important
I agree. But it also makes it less viable. If script kiddies can find and exploit zero days, the capacity to co-ordinate breaks down.
There was always a guild ethic that drove white-hate (EDIT: hat) culture. If the guild is broken, the ethic has nothing to stand on.
How do you know? If the people who like to crow about vulnerabilities aren't doing it, it doesn't mean that the people who are actually in a position to exploit them systematically and effectively aren't doing it.
Those embargoes have always been dangerous, because they create a false sense of security. But, as you point out...
> With AI, anyone can do this to any software.
Yep. Even if it hadn't been true before, it's clear that now you just have to assume that everybody relevant will immediately recognize the security impact of any patch that gets published. That includes both bugs fixed and bugs introduced.
... and as the AI gets better, you're going to have to assume that you don't even have to publish a patch. Or source code. Within way less time than it's going to take people to admit it and adjust, any vulnerability in any software available for inspection is going to be instant public knowledge. Or at least public among anybody who matters.
Shouldn't this naturally lead to a state where all (new) code is vulnerability-free? If AI vulnerability detection friction becomes low enough it'll become common/forced practice to pre-scan code.
Scanning every commit doesnt automatically make everyone in the world patch immediately, vulns can still be found from commits and diffs and used against those who havent patched yet.
"How likely is this diff a patch for an existing vulnerability?"
Seems to be an easier question to answer than
"Are there any new vulnerabilities introduced by this diff?"
In other words identifying that a patch is for a vulnerability is typically easier than finding the vulnerability in the first place.
You'd think.
But then you'd think people would do a lot of other things too. I hope, I guess.
The other danger is that "the cloud" may become even more overwhelmingly dominant. Which of course has its own large security costs.
Obligatory xkcd https://xkcd.com/605/
We know because we could see the effects of the average rate of vulnerabilities discovery and exploitation, and it's definitely going up very fast. Until recently, vulnerabilities were relatively hard to find, and finding them was done by a very restricted group of people world-wide, which made them quite valuable. Not any more.
It could equally be argued that the AI slop that's being produced makes for a lot more vulnerabilities being shipped. The bigger target makes for the easier discovery.
But that's not accounting for all of the discoveries, not at all.
I've also seen the npm people talking about the surge in AI code overwhelming the ability to properly review what's being distributed, and a large number of vulnerabilities being attributed to that
Total number of vulnerabilities likely goes up over time weighting all projects equally, but goes down over time weighting by usage.
This is the basis for "correlation points to possibly fertile grounds for an investigation"
Pragmatically, correlation *is* evidence of causation in favour of the best explanation, until somebody finds a better explanation.
> It could equally be argued that the AI slop that's being produced makes for a lot more vulnerabilities being shipped.
This is also true, and does not exclude the other, because for the moment the vast majority of production software in the world (and therefore the bulk of enticing targets) was written before AI. If LLM software will become prevalent in commercial setups, then LLM-generated code will eventually become the majority of targets.
Uh, no.
Correlation is only ever one thing - cause for investigation.
Everything based on correlation alone is speculation.
You can speculate all you like, I have zero issue with that, but that's best prefaced with "I guess"
edit: Science captures this perfectly, and people misunderstand this so fundamentally that there is a massive debate where people who think they are "pro science" argue this so badly with theists that they completely hoist themselves with their own petard.
Science uses the term "theory" because all of our understanding is based on "available data" - and science biggest contribution to humanity is that it accepts that the current/leading THEORY can and will be retracted if there is compelling data discovered that demonstrates a falsehood.
So - because I know this is coming - yes science is willing to accept some correlation - BUT it's labelled "theory" or "statistically significant" because science is clear that if other data arises then that idea will need to be revisited.
However I wanted to also point out that relying only on educated guesses can lead us into a position where we are "papering over the cracks" or "addressing the symptoms", not the "underlying cause"
Yes, sometimes that's all that can be done, but, also, sometimes it can be more damaging than the cause itself (thinking in terms of the cause continuing to fester away, whilst we think it's 'solved')
No. You kept blabbering about "science" when most uses of knowledge are not about science. The original topic was also definitely not "science": it was about having a reasonable opinion about whether, empirically, the rate of discovery of vulnerabilities is increasing or not.
My point remains: if we misattribute the cause of the rising vulnerability rate (discovery vs. creation), our 'educated guesses' will lead to solutions that address the symptoms while the underlying problem continues to fester. Calling precision 'blabbering' is exactly how we end up with the 'false sense of security' mentioned earlier.
Exhibit A:
ragall 2 hours ago | root | parent | prev | next [–]
> How do you know?
We know because we could see the effects of the average rate of vulnerabilities discovery and exploitation, and it's definitely going up very fast. Until recently, vulnerabilities were relatively hard to find, and finding them was done by a very restricted group of people world-wide, which made them quite valuable. Not any more.
Exhibit B:
ragall 2 hours ago | root | parent | next [–]
Very often you only have limited time for investigation and you have to act now. Action is almost always based on educated guesses. reply
I would like to see actual evidence of this, not.. vibes
I mean, this reeks of "Anyone is a Principal developer now" when the truth is there is still work to do.
I haven't seen this kind of thing and I get the impression, despite all the hype, that this will be a frequent phenomenon now thanks to LLMs.
So it's not surprising Dirtyfrag was disclosed by a fix in the Linux kernel. [2]
[1] https://www.zdnet.com/article/torvalds-criticises-the-securi...
Much of today's workloads are containerized and run on roughly ephemeral nodes that can be switched out easily- K8s version upgrades force this more or less. We tent to run more and more of-the shelf hardware and worry less about individual node failures now.
In-memory updates also not magic , and can be limited as they requires data structure semantics to not really change and can create its own class of issues/bugs including security ones.
While am sure there are still use cases which dictate this type of update, the need is lot less than 15 years ago that the patent expiry will do much to the ecosystem.
The US is at war. Much of the world is at war at the cyber attack level right now. The US, the EU, most of the Middle East, Israel, Russia... Major services have been attacked and have gone down for days at a time - Ubuntu, Github, Let's Encrypt, Stryker. Entire hospital systems have had to partially shut down.
Now, in the middle of this, AI has made attacks much faster to generate. Faster than the defensive side can respond. Zero-day attacks used to be rare. Now they're normal.
It's going to get worse before it gets better. Maybe much worse.
How is it going to get better?
One of the big challenges with cybersecurity is that attackers only need to find one exploit, while defenders need to stop everything. When you have a large surface area and limited resources, it's much easier to be the side that only has to succeed once. AI eliminates the limited resources problem.
...so if we assume a halting oracle?
In time most of the bugs AI can find will be fixed, and things will calm down. Some bugs will be left, but will be too complex to find and weaponise (or rarely).
Alin short, attackers have advantage for a brief time now, but ultimately defenders will win. I guess this "fight" might be over before the end of the year.
Right now all that stuff is optional, so most companies don't do it, which makes more security holes and it takes longer to patch.
E.g. you build an app, it stores data via api etc. etc. You can test in sandbox. The cloud deploys for customer who paid you via that cloud and you work at arms length. You may not even know their name. You just get the pro subscription fees.
The idea bubbling in my head would be an app store for cloud products. But with competition i.e. you use Railway or Heroku or AWS for the best deal.
Be gentle this is an idea in my head I am sure it can be torn down by a retort at this stage. But this exists in forms and I think it will emerge. It is inversion of control at the entire app level.
This is similar to buying a hammer. If you make hammers you sell them to a store, the store knows the customer and only the customer can see the nails.
No, it's similar to letting someone else do all your hammering because using a hammer is too dangerous. And then, to make the process more efficient, letting them take control of your home to be able to provide hammering services while making sure you can't touch the hammer.
Unknown chance of having to pay out x large payout amount if you do = very very high premiums. Or not being willing to underwrite the insurance at all.
Remember, insurance is just gambling. The company is betting that the amount of money they'll make from everyone's total premiums added together is greater than the amount they'll have to pay out. Dumb gamblers don't last long. Smart gamblers will evaluate the risk and say "Okay, that'll be $X million a month in premiums", or even "Nope, we won't cover you". Can most open-source projects afford that?
However, half of the vulnerabilities are logic errors in terms of what I would call RBAC enforcement, incorrect access permissions, and so on. Rust won't help at all with any of these.
It’s full of bugs and has had a string of RCE vulnerabilities published recently, probably because of Mythos.
Working with it day to day I get this feeling that the tech stack used results in a system that’s… clumsy and constrained.
Little things give me that impression, and I can’t quite put it in words, but it’s thirty years of experience working with dozens of languages and platforms speaking here.
Using C makes you clumsy.
It makes you trip over things other languages don’t.
It makes it obscenely difficult to do even simple things. It’s like trying to put a delicate ship into a bottle while wearing oven mitts.
Switching to a better language isn’t just about the specific capabilities of its compiler, it’s also about what it enables in the humans using it.
Security researchers should report their findings to a committee that includes some big companies (IBM and Oracle seem like trustworthy choices here, but ideally we should find a way to get Microsoft included). Those companies would apply the security patches and distribute binary builds of Linux to their customers. Users fortunate enough to have a business relationship with those companies would be protected immediately. The source would still be published after 90 days for educational purposes and for anyone who doesn't appreciate the security benefits of this approach.
"But even if you could convince people to collaborate like this for the greater good, the GPL makes it legally impossible", you say. Ah, but the GPL only says you have to make the source available for a minimal monetary cost, it doesn't impose a time limit. Traditionally, responding to source code requests with a snail-mailed CD is good enough. No judge in the US is going to rule that a short administrative delay in sending out those CDs - in the name of everyone's security, after all, and 90 days is nothing to the judicial system - violates a nebulous licensing agreement from a different era.
Why?
> Additionally, having AI evaluate each commit as it passes is increasingly cheap and effective
This is the key. With AI, the “people won't notice, with so many changes going past” assumption fails.
Soon, there will be no such thing as a safe way to disclose a vulnerability in an open source project. Centralized SaaS will have a major security advantage here.
Edit: Because an RCE in a open-source dependency means you are just as vulnerable when the security patch lands? I don’t see the controversy.
They are basically slot machines, sometimes you win a little bit and sometimes you win a lot but usually you just burn a ton of time and money sitting and staring at a screen (and frying your brain).
https://blog.mozilla.org/en/privacy-security/ai-security-zer...
for those (like me) who hadn't seen it before.
I agree it is not much additional evidence! If someone wanted to try running the same test on a series of N commits from that list including this one I'd be very curious to see the answer!
This is an important facet of the problem space: security risks turning into an arms race for who wants to spend more tokens.