Posted by dangelosaurus 14 hours ago
What are the mitigating factors that prevent this level of vulnerability from being an existential threat to humanity?
Hopefully this type of work to automatically discover & patch vulnerability leads to much more reliable software, and the remove the need for cyber security.
[1] https://www.cyberdefensemagazine.com/zero-day-market-the-gov...
2026: https://medium.com/@hammaadmughal/the-invisible-bazaar-insid...
etc. Concept of governments being main buyers doesn't change by year.
To explain more seriously, almost all visible cyberattacks are carried out by profit-motivated criminals. This has a very clear cost-benefit analysis; your revenue is hacking organizations and extorting them for available money, and your costs are developing and deploying attacks. The ROI is outrageously good easily being on the order of 10x to 100x. If it is so easy and so profitable why do not they just hack all of the millions of organizations on the planet with millions of dollars of available cash for trillions of dollars of revenue? Sure, all they have to do is create a trillion dollar revenue commercial organization bootstrapping themselves with no outside investors, loans, or access to financial resources or the financial system.
I mean, it took Mark Zuckerberg 20 years and a metric ton of VC investment and public investment to make 200 billion dollars in revenue. So what kind of loser can not make a commercial organization 5x larger with a trillion dollars in revenue in less time with no outside investors? Are they even trying? Just make more revenue than any other commercial organization in the entire world and extract that money in a freely usable form, how hard can that be?
They are not there yet, but they are trying. If you look at cybercrime reports you see something like 3x-10x growth year over year. Ransomware reports from a few years ago were showing a 3x year over year growth in number of attacks and simultaneously a 3x year over year growth in ransom amount with the averages being on the order of 100,000 dollars. Chump change for any serious organization, but give that a few years of 3x and that starts to look like real money. However, as it becomes real money it becomes harder and harder to launder it into a usable form. Sure, you could hit someone for 30 million dollars, but how are you going to get that into your bank account? Cryptocurrency helps some, but even there the big money tries to steer away from large overt crime. Maybe you could launder a few tens of billions a year that way, but that is still a far cry from the trillions for everything everywhere.
And of course even assuming you can scale your organization 10x year over year it still takes a decade of sustained growth, scaling your hiring and organization across 9 orders of magnitude, to get to billions per year. A literal decade even if you were growing at a outrageous pace for a outrageous amount of time just to reach a fraction of the total addressable market. A decade of that and you are still nowhere near blowing up all the power plants. Give it time, we need a lot of bears to get there.
From his talk, AIs can't reliably deduplicate or count vulnerabilities, dash for easy and meaningless escape hatches like malicious filesystems, and report the same things as found by everyone else. Indeed, this post literally says: "Within 10 days while the scan was running, we observed over 100 Linux LPEs had been patched or had staging patches from other contributors, with ~300 more being publicly discussed."
A few ready takeaways: this is just hype. The community is already doing a great job at finding and patching the actual vulns without the attention-desperate big labs. And no, you absolutely do not need to pay them a million dollars each year for the privilege.
This behavior is the same thing that OpenAI has been doing to math -- produce tons of slop that are useless to humans. But this time, we don't even get to see how bad it is because they need to hide it to protect our precious little minds. We have got to be afraid though, very afraid.
Our field really needs someone like Terry Tao to unambiguously stand up to this behavior and say that they are not welcome to take such dumps on everyone.
It is however reasonable to assume that a lot of bugs that are being found by the community these days were found thanks to those people using AI to aid them.
See for example what a PostgreSQL developer is saying:
> All software contains bugs, and PostgreSQL’s code base is large, complex, and old. Our git history – converted from CVS – now reaches back more than 30 years, and the first code was written a decade before that. There’s a huge amount of code, and that code contains a huge number of bugs. That’s always been true, and it hasn’t stopped PostgreSQL from being popular or successful. What’s different now is that it’s become vastly easier to find those bugs. […]
> […]
> […] a huge number of people on the Internet are now reporting bugs in PostgreSQL that they found using LLMs, and the PostgreSQL committers have been very, very busy fixing them […]
http://rhaas.blogspot.com/2026/10/postgres-19-is-late-and-it...
Why not look at something like SQLite instead and conclude that things are great?
The Linux kernel is also a huge amount of code that has been worked on by a lot of people for many many years.
There are certain types of bugs that are too obscure to realize are even there for a person alone. And there are types of bugs that even if you see that something is funky it is too time consuming to dig into for a human.
I am not saying that the Linux community is not doing great. They are doing a fantastic job, and have been for years, and they are giving all of that away for free too which is nothing short of incredible.
What I am saying is that I think AI allows everyone to find even more bugs than was humanly possible without AI.
One of the most important paragraphs from the article. Real implications for software development from a security perspective for new and existing software. Every software developer and vendor has a choice... Submit your code base to a frontier model to have it reviewed and fix all the bugs. Each time a new model is released, this process must be done again. If your company is against submitting your code to the frontier model companies for the many good reasons to avoid doing it, your software will eventually become a target if it isn't already. Then your company will be forced to scan the software and deal with the fallout from getting hacked.
If the Linux kernel has this many exploits, and it has been audited more than most other software... Then it seems obvious that all other software is even less secure.
As models get better and porting costs get lower, maybe it's better to bite the bullet and RRiR instead of having a minor bugpocalypse every 3 months, as each subsequent generation of frontier models finds sev-high CVEs?
"you're going to need to do multiple rounds of scanning so we recommend budgeting $3.6 million a year for buying tokens from us not just $1.2 million.
Dont worry just give us a few million dollars and you'll never get hacked, you can cut your security staff in half to pay for it! We definately wont cause more bugs!"
Yeah yeah we're all going to be left behind if we do not jump off the same cliff everyone else is jumping off.
It is wrong to try to take your own lessons from this and do the thing but in a less harmful way. Just do exactly what the labs want you to do.
Jesus man
The other issue is GitHub allowing 1000's of black hat open-source exploit malware swarm tools being forked.
The future internet is going to be perilous.
Within 12 hours, the analyst agents produced 2,308 candidate findings after deduplication. This stage cost $316,549 in API credits.
You don't need SOTA models to pend test your systems.
With enough effort put into proper harnesses, you can use good open-weights models and achieve a lot.
You can also use coding agents to clear out technical debt, or improve the security architecture.
Don't let yourself get paralyzed by lack of access to the latest-and-greatest models.
You can the same $20/mo Claude code plan you've been using to build your system to red team attack the system you've been building with it and probe it for holes, and it'll do a decent job of making sure there aren't any glaring issues.
Imagine if open-source organizations other than the Linux Foundation (and Mozilla I guess) could throw $1.2M of API-rate LLM usage at security.