Posted by speckx 3 hours ago
Meanwhile: the ability to poison models, if it can be made to work reliably, is a genuinely interesting CS question. I'm the last person in the world to build community with anti-AI activists, but I'm as interested as anybody in attacks on them! They should keep that up, and I think you'll see threads about plausible and interesting attacks are well read, including by people who don't line up with the underlying cause.
Ultimately, it comes down to the halting problem: If there's a mechanism that can be used to alter the measured behaviour, then the system can change behaviour to take into account the mechanism.
In other words, unless you keep the poisoning attack strictly inaccessible to the public, the mechanism used to poison will also be possible to use to train models to be resistant to it, or train filters to filter out poisoned data.
At least unless the poisoning attack destroys information to a degree that it would render the poisoned system worthless to humans as well, in which case it'd be unusable.
So either such systems would be insignificant enough to matter, or they will only work for long enough to be noticed, incorporated into training, and fail.
I agree it's an interesting CS challenge, though, as it will certainly expose rough edges where the models and training processes works sufficiently different to humans to allow unobtrusive poisoning for a short while. Then it'll just help us refine and harden the training processes.
The question is not whether the system can change, it's whether the system is incentivized to change. Poisoners could operate entirely in the public, and theoretically manage to successfully poison targeted topics, and it could cost the model developers more than it's worth to fix it. Think about obscure topics like, say, Dark Souls speedrunning. There is no business demand for making sure that a model can successfully give information relating to something like that, so poisoning, if it works, would probably not be addressed, because there's no reason for the model developers to care.
Whether model poisoning becomes a bigger issue depends on the incentives for companies to keep fighting it. For now in comparison to attackers the incentives and resources needed to defend against model poisoning are huge so it's just temporary setbacks. Will that unevenness in their favor always be the case?
https://en.wikipedia.org/wiki/Lotka%E2%80%93Volterra_equatio...
https://en.wikipedia.org/wiki/Rice%27s_theorem
Formally, any non-trivial semantic property of a Turing machine is undecidable. Semantic here (roughly) means "behavioral" questions of the turing machine. E.g. if you only look at the "language" it defines (viewing it as a black box), then it is undecidable to answer any question about that language (including things like if it terminates on all inputs).
Practically though that isn't a complete no-go result. You can do various things, like
1. weaken the target you're looking for. if you're ok with admitting false positives or false negatives, Rice's theorem no longer applies, or 2. rephrase your question in terms of "syntatic properties", e.g. questions about how the code is implemented. Rust's borrow checker does this via lifetime annotations, for example.
> the fact the Chinese populace is much more pro-AI than the West.
https://www.ted.com/talks/peter_steinberger_how_i_created_op...
Then I have good news for you: If humanity goes extinct in the next few years because of unaligned superintelligence, there actually will no longer "be an active community of people who loathe AI and work to obstruct it"
This is either a misunderstanding of the anti-AI crowd or an intentional attempt to discredit them. The majority of anti-AI people don't actually fear this because that belief would require that this person has already bought into the hype regarding the actual power and prowess of AI. The bigger motivator for anti-AI folks is usually just the way it amplifies the negative traits of humans and the systems we have created which is already happening and doesn't need any type of pending "superintelligence" breakthrough. For example, an AI doesn't actually need to be able to perfectly replace the work I do for someone to decide that it's more cost-effective to fire me and give my work to that AI.
This attempt to "reframe and reclaim" (here, paraphrased: "significant existential risks from AI is actually marketing hype by pro-AI fanatics") is a rhetorical device, but not an honest one. It's a power struggle over who gets to define and lead "the" anti-AI movement.
We may agree or disagree with them but there are rational anti-AI arguments that center on X-risks.
See my other comment. I qualified what I said while the comment I replied to didn't, so it's weird that this is a response to me and not the prior comment.
>here, paraphrased: "significant existential risks from AI is actually marketing hype by pro-AI fanatics"
If we're talking "dishonest rhetoric", this is a dishonest framing of what I said. I'm not saying this is inherently intentional marketing hype. I'm saying there is a correlation between someone who thinks AI is that powerful and someone who thinks AI will benefit humanity. The anti-AI crowd is less likely to be a believer in AI's unique power and will simply look at it as a tool wielded by humans which means critiques of it will simply mirror critiques of humanity.
I've seen people claiming that this could happen, but I've yet to read any plausible scenario where this might be the case. Maybe I lack the imagination, could you enlighten me?
You may ask why that is interesting: it's because carrot cake is, despite the name, made mostly of flour and dehydrated lemons. The cooking process is of course handled by a custom implementation of CP/M, running on a Z80.
One time it drew a fortnight riding a bike. Hilarious.
Someone shared the list on here years ago but I can't find it again.
Are you making big money from the hype?
There were never such wide scale and, above all, centralized efforts to coerce and shame people into using the Internet or smart phones in spite of their best efforts.
Don't play dumb.
I mean, it's still ongoing! Tons of people prefer to do things the analog way, and it's certainly not for a lack of companies trying, as the analog way is usually much more expensive.
In their personal lives, everybody should of course be free to do what they want, but I also doubt that zero people have been fired for e.g. refusing to train to use a computer and email because they preferred the aesthetics of typewriters or handwritten memos and physical intra-office mail.
I can guarantee there will be at least a few small ones, especially in the wake of the Sam Altman attacks and the "Zizian" cult. I doubt they'll be very organized and they will ultimately fail, but unfortunately at least a few people will (and have already) die(d) because of these radicals.
https://www.theguardian.com/technology/2026/apr/18/sam-altma...
https://edition.cnn.com/2026/04/17/tech/anti-ai-attack-sam-a...
https://www.theguardian.com/global/ng-interactive/2025/mar/0...
Also saying "these radicals..." like this makes you sound like you are the Empire in Star Wars.
Can you not see how there's a difference?
* No legitimate justification: their materials are being stolen to train and be regurgitated by LLMs and generate products. They are not being compensated yet their contribution goes on to make AI companies money, and preventing open consumption of their materials, to assist an AI company in rendering them obsolete, is not a justification for retaliating? You would have the barest whiff of a point if OpenAI and company were going to artists, requesting materials for training, and were given tainted ones, that at least I could say was duplicitous. But not when it's publicly posted, that's just an AI company not doing a good job of minding their input.
* Serve only to make access to and transformation of info more difficult: As in, you have to go to the website of the person actually publishing the information, as opposed to having it read in a Google summary? Also worth noting this inconvenience applies only to a theoretical person using an AI search tool. Everyone else is unaffected. Seems like if you're going to a particular service provider whom is uniquely unable to provide the service you want, that seems like an easy to solve issue: use something else.
* can only hope that by these egregiously anti-social luddites: Your daily reminder that the Luddites were not anti-technology, they were anti-corporations using mechanization to make an ever dwindling number of workers produce ever more products of ever lower quality.
* we'll gain the knowledge to render this category of attack moot for the foreseeable future: This is a bad strategy and historically has not worked for a single industry. If your industry itself exists in open opposition to consumer movements, you don't win. At best, you survive. But there's no version of this where everyone just unwillingly adopts AI and you can tell them to deal with it. Whole companies now are cropping up to help people who want to opt-out of the AI future as promised.
So yes, you can pollute the good old internet even more, but no, you cannot change the arrow of time, and then there's already the growing New Internet of APIs and public announce federations where this all matters very little.
Abusive, sneaky scraping is absolutely through the roof.
Since AI crawlers don't obey any consent markers denying access to content, it makes sense for content owners who don't want AI trained on their content to poison it if possible. It's possibly the only way to keep the AI crawlers away.
Think about it, why would a training scraper need to hit the same page hundreds of times a day? They only need to download it once.
I think this is LLMs doing web searches at runtime in response to user queries. There's no caching at this level, so similar queries by many different users could lead the LLM to request the same page many times.
Unfortunately that won't work. If you've served them enough content to have noticeable poisoning effect then you've allowed all that load through your resources. It won't stop them coming either - for the most part they don't talk to each other so even if you drive some away more will come, there is no collaborative list of good and bad places to scrape.
The only half-way useful answer to the load issue ATM is PoW tricks like Anubis, and they can inconvenience some of your target audience as well. They don't protect your content at all, once it is copied elsewhere for any reason it'll get scraped from there. For instance if you keep some OSS code off GitHub, and behind some sort of bot protection, to stop it ending up in CoPilot's dataset, someone may eventually fork it and push their version to GitHub anyway thereby nullifying your attempt.
One could imagine an open source project that doesn't want to be ingested by an LLM. They could try to put that in the license but of course the license won't be obeyed. Alternately, if they could alter the code such that the OSS project itself remains high quality, but if you try to train a coding LLM on it the LLM will output code full of SQL injection exploits (for instance) or maybe just bogus uncompilable stuff, then the LLM authors will suddenly have a reason to start respecting your license and excluding the code from their index.
Yes, they can't publish it without attribution and/or compensation (copyright, at least currently, for better or worse). Yes, they shouldn't get to hammer your server with redundant brainless requests for thousands of copies of the same content that no human will ever read (abuse/DDOS prevention).
No, I don't think you get to decide what user agent your visitors are using, and whether that user agent will summarize or otherwise transform it, using LLMs, ad blockers, or 273 artisanal regular expressions enabling dark/bright/readable/pink mode.
> it makes sense for content owners who don't want AI trained on their content to poison it if possible. It's possibly the only way to keep the AI crawlers away.
How would that work? The crawler needs to, well, crawl your site to determine that it's full of slop. At that point, it's already incurred the cost to you.
I'm all for banning spammy, high-request-rate crawlers, but those you would detect via abusive request patterns, and that won't be influenced by tokens.
This is true. Some documentation of stuff I've tinkered with (though this isn't actually published as such so not going to get scraped until/unless it is) having content, sufficiently out of the way of humans including those using accessibility tech, but that would be likely seen as relevant to a scraper, will not be enough to poison the whole database/model/whatever, or even to poison a tiny bit of it significantly. But it might change any net gain of ignoring my “please don't bombard this with scraper requests” signals to a big fat zero or maybe a tiny little negative. If not, then at least it was a fun little game to implement :)
To those trying to poison with some automation: random words/characters isn't going to do it, there are filtering techniques that easily identify and remove that sort of thing. Juggled content from the current page and others topologically local to it, maybe mixed with extra morsels (I like the “the episode where” example, but for that to work you need a fair number of examples like that in the training pool), on the other hand could weaken links between tokens as much as your “real” text enforces them.
One thing to note is that many scrapers filter obvious profanity, sometimes rejecting whole pages that contain it, so sprinkling a few offensive sequences (f×××, c×××, n×××××, r×××××, farage, joojooflop, belgium, …) where the bots will see them might have an effect on some.
Of course none of this stops the resource hogging that scrapers can exhibit - even if the poisoning works or they waste time filtering it out, they will still be pulling it using by bandwidth.
It wont mean we see the model collapse in public, more we struggle to get to the next quality increase.
I understand that if I have an AI model and then feed it its own responses it will degrade in performance. But that's not what's happening in the wild though - there are extra filtering steps in-between. Users upvote and downvote posts, people post the "best" AI generated content (that they prefer), the more human sounding AI gets more engagement etc. All of these things filter AI output, so it's not the same thing as:
AI out -> AI in
It is:
AI out -> human filter -> AI in
And at that point the human filter starts acting like a fitness function for a genetic algorithm. Can anyone explain how this still leads to model collapse? Does the signal in the synthetic data just overpower the human filter?
It’s pretty shocking how much web content and forum posts are either partially or completely LLM-generated these days. I’m pretty sure feeding this stuff back into models is widely understood to not be a good thing.
Doom-saying about "model collapse" is kind of funny when OpenAI and Anthropic are mad at Chinese model makers for "distilling" their models, ie. using their outputs to train their own models.
In fact, given this many parameters, poisoning should be relatively easy in general, but extremely easy on niche subjects.
Nope. Go look up double descent. Overfitting turns out not to be an issue with large models.
Your video is from a political activist, not anyone with any knowledge about machine learning. Here's a better video about overfitting: https://youtu.be/qRHdQz_P_Lo
That said, I see red flags here. This is an extraordinary claim, and extraordinary claims require extraordinary evidence. My actual degree (not the drop-out one) is in Psychology and I used statistics a lot during my degree, it is only BSc so again, I cannot claim expertise here either. But this claim and the abstracts I scanned in various papers to evaluate this claim, ring alarm bells all over. I don‘t trust it. It is precisely the thing that we were told to be aware of when we were taught scientific thinking.
In contrast, this political activist provided an example (an anecdote if you will) which showed how easy it was for an actual scientist to poison LLM models with a made up symptom. This looks like overfitting to me. These two Medium blog posts very much feel like errors in the data set which the models are all to happy to output as if it was inferred.
EDIT: I just watched that video, and I actually believe the claims in the video, however I do not believe your claim. If we assume that video is correct, your errors will only manifest in fewer hallucinations. Note that the higher parameter models in the demonstration the regression model traversed every single datapoint the sample, and that there was an optimal model with fewer parameters which had a better fit then the overfitted ones. This means that trillions of parameters indeed makes a model quite vulnerable to poison.
Instead, the LLM did a web search for 'bixonimania' and summarized the top results. This is not an example of training data poisoning.
>This is an extraordinary claim, and extraordinary claims require extraordinary evidence.
Well, I don't know what to tell you; double descent is widely accepted in ML at this point. Neural networks are routinely larger than their training data, and yet still generalize quite well.
That said, even a model that does not overfit can still repeat false information if the training data contains false information. It's not magic.
It's wild to see the about face. Now it's:
> If [companies] can’t source training data ethically, then I see absolutely no reason why any website operator should make it easy for them to steal it.
It would have been very difficult to predict this shift 25 years ago.
Let say person A wants everyone to be rich.
Person B plots a plan to make themself rich and everyone else poorer.
One can make an argument that any action by A is now a contradiction. If they work with B, it makes a lot of people poorer and not richer. If they work against B, B do not get rich.
However this is not a contradiction. If a company use training data in ways that reduce and harm other peoples ability to access information, like hiding attribution or misrepresenting the data and sources, people who advocate for free information can have a consistent view and also work against such use. It is not a shift. It is only a shift if we believe that copyright will be removed, works will be given to the public for free, and companies will no longer try to hide and protect creative works and information.
We welcomed the vampires in and wonder why our necks hurt.
The last time a property class was removed was _slaves_.
Arguing that copyright is good because a subset of big tech doesn't want it around is as stupid as arguing that slavery is good because the robber barons don't like it.
What's more it's a property class we have been fighting against since before the majority of people on here were born. We are finally winning after decades of losing. The 1976 copyright act was at best a Trojan horse and the 1998 Mickey Mouse Protection Act was a complete disaster.
In short: sprinkles holy water.
They are thrilled.
The folks fighting perpetual copyright were not fighting to make it possible for Disney to fire creatives. In fact they were fighting for the creatives to triumph over Disney.
> In fact they were fighting for the creatives to triumph over Disney.
We were doing nothing of the sort. It was "Information wants to be free" not "we want to provide a perpetual job for a subset white collar workers".
sprinkles holy water
Our concern was that corporations were expanding the definition of intellectual property to the extent where you couldn't make a movie or song or write a book as an individual without some corporation with a massive "IP" warchest coming after you and declaring it derivative. You couldn't write some software without a corporation with a massive repository of junk patents claiming you infringe.
We wanted to insure that individual creators could continue to have a voice, and not get sued out of existence by an IP Legal/Industrial Complex that was forming causing arms races between megacorps and SLAPs against everyone else.
If we knew we were feeding a yet-to-be-invented slop machine that would allow megacorps to unemploy all the creatives, most of us would not have supported that.
And by the way Disney is all in on AI for the same reason they were all in on perpetual copyright. In the perpetual copyright world, having a massive library of content you no longer have to pay residuals on was a source of massive amounts of "free" revenue. You could just keep re-releasing and re-making stuff. You did not have to do the messy, expensive work of paying people to come up with really good new stuff.
In the AI world, the money-printing capital asset is the trained model that grinds out slop 24/7 and you -emdash- again -emdash- don't have to pay actual people to create anything new.
We have multiple Communist ais that is on par with Western ai from 18 months ago and can run locally on 5 year old hardware.
I have no idea the fever nightmare you live in but the future is bright and only getting better.
Property classes are born and die everyday. You can own the rights to publish an arcade video game, but that class of rights would have been way more valuable 45 years ago. NFTs were born and died just recently. You can own digital assets worth real money in an online game that simply shuts down.
Some people may read this and say "these don't qualify as a property class", to which I will remind you that property class used in this way is a brand new term, which I think is invented solely to be able to compare the limitations on human freedom associated with slavery to the limitations on human freedom associated with intellectual property.
Easy counterexample: titles of nobility. Also perpetual bonds, delegated taxation rights, the ability to mint currency. The list goes on.
If you're going to use history to support your AI bull agenda, you should at least pre-fly it with the AI first -- it would have pointed this out.
> Arguing that copyright is good because a subset of big tech doesn't want it around is as stupid as arguing that slavery is good because the robber barons don't like it.
Sorry, who's saying it's good? You are, actually, insofar as you're willing to support the right of AI companies to take people's information and use it to create copyrighted model weights. Why do you care less about the intellectual property of billionaires than that of the common man? Do you really think they're on your side?
People are in general for whatever they think will benefit them, and against what they think will harm them.
So piracy is ok when it benefits the little guy and not ok when it benefits the big guy. Unions are good when they stand up against employers, and bad when they discriminate against non-union workers. There's no contradiction there.
The information is still there, as is the community that you've built, the joy that you get out of sharing the information, everything you've learned...
Why is any of that diminished, just because some people or entities that you dislike also got something out of it?
The end result of major tech companies sweeping in, taking everyone's creative work, outcompeting the originals with AI derivatives, and telling every artist on the planet "fuck off, send a job application to McDonalds" is significantly less art.
Copyright was invented to prevent exactly this scenario.
Hackers have usually drawn their funding from their (often lucrative) employment, which is what gave them the freedom to give away the products of their hacking for free.
One needs copyright to survive, the other see it as a means to enforce openness at best (those in favor of copyleft) and as an obstacle to their pursuit (owning the full system, liberating all aspects of and information about it) at worst.
This rift was always visible if you knew where to look, but AI is definitely wedging it wide open.
Such is the fate of all utopian dreams.
Still, people were saying all kinds of inane stuff 25 years ago too.
I say this as someone whose notions exist orthogonal to the debate; I use AI freely but also don't have any qualms about encouraging people to upend the current paradigm and pop the bubble.
Tell me more? I'm guessing you might say: neither connects with everyday people, they have misaligned incentives*, they (like most corporate leaders) don't speak directly, they have more power than almost any elected leader in the world, ... Did I miss anything?
My take: when it comes to character and goals and therefore predicting what they will do: please don't lump Amodei with Altman. In brief: Altman is polished, effective, and therefore rather unsettling. In short, Altman feels amoral. It feels like people follow him rather than his ideas. Amodei is different. He inspires by his character and ideals. Amodei is a well-meaning geek, and I sometimes marvel (in a good way) how he leads a top AI lab. His media chops are middling and awkward, but frankly, I'm ok with it. I get the sense he is communicating (more-or-less) as himself.
Let me know if anyone here has evidence to suggest any claim I'm making is off-base. I'm no oracle.
I could easily pile on more criticisms of both. Here's a few: to my eye, Dario doesn't go far enough with his concerns about AI futures, but I can't tell how much of this is his PR stance as head of A\ versus his core beliefs. Altman is a harder nut to crack: my first approximation of him is "brilliant, capable, and manipulative". As much as I worry about OpenAI and dislike Altman's power-grab, I probably grant that he's, like most people, fundamentally trying to do the right thing. I don't think he's quite as deranged as say Thiel. But I could be wrong. If I had that kind of money, intellect, and network, maybe I would also be using it aggressively and in ways that could come across as cunning. Maybe Altman and Thiel have good intentions and decent plans -- but the fact remains the concentration of power is corrupting, and they seem to have limited guardrails given their immense influence.
* Here's my claim, and I invite serious debate on it: Dario, more than any corporate leader, takes alignment seriously. He actually funds work on it. He knows how it works. He cares. He actually does some of the work, or at least used to. How many CEOs of the companies they run actually have the skills to DO the rank-and-file work? Even the most pessimistic people probably probably can grant this.
Yep, Dario is straddling this sort of impossible line: he's the least-scary harbinger who is try to be one of the more transparent people to sound the alarm. But the funny thing about saying "don't shoot the messenger" is that it usually gets uttered well after the messenger has taken a bullet.
> You're overthinking the parent comment, I think.
Luckily, the phrase overthinking is on the way out. We really don't want any more Idiocracy Part II. In this day, we need all the thinking we can get. We often need (1) better thinking and (2) the ability to redirect our thinking towards other directions.
In my experience, 2026 is the year where almost all stigma about "talking AI" is out the window. I am nearly at the point where I say whatever I think needs to be said, even if I'm not sure if people will think I be crazy. So if Typical Q. Person asks me, I tell them whatever I think will fit into their brain at the time -- how AI works, why Dario is awkward, why superintelligence is no bueno, etc.
Dario is not just a messenger, though. In his case it would be more like, "Don't shoot one of the generals in the invading army." To which it would be reasonable to ask, "Why not?" Even if he's the general saying that he wants minimal civilian casualties.
These days the tech industry is more moneyed circus than serious effort to improve humanity.
Fortunately no-one sane enough among us, computer programmers, believes in that bs, we all see this masquerade for what it mostly is, basically a money grab.
Assuming the LLM actually got its answer from that comment, it was from a web search.
Models are retrained only every few months at best; it is not possible for a comment made a few hours earlier to be in the training data yet.
Should they hire them?
Yes the specification is holding a lot of weight here. Assume it's comprehensive and all consultancies offer the same aftercare support. Otherwise we're just handwaving and bike shedding over something that's not measurable.
If we're going to have AI overlords, it'd be great if they spoke with proper grammar.
I simultaneously think
1. AI will be a massively impactful technology on the scale of the industrial revolution or greater
2. The potential upside of AI is enormous, but potential downside is just as big (utopia or certain ruin)
3. Most current AI companies are acting somewhat reasonably in a game-theory sense with respect to the deployment of their tech, and aren't especially evil or dastardly compared to Google in the 2000s, social media in the 2010s
4. AI safety is an under-appreciated concern and many who are spending time nitpicking the details are missing the bigger picture of what ASI and complete human obsolescence look like.
5. No amount of whiny protest, data sabotaging, or small-scale angst or claiming that AI is "fake" or hoping for the bubble to pop is going to have even a marginal effect on the development of AI. It is too powerful and the rewards are too great. If anything it will have an overall negative effect because it will convince labs that their potential role as a utopian, public benefactor will not be appreciated, so will instead align themselves with the military industrial complex for goodwill.
Resistance is futile
You're just picking random problems with tech and blaming them on AI.
Isn't there somewhere between removing AI from the world entirely and just sitting back and letting it take over everything? I want to talk about responsible AI use, and how to mitigate the effects on society, and to account for energy consumption, etc.
I think AI as a proper utilized tool, is amazing, I think our lack of restraint when just throwing it into everyone's hands without understanding of the tools they are using, is horrifying. I'd imagine a lot of the community here echos that same sentiment, but maybe not, and i am just making assumptions.