Posted by softwaredoug 6 hours ago
There are millions of samples available on huggingface and models explicitely trained on output produced by fable. There has been no action taken against them.
Another example is that it appears that the upper limit of what you can do is ultimately dependent on people working on the model, otherwise grok would be a LOT more competitive pre-cursor acquisition.
And lastly, kimi architecture is vastly different than that of fable as it uses mechanisms developed by... kimi themselves. US AI labs are inspired by opensource advancements just as much as open source labs are inspired by traces from models such as fable.
Claiming in any shape or form that fable disillation is one of the primary reasons why kimi k3 is so competitive is slandering the work of other labs that cooperatively push the open-source models forward.
edit: (moved this to bottom) The only argument they have here is that they use GB300 GPU's which for some reason should not be available to chinese citizens.
ed: to clarify, I totally agree that a huge chunk of the value in LLMs is coming from the source material. My point was just that training an LLM takes more resources and expertise than distilling from an existing LLM so I don't think the equivalence between training and distilling is entirely justified.
It's the most CS-major take ever!
The LLM output, is not the same as the input - there is value add.
Of course works used as raw inputs to LLMs required work and are reasonably subject to IP concerns - but they are different.
It's possible that the LLM makers 'owe' the content creators that created the content they used to make their products - it's an interesting but separate question.
We could very well end up where content IP is protected, LLM output is not and visa versa with reasonable legal founding, doubtful but plausible.
How, and why?
> We could very well end up where content IP is protected, LLM output is not and visa versa with reasonable legal founding, doubtful but plausible.
That is the current state of legal rulings - LLM output is public domain, not copyrightable.
This is not automatically true. Training and distillation use the same underlying infra and method and there is no intrinsic differences in between.
Writing books, building Wikipedia, and answering questions on online forums takes a lot of resources and expertise that scraping didn't. So at the very least, we're already one rung down the "maybe you should've asked" ladder.
It took me a year to write a book. It took OpenAI and Anthropic a fraction of a second to ingest it. Do you understand now why I give zero shits if it takes Anthropic a billion to train a model, and Moonshot 10k in API cost to distill it?
If the distilled model is cheaper, then it's just LLM's getting LLM'ed.
If that is the whole point you need to clarify why this is the case on an objective level.
I would say building a comparable model using any means necessary (just like what Anthropic and OAI did) at a lower cost is actually more valuable to soceity and Monshoot is arguably generating more value with less.
Raw materials vs. Value add.
They are different things, like ore and metal.
Distillation is a new thing we need to understand, it's probably closer to IP than not.
We're talking about things like text people wrote, not some kind of raw data floating out in the ether.
That would stun me, but it's a little hard to read.
The recent announcement that AI-assisted research produced a counterexample to the Jacobian conjecture--a long-standing open problem in algebraic geometry--shows the original value AI can create. The result was not copied from a textbook; it emerged from AI learning from existing material, much as a human does, and then applying that knowledge in a new way. If that's a violation of copyright, then a human doing the exact same thing would be a copyright violation too. But it isn't.
It’s massive copyright infringement.
The human buys the books.
>Claiming in any shape or form that fable disillation is one of the primary reasons why kimi k3 is so competitive is slandering the work of other labs that cooperatively push the open-source models forward.
If the distillation is irrelevant to why it is competitive, why do they do it then? Obviously is helps improve their benchmarks/performance to some degree, otherwise they wouldn't need to do it.
Although I will reiterate the fact that distillation is not the primary reason why these models are performing so competitively.
If Kimi k3 really were above Fable 5 then there invariably the USG would have to consider their restrictions on model capabilities excessive, or one would have to admin closed source models are held to more restrictive safety standards than open source models.
>Although I will reiterate the fact that distillation is not the primary reason why these models are performing so competitively.
How would you know this? How could you ascertain exactly how much performance is attributable to their unique engineering/research? If they really were so competitive they could surely make a model that isn't dependent on distilling Fable or other frontier models.
Kimi specifically relies heavily on reasoning traces which is largely due to their training strategy and will perform poorly when thrown into a conversation from another model. Another fun advancement is that they simply ctrl+c ctrl+v'd attention which means that the model can steer where to look in the context window without ever producing an output token increasing token efficiency and attention accuracy as a side effect you end up with weaker prompt adherence.
None of these 'issues' manifest in US models which proves that kimi has diverged and is achieving these capabilities seperately from the architecture that US labs rely on.
I would agree with you during the Deepseek R1 era, but US labs were heavily inspired by open research at that point as well so I wouldn't give them too much credit.
In other words: even if the US (somehow) denies them access to the current OpenAI/Anthropic models, they'll be able to improve based on what they already have.
Actually, that has already happened in many domains, it's just that most western people (USA especially) won't admit it.
at the same time, I don't buy the idea that distillation is unimportant in assessing what Chinese labs are capable of. If it wasn't, why did Kimi's release timing coincide so well with Fable's launch?
and if Anthropic hadn't released Fable, would we have Kimi today? If the answer is no, then I think that's still a very important point to consider.
For some reason folks seem to think that China can take action and then other countries can't also take action or respond to that action and it comes up again and again. China has hypersonic missiles! Pack it up boys time to go home. Nothing we can do. Dang shucks. China distilled American AI models, welp time to just close it all down and let's just write off those trillions of dollars and all the literal geniuses financing and building these things. Oh well China can just copy American models while we spend all the money! Ok we just stop developing models and we'll just copy their models. China will flood the market with their cheap products! Nope can't do anything like, oh, idk, not buy any of those products or just raise the prices on them in local markets. It's never-ending. I don't understand the lack of capacity to reason about other actors that takes commonly takes place. And that's just China, never mind other general issues.
That works both ways, competition and performance spur new developments. You don't think the American labs are looking at Chinese research on how to reduce compute per token?
Let's not forget how much people talked about "prompt engineering" before Deepseek mainstreamed the idea of thinking mode which is now universal
Note that Chinese companies are free to rent from GB300 clouds internationally. There are large datacenter hubs in Singapore and Malaysia serving chinese and other customers.
Though there is also reported [1] significant smuggling of Nvidia chips into China as well.
The word I would use is inevitable. It reminds me of the (PC) clones wars…
Also, companies that use distillation may be competitive but seem unlikely to surpass the companies that are training these models from scratch.
US dominance is also important for approaches to safety, especially political approaches. If the frontier models are all US-based, safety might be tackled via internal US policy. If other countries can independently train competitive models, international cooperation is required.
Edit: It is also important for the business model. Companies won't be able to justify tremendous training costs if competitors can replicate their product much more cheaply via distillation.
I don’t think China’s necessarily any better, but I’d rather have the most powerful models be open rather than under the exclusive control of the US executive.
No, no we do not.
We already know it's false because you would have hundreds of competitors if it was that easy.
The reason why these Chinese labs are releasing good models is simpler, they have access to a tremendous pool of talented people.
Correct, but it at least helps answer the question of "how do they make such good models for a fraction of the price???" The answer is someone else spends the untold billions and Chinese labs do a little tweaking.
It's showing that 'distillation' is a viable way to reclaim all of what they stole and hoard, and with enough luck their debts will come due in time for them to feel it.
All LLMs are trained on the corpus of humanity's knowledge, the legacy of everyone who's ever lived and our civilization as a whole.
Anything that prevents or circumvents the accumulation or gatekeeping of this knowledge and puts it in the hands of more people (that are not AI company shareholders) is a good thing. Whether that is done by open sourcing the model weights, the training set, or by making the output better and cheaper, it is all fair game and is, as another poster mentioned, inevitable in the long run.
> Distillation is not illegal by every definition of the word
Note that Anthropic (and USG) alleges [0] not only that Kimi was distilled, but that they actively circumvented measures intended to stop distillation. There are multiple ways that's illegal, including:
- Civil breach of contract. Anthropic's TOS explicitly say you can't do what Kimi is alleged to have done.
- Economic espionage: 18 U.S.C. §1831 criminalizes obtaining a trade secret through theft, fraud, or deception while intending that it will benefit a foreign entity.
- Trade-secret misappropriation: if Anthropic could argue industrial-scale querying reconstructed proprietary aspects of Fable (like by showing it produces similar outputs, as others have done) then it's illegal under 18 U.S.C. §1832.
- California computer-access statute §502 bars knowingly accessing a computer system and, without permission, taking, copying, or using its data.
- Computer Fraud and Abuse Act protects against the case where restrictions against an activity are circumvented (like Kimi is alleged to have done).
> There are millions of samples available on huggingface and models explicitely trained on output produced by fable. There has been no action taken against them.
A lack of prosecution does not make something legal. There is also the scale/commercialization thing, which isn't an issue with random tiny HF datasets/models. Remember: Kimi also sells K3 inference.
> kimi architecture is vastly different than that of fable
How do you know that? Do you work for Anthropic? Also, this has nothing to do with architecture, we are talking about data.
> US AI labs are inspired by opensource advancements just as much as open source labs are inspired by traces from models such as fable.
Cool. The difference is that one of those things is legal (because they chose to open-source) and one of those things is illegal theft of trade secrets (because it was stolen).
> Claiming in any shape or form that fable disillation is one of the primary reasons why kimi k3 is so competitive is slandering the work of other labs that cooperatively push the open-source models forward.
1) this has nothing to do with other labs, just Moonshot (and Z.ai, MiniMax, DS)
2) slandering or not it happens to be completely true, so, there's that
[0] https://www.anthropic.com/news/detecting-and-preventing-dist...
In this way, it is different from literal theft. Stealing money/objects from a thief and keeping them is not justified.
An analogy might be a baker stole 20% of the flour used to bake their special bread, which was then stolen. Both thefts are obviously wrong and bad.
"Judge approves a $1.5B Anthropic settlement over pirated books used to train the Claude chatbot"
https://abcnews.com/Technology/wireStory/judge-approves-15b-...
In this case: resolve the theft claims against the US frontier labs, and only then let them make claims against third parties. It would be totally unreasonable for (say) OpenAI to extract a settlement from Moonshot and use that to pay its own claims. Ordering matters.
Same thing here. This whole situation is just comical.
Now, if that kid were to print the bootleg translation and sell it to schoolmates, that's worth a slap on the wrist. The kids willing to pay would likely have paid for official copies.
When these LLM labs download our works, feed them into their models, and sell the output to people that used to pay for our work, that's worth a very hard slap. I honestly have less of a problem with the open models.
It seems like you think theft is bad everywhere except when Anthropic does it.
When I said "Does this matter?" I specially meant that distillation in itself, the data you get from distillation is first and foremost not owned by anthropic nor is it copyrightable. If a user willingly gives up their anthropic reasoning data/traces that is 100% legal no matter what the "terms of service" say as it's not enforceable and would fall apart in court.
And what I explicitely pointed out that focusing so much on distillation is an attack on open research and claiming that the majority of advancements are thanks to US labs which is simply not true (at least not anymore this was somewhat true during deepseek R1 era), but that in itself was inspired by open research.
> How do you know that? Do you work for Anthropic? Also, this has nothing to do with architecture, we are talking about data.
Because anthropic would be the first ones to make that information public and the architecture is unique to kimi... They made it, they wrote papers on it, it's their research.
P.S. none of the quoted laws apply here since no trade information is stolen, the one about circumventing distillation protection might hold up in court although unlikely.
Agree, and this is exactly what Anthropic is alleging.
> data you get from distillation is first and foremost not owned by anthropic nor is it copyrightable. If a user willingly gives up their anthropic reasoning data/traces that is 100% legal no matter what the "terms of service" say as it's not enforceable and would fall apart in court.
It's important to note this is NOT what happened. Anthropic was able to trace data directly back to employees at the company: "We attributed the campaign through request metadata, which matched the public profiles of senior Moonshot staff."
> none of the quoted laws apply here since no trade information is stolen
There is a lot of work showing Kimi models produce similar outputs to Anthropic models, which constitutes trade information. This is not dissimilar to past and ongoing IP suits against Anthropic and OpenAI by showing the models would recreate images of Mickey Mouse/NYT articles etc.
For the record, I'm a researcher myself and I'm well aware how competent the researchers are at the open-source labs/how much they've contributed. But that's not at issue here, my disagreement with you is specific to your arguments about legality; you're conflating what you think should be legal with what actually is legal.
Everything else is simply justifying why it shouldn't, the specifics don't really matter as there is no legal framework to stop china from continuing to distill models and anthropic has proven they cannot use software solutions to stop it either as distillation is still a problem. But I do still believe it wouldn't hold up in court either way as stopping companies from generating training data which was trained on the entire human knowledge corpus is just stealing from thieves and making it 'open' once again so the argument only gets weaker.
This is true, but Kimi also has a variety of defenses. Kimi can't raise unclean hands if Anthropic systematically violated others' terms of use, but it can raise copyright misuse (which is similar in some respects to unclean hands) as well as lack of standing to enforce restrictions in the contract due to the third party beneficiary principle (i.e., Kimi would argue that Anthropic cannot sue Kimi for derived IP that rightfully belongs to third parties whose terms of use were violated by Anthropic, and the proper party to sue Kimi, if any, would be those third parties). That latter argument usually fails in small-scale cases (ProCD) but has been successful in larger ones where the alternative would be anticompetitive.
Ah yes, I remember when Anthropic crawlers abided by the TOS of the websites they slurped up.
All your other points are downstream from this, which makes them pretty tenuous. Labs don't think that ToS or other explicit wishes of content providers apply to them, but they expect everyone else to abide by theirs.
US and CA law really don't care that Anthropic violated IP law elsewhere.
Plainly who gives a flying fuck. The US can claim whatever rules they want and so can China or any other country. On international level all those rules are artificial constructs unless they can be enforced. China can just say for example that they do not recognize copyrights /patents / whatever so it is "legal" for them.
1) its not illegal (it is)
2) it shouldn't be illegal because Anthropic stole training data (thats not how the law works)
I am a practical man. From what I see laws are mostly for common folks and often do not even serve real justice. The higher one goes and the amount of money / power involved the more the laws bend and on international level the only law that matters is the size of one's club and willingness to use it. And when the country with supposedly biggest one starts crying I find it laughable.
The reason why the United States government is weighing in is because it's in the national interest of the US to have supremacy in "AI".
Legality or lack thereof is one of many data points about whether a thing is noteworthy.
Moonshot performing distillation is rational from their point of view. Reducing costs is in the interest of businesses. It's also rational for frontier labs and the US government to add obstacles to this process.
As consumers this is probably a positive development.
And OpenAI scraped and distilled that answer and gave me nothing
I would prefer some sort of democratiziation of the money made from the democratization of information as well
Circumventing costs.
I mainly focus on the last.
It will be hard for a frontier lab to justify spending the compute and data curation needed to advance AI further if that expenditure can be assimilated into your competitor's products within months/weeks. So reality will present labs with three choices:
A. Cease spending massive amounts of money and compute improving those models.
B. make those improved models more difficult to distill from, either through some regulatory regime, or some technical solution, which seems unlikely to me.
C. making the best models available only to select partners and government.
In all these potential outcomes, China, which lacks compute that U.S. labs enjoy, will likely stop seeing massive improvements in their AI models. Improvements to be sure, but right now they are enjoying gains from distillation AND their own model innovations, and these potential outcomes would largely stop one of those sources.
part of the definition of theft is that the original owner is deprived of it, which does not apply to copyright infringement.
You can only argue with damages from the perspective of potential profits, still not theft though.
Or with services, if a barber cuts your hair and then you run away without paying them, do you not consider that theft, even though there's no change in ownership occurring?
I think you're wrong: there is absolutely damage to the authors and publishers from what the AI companies have done.
I mean honestly if they did that why should I care? I'm happy to see copyright violated in a manner that leads to the creation of new technology. IP law exists strictly for the benefit of society and by all appearances AI is an incredibly powerful tool.
Also while I'm at it libgen is a gift to humanity. Information wants to be free. Spreading and preserving knowledge is generally one of the most wholesome activities anyone can undertake as far as I'm concerned.
How did Moonshot "distil" a huge model in such short time and still had time to run the benchmarks and do the usual release thingies?
I think Anthropic is desperate to stop foreign competition and the administration is happy to help because they too are heavily invested in these companies
[Fable fires up a ton of subagents. Their reasoning traces are horrific but somehow K3 learned something.]
Even by San Francisco standards, it is amazingly whiny and pathetic for Anthropic to complain about stuff like this. Dario et al violated copyright, stole your GitHub repos, and now they're burning billions of dollars trying to outcompete you. They're real vampires. OTOH Moonshot violated Anthropic's TOS and are, at worst, moochers. But Fable's output is not actually copyrightable.
We should do more distillation and figure out how to create faster leaner and better models.
Even an openai's guy (head of something made up) called bs on the idea you can train something like k3 by distillation.
Anybody I know who works in LLM research says that distillation is either useless or merely useful in post training to show "correct" behavior.
And even then you don't get a competing model, if RL on good prompts was that useful, all labs would've long skyrocketed in capabilities just by looping on increasingly better prompts, yet that doesn't work.
I don't know what this guy thinks AI is, but this strikes me as delusional.
In my view, AI (LLM) is two things mixed together:
1. A reasoning engine on top of relatively rich fuzzy modal logic, implemented through variety of rules, which implement very common concepts.
2. A huge dictionary of words defined (with lot of detail) in the said logic, together with many known facts about them. Maybe bigger than Wikipedia.
Now, how on Earth do you want to gatekeep either of this? You can't gatekeep the 1st, logic of common sense, that's almost as difficult as gatekeeping a Turing machine (a concept of a computer). And gatekeeping the 2nd is ridiculous too, as it was built mostly from already published sources like a giant Wikipedia.
If anything, the opposite, to gatekeep AI is actually dystopian. It would mean end not only to right to compute, but also end of right to scientific knowledge.
(And I think, honestly, Chinese understand this. Trying to control-export AI makes as much sense as trying to control-export an English dictionary.)
Wow, just wow. He is not even subtle about it.
Obviously, it's not K3 level. But Fable did just put itself out of a job in this case.
So here robbers are blaming robbers?
These claims are just pointless, everytime
I see no problem with distillation, on the other hand the complete dismissal of copyright by AI labs is pretty bad, I don’t think we should put them at the same level
Courts keep ruling over and over that an LLM trained on copyrighted works qualifies as a transformative work and is therefore fair use. They don't have to dismiss copyright law, this has always been allowed.
The only thing they get in trouble for is pirating the works to get their hands on them.
This will have to wait for the Supreme Court. OpenAI and Microsoft 100% deserve to lose, even without OpenAI allegedly hiding evidence.
It takes two parties to agree to a settlement. That the other party agreed to a settlement instead of taking it to court implies this was not the slam dunk you may think it was.
I generally agree, in the same sense that it's "fair" for the US and China to spy on each other. It's not a moral outrage, but it is something that the targets can and should try to prevent.
Regardless, it was always inevitable—will continue to happen.
Technically, providing better value from your competitor's private holdings could be theft (of trade secrets), but might it also be fair use? "Schrodinger's IP" be damned.
I don't think the 1.5B settlement has resolved this. The 2 cases need to be merged!
Meta, for examples, doesn’t want employees to use Claude Code due to distillation risk.
It's also about the larger companies explaining why they can't be as efficient, of course they can't, they're not just ripping the outputs of another model that someone else invested billions to train.
True, they're simply ripping the inputs that humanity invested thousands of years and trillions of dollars to produce.
If they payed for inference, doesn't they own the output? So if I pay for a model to generate code, isn't that code mine to do with it whatever I want? Just curious.
It would be one thing if Moonshot was breaking into OpenAI servers and stealing trade secrets, but the only thing they are doing is looking at the output of the program, which is exactly the service that OpenAI offers. So, at best, this is a ToS violation. Sucks for the frontier labs I suppose, but live by the sword - die by the sword.
The issue seems to be the US only likes competition when it is winning. Markets in Asia are meant for cheap labor and resources, they're not meant to actually compete. /s
This. Free markets for everyone when they're the dominant economic force. Protectionism, tariffs and import/export controls when they're not.
Disgusting.
Like the US did when it "stole" the textiles IP from the UK in order to kickstart its own industry?
> The industry cannot sustain itself if that’s the model
Then let it fall apart.
Anthropic just settled a $1.5B suit over it!
It's like they've read the history of the US and how it got to where it is in the first place.
It is what built and sustains the movie and music industries. See: work for hire and 100+year copyright length
The tech industry: see: copyright and patent assignment from discoverer to corporation.
I know that corporations forcing me to assign patents and copyright to them was an incentive to take published works from "software practice and experience" and other technical journals, use them as the core of my work, and disclose that source to the company I worked for. Didn't stop them from applying for patents, however.
I think the discussion of copyright needs more refinement. We need to separate the discoverer's need for acknowledgment of development effort from the rent-seeking core of copyright.
You're right to call out the nasty environment surrounding intellectual property in the US and the exploitation of ideation in general, you just needed a correction on that. Someone else in this chain said virtually the same thing, which is a weird coincidence of historical ignorance. Not too weird, people tend to forget the 18th and 19th centuries happened, and much of the causally important wheels of the world are in the unsexy grease pits nobody wants to think about.
Then there is Alexander Hamilton's advocacy for importing foreign technicians that bring back IP and reproduce it here in the states. Best of all was the patent act of 1793 which like with the literature copyright ignoring, let us citizens patent inventions from the other side of the pond.
The founding fathers definitely had the right idea on IP.
The economic viability of Anthropic and OpenAI rely on their being able to charge more for model access than their R&D and inference costs. If the market price for SOTA model access drops below that level, then these businesses will have to decide whether to continue to lose money or to reduce spending on R&D.
Moonshot's papers [1] claim that their training load was primarily from synthetic data and model self-teaching rather than RLHF and therefore keep their costs low. If Moonshot genuinely does not rely on human-led training, they will surpass US closed-source model providers. The United States government considers US supremacy in "AI" as a national security consideration.
This announcement is noteworthy because it implies that Moonshot's success is in fact due to distillation. It's in the interest of US frontier labs to place barriers to this if they find themselves in the position of subsidizing rival labs' research.
1. Kimi K2, https://arxiv.org/html/2507.20534v1
Lots of things are impossible or very difficult to stop completely but measures can be taken to reduce their prevalence.
What hurts other people too?
You don't get to call Moonshot's a "claim" and this political hack's an "announcement." They're the same thing. Treat them the same. Diction designed to favor one of two equal positions is some weak sauce.
> "Well, Steve [Jobs]… I think it’s more like we both had this rich neighbour named Xerox and I broke into his house to steal the TV set and found out that you had already stolen it."
Source: https://www.goodreads.com/quotes/824084-well-steve-jobs-i-th...
Wow.
> Aren't consumers benefiting from this practice by getting better cheaper models as a result?
Aren't consumers benefiting from cheap chinese batteries, EVs, and drones?
Routers have now gotten the same treatment. So yes, consumers have been historicaly benefiting from all these things, and those benefits are about to evaporate as we lose access to cheap and high quality Chinese products before any domestic equivalents exist. And IIUC banning the use of Chinese LLMs for consumers and/or businesses in the US is now being discussed at the highest levels of government, with the "encouragement" of US AI firms.
I don't think any of these people care that America consumers are increasingly going to feel like they're living in a sanctioned country. It's all about the defense and b2b segments.
I mean not really. A quadrocopter is a remarkably simple thing made out of extremely generic parts: 4 DC motors (and ESCs), a radio, a computer, a battery and an inertial measurement unit. Anybody with a rudimentary amount of electronics knowledge can build one, the components are extremely widely used. The most unique parts about them are the frame and propellers, which are pretty easy to fabricate.
Source: I've been flying R/C aircraft of various types for over 30 years. Last year I built, I think, at least 7 FPV drones (a mix of fixed wings and quads).
Yes?
Not remembering my economic theory here, but it is likely more efficient/expensive to simply have the federal government cut checks to our moribund industrial sector companies and let consumers benefit from modern technology.
Cut GM/Ford/Stellantis a $20B check each, let consumers save (conservatively) $200B annually on new car purchases + downstream benefits. Huge win for consumers & taxpayers.
If it's not worth subsidizing explicitly like this, then we also should not subsidize by banning Chinese imports, which also ensures US drivers have less access to modern vehicles. (And downstream ensures US auto designers are less likely to have had contact with modern vehicles, making it less likely that they will be able to design future generations well.)
The banning or effective banning through tariffs of products like EVs is a pretty dumb economic strategy that rarely works out in the long-run.
Depends on what country you live in I suppose, but likely a spectrum of yes than any outright no. For example, Chinese EVs are using a different battery chemistry and not putting demand pressure on the more expensive chemistry western manufacturers use
They are a petroleum products importer, so it’s a big win for them.
cf https://www.reddit.com/r/codex/comments/1uyj6pq/kimi_k3_is_1...
"What was I supposed to do? Call him for cheating better than me in front of the others?!"
Said in response to being out-cheated at a high-stakes poker game.
Except in this case, it sounds like that's exactly the path they have chosen.
https://getyarn.io/yarn-clip/7612c4ce-1077-479f-a7bf-617dbc6...
none of the frontier labs provide probability distributions over the tokens which is the actual method of distillation you use to train a smaller model based on a larger one. they don't even provide all the tokens.
therefore this so-called distillation the frontier labs whine about is just a set of clever methods to work the existing LLM into the training process for a new model. methods like having the existing model grade the output of the new model and work those grades into the RL method. give the new models structured tasks and use the existing model as a source of truth for those tasks and a myriad of other hacks.
efficiency scales with the gap between the models and generally allows an efficient bootstrap process. the implication that distillation wouldn't allow further advancement is false however, you can then start doing the same thing the frontier labs have been doing: dumping cash on humans to provide the signals or burning tokens on exploratory paths and grading the results.
what openai and anthropic don't like is that fact that all the cash they burned can be used to benefit everyone and not just them. and that no matter how much more cash they burn to build up the gap it will closed at a small fraction of the price.