Posted by AnhTho_FR 6 hours ago
Is it really one big computation f(g(h(x)))?
Each token prediction is one big function call. Then you just recursively generate more tokens until run out of context or the model predicts a next token indicating end of sequence. Technically the model outputs a matrix where the last row is a probability distribution, but I’m counting sampling from it as part of the chain. Hundreds of billions of dollars has gone into just making the function fatter and gradually changing pieces here and there.
You know, I haven’t kept up with MoE and etc where there’s a bit of selection going on, so I should probably be more humble. I think new work has only added different “paths down the same hill” though (no recurrence, just select different matrices), but could be wrong there. I don’t think I’m wrong on my general intuition, just want to be epistemically honest!
Is it all one big computation? Its turtles all the way down.
Original attention involves (very crudely) an approach of scanning how every token (roughly a word) relates every other token and training a classic neural network on related tokens - to get either language translation or next word prediction (and next word prediction is what "seems intelligent" in LLMs). [1]
The problem is that since original attention is "everything to everything else" it scales quadratically (O(n^2)) with the size of the train set (or train set window) and so basically even the largest data center can use that once a truly vast training set is accumulated. Which is to say that "dirty little secret" of LLMs following the "Attention Is All You Need" paper don't actually scale. That model (in my crude, amateur understanding) is elegant for allowing every word's connection to every other word to be weighed and still brute-force for not starting with or achieving "understanding" of the words [3 give only some background but also why "full" attention is powerful].
Linear attention is a way around the quadratic quality of original attention so everyone is naturally using clever approaches to make it work. Simplifying terribly - you're trying to determine the value of word before you see in context. But my intuition is that since (Everything X Everything) is inherently a quadratic relationship, none of these can capture their expanded data set in the way original LLMs did - not they are worse but all the models seem likely to hit diminishing returns in terms of blindly capturing meaning from all-the-world's text (and data).
Background and notes: [1] https://en.wikipedia.org/wiki/Transformer_(deep_learning_arc... [2] Linear Transformers Are Secretly Fast Weight Programmers: https://proceedings.mlr.press/v139/schlag21a/schlag21a.pdf [3] Transformers are Deep Infinite-Dimensional Non-Mercer Binary Kernel Machines: https://arxiv.org/pdf/2106.01506
Yep! I know some of these words.
The rest of us reading this are like, holy batman, what the fuck was that?!
You could have your own hacker news, it's just a textbox, a bunch of tables and headings! Once you add these, it'll be abundantly clear that you also need a database. It should now be obvious that you also need a user system and it can be easily shown that needs a backend. Admin tools, tests, statistics, performance checks and so on can easily be derived from such backend.
A piece like this which uses it in a headline but in no way makes an average reader feel like they could have come up with it is just badly misjudging how good of an explanation it is.
“You Could Have Invented Parser Combinators”
“You Could Have Invented Container Runtimes”
“You Could Have Invented Git”
Given the references to “mathematicians”, I think this reaction is more about an unfamiliarity with the concept of applied mathematics, which is ironic for practitioners in a field containing so much that is (or should be) regarded that way.
Software used to be all about “discrete math”, logic, maybe a bit of queuing theory…and now suddenly linear algebra and statistics became important. Don’t panic, it’s just another textbook on the shelf.
What this particular mathematical writeup seems to address is ‘how does someone who is already up to speed on the mathematical underpinnings of attention kernel neural networks get brought up to date on what they need to understand in order to see this new algorithm as obvious’
Which is a smaller audience to be sure, and - given how fast this field has grown - somewhat disingenuous in its implications.
This kind of title is appropriate for a piece aimed at helping someone who is a little behind the curve catch up. In the field of deep learning it should be obvious even to the people inside it riding that curve that it is moving really fast and that this kind of content only works for people who are running to keep up.
[1] https://medium.com/@gtrevorjay/you-could-have-invented-conta...
I often find people get annoyed at mathy stuff because they seem to think that they should be able to read it like a (comparatively low information dense) newspaper article or something similar.
Math isn't like that, it usually has high information density and you need to parse every single symbol. And also people make this mistake where they gloss over stuff they don't get because they think they'll just understand things from context. Works great in normal literature - but math ain't like that. If you don't understand something, go back to the definitions.
Writing a detailed technical post behind the news of Kimi K3 and KDA algorithm with an audacious title like "You Could Have Invent Breakthrough It too" they are pre-filtering out the ones who couldn't comprehend with quick read (myself included) and attracting the ones who agreed with the blog post. At the end with a strong CTA to promoting their 10x cheapter open weight model AI inference and hiring too.
Good job Doubleword, I see what you are doing there.
We're just a group of guys and gals who like inference!
Nothing complex
"New algorithm, or fmadd?"
"... fmadd."
"You could have come up with Kimi Delta Attention, but you didn't, did you."
I always thought that the jump from LSTM/GRU -> Attention wasn't a particularly big one. Instead of partial unroll, do a full unroll. Why not (because it's too expensive, that's why not). Every component was known, and everybody anywhere near ML knew perfectly well why NOT to try that: because you just don't have the compute to fully unroll an LSTM. From that point attention is optimized (they key-query mechanic). The big innovation is not so much the mechanism itself but realizing the parallelize-ability of it.
It's sort of like if one would today make the "improvement" to attention to replace they key-query-value mechanic by just dropping it while making the entire context the latent space. That will outperform attention, nearly guaranteed. It'll also make even Google's cluster networks meltdown. Attention is one of those innovations that came mostly from realizing you had better hardware than everybody else and asking yourself how to use it. It's still quite the accomplishment, they had to get it working. But nobody else was really capable of making this leap.
What do you mean by this? Like concatenating all token embeddings into one large vector?