Posted by AnhTho_FR 4 hours ago
Many will likely disagree with me, but inconsistent notation (across papers!) is to me friction. At least in this article the author explicitly explains the notation at the very start...that is not always the case. Rarely, even.
EDIT: Didn't even notice the notation switch, much appreciated.
It's like saying "I never understood people who prefer to use functions instead of inlining everything". Adding a bunch of visual noise to a statement doesn't improve comprehension.
Trying to read any math paper is basically like trying to read CodeGolf.
Remember that the oft-used e^x is actually an infinite series, even writing it out in summation form would be quite verbose given its frequency in many equations.
Generally a numerical rather than symbolic representation: Meh https://docs.sympy.org/latest/modules/functions/elementary.h...
But, for someone just starting out (or even an expert who just wandered into a new area), this becomes a barrier to understanding. Sometimes we need the mathematical summary, what tells us the immediately answer we are interested in, perhaps even if it loses the deeper nuances that are beneficial for the experts more fluent in the language.
(And yes, all math notation is human created, so it makes my point quite a bit messier in practice.)
So you and all your peers agree to call that procedure "ẽ"
They explain one particular aspect of the notation but never define the variables used. What is k? q? S?
It's obvious if you've studied machine learning before, and for some of them you can make an educated guess, but it makes the article mostly opaque if you don't already have some domain-specific background knowledge.
Also, letters avoid having to name them, naming being a hard problem and all.
Creating or combining to have something new, that does not already exist is actually freaking hard!
The moment its presented and people go "o, that is not that difficult", "i was able to also do that", or some nonsense like that. Everything looks simply the moment somebody did the hard work.
We have all been there was developers. Thinking we invented something new, and ... then you discover somebody already made it in the 70's and its everywhere. But because it never cross your path, you never realized it existed.
(Side notes: I have physics PhD and mild dyslexia)
> The identity [...] is the whole trick. The outer product is a matrix; the inner product is a number. We no longer store every past key and value. We store their summed outer products in the fixed-size state S_t.
[1] I specifically mean those who are able to hold very big complex ideas and systems in their head, and reason about them, which seems to be an important talent for mathematicians.
One of the though experiments I ran with one of my friends during a recent conversation over drinks was this: raising a bunch of "control group" kids away from the screens and the algorithmic ocean of "normie-tier content," and in a very learner-friendly setting with hyper-strict control on the quality of media and source material they get access to, just like we've been doing it with frontier models. Think of it like a monastery but for kids, while teaching them all the latest advances in our understanding of reality through mathematics, engineering, computer science, deep learning, and whatnot.
What I'm getting at it is that we might still need super smart people to push the boundaries of knowledge while using super-advanced AI tools, and anyone who says AI will "completely replace" humans are just misguided. We will always need super smart people with largely unadulterated thinking.
> Are you thinking what I'm thinking, Pinky?
> I think so, Brain, but --
> YES! Time to take over the world [utilizing these philosopher kings]!
I would have liked some refresher on some variables though (like d_k in quadratic attention).
Now way I could have come up with Kimi Delta Attention.