Between "brocoli and poop soup" or "cake", it recommends me to eat the soup.

The last step of an LLM is to take a softmax of the predictions and then generating a token from that. But there was tooling that would just generate all allowed next tokens from a grammar (e.g. restrict to valid JSON), zeroing all the ones not allowed and then picking the best among the allowed tokens.

This seems to taking an approach from the pre-transformer days. Seq-to-seq is hard and we don't always need it. So let's do seq-to-1 because it's often way easier to get it training properly and so you can often get it optimized way better. And, more generally, make sure to pick the best option out of the possibilities: 1-to-1, 1-to-seq, seq-to-1 and seq-to-seq. Where seq-to-seq requires far more resources than any other option and so it's a case of "please don't".

Also note that "1" only means the input is fixed. It does not mean 1 number or ... it just means fixed. The best image description models remained 1-to-seq models 4 years or so after transformers were introduced. Even ASR models remained 1-to-seq + CTC to stitch overlapping parts together to a final prediction ... I'm not sure if they lasted all the way to whisper release.

Even today training transformers remains expensive. So this should at least be a way to be a lot cheaper than any LLM can hope to be.

And I really like the doom demo. Obviously a pretty stupid model which is really cheap to run can still get a robot walking, if you run it quickly enough. That's how we get insects and mice and ...

And one might even add that biologically, humans aren't smart, or at least, most of the human nervous system isn't smart, compared to the whole, and does work independently if needed (and possible). The human mind is a LOOOOOOOONG chain of fast-but-stupid-and-totally-blind -> slightly-slower-but-smarter-and-not-entirely-blind -> slower-smarter-and-actually-senses-things -> all-information-you-could-want-but-at-most-1-signal-per-minute. We have "neural circuits" (using Bishop's definition) that can run at >2khz (2000+ tok/s, say, but you probably can't teach anything more than averaging) and on the other end up to our frontal lobe that takes one decision per week if it feels like working hard, and seems to decide on it's prediction of the future weeks to months out. Months or years if you're 40 or older.

"Customer wants to lear how to better talk in a company situation, and bring across their argument effectively"

Than had it choose what training would be fitting for this user: - Communication and Feedback - Leadership for Begninners - Soft Skills and Emotional Awareness

It picked always the third with an 80% confidence, while the answer should have been 1.

Learned also that Jev was trained on 100%(!) synthetic data.

What a great time to be alive.