This is such a smart way to demystify LLMs. I really like that GuppyLM makes the whole pipeline feel approachable..great work

I like the idea, just that the examples are reproduced from the training data set.

How does it handle unknown queries?

Could it be possible to train LLM only through the chat messages without any other data or input?

If Guppy doesn't know regular expressions yet, could I teach it to it just by conversation? It's a fish so it wouldn't probably understand much about my blabbing, but would be interesting to give it a try.

Or is there some hard architectural limit in the current LLM's, that the training needs to be done offline and with fairly large training set.

Wow that is such a cool idea! And honestly very much needed. LLMs seem to be this blackbox nobody understands. So I love every effort to make that whole thing less mysterious. I will definitely have a look at dabbling with this, may it not be a goldfish LLM :)

I am trying to find how the synthetic data was created (looking through the repo) and didn't find it. Maybe I am missing it - Would love to see the prompts and process on that aspect of the training data generation!

> you're my favorite big shape. my mouth are happy when you're here.

Laughed loudly :-D

Does this work by just training once with next token prediction? Want to understand better how it creates fluent sentences if anyone can provide insights.

Building it yourself is always the best test if you really understand how it works.

Forked. Very cool. I appreciate the simplicity and documentation.

This is so cool! I'd love to see a write-up on how made it, and what you referenced because designing neural networks always feel like a maze ;)