Top
Best
New

Posted by b44 4 hours ago

Show HN: Microgpt is a GPT you can visualize in the browser(microgpt.boratto.ca)
very much inspired by karpathy's microgpt of the same name. it's (by default) a 4000 param GPT/LLM/NN that learns to generate names. this is sorta an educational tool in that you can visualize the activations as they pass through the network, and click on things to get an explanation of them.
62 points | 5 comments
kfsone 1 hour ago|
Minor nit: In familiarity, you gloss over the fact that it's character rather than token based which might be worth a shout out:

"Microgpt's larger cousins using building blocks called tokens representing one or more letters. That's hard to reason about, but essential for building sentences and conversations.

"So we'll just deal with spelling names using the English alphabet. That gives us 26 tokens, one for each letter."

mips_avatar 21 minutes ago||
Using ascii characters is a simple form of tokenization with less compression
b44 1 hour ago||
hm. the way i see things, characters are the natural/obvious building blocks and tokenization is just an improvement on that. i do mention chatgpt et al. use tokens in the last q&a dropdown, though
msla 1 hour ago|
About how many training steps are required to get good output?
b44 1 hour ago|
not many. diminishing returns start before 1000 and past that you should just add a second/third layer