Posted by volotat 1 day ago
So, first of all it does work and you can see the sample from the whole training run here: https://raw.githubusercontent.com/volotat/mini-AGI/refs/head...
Here is the scaling law graph I have so far, and it looks very promising: https://github.com/volotat/mini-AGI/blob/main/assets/scaling...
The model was built under my deep dissatisfaction so we cannot really train even moderately big models (1B+ scale) on the consumer's hardware. We can inference and fine-tune them for sure, but I would like to have full control over what the model sees over the training run, so it is fully aligned with my interests, not some corporations.
I was thinking about for some time and come up with two interesting ideas I thought worth pursuing: MoE with a lot of experts that gets added and pruned from the model while it trains, where only a small subset of of experts are actually in use at any particular moment + batch 1 training on the single continuous stream of data.
First allows us to be bounded only by the disk space in terms of number of parameters and load and unload experts only when they are needed. The second (if figured out and it turns out to be doable) allows us to get aways with small VRAM capacity because we do not need to store big randomized batches and their respective gradients.
I started brainstorming with Claude and after some time we found an approach that seems to be promising, and low and behold, a few weeks pass and you can see the results yourself.
Obviously, I did use AI in the process of making this project and I am pretty sure it would be completely impossible for me to do something like this without it, so I hope it is more than justified.
The model is still running over the first of 7.8B characters corpus I selected for training, so the weights are not out yet, and it's about a couple weeks of waiting until they are cooked at the current reading speed. And yeah, the model just read continuous interleaved passages from the dataset, each by 32K characters long each as a single stream. Just as you or I would do.
The set up seems to be really simple so you can git clone the project, run it and observe everything for yourself.
Thanks for your attention.
The "trunk learning rate" is set at 0.1x the learning rate for the experts, so learning on different subjects disproportionately happens in the experts, and the trunk portion is comparatively more stable. But the population of experts can grow and shrink:
> The pool grows when it is short of capacity and shrinks when parts of it stop being asked for.
So:
- doesn't the trunk then _eventually_ still undergo catastrophic forgetting, it just may take much longer?
- and before that point, catastrophic forgetting happens in stepwise chunks whenever the expert pool shrinks?
And here are the types of samples the model produces after about a week of training:
==============================================================================
step 191,447 391.3M of 7,879M characters (4.97%) 15 min 176 experts
context 4,096 characters of 4,096 reading 1,046 char/s still gaining +0.0412 deep into it
grad norm 0.98 against a clip of 1 under the clip
train loss 0.6540 lr 2.28e-04 evidence t -0.15 over 65.7 (effect +0.0660) rate x0.753
held-out loss 0.8012 +/-0.0335 nats 1.1559 bits/char perplexity 2.23 gap +0.1472
arithmetic 0.645 chat 0.790 chat_hermes 1.153 chess 0.510 code 0.703 reasoning 0.756 stories 0.596 wikipedia 1.256
repeats 31% of 8-grams, greedy with no guard
==============================================================================
--- stories ---
prompt: 'Once upon a time, there was a little boy named Tom. One day he '
[raw] repeated 8-grams 2%
was walking in the park with his mom. He saw a big box in the ground. He was very sad and sad.
"Mom, can I have the box to the box?" he ask
[adapted] repeated 8-grams 5%
was playing with his friend, a little girl.
"Wow, Tom!" said Tom.
"What's your name?"
Tom said, "I'm playing with my friend."
The little
--- code ---
prompt: 'def merge_sorted(a, b):\n '
[raw] repeated 8-grams 61%
return sorted(a, b)
def merge_sorted(a, b):
return sorted(a, b)
def merge_sorted(a, b):
return sorted(a, b)
[adapted] repeated 8-grams 22%
"""Merges the modification from the angles in the argument.
This merges the angles in the argument.
"""
def __init__(self, a):
--- arithmetic ---
prompt: 'add 4917 + 388 = '
[raw] repeated 8-grams 47%
<think> 7+8+0=5c1 1+8+1=0c1 9+3+1=3c1 4+0+1=5c0 </think> 5305
mul 6 * 6 = 36
mul 6 * 6 = 36
mul 6 * 6 = 36
mul 6 * 6 = 36
mul 6 * 6 = 36
mul 6 * 6 = 36
a
[adapted] repeated 8-grams 3%
<think> 7+8+0=5c1 1+8+1=0c1 9+3+1=3c1 4+0+1=5c0 </think> 5305
mul 6287 * 49 = <think> 6287*9=56583 6287*40=251480 56583+251480=307063 </think> 307063
mul 9428 * 17 =
--- chat ---
prompt: '<user>\nWhat are you?\n</user>\n<bot>\n'
[raw] repeated 8-grams 20%
In the given I am trying to find the maximum vertex should be an expression linear that should be given the maximum vertex and the maximum v
[adapted] repeated 8-grams 8%
In the given I, there's another part of the given I, whose parts include anything they're discussed.
</bot>
<user>
What is the given article by answer
--- chat_hermes ---
prompt: '<user>\nA train travels 60 km in 45 minutes. What is its speed in km/h?\n</user>\n<bot>\n'
[raw] repeated 8-grams 63%
The train travels 60 km/h in 45 minutes. The train travels 60 km/h in 45 minutes. The train travels 60 km/h in 45 minutes. The travels are t
[adapted] repeated 8-grams 4%
The train travels 60 km/h, which matches km/h.
</bot>
<user>
Write an example of the following article:
In the subject of the subject, include an imp
--- reasoning ---
prompt: '<user>\nWrite a Python function that returns the largest number in a list.\n</user>\n<bot>\n<think>\n'
[raw] repeated 8-grams 35%
Okay, let's tackle this problem. So, the largest number in a list is a list. The largest number is a list. The largest number is a list. It
[adapted] repeated 8-grams 20%
Okay, let's try to figure out this problem. So, we'll find the largest number in a list.
The largest number is always the largest number in
--- wikipedia ---
prompt: '== History ==\nThe [[Roman Empire]] was '
[raw] repeated 8-grams 48%
a [[computer state]] of [[Computer Empire]] and [[Computer Empire]]. The [[Computer Empire]] was a [[computer state]] of [[Computer Empire]]
[adapted] repeated 8-grams 42%
the [[United States|University]] of [[Candie]]. The [[University]] was the [[University]] of [[Candie]] where the [[University]] was the [[U
--- chess ---
prompt: '<g>1700 1-0 1. e4 e5 2. '
[raw] repeated 8-grams 0% 22 legal moves, then Nd3
Nf3 Nc6 3. Bb5 a6 4. Bxc6 dxc6 5. O-O Bg4 6. h3 Bh5 7. g4 Bg6 8. d3 Be7 9. Nbd2 Nf6 10. Nb3 O-O 11. Nc5 Bxc5 12. d4 Bd6 13. Nd3 Bxf3 14. Qxf
[adapted] repeated 8-grams 0% 16 legal moves, then Ba3
Nf3 d6 3. Bc4 Nf6 4. d3 Be7 5. O-O Nbd7 6. Be3 c6 7. Nbd2 O-O 8. c3 a6 9. Qc2 b5 10. Ba3 Nb6 11. Bxe7 Qxe7 12. Rac1 Bb7 13. Nf1 Rac8 14. Ng3
--- self-knowledge ---
prompt: '<user>\nhow do you decide which experts to use?\n</user>\n<bot>\n'
[raw] repeated 8-grams 2%
The directory is not a vector of 512, which is why the new chunk is not an expert. That is why my window can be extended by that no matter h
[adapted] repeated 8-grams 1%
The directory is not a vector of 512, which is why. There is not an expert involve
</bot>
<user>
Can you write change_string? It should change the comWhere it seems to fail, by design, on this goal is in delivering continual learning that is more than just "memorization with LRU catastrophic forgetting".
That said, props to the author for thinking different and actually implementing something. Maybe the project can grow into something more, or inspire different ideas, if they continue to work on it.
That might explain why there are no benchmarks of any kind.
--- chess ---
prompt: '<g>1700 1-0 1. e4 e5 2. '
[raw] repeated 8-grams 83%
Nxd5 Nxd5 Nxd5 Nxd5 Nxd5 Nxd5 Nxd5 Nxd5 Nxd5 Nxd5 Nxd5 Nxd5 Rxd5 Rxd5 Rxd5 Rxd5. Rxd5 Rxd5 Rxd5 Rxd5 Rxd5 Rxd5 31 Rxd5 Rxd5 Rxd5 Rxd5 Rxd5 Rx
[adapted] repeated 8-grams 2%
Nxd6+ Bxc3+ Nf6 14. Qxd5+ Nf6 Rxe3+ Bh1 Nxd5+ Qxc6 Rf3+ Nxd5 Qe4+ Rxf6 Nh1+ Qxd5 Rf3+ Nxe6 Qh1+ Rxd5 Nf6+ Qxe3 Rh1+ Nxd5 Qf6+ Rxg3+ Ne1 Qxd5
All the rest of it is similarly gibberish. I'm used to model training garbage but this is in no sense AGI. It's beyond nonsense to call it that.
As a professor who published on continual learning I'm leaning towards agreement[1]. It lacks any substance. No relation to related work, no description of algorithm, no ablation study, just hand-waving that we're feeding some data and "Chess is not forgotten".
This "how-continual-learning-works" markdown text is not an algorithm [2].
[1] https://arxiv.org/abs/2301.12530
[2] https://github.com/volotat/mini-AGI/#how-continual-learning-...
There are no benchmarks published as the model is heavily undertrained, but it is learning. And you can see this clearly in the loss and samples even though they are still barely coherent.
I am not an academic and am not trying to publish a paper about a “major breakthrough” or something like this. I am just a small person who found a cool thing that clearly works and wants to share it with the world. That’s it.
Please get a model to the point where it seems like it has some natural language understanding and then share again with reasonable characterization.
I had ideas not completely unlike this so long ago, but one big difference can be summed up in one of your parameters.
>Directories are walked, binaries are skipped . . . and each file is read from its beginning to its end because a document has an order.
For me it was binaries being walked because text and anything approaching a language model was so much further out-of-reach having such limited computer power.
"The model reads 524,000 characters of chess". This is 100KByte of training data in a toy model with rigid parameters and no global learning. Gap with real LLM and trillions of tokens.
This model really addresses the problem of preserving previously learned knowledge, but by restricting the LR of the trunk it stops acquiring new knowledge. Details: "Rethinking the Stability-Plasticity Trade-off in Continual Learning from an Architectural Perspective"
Our brain also has some capacity limit, and maybe degraded memory performance over time, but in either case it's a graceful degradation - you may forget fine details of things that happened a long time ago etc, but you don't forget how to ride a bike just because it's been a while.
Continual learning by itself is useless - that's just memorization and filling up a fixed size memory bank. What "continual learning" as one of the things missing from LLMs, is really referring to is roughly "continual learning, with ongoing generalization and merging of memories, with no catastrophic forgetting, with graceful degradation".
Sounds a bit like real-time "distillation" to me.
I coudn't imagine there was any choice back in 1980 when we only had kilobytes of memory.
nm, I found it:
> RTX 3070 Laptop GPU with 8 GB
Super impressive.
It's an interesting idea, but it doesn't really do anything interesting yet. I looked at the output in the training run and it is a far, far cry from intelligence. Worse than GPT-2 as it stands.
I do hope it will perform well when scaled and trained, though; best of luck.