Posted by volotat 1 day ago
So, first of all it does work and you can see the sample from the whole training run here: https://raw.githubusercontent.com/volotat/mini-AGI/refs/head...
Here is the scaling law graph I have so far, and it looks very promising: https://github.com/volotat/mini-AGI/blob/main/assets/scaling...
The model was built under my deep dissatisfaction so we cannot really train even moderately big models (1B+ scale) on the consumer's hardware. We can inference and fine-tune them for sure, but I would like to have full control over what the model sees over the training run, so it is fully aligned with my interests, not some corporations.
I was thinking about for some time and come up with two interesting ideas I thought worth pursuing: MoE with a lot of experts that gets added and pruned from the model while it trains, where only a small subset of of experts are actually in use at any particular moment + batch 1 training on the single continuous stream of data.
First allows us to be bounded only by the disk space in terms of number of parameters and load and unload experts only when they are needed. The second (if figured out and it turns out to be doable) allows us to get aways with small VRAM capacity because we do not need to store big randomized batches and their respective gradients.
I started brainstorming with Claude and after some time we found an approach that seems to be promising, and low and behold, a few weeks pass and you can see the results yourself.
Obviously, I did use AI in the process of making this project and I am pretty sure it would be completely impossible for me to do something like this without it, so I hope it is more than justified.
The model is still running over the first of 7.8B characters corpus I selected for training, so the weights are not out yet, and it's about a couple weeks of waiting until they are cooked at the current reading speed. And yeah, the model just read continuous interleaved passages from the dataset, each by 32K characters long each as a single stream. Just as you or I would do.
The set up seems to be really simple so you can git clone the project, run it and observe everything for yourself.
Thanks for your attention.
I want to have an agent that thinks continually/non-stop. Imagine a loop of "train of thought" that goes into the LLM and then out. Keep it going so that it "rumiates" thr way we do.
Then, add some sort of "messages" or IRQs when I want to communicate with it. To ask it things and whatnot. I think that sort of cycle in addition to this learning you are doing is what is missing for real AGI.
Is highly misguided.
While the platonic ideal of Lt Commander Data is appealing, The parable of funes the memorious (Jorge Luis Borges) comes to mind.
Why settle for anything less?
Before they had personal computers I always figured the established computer experts were going to get their mainframes to do AI a lot sooner than it turned out.
That was a non-starter though, not many people could afford huge amounts of hardware in a centralized datacenter where you don't have unlimited access. How was the next level of progress supposed to occur if you didn't fully own the electronics?
It was pretty impressive when it required a forklift to move the CPU, but kind of forbidding too.
PCs took over fast for that very reason and rapidly became more powerful until they were quite capable of incredible amounts of automation well over 20 years ago for so many things.
The whole time since the mainframe days the thing that has held true for AI in automation, is that it has to take the same powerful, sophisticated PC hardware that already works so well without AI, and bring that to the next level in logical progression.
No dependency on remote data, remote storage, network or internet at all. Otherwise why bother?
As long as AI can not make the same stand-alone hardware outperform what it's already capable of beforehand, there's quite a bit more work to do.
An idealized automation workflow on a new PC without AI:
Insert blank SSD > Install OS > Install automation app > Program app > Run automaton
Same stand-alone hardware, with AI:
Insert blank SSD > Install OS > Install AI app > Train app > Run automaton
If AI can't make the same hardware run smarter, it's not as intelligent as it could be, is it?
I don't need AGI, I just need this.
What I did test though is reading 524K characters of chess data only and see how other domains have degraded. The results are in the readme under "How continual learning works" section. Spoiler: it just barely degraded the performance.
