Posted by mrashad74 8 hours ago
So to get any model working on $OLD_HARDWARE, the hardest part of the task is now sourcing the hardware itself and giving the agent ssh access -- a modern agent will be able to source a compiler, write the implementation, optimize it, test in qemu (or any other simulator) and push it onto the device.
Example prompt: "write an inference engine for qwen3 0.6b in pure assembly that runs on this device. Write a script to attach the weights to the resulting binary. Prefix is supplied as a stream of u32 token IDs on stdin. The engine streams u32 token IDs on stdout."
I'm treating it as a search algorithm project, where I develop the methdology and search for pushing the limit of devices and searching in a big space of possible kernels and so for what works for a specific hardware.
so I'm looking at it from a different prespective, it takes a lot of brute forcing, and claude can do this well, the whole game (to me) is which directions you push the model into to reduce the search space. and how that learning from very small and tight devices you can scale these techniques.
But in essence, I don't disagree with your sentiment - the goal is not 0.6B models, these are extremely trivial as you mentioned and as I showed in the experiments, the goal is 16B+ models, and this as far as I've seen, requires smart methods to search for optimizations and customizations both to the model and inference engines.
"claudia slop me up some 600m, commit, push, make 500 pages of slopped README.md, github stars+++"
great, you should definitely do this, there will be absolutely no problems whatsoever and it'll also run at peak speeds while of course being completely unusable because who wants to run a slopped 600m
t. at 1200+ commits on my "oneshot"
It's fun times actually. And I think such experiments and others posted are both showcasing the limits people are pushing.
Now Reached 16B models running on iPad Air 1st gen and xiaomi tv box, both have less than 1GB of usable memory, and reached some decent speeds given the hardware.
I'm open sourcing the effort done, kernels developed and results for everyone to re-produce and push more, also the modified models are on hugging face : https://huggingface.co/hyperionaidev
I want a more challenging device, what do you think?
A Playdate
I hold A to speak. B to cancel. Turn the crank really fast and really slow → send the prompt!
I still do use it sometimes.
The app talks via serial and WiFi.
Perhaps the real value with AI is going to come from...surprise here...a whole bunch of different low resource AIs networked together living and working and yes playing thru a networked "society"
Sounds kinda familiar doesn't it?
I'd thought of the ESP32 devices but their RAM is too small
It also runs on MS-DOS 80386 2MB RAM, so...
And iPhone 1. And Digital UNIX.
Have you got a write up for it?
Between the VUs, the R5900's SIMD (MMI), the GS and the I/O buses (FireWire, Ethernet, IDE, Memory Cards...) to stream model weights, the smallest Bonsai models (or aggressively quantized Qwen3.5-0.8B) and some really cursed programming might just eke out something the PlayStation 2 has no business running.
One of those vapes with a screen on it.
I’ve been working on small local models and it’s amazing how fast the tech has progressed
A year ago, I was creating conversational characters by finetuning a qwen 2.5 0.5b param model, and it worked pretty well. But no reasoning or math capabilities, and still required some decent specs to run; could maybe do well enough in a beefy raspberry pi
But now, I have a locally trained 6.5M param model that does reasoning, solves multi-step math problems, does simple physics problem simulation, tool calling, RAG and conversational chat. Total size about 30MB packaged for ONNX, and runs with <10ms latency in the browser
I believe in a 6-8 months it will get even better and my TV will be happy.
I didn't try fine tuning so small models honestly but looks interesitng to try.
It's all quite clearly collusion among a few dozen, hundred. No way their education/experience or data analysis engines have not made them aware.
They’re just used to Boomer and Silent Generation politicians blithely enabling them cause line go up for them when they do!
Curious to see how well it works on Millennials and GenZ; the elders were educated before STEM education had matured in a lot of ways. Younger generations have not had debunked fantasy rammed down their throat as scientific fact; Nostradmus types of the mid-late 1900s pointing at some ignorant math assuming it's true because it's written in a scientific notation! Duh!
Science education of the past gave rise to thinkers like the Oceangate CEO. Nihilists who figure there's no such thing as forever and we all die anyway; so YOLO no matter who I take down with me.
In these settings, data is insanely messy beyond your wildest nightmares and hidden away in free-text fields that small models like this are great at cleaning up, normalizing, parsing, categorizing, etc
This alone is quite useful in small/limited devices/contexts.
Realtime log parsing and summarization is another application I've seen for little LLM's..
Basically, anywhere you'd use an LLM, but at much, much smaller scales of ontology.
On other notes : there is no free lunch, you have to sacrifice something for the other
https://github.com/slvDev/esp32-ai
It shows using Per Layer Embeddings to run models that normally take 100s of MB of ram run efficiently in just a few KB of ram and keeps most of the model on disk. It’s somewhat similar to MoE so it doesn’t stream in the whole model from flash for every token, but takes it even farther. If some of these devices have 1GB of ram and a few GB of disk there might be some interesting options here.
I’ve been investigating getting something like that running on the Wii too, though I haven’t made much progress in a while. Some basic calculations says a 1B model might be possible, at least a few hundred M
I'd be interested in seeing some older flag ship mobile devices