as of now, it's just for novelty and fun, without serious fine tuning on domain data, you can't really do much with the smaller models.

well, the whole idea honestly is curiousity about limits of hardware more than actual application today plus the idea that in the future people can run more capable models (in terms of capability @ small parameters) on such devices.

It makes more practical sense today to just go into 4gb GPUs and many people are working on this already.. on my side of things, it's more on the exploration side for now, until we see what good models get released that can use this capability.

There are many people on github and x running decent models especally MoE models on older GPUs and it's doing nice, I'm personally tinkering with this at home.

Unless you have electric resistance heat, and it's cold.

no electrons were hurt in the process of making this :(

Funny, I use that exact website to find passively cooled hardware to use as a server.

I did get Qwen3.5 0.8B working at 5 tokens per second on the iGPU of a Dell Wyse 7020. I mostly use it to complain to; I have not yet found an agentic task for it. But also the CPU does plenty of small services and terraria servers and such.

https://nortyspock.github.io/blog/running-a-language-model-o...

Out of interest, what have you been doing with them?

Thanks for your kind words :) hope to do something impactful one day with it

That's programming in 2026. Right now, Claude it doing my job for me, but if it goes well enough I'm the one who's going to be promoted.

most of the problem is actually brute forcing, so I'm using a brute forcing machine and just giving it pointers to reduce the size of search space for tricks, still fun, nothing frontier I know, but it's nice to see what we can do with our little brute forcing machines.

well, yes claude is doing that, and I'm directing it and building the experiments and direction only, I didn't hide it at all :)

Everyone is using LLMs now, so it's acceptable to use it for a fun experiment too I believe.

challenge accepted, I have one lying around, will see.