Posted by neomindryan 1 day ago
FWIW I'm getting a hardware max of 20 tok/s (approx topping out the GPU's compute) on my custom local diffusiongemma port running on an M3.
For CPU inference on old hardware I don't think q4 offers any benefit over q8 since the AVX unit doesn't support such small floats. I don't even think AVX supports 4-bit int math. IIRC AVX2 does.
A 10 year old Xeon is all you need
I've got a 14 cores / 28 threads Xeon from 2015 that I use as a server at home (ZFS / VMs).
It's really a sweet machine.
For ricing I've got a semi-recent AMD 7700X / DDR5 RAM (from 2023 ?) which is my main machine but the real deal is my old and trusty 10 years old Xeon server.
DDR4 ECC is pricey too atm but a 10 years old Xeon is basically free now.
A 20 cores / 40 threads costs maybe 20 USD (for just the CPU). Slap that in a $100 old HP Z440 workstation and you're good to go for quite a few workloads.
Mine is only on when I'm at my computer: it's not turned on 24/7 but more like 8/7 so the entire "but it consumes energy" point is moot.
Whether the writer's setup affects that choice I don't know.