At 20 kW average consumption per WSE-3T wafer this gives roughly 1.18 MW for the whole system. Assuming 1500 tokens per second and 190 concurrent requests this gives 4.1 J/token. Which is good I suppose for the size of the model?
drum55 5 hours ago||
For a fairly weak comparison, modern Bitcoin mining chips are around 13J/TH, so around 0.32TH per token, or 320 billion double-sha256 mining operations per token served by this system. Peanuts are also about 4200 Joules, or almost exactly 1000 tokens per nut assuming you could efficiently turn it into electricity to run this chip. Just as a frame of reference, I don't think this actually serves to make any point.
shrubble 3 hours ago||
Each Nvidia rack is 120 kilowatts, so the power consumption is about the same as 10 racks of GPUs, FWIW.
xnx 4 hours ago||
Cerebras is the Transmeta of the AI wave.
lifeisloving 1 hour ago|
Not sure why people would downvote this. Your not allowed to criticize any AI company on this site it seems.
russli1993 3 hours ago||
could push kv-cache to off chip cheaper storage like dram. Inference engine manages decode for a single user to run to completion before decode a different users so kv-cache fetch from dram to sram is limited.
R1ck404 1 hour ago||
[flagged]
zhyblife99 7 hours ago||
I'd want to know is what a 59-wafer pipeline costs in failure domain. The GPU deployment can lose a node and degrade; Because, here the model is the rack, so one bad wafer takes the whole thing down, and a pipeline that has to be live end to end doesn't get a rolling upgrade. How do you drain one?