Posted by martinald 1 day ago
I don't know about that but based on my own experience with Deepseek v4 Lite alone (with high effort) I have no doubt in my mind that anyone claiming such great things about GLM 5.2 must be true because Deepseek v4 already is really awesome.
Because accelerators like H200, B300 etc. are highly parallel and designed to run like 200 or maybe 300 sequences at once (depends on the model, just guessing). I assume they finance the hardware and that cost per device or rack is the same whether each unit is handling 10 requests or 150 requests (aside from electricity).
And probably international customers factor into it to get good utilization over more of the night time. And it likely is something that they look at quarterly more seriously than monthly. The biggest risk to profits might be a downturn in business that causes some portion of the financed AI accelerators to go idle or get low utilization for some weeks (that they can't sublease).
But let's say they could someday scale that up to a much larger model, 72 large chips per wafer and each chip can do 1000 LLM requests at once (Vera Rubin?). So it's roughly the equivalent of an NVL72 rack.
You might be able to serve something like 50000-60000 requests at once. So I think it's more like handling a small city's worth of customers per wafer than the world if you had that.
I believe in less than 5 years we will get to that, but the model size and/or number of agents is going to keep going up also.
What I actually want is an FPGA board with a very large number of DDR3/DDR4 RAM slots arranged in banks (2, 4, 8 or even more banks). I want an FPGA board that can hold 1TB of DDR3/DDR4 RAM.
The throttling point right now is not RAM, it's bus speed. Having different busses for banks of RAM alleviates that.
LLMs need retraining to incorporate new knowledge.
Baking them into wafers means they will be out of date by the time they finish the first wafers.
I don't see the C++ compiler standards or Newton's laws changing every day.
That’s like a gas station saying they have 90% margin over pumps but still losing money.
Deepseek's 0.86 or whatever is likely subsidized but alternate providers offer it for a price comparable to glm-5.2.
They have published tons of articles dedicated to performance and efficiency engineering. Feel free to have a look...
Does ananyone outside deepseek have a working code for the v4 compressed attention mechanism?
Has any other provider managed to bypass CUDA and program the compute engines in their native assembly language to get 10% more performance out of them?
There is your answer.
It doesn't need to pass whole conversation history as context (unlike /model), you can ask follow up to that forked model (which sub agents in claude doesn't support AFAIK), and you can ask models from opencode while using claude.