Posted by pella 13 hours ago
Otherwise we have a world of attackers using open and closed source models against a much smaller group of maintainers that are likely heavily dependent on Anthropic and OpenAI and for whom it may not be a simple matter to just get approval to start using the open model flavor of the month.
The maintainers don't need approval to use GLM.
There will always be a market for frontier labs in the cloud based models - these models will always be able to be bigger, and that will likely translate to doing things local models can’t.
Logically also we’ll likely get to a point where RAM drops in price as production ramps up, and local LLM is both capable and cost effective. This feels like it is coming for Siri / Gemini / Alexa personal assistant type use cases.
So I think the local LLM will become a thing in laptops and phones in a year or two, offering PA type use cases. Professional LLM services will likely remain at the frontier (and in the cloud) for the foreseeable.
There are workplaces that refuse to use LLMs because they fear the devs will expose sensitive data without care.
We will get to a point where prosumer laptops that etch SoTA LLMs in removable silicon will be as expensive as cars.
But there's a significant difference in hardware required between running a 3B parameter model and a 700B-1T+ parameter model.
This reads like a hallucination. What does native speed even mean?
(fable/sol are ~60 t/s, and OpenAI just announced their Cerebras partnership(?) for "ultrafast" mode of 750 t/s)
models aren't able to run that fast right now on our consumer/prosumer hardware. M5 Max for example has a memory bandwidth of 600 GB/s. a 5090 has 3x that, so running the same model on a 5090 is that much faster (provided the model is within 30GB).
running a bigger model on an M5 Ultra is still much slower than running it on a Blackwell chip with sufficient vram, CUDA being a major difference. if apple can bridge this gap, interesting things will happen... and just imagine if M7 Ultra has comparable speeds to Blackwell (or even Rubin)!
GPU Memory Bandwidth: 7.1 TB/s Interconnect Bandwidth: 900 GB/s bidirectional
https://pi3g.com/nvidia-gb300-specifications-including-memor...
If you think M7 will hit even 15% of these speeds you're very optimistic.
He did answer you though. Native is x10 the non-native speed. 50/50 that's not a bot; though it could be a meat-proxy
OC was registered in 2016 though? What do new accounts have to do with this?
The latest on Apple is TSMC is stuck on the next iPhone due to lack of RAM. Good luck getting any Macs. Memory shortage is getting worse.
It is so valuable in practise to be able to have the models see screenshots - I guess if they aren't in the benchmarks then nobody will focus on it. But it completely nixes these for some of my main use cases.
I break out Claude when I hit some serious roadblocks, but that doesn’t seem to be happening much after the last deepseek flash release.
Deepseek prices just went up, but are still low.
I will def try GLM on my next project
I remember trying their Coding Plan out before the change and the 5 hour limits felt too restrictive then even for light/medium work, especially cause of the whole peak and off-peak thing: https://blog.kronis.dev/blog/z-ai-s-glm-5-2-is-a-great-model...
Nowadays, I’d probably go with their Max plan if the rate limits are okay? Anyone using them now?
Oh also unrelated but ZCode was surprisingly good, which is surprising for a tool that came out of nowhere - even some of the critiques in my blog post have been patched out. Sadly they don’t support using Claude Code as an agent so can’t use it like Paseo or Kepler or Agent Orchestrator.
You're gonna have a had time getting straight answer to that out of the internet. There are now 4 different flavours of the Max plan floating around (Legacy V1, Legacy V2, New plans, and the current credit ones). And on top of that they have peak times. So ~8 scenarios, 24 in total across all feedback for their coding plans.
So when someone tells you they're having a good time on a GLM coding plan it's damn near unusable as a datapoint unless both parties are very clear about what precisely is being discussed
[It's been good for me though...V1 Max off peak...which is basically the best of the 24]
I haven't tried it in a few months, but it went from amazing to unusable really fast.
Currently 20% of my 5 hour limit and 4% of my weekly limit.
Total: 58.46M
GLM-5.3 Cached: 56.91M
GLM-5.3 Uncached: 1.23M
GLM-5.3 Output: 315.18K
Cache hit rate: 97.9%
Full 5-hour Full weekly
Total 292.3M 1.461B
Cached 284.6M 1.423B
Uncached 6.15M 30.75M
Output 1.576M 7.88M
Their own estimates are quite different, probably due to their conservative caching estimates vs what I normally get on longer form work: https://docs.z.ai/devpack/overview#estimated-token-allowance
export ZAI_ANTHROPIC_BASE_URL="https://api.z.ai/api/anthropic"
export ZAI_ANTHROPIC_AUTH_TOKEN="$ZAI_API_KEY"
claude-zai() {
{ local -; set -x; } 2>/dev/null
ANTHROPIC_BASE_URL="$ZAI_ANTHROPIC_BASE_URL" ANTHROPIC_AUTH_TOKEN="$ZAI_ANTHROPIC_AUTH_TOKEN" claude "$@"
}
$ claude-zai
Got reminded now of glm-5v-turbo - that 'v' was for vision - will try assign it role [vision] now in omp. See what happens. :-) Often times it's handy when describing gui problems if the harness/model 'can see'.
I am talking about ZCode, the program, being unable to delegate to other harnesses, like using Claude Code (or even OpenCode) within their UI, so that an Anthropic subscription can be used, because Anthropic don't let you use 3rd party harnesses directly.
It's basically what Paseo: https://paseo.sh/ and Kepler https://www.gitkraken.com/kepler and Zed https://zed.dev/ support doing.
ZCode doesn't seem to work at that level, it instead feels comparable to OpenCode or Codex or Claude Code directly, while also being desktop oriented - you just make API calls directly within it.
It's okay if it's not a goal of theirs, it's just that their UI is really really nice and that would be a cool direction for them to also go in some day.
Paseo had a really nice UI/UX, except sometimes sub-agents within OpenCode sessions would hang. Still, quite pleasant if you want something like the Codex or Claude Code desktop apps, but across various providers.
Kepler integrates with issue trackers like GitHub, you can just create a worktree from a ticket and let it churn, seemed like the second most polished option I tried, but there are obvious gaps - like moving cards manually, some missing UI options etc., which I'd chalk up to either the software just being that new or maybe being a little bit vibe-codey. Either way, one of the more promising options if you want something like Kanban board for agents.
Zed is mostly just a (really nice) text editor with some AI integrations, though it seems like they're also building a more agentic product as well - https://delta.dev/ haven't used that one much and am not in circumstances where I'd collaborate with people that closely, but there was a pretty cool podcast episode with the creators recently and it seems like it works pretty nicely for them! As an editor though, it succeeded where Fleet failed and has mostly replaced Visual Studio Code for me. Nothing against VSC, Zed just does most of the stuff I actually need out of the box.
Some of those tools interacting with Claude Code instead of trying to replace it is more or less the way to get Anthropic's models in other tools while still on a subscription (at least for now). How it works under the hood, go figure, there's ACP https://agentcommunicationprotocol.dev/introduction/welcome but also any number of hacky approaches.
To be fair, you can use Anthropic's models in many other harnesses directly, it's just that it then counts against API billing instead of your subscription, which ends up being way more expensive for individuals, but is kinda what you're supposed to do as a company.
But Neuralwatt significantly raised their rates since then.
yet but it is still contain a lot of trash. you need better models to process those trash and create a curate dataset. this will happen again and again until there is no more juice to squeeze. and I'm sure we are still not done with it.
> post training
yeah this will be crucial. the big models are already too capable, they are just not that aligned with current agent tasks.
> parameter count doesn’t seem to be a direct correlation anymore
I don't think so, remember that chinese labs do not have as much compute power compare to US frontier labs. that's why deepseek v4 flash had that huge jump and deepseek v4 pro is kinda a disappointment, they just do not have the compute power to proper posttrain the pro model like they wanted. glm is also a relative small model so you also can see the huge jump with just post training. so it does not mean the size does not matter, it is just mean that the chinese labs currently only capable of training smaller models effectively.
LLMs are still surprisingly “easy”. You need maybe a couple dozens of right people, a lot of good quality data and a lot of GPU that you know how to operate. There’s relatively little “secret sauce” needed.
The data is mostly coming from places like Scale/Mercor/etc and net dumps with some filtering and batch prioritization, and RL on verifiable domains like code/math/games.
10 years ago it was super hard to have usable “frontier” ML. You needed very complex data warehouse, feature engineers, feature stores, multi level ranking, calibrations, tons of different model architectures, etc, etc. Each by itself was extremely hard engineering problem and really only handful of companies could deal with that complexity.
With LLMs, 95% of that is gone, infra to support them is greatly simplified. Of course, to make really reliable, performant, user friendly, etc - you still need to a lot of engineering. But it’s very different challenge.
Despite this being the topic du jour of 2025, it was never true. Most of the "we've hit a wall with data" came from communicators / media and not researchers. It got popular because negativity sells. It's a false premise for a number of reasons:
a) Data curation is as important, if not more important than bulk data. Models becoming better at classification leads to better curation leads to cleaner data. Throwing common crawl and pray is so 2023. We've known this since llama3 days, it worked then, there's no reason to think this will not continue to work as the models imrpove.
b) Models are today good enough that you can augment / multiply your data easily with enough compute. You can now have a model take "authoritative content" and create more data from that + scenarios. Say you take a book on computer architecture. You ask models to break it down. Then you ask models to find examples for each topic. Then you ask models to ask questions and offer answers from several viewpoints. Then you take each of those and ask other models to flag inconsistencies. And so on. But you can whateverX your data from one authoritative source + bulk data into 5x - 10x "scenarios".
c) RL is really really really powerful. It's hard to do right (reward hacking, instabilities, etc) but once it works it "keeps" on working. Again, we knew this to be true a few years ago, ever since models really started to do well on math (highly verifiable). It only follows they're getting better on cybersec and other verifiable tasks. But now, with models improving, you get the same data augmentation pipelines as above, just better because they're also verifiable. For example, the way cursor augments their data: take a repo, ask an agent to identify a feature (it can be a large multi-file feature). Remove all code relating to that feature, but keep the original tests in the repo. While training, that becomes a RL scenario: implement this feature in this repo. Verify it with the original (hidden for training) tests. Reward appropriately. Now you can get 1 repo -> 20-50-100 scenarios. Instead of "feed everything into the pretraining", you're now creating scenarios, verify them w/ existing tools, and get your scoring function for the rewards. And, importantly, as the models become better in general, they also become better at this pipeline building exercise. So the next iteration gets trained on more scenarios, better scenarios, and so on.
> how will models continue to get better?
Probably the same. No one can know for sure, but at the moment, despite all the "walls this, slowdown that, plateauing" and so on, there are no signs of slowing down. And, as you noted, this works across the field of model sizes. There are, of course, theoretical information-based limits on size, but smaller models also improve, once "bigger" models can be used as training data generators, oracles for verification, rubric verifiers for open ended questions, and so on.
And smaller models (i.e. cheaper to serve) get to generate more traces during RL, and more rollouts give you better training, and so on. Next up - hardware optimised inferencing (ASICs basically). Once you have that, we can expect another wave of improvements. And so on.
Big models are going to "tap out" on non verifiable fields within ~2 years, just because the pool of experts able to reinforce the models is going to get very small, and as the nuances get finer, the signal from reinforcement is going to get progressively less aligned with the intent. Math and code will be mostly tapped out in that time frame as well, even though we can technically scale them "infinitely," just because the cost benefit won't line up. At that point, most RL will be "gyms" with games that are designed to model designated valuable economic activity.
In the next few years, we'll get small domain specific distillates that are ridiculously smart in their domain (imagine if Qwen 3.X 27B went super saiyan), and even frontier labs will be routing to experts/orchestrating because the cost to serve/TPS difference is huge. They'll still train the god models for PR/marketing, c-suite use and distillation, but using them for day to day work would be like making houseware out of solid gold.
A positive feedback loop then. RL->better model->better RL pipeline -> better model…
And we’ve only recently started getting into the much better RL pipelines
Not really, but a lot of what isn't used isn't very good.
More important is synthetic data. Use a teacher model with RAG with a huge reference library to write synthetic transcripts of idealized behavior for the model. Use models to judge and correct these transcripts. Train on the good ones. Use bad traces to train the model to correct its own errors (e.g. don't train it to produce a bad transcript but if it finds itself in the middle of one train it to self correct).
Similarly, for tasks that can be closed loop evaluated -- e.g. running computer software and programming, unlimited amounts of novel training data can be generated... including for highly original tasks: e.g. run publications in any domain through a model prompted to look for programming problems suggested by the material. Then write/judge/improve transcripts of solving those novel problems.
I expect in the future smaller models won't be directly trained on any internet data at all-- but entirely on simulations of idealized expected behavior from the model under construction. Raw internet data in that case would show up in prompts, but never in the target output (except of course for prompts that are asking it to copy the input).
Hopping for an AgentWorld variant from Qwen but I guess, I have too high expectations.
We just had amazing releases this past two months
kimi k3, glm5.3 qwen3.8 and now glm5.3
These open models are getting really good
I know OpenCode Go has both, but their limits seem kinda low, so I'm not sure how feasible it is to run such a task with them.