Posted by ModelForge 20 hours ago
From what I gathered, LLM inference is bottlenecked on memory, right? Which implies there's "spare" compute we haven't been using? Does reusing the weights like this allow us to utilize it? (Do more math per unit of memory?)
I don't think this is explained by the model simply being more capable and therefore achieving more per token: the usage of recurrent depth (Neuralese) is exactly predicting less CoT monitorability even at equal capability.
Better Multi hop reasoning is one of the most notable improvements of the architecture. The tricky part is figuring out a way to optimize the number of times you loop as it varies between tasks. Too few and you leave performance on the table too many and performance begins to drop.
This guess/explanation predicts that most users won't match what the initial social media hype claims, doesn't discount user experience as simple habituation nor does it assume companies are lying when they say there have been no changes to the model itself (quantization included).
I also think there's an aspect where initial testing is more forgiving because the more persnickety polish bits can be ignored and tests are likely to have similar structure to things that can be trained for. Meanwhile, actual specific work items are a broader unusual distribution with more stringent acceptance criteria.
Personally, I can detect a separation between Sol and Astra (but not as large as that between Opus and Fable). While they can solve most of the same problems, Astra takes less time, is less frustrating to talk to, is cleaner, notices more, spins wheels less and requires less corrections.
IMO this is 90% of it (as someone who has a bit of a different interaction style and runs these things less autonomously, and hasn't generally seen the claimed regressions). Day 1: throw new stuff at it that failed badly, exciting to see something make more progress! Day n: reality sets in that it still wasn't perfect the first time.
So it may be a widespread hallucination. But there's no evidence of that either.
If they have a "hey we're being benchmarked" mode, which is not hard to imagine, avoiding tripping it is going to be annoying and difficult to prove.
> "Demand for Astra is really unprecedented. We're pulling all the levers possible to sustain the demand, but I've not seen anything like it until now and we went through very steep growth before. Priority will always be to keep excellent service for existing users, but we might have to pause new Pro subscriptions for a bit if this continues."> We've made some improvements that improve usage on the long tail for power users of Astra when logged in with your ChatGPT account.
> No change in quality and a pure win that on the long tail can result in up to 3-4X less usage being drawn from the subscription.
https://x.com/thsottiaux/status/2096717905614524491 (https://xcancel.com/thsottiaux/status/2096717905614524491)
You can't fool everybody all of the time, but you can fool almost everybody most of the time.
But most of all, it's easy to fool yourself.
Point is, I really don't buy all the stories about a model suddenly being downgraded without at least a modicum of substance. People are grasping at straws in the noise.
Then after a few days you notice the prompts that it does badly on that the old ones did fine with and everyone is convinced there's a regression when it's just a different part of prompt space
It prioritises getting something working over making something good during the 1-shot phase and outputs maximum slop.
I’m working on hard things, it is very noticeable when it is hums through something and then falls over on something it should not
I can tell by analyzing my own prompts to look at when I get frustrated ;)
It can very well be Sol, no? What stops them from using cheaper model for some requests during "rush" hours or simply use cheaper model for every Nth request.
That would trigger a full prefill (context recompute) every Nth request because cached tokens aren't interchangeable between models, and that would require way more compute than just staying on Astra.
To avoid full recompute, you could prefill a cheaper model's context incrementally by always feeding it Astra's outputs in the background (and vice versa), but then that would require 1.5-2 more VRAM for each session + the complexity of keeping them in sync.
If the rumors are true that Astra is a looped transformer, a more practical approach would be to dynamically adjust the loop count during peak hours.
Alternative theory - it always seems amazing when it first comes out then the novelty wears off and we’re just meh about it. New model is a model is a model. I bought a PS5 Pro and was genuinely blown away by it at first…few weeks later I’m just like…eh it looks pretty good I guess? It’s still the same, I’m just used to it now and the wow factor along a new thing is going. Kinda like that.
Or they are just compute constrained so they have to serve a shittier version. Who knows?
I hate how opaque these companies are. It feels deceptive and evil.
It seems to overengineer really bad and it is also very slow due to it "thinking" too much I feel like.
One example is that I asked it to implement a new functionality inside an existing App of mine and if I had written it myself it would have been like a ~50 line diff. Astra took like 10 minutes to write ~400 lines, most of them useless and also in pretty bad style, barely readable code.
Maybe I am bad with prompting but I didn't have these issues before, not even with 5.6 Sol on max reasoning.
Also, Astra overlooked, in my opinion, a serious flaw in its approach for something I was working on recently, which really surprised me.
Reading between the lines, there were some breakthroughs with Astra, which I'm sure is why OpenAI released it so quickly after Sol, but probably not in the ways the traditional OpenAI customer wanted.
However for typical low to medium difficulty code, it will often either overengineer stuff, create massive functions instead of organized code, and just write very hard to read code. It literally looks like minified code. Clearly they trained it to reduce the number of output tokens and in turn the code is often atrocious. I'll keep trying Astra but I might actually go back to 5.6 sol for many tasks if I keep getting these results.
I gave Astra a pretty straightforward bug ticket yesterday. The bug involved an edge case that could sometimes result in an invalid value getting stored in a user profile field. Pretty harmless, no crash or anything, just annoying.
Based on past experience, I don't trust OpenAI, so I decided to watch Astra as it worked. About four minutes in, it convinced itself that it should also check the prod database to see "how far the corruption has spread" and attempted to SSH into the hosting provider. This resulted in my 1Password to prompt me, which I of course denied. Then I stopped Astra, closed the ChatGPT/Codex app and gave the task to Opus 5. Suffice it to say I will not be renewing my subscription, because "you have to watch it like a hawk" is the opposite of agentic engineering.
This isn't intended to stop a model like Astra hacking its way out of course, it's more like guardrails on a staircase.
My personal container manager tool has an intercepting SSL proxy and small Javascripts on the host can rewrite or block HTTP requests. The agent gets its own isolated home directory and can't tamper with mine. Local caches like Maven are mapped read/only with a write layer on top.
Negative feedback filed and ChatGPT uninstalled.
If enraged_camel had been doing something else involving the production database at the wrong time, they might have accepted the 1Password prompt.
The way I use it now is I'll ask a chat 6 Pro session to make a plan and then have Sol implement it, then 6 Pro reviews it. This seems fine and it doesn't use my Codex minutes, so I'll use Astra. But on the metered tasks I don't see the utility.
This is a problem for OpenAI because if Sol is good enough, and they don't have a moat, then it's only a matter of time before Sol-level models are open sourced and running locally. I know I'll be doing that as soon as I can.
Much the same as you're saying, I never got around to verifying how much of that was because of Astra being better vs just being a different model sent specifically to those tasks because the token usage didn't make sense to spend unless it was something not working in Sol. So even if it was all due to Astra being fantastic I'd still not like to use the model for the cost being even more fantastic.
previously, conversation might have 50k tokens spent on reasoning. the next turn takes all the previous tokens as well (if you wanna preserve prompt caching) which is not ideal. this new method skips that so you get more free context until compaction kicks in.
is this true? if so its a huge deal. why is it not spoken about? its one of the main reasons i don't use High or Max