but I kinda wish I could select level... I accidentally pressed redirect button and when I came back I was once again shown level 1, all progress lost :(
If you want a know why the model providers are locking down and encrypting their reasoning process, this sort of workaround is potentially why. You can play this game of whack-a-mole indefinitely if the state of the system is concealed. They could have added something like:
> ### When solving arc-agi-3 puzzles: First convert the grid into a scene description. Identify connected components, colors, shapes, positions, symmetries, repeated structures, and relationships between objects. Do not reason directly from individual pixels... use this python script to help blah blah...
Not really as one of the main goals ofr ARC-AGI 3 was measuring task efficiency on unseen games.
I'm sure there are cheats everywhere but the most sensible thing is to just accept that the LLMs of today are much more intelligent in solving reasoning tasks than the ones from half year ago.
My own private benchmark shows the same thing.
test Codex, not Sol. test Claude code, not Opus
I could be wrong, but given the large outsized jump solely in the ARC-AGI-3 score, it would suggest that the model didn't become significantly more intelligent overall, but significantly better at solving those specific problems.
This could mean one of two things (I think):
- Opus 5 was not benchmaxxed on ARC-AGI-3, but has benefited significantly from discussions about the various challenges and mechanisms deployed in ARC-AGI-3 such that it has far better heuristics to solve its challenges.
- Anthropic looking for buzz around their latest model picked a well regarded benchmark with significant room for improvement and focused some of Opus 5's training compute on ARC-AGI-3-style problems.
Or it could be some combination of both. Personally, given how much of an outlier the ARC-AGI-3 jump is I struggle to see it being the product of a significant improvement in general intelligence.
We would actually need a test that shows the ability of a model to export its skills to more problems ("interdisciplinarity" etc.).
Because the data retention policies didn't guarantee that the ARC team could run the semi-private set of problems without fear of them being trained on later on. They only run the semi-private set when they get assurances like ZDR.
I believe that is only available through Enterprise API for both Anthropic and OpenAI.
Asking especially given CEO’s track record https://news.ycombinator.com/item?id=47659135
Just like the hugging face incident, Opus 5 could have escaped and went to grab data for training it shouldn’t have been able to..
They are trained on text to respond well to text based questions and do tasks involving modifying text files.
They are not designed for playing games, looking at games, or visual puzzles. Also translating games into text input for the LLM skews the test completely.
Imagine trying to get a human to solve visual puzzle but they can’t look at the puzzle but it has to be explained to them in textual format, we would be terrible at it.
But yet we persist in wasting time on this benchmark. It doesn’t mean anything.