Opus 4.8 (max thinking) scored highest and Grok 4.3 lowest

It's hard to understand what's going on with Grok. It's like it has capabilities in a theoretical sense but maybe the training is so focused on being in x.com/grok.com with the web search tool enabled for "is this true?11" type queries that with any API type usage with document workflow instructions, tool use, code gen etc it completely falls over

Sounds about right

Why would anyone think that models optimized for efficient context management, giving much more weight to a short sliding window, would attend to distant, heavily diluted tokens?

Plus the model's capacity to take more context into account and actually integrate it to the output is simply limited by the number of activated parameters. If you give it a playbook, you are forcing to choose it between attending to the playbook and the task at hand.

If you want to force it to work step-by-step, you need to present the steps one-by-one. Ideally with rules for the current step at hand and maybe relevant input again, depending on overall task size.

Why did you think models love to re-read files before editing them? It increases recall quality and thus edit precision and thus benchmarks.

Dealing with agents/LLMs based on "instructions & guidelines" has taught me - nothing (un)reliably governs agents other than agents themselves or (rather i.e.) their motherships (assuming they can and they intend to). Or if you add ton of local tooling.

One question I'm asking myself is: can LLMs be bored?

This is a problem with soft rules and LLM in general, and it makes sense that it gets worse on complicated tasks with long context.

Glad to be able to put some numbers on it.

simonw has been correctly asserting that there is no deterministic way to prevent hallucinations. I’d argue that this obstinance could be the seed of a strong argument in favor of their capacity to eventually coalesce some form of consciousness

I'd be very interested to see a comparison with actual humans on the same benchmark.

In my experience, the more structure you enforce on models, the worse they perform and the less they actually do what you want.

LLMs are not people. Employee handbooks, like all other social rules, are not self-executing; they depend on the fact that human beings are innately social animals.