I wrote about this recently here: https://fabraix.com/blog/adversarial-cost-to-exploit

I think the core issue is in static benchmarks and the community needs to start moving beyond measuring pass/fail (which worked when agents were incapable of doing much of the work) to dynamic evals that simulate more how we evaluate humans.

The "private benchmarks" suggestion comes up every time, but I think there's a more interesting axis: benchmarks built on top of already-public, already-stable test instruments. SWE-bench is fundamentally a corpus that lives on GitHub — once it ships, it leaks into training data automatically. Benchmarks built on contested qualitative instruments (psych tests, opinion surveys) have a different contamination profile because the correct answer doesn't exist in the training corpus to memorize — only the question does.

That doesn't help for measuring coding ability specifically (you fundamentally need a code-correctness oracle), but for capability axes where the "answer" is a stated position rather than a verifiable fact, public + stable can still be useful. The SWE-bench problem isn't really "public", it's "public + has a fixed correct answer".

This feels very much like "we are now moving the goal posts".

It’s very hard to encode the properties that matter most in code in tests. [1]

[1] https://voratiq.com/blog/your-workflow-is-the-eval

AI labs should compete on a bench that's adversarial, such as go or Starcraft

> We have incorporated these findings into our recent evaluation efforts. In the last months we’ve chosen to report results from the public split of SWE-Bench Pro. We recommend other model developers do the same. SWE-bench Pro is not perfect, but empirically seems to suffer less from contamination issues.

https://arxiv.org/pdf/2509.16941

The timing makes me wonder if this is a direct response to Deepseek V4 having performance comparable to SOTA models.

It's been fun benchmarking AI investigations at botsbench.com . Part of it is checking for these kinds of issues - we recently started seeing contamination in our first generation challenge, and less obvious, agent sandbox escapes for other kinds of cheating. Fun times!

more context in small writeup + we interviewd the team behind this when it was announced: https://www.latent.space/p/swe-bench-dead

Without SWE-Bench though, how will AI models properly game their results to show ~5-10% gain each iteration?

Once a benchmark is known and there's billion of dollars on the line, obviously every company will game them.