Posted by Edward40 3 hours ago
It’s unclear whether that’s supposed to be surprising against the selected models for the benchmark. I suspect it’s not and the headline is written to pander to “wasting money on AI” upvotes.
Edit: they changed the title. The one used to get to the front page was something like "Across N agent runs 64% of agent cost went to failing runs".
Edit2: since the link is poor quality, I tried to answer some of these questions myself using 927k agent runs vs. this blog post's 27k: https://micjm.com/blog/on-model-harness-interactions/. The upshot: there's not a lot of harness x model interactions. OpenAI models do seem to have a preference.
A serious effort here would use a specific set of terminal bench questions. Instead these look like cherry picked questions. How were they picked and why? Without an answer we should assume they were picked to produce some result.
For scoring, final passes are divided by scored attempts, where infrastructure errors are excluded from the denominator, timeouts are still graded by the verifier, and reward hacking (such as looking up solutions online) counts as a failure. Costs are calculated using list price multiplied by reported tokens.
https://hub.harborframework.com/datasets/frontier-harness-ev...
The biggest difference is that I, as a human, get mentally exhausted. but modern AI (when given unlimited tokens and tools) is like if Stephen hawking had a meth habit and the tenacity of Sisyphus with a bad case of syphilis after being told he can be cured if he can get that boulder over the hill
And even if they did try to test them on a lot of gotcha tasks; I'm perennially reminded of the ever-relevant quote about AI and other tech: "this is the worst it will ever be"
My Windows work desktop has two orders of magnitude more memory and more processing power than the last time I used Windows, yet boots slower and is laggier than any Windows I've ever used.
Almost all useful information online is concentrated into YouTube and Reddit, everything is advertising and trying to steal attention. I'm not even welcome on Reddit anymore due to the media campaigns against my country and race.
We've definitely passed the point where the derivative has gone negative.
We kept the category unified so we have tasks for every domain like ML , science and others.
For example, on bun-sourcemap-leak (https://hub.harborframework.com/tasks/terminal-bench/bun-sou...), all 440 trials finished cleanly with zero infra errors or timeouts, yet 0 out of 440 passed all hidden-variant tests (the best was 35/36 across three DeepSeek V4.1 Flash runs).
Some of these tasks take up to 8 hours per run, so if you are not careful, you end up burning massive amounts of compute and API spend just to get an accurate evaluation signal.
For harness developers: cut steps, not just prompt tokens. Prompt size, tool count and cache hit rate did not predict cost per pass. Steps did, because every step sends the whole context again.
Incredible, none of these harness developers considered just asking the LLM to solve the problem right away! Don't siege Leningrad, take it immediately!
The places where you let an agent do its thing is bound by the (implicit) interface your prompt creates. In an ideal situation this interface is a simple isolated thing in your application.
If you just let it do its thing, expect a monolith. You shouldn't rely on it coming up with interfaces that mesh well with what you had in mind, and each of those interfaces represents a decision that it took, restricting future iterations of the code. So a monolith is not only what you can expect, but also the best you can hope for.
If you want to let it cook, create more places for code that does a thing and you don't care how.
With some up front care you can create an environment where they mostly do what you want when you let them rip without oversight. As the codebase grows it becomes less likely they'll see any one example, and more rarely expressed ones can get missed. And then codebase predating LLMs or designed without this consideration you just have to accept what you get or wrangle them manually.
The "objective" nature of code is revealing a flaw in the human division of labor.
I know why the US frontier labs push it of course, you can't push a trillion dollar plus idea if you also say it should be a cheap commodity and the best and latest are now the answer, a bazillion mid levels are.
But most work of value I do is with a swarm of "not the latest nor the brightest" models, with a smart one controlling the dance. Most decisions or review is better done through consensus. The myth that "if we keep going it will reach a point where it's always correct" seems so obvious to me, it doesn't work like that, for that matter we don't work like that.
This is not a jab at it. I work with it. I love it. But I think the story imperative are a mismatch for the actual needs of most companies, and it's hiding between the "frontier needs" of special research & co, or the governemental needs.
On the same line of thought: it's pretty obvious as great as these things are, no they're not going to "replace all employee", and no employee without them won't be able to compete either. They're a supercharger, a self multiplying smart assistant with capabilities in any field, and it feels so wrong to me that they've been presented in a complelty different way.
The parallels with self-driving cars are interesting. We demand that agents don’t just produce better code than most humans (which the frontier models do), but that they perform much much much better.
To my surprise, none of them came close to producing an acceptable PR. Quality varied, but even the very best results were missing significant pieces.
My takeaway is that the human in the loop is still massively important for many things. And that makes it really hard to come up with a useful benchmark.
Otherwise you still end up with balls of spaghetti where each change takes longer and longer to work into the codebase. With a bit of planning, you can keep tackling changes and they keep taking the same amount of time more or less.
The requirements will be more stable than the code. Focus on them instead.
These kinds of changes to the way teams are formed and managed, and how products are developed is what really fascinates me about the impact of coding agents.
Even if it works, for some definition of works, to dump a huge spec into a coding agent context window, I suspect this is where a lot of horror stories about agentic coding come from.
Teams, and the management structure above them, are going to have to get used to more collaborative and iterative product definition and development in order to extract the most value from coding agents.
Wouldn't count on it. There likely is a ceiling, we might or might not be near it.
I have code I wrote in 2006 which is still in daily use adding business value at my company. Why would it need replacing?
Code has always been fungible, in the sense that it could be refactored.
> you should expect any code you write today to mostly be replaced in the near future
Why on earth do you think this? It does not follow from your other points. Just because you could turn your business into a rube goldberg code juggling machine, why would you?
I recently used agentic processes to bring a young codebase to a functioning state very rapidly. This forced discussions about design principles that were folded back into a short document in the repo which then led to a major refactoring plan, and a massive pruning of code that was written the week prior. There are solid engineering approaches in the codebase now that wouldn’t have been achieved without this discovery process.
I find that only works with topics I know very little about.
- tongue in cheek: in topics you know very little about, you don’t have the ability to tell whether the LLM output is trash or not
- very serious: LLMs allow you to create things you could not have created before due to not having the knowledge
Sort of like a Dunning-Kruger or Gell-Mann amnesia.LLMs are allowing people to create things with very little effort compared to before, and those creations are of questionable quality - indeterminable by the LLM-user due to a lack of experience.
I think the quality of the results vary with the rigor of the process. If you one-shot prompt a new service (even with Fable5) "build me an airtable competitor, no bugs", you will get some real code camp quality results, but a well defined set of prompts and a workflow that leverages multiple coding agents reviewing each others work, and implementing test cases at a relatively granular level yields radically different results.
Couple that with a rigorous design/spec session, and multiple checkpoints (say at release milestone) to monitor drift from the original design makes the process a bit more intentional and measurable, while still being far faster than a typical engineer can build.
For a given model, the best harness depends on the problem. As models become increasingly capable, the harness appears to matter more than the model. This is perhaps controversial but allow me to explain.
Most of my problems are in a single domain, using specific tools and programming languages. I give whatever model de jour the specific tools it needs to solve the problem. I don't let it execute arbitrary scripts or programs (though it can, if it really tries). I give it a restricted number of calls and I feed that information back into it (budget). I let it farm out work to other models, using the same harness. I give everything a budget, which is tracked and enforced, but it's not real, and refills at different times.
This bespoke, specialized harness absolutely smokes any other mainstream harness for these tasks. Significantly faster, cheaper, more correct.
But I can't build a mobile app with this harness, or make a movie.
These tests are doing the same thing. They measure a handful of scenarios. I can't easily tell if those map to my needs or not. In my experience, very few of them do.
Can these general purpose harnesses solve my problems? Yes, if I'm willing to let the model churn for a long time and burn a lot of tokens. But that doesn't feel like a sustainable practice.
Maybe I'm too early, maybe it's premature to optimize the harness this way, but I'm spending fewer tokens than my peers and getting a lot more done, and that matters.
I've been tinkering with harnesses recently but have mostly used github copilot on vscode mostly thus far as my ai tool and feel like the harness setup on there is lacking (or really confusing...). many thanks if so :)
just trying to learn without googling "how to setup a harness" and get bombarded by a trillion yt vids or competing harness providers lol
However, I am hesitant on the correlation between pass/fail in evals which are a binary 0/1 determined by tests, and real world usage, where users can steer, run code review, and care about more than just test pass/fail.
I wrote about the problem of determining "success" for coding agents, and a lot of it boils down to the fact that agents can write dumb code that passes tests, and write smart code that fails tests, and we need to shift to rewarding quality of code and alignment with intent rather than just signal on held out tests (which is still important!)
Cost per pass is simply the average cost per task divided by the pass rate.
If you draw a line from the origin to any harness on that chart, its slope (pass rate ÷ cost) represents passes per dollar. Cost per pass is the inverse of that slope, meaning the lowest cost per pass gives you the steepest line.
To illustrate the difference, the author of FrontierHarness included "Figure 3. Cost and pass rate of each harness with each model", in which we can see that Codex CLI has $0.937 cost per pass, but a $0.573 cost per task. Measuring cost per task also changes the winner to OpenCode
I think this is not too different from physical products, either. You often hear of companies spending multiple years and iterating through dozens of prototypes in order to bring something new to the market.
I don't think there is a shorter path: you never truly understand a problem until you take up the engineering work to solve it.
GitHub and Copilot CLI have started to do the same btw (not sure if they publish the traces/reward hack results). But we should normalise this as consumers shouldn't be forced in to playing this game.
Let's not forget multiple departments like the Pentagon, FAA, Homeland Security etc, have been raising alarm that things like this were becoming a heightened risk since Trump and Musk cut ~300k jobs with DOGE, and slashed civilian contractors across nearly every department.