Top
Best
New

Posted by Edward40 4 hours ago

What is the best harness for each model?(runta.com)
218 points | 86 commentspage 2
samueloph 1 hour ago|
Am I'm missing something or at this point the distinction between "harness" and "agent" doesn't exist anymore because too many people use them interchangeably?

Is this post mixing them up or is my understanding of these concepts off?

layer8 1 hour ago||
harness + model = agent (plus sub-agents)

Do you have a different understanding?

Edward40 18 minutes ago||
[dead]
ogmutlu 1 hour ago||
They mean the same thing?
cronin101 2 hours ago||
I'm far behind the curve with local/open weight models, but I thought it was common practice that you should do some last-mile fine-tuning on the particular harness that you're using and would get a reasonable return on investment for doing so (particularly on quantized models)?

Can anyone confirm/deny this is the case?

If so, it seems a shame not to include this practice into the reporting methodology for full transparency.

bisonbear 1 hour ago||
In MiMo's v2.6 public RL training run, they explicitly trained on 23 different harnesses in order to minimize overfitting on any single harness. I imagine that this technique has to be common in RL training runs given that model providers have little control over where their models are used.

https://mimo.xiaomi.com/rl/#overview

hedgehog 1 hour ago|||
It's typically easier to tune the harness around the models' existing behavior. In my experience the primary failures in models > 5B params on coding tasks are just that they aren't able to translate instructions into a correct description or implementation of the problem (will write wrong code, wrong tests, and wrong documentation). The basic tool calling and syntax of generated code is fine.
ford 2 hours ago|||
I don't think this is common, though it would be interesting if ThinkingMachines/HuggingFace/OpenRouter/BaseTen offered fine-tuned variants of the most popular open models that worked with specific harnesses
cronin101 2 hours ago||
In hindsight I think I overestimated how common/achievable this is with the logic that if PewDiePie could do it, so could a layman https://felloai.com/pewdiepies-ai-model-beats-gpt-4o/

However, the hardware requirements are still quite enormous.

I could see a gap in the market for crowdfunded (gofundme?) post-training of open-weight models, per harness. I estimate that the one-time compute cost is probably $x000 so this scales nicely with a large number of interested developers that could reuse the weights.

jrmg 2 hours ago||
What does "last-mile fine-tuning on [a] harness" mean?
cronin101 2 hours ago||
Taking the open weight model and altering those weights in order to overfit for success on the tool shape / quirks of the harness you are running the model within.

As with most post-training this would be done via some scored/verifiable problem set (we can probably tell if the tool calls blew up during the coding trajectory) e.g. a known set of input tasks where solutions can be verified.

Last-mile aka doing in yourself (on box) vs the compute cluster that created those open weights.

https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)

I believe (at least this used to be the case) that heavily quantized models that are designed to fit on consumer hardware have inflated failure rates and this can claw back quite a bit of reliability.

augment_me 1 hour ago||
This is completely expected. Agents are pseudo-greedy search functions that brute-force solutions using their context.

If you want perfect search, you need lossless compression, which defeats the point of the current systems that are lossy L2-optimized representations of all information.

Edward40 1 hour ago|
[dead]
bob1029 1 hour ago||
The entire point of this AI thing is that the cost of failure is now ~zero. You can try again in a few minutes / hours. A traditional software business might only complete one iteration before time runs out.
pitabas106 1 hour ago|
[flagged]
worik 21 minutes ago||
Given the variation between results for different, but similar capability models it is hard to take this seriously
Majromax 2 hours ago||
Saying that 'steps per pass' explains cost seems nearly tautological. Run costs are proportional to the number of tokens produced, and the number of tokens is strongly correlated to the number of steps.

The 'steps per pass' criterion is also not something under the harness's direct control in the way that its tool collection and system prompt (/start tokens) are. Even the cache hit rate is hard to control beyond a limit, since modern harnesses rarely rewrite history.

I'm also suspicious about the run-to-run variance between models. Given the high variability within the presented results and the observation that failed runs (which can spin for a long time before delivering a wrong answer) account for a majority of the cost, I'm not sure that many of the conclusions here would hold up under a second sample.

Edward40 2 hours ago|
[flagged]
jrmg 2 hours ago||
"In 27,280 coding agent runs, 64% of inference spend went to runs that failed" (the title this submission was submitted with) is a pretty editorialized title.

The page's title is: "Introducing FrontierHarness Eval v1.1"

Edward40 2 hours ago|
[dead]
andai 2 hours ago||
>Through our experiments, we found that you should consider switching harnesses when switching models to maximize your experience.

I've seen this for years, and seen several papers that studied it: the same prompt will help with one LLM and harm performance for a different LLM on the same task.

And on benchmarks which test LLM x Harness, they find that LLMs tend to do way better with their "own" harness. (Or in the case of Chinese LLMs, the harness of the LLM whose training data they borrowed :)

colinmarc 2 hours ago||
How are people even using Baseten for coding agents? I am using Pi (so very small requests) in a "normal" single-agent way and I hit the rate limits on throughput immediately.
Edward40 2 hours ago|
[dead]
WorldPeas 1 hour ago|
evolving harnesses are also a cool development I've seen recently though I feel they'll have diminishing returns https://www.alphaxiv.org/abs/2609.01437
More comments...