Top
Best
New

Posted by Edward40 4 hours ago

What is the best harness for each model?(runta.com)
233 points | 90 commentspage 4
underdeserver 2 hours ago|
Wait till you hear what the human success rate used to be.
johnrc 26 minutes ago||
[flagged]
guanlan 4 hours ago||
Hi HN, I’m Guanlan, founder of Runta. The eval runs on our runtime, so weigh this accordingly.

The short version: when you change the model, think about changing the harness too. On any one model, cost per pass varies by up to 4.4x

What we ran: 22 harness configurations × 5 open-weight models × 62 terminal tasks × 4 attempts. 27,280 runs, 83B tokens. Each harness ran with default settings at maximum reasoning effort, each trial in a fresh sandbox. Costs are Baseten list prices times reported tokens.

What we found: The lowest-cost harness changes with the model. Codex CLI ranks 1st on cost with GLM-5.3 and 15th with Kimi-K3. Paying more doesn’t buy passes. The harness with the highest pass rate ranked anywhere from 4th to 19th on cost. The model still sets most of the bill (93% of cost variation). The harness is the second lever. Starting prompt size doesn’t predict cost. Output tokens do. 64% of inference spend went to runs that didn’t pass.

radlad 3 hours ago||
Hi Guanlan, I vouched for your comment since you involved in this post, but you should be aware that all of your posts have been flagged, likely for AI patterns in your writing.

> Don't post generated text or AI-edited text. HN is for conversation between humans.

https://news.ycombinator.com/newsguidelines.html

latentsea 3 hours ago||
This is why for open-weights models at least harness aware post-training is becoming a thing. Also closed source harnesses that aren't fully editable are basically an anti-pattern. The labs should just hurry up and productionize self-evolving harnesses, and have the models internalize harness engineering, so they can optimise the harness for the given task. These are solvable problems.
shiqimei 4 hours ago|
[dead]