Posted by Edward40 4 hours ago
Is this post mixing them up or is my understanding of these concepts off?
Do you have a different understanding?
Can anyone confirm/deny this is the case?
If so, it seems a shame not to include this practice into the reporting methodology for full transparency.
However, the hardware requirements are still quite enormous.
I could see a gap in the market for crowdfunded (gofundme?) post-training of open-weight models, per harness. I estimate that the one-time compute cost is probably $x000 so this scales nicely with a large number of interested developers that could reuse the weights.
As with most post-training this would be done via some scored/verifiable problem set (we can probably tell if the tool calls blew up during the coding trajectory) e.g. a known set of input tasks where solutions can be verified.
Last-mile aka doing in yourself (on box) vs the compute cluster that created those open weights.
https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)
I believe (at least this used to be the case) that heavily quantized models that are designed to fit on consumer hardware have inflated failure rates and this can claw back quite a bit of reliability.
If you want perfect search, you need lossless compression, which defeats the point of the current systems that are lossy L2-optimized representations of all information.
The 'steps per pass' criterion is also not something under the harness's direct control in the way that its tool collection and system prompt (/start tokens) are. Even the cache hit rate is hard to control beyond a limit, since modern harnesses rarely rewrite history.
I'm also suspicious about the run-to-run variance between models. Given the high variability within the presented results and the observation that failed runs (which can spin for a long time before delivering a wrong answer) account for a majority of the cost, I'm not sure that many of the conclusions here would hold up under a second sample.
The page's title is: "Introducing FrontierHarness Eval v1.1"
I've seen this for years, and seen several papers that studied it: the same prompt will help with one LLM and harm performance for a different LLM on the same task.
And on benchmarks which test LLM x Harness, they find that LLMs tend to do way better with their "own" harness. (Or in the case of Chinese LLMs, the harness of the LLM whose training data they borrowed :)