Well, Greg KH emphatically states that local LLMs are more than good enough for bug finding in the Linux Kernel. [0] Seems strong enough for the task to me. [1]

He goes on to recommend that one uses local LLMs exclusively for bug finding because he and his colleagues have seen multiple occasions where folks who used the LLMs provided by the Major LLM Manufacturers had the stuff they were working on get "mysteriously" leaked.

Even if that sort of thing were not known to happen, it's just smart to not process data you want to keep secret in systems that are known to straight up retain user data and/or "anonymize" user data and send it back to their HQ for use in production of newer versions of their products.

[0] IIRC, it is this video: <https://www.youtube.com/watch?v=NnV_cWeoo5Q>

[1] And even if they're six months behind the latest publicly-available tools provided by the Major LLM Manufacturers, six months ago a notable subset of folks were on here screaming that their latest tools enabled fully-automated hands-off coding and had ushered in an era of Software Factories that were rapidly going to make all programmers obsolete. ;)

I've built an RSS reader and I'm doing a half assed reverse engineer / web version of Tapped Out (a silly little Simpsons mobile game that EA killed a while back). I've had to go to Claude a couple times when it got stuck on something curly, but I think that was mostly just loading a fresh context and I could have solved it I'd I really tried.

I've been using Qwen 3.8 and more recently Next for the last two months almost exclusively on home projects (I have access to a system with an RTX 6000 for free). It does everything I want it to! I have no issues believing it would be fine for more than just pet projects.

Well in large corp this takes time, even if you have an army of Einsteins at your disposal (just like we saw with Elons Doge) because there will be 16 different spreedsheets/dbs talking abt the state of the system that all contradict each other. While the 6 people who maintain their own state to keep the lights on, will have job titles that have nothing to do with system state maintenance.

Small companies its hit or miss cuz they are overwhelmed by the pace of change. Basically no one wants to hear things will take time. Intelligence is just one system variable and others dont just auto sync.

This is a benchmark that solves ten hard problems in five passes. It scores 9.8 against the authors' 9.6 and Fable 5's 8.0. Memdoor workflows are the harness that replicates the original GVS5H thread harness posted in HN last week. Raw results (i.e. each call's tokens, cost and host) are in the bench folder for verification.

will definitively try it. Overall with a smarter model the improvement is in the token usage and latency. Tbh I think i could get a 10 score if i tuned the prompt and temperature a little more. The original author harness is already giving 9.6 score which is pretty good. However this harness loop can take more than 1 hour on the hardest problem at 100 token /sec. A smarter model such as 3.8-flash-next could get 10 score and resolve the hardest problem faster. That's my estimate i'll give it a try. Thx

I would not think in terms of "how much" context, but really just how the harness manages it. I've been fiddling around with harnesses earlier this year and the rabbit hole goes very deep. I eventually had to drop it because it consumed me, and I can only do one thing well. But what I learned is that there is a lot on the table at the harness level.

The elephant in the room and the constant tension is that the models do get "used" to a certain harness (trained for it by the model labs), but they also generalize well. So one really advanced mechanism could fall short because the model could fail to generalize, whereas in other cases the model might handle large deviations from how a conventional harness works better than expected. This is where the gains are.

"think", "believe" - maybe, but this is why we need better measurability.