Instead, use them in adversarial mode - run QA scenarios using LLM agent as a substitute for end user to do bug discovery.

The models write good/great code. I'm very happy to never write code again but models are no where near good enough to run off on their own without a human in the loop. Models LOVE to cheat. Just peek at the tests they write.

And before you come at me, I have built 7 products in the past 1.5/2 years with agentic engineering, all with users. One of those projects is _dead_ because I let the vibe go too hard at the same time Anthropic decided to nerf both their harness and their models silently. If you care to look at the source: https://github.com/Robdel12/OrbitDock I spent a week or so and like a billion+ tokens trying to refactor and save it. It just wasn't worth it.

I wish people would be pragmatic about this. I get the dream is to let it do everything and not to be in the loop, because being in the loop is exhausting. But if you want to make whatever you're building be robust and survive more than 6 months, you have to. I don't care how good your tests, plans, skills, etc are. At some point the model will have to decide something and it'll be the wrong one. Compounding the slop from there forward.

I disagree with the way that big models are trained on noisy relationships and RL is applied to tone it back down, it represents a stupid amount of compute thrown at this problem at a scale that is often unnecessary.

The rest of it is on point.

We may choose to disagree but thats the point of healthy debate based on clearly expressed opinions.

Key point is I don't think this is AI slop which is way too common in long form articles these days and in keeping with the whole point of his article.

The one downside was that it required their computers to install dev dependencies, postgres, infisical for secret management, etc.

That's the next frontier. What I'm working on now independently. I plan to open source this solution, but again I'm not a savant genius. I guarantee many people are working on the same thing it's converging. Our job as engineers is becoming more of a higher level facilitator and AI "plumbing" maintenance work. How can we get the AI to empower everybody at the company while keeping the wheels turning. That's my aim.

Also it's missing the point of a software factory concept

A software factory will not work infinitely forever for everything. A software factory isn't a solve anything button (aka god).

In a conventional factory, things break and fail. Process machines get poluted. Extruders get jammed.

You still need to establish intent, define what you care about, define guardrails, and of course manage the factory.

Its crazy to me people write these articles and create standards like this is some kind of engineering standard with years of research and experience

This is like calling these folks the experts on aviation: https://youtu.be/M9Yww9LG3gw?is=xgtA-xMpNy-09Asu

Its still so early in the game for de facto standards - engineering teams need to experiment and see what works for their own quality metrics not just parrot “standards and methodologies”

This is still the very early days of AI and AI engineering

This was easy to find out I thought. And just with an old-fashioned google search too, no deep research agent needed. See here: https://diffusion.io/

Seems like it went pretty well if a consulting company is now being started.

I agree with a lot of what Dex Horthy is saying here but on some fronts I feel like he's missing something. Coding well with LLMs, it's not a skill issue, it's an effort/laziness/rigor issue.

In order for coding with LLMs to go well, there has to be more rigor, more discipline, more good engineering hard-assedness. To reiterate, the teams seeing the best results with AI were already high-discipline and high-hygiene.

AI works on data. The better the data, the better the likelihood of a desirable outcome. Code is data. If you have bad code, no matter how awesome the model you let loose on it, you can't get as good a result as if you had good code to start with. This principle has been well known in AI/ML circles since the 20th century.

e.g., if you are doing spec driven development and not seriously investigating formal verification, IMHO you will come up short. Prompts are simply not enough to steer a coding agent to the level of precision needed. Without deep programmatic verification - at all levels, formal verification is just one slice - the solutions the agent produces will always be just slightly (or very) out of true.