Posted by wilhelmklopp 5 days ago
> It’s actually fine if agents make a lot of boneheaded mistakes. What’s not ok is if they keep making the same mistakes.
It's simple: when a mistake happens, you run your CAPA process (Google CAPA form and see examples to extrapolate what that process might look like) and determine the root cause and the correction to the process that allowed the mistake to happen in the first place.
(At least as a SaaS vendor in life sciences, when we had a CAPA (e.g. after a SEV0 failure), it would be folded into our SOPs and then we would be required to retrain on the SOP. Auditors would want to see our evidence of CAPAs, the versions of our SOPs, the records of training. All to extreme for most shops, but I add this for context/color)
This is something most eng shops do not have the discipline for since it requires some diligence.
Should it be fully agentic? Should there be human intervention here to approve the CAPA? Open questions to be answered.
It’s all very well having a list of actions to avoid but that doesn’t help if your agents won’t reliably follow it.
You need two pieces:
a) prompts, that tell the agents what to do and how to do it (and ideally, the why, where, etc, the full picture) - that's the positive half, that drives behavior the way you want it.
b) deterministic tooling that prevents negative outcomes, like linters, compilers, static analysis, fuzzing, testing, the more the better. This side should either be firewalled off from the AI or very carefully watched so that it doesn't drift.
The part that you put in the deterministic side is the "never do x" stuff - I have lint for long comments (which AI hits every single time it commits), all my dev scripts are in typescript, precommit hooks, massive CI, and I lint even for things like redirecting error to standard out, tiny stuff, and also e.g. static migration analysis so the AI never ships an exclusive full table lock in a migration, for example.
You can’t keep humans from making those errors either but you also don’t let an error prone human crank out 20k LOC per day without forcing other humans to understand it.
C# Roslyn Analyzers[0], for example, are quite powerful and can identify complex patterns in code. One approach to deterministic enforcement would be to ensure that the project is set up with an analyzers library and mistakes that can be deterministically flagged are
[0] https://learn.microsoft.com/en-us/visualstudio/code-quality/...
Each of these are just layers of control at different lifecycles of agent code generation. Analyzers are nice because it gives targeted, static analysis that can prevent certain classes of errors very early and at lower iterative cost (e.g. a build)
“You can’t deterministically keep them from making even a tiny fraction of all the possible errors they can and do make though.”
And you replied
“This may be platform dependent.”
I’m unsure how else to read that other than an implication that this might be possible on some platforms.
I've been using languages with stronger type systems and that's also a huge boon.
Why would that be the case? You can run human written code through the same “linters, compilers, static analysis, fuzzing, testing” as you do agent produced code.
Agents can also do all of those things, but they are generally more compliant to instruction.
Agents require far stricter guardrails than humans. Without linters, tests, static analysis, oracles etc… no agent can create a large program.
Even if you’re correct, you just build those checks into CI so that neither humans nor agents can skip them.
In test cases i can do anything, a test framework is just a way of discovering and then scheduling functions to run. I can emit useful instructions to the agent from the failed test case: "After walking the AST of all use of state machine X, a branch was found at Y which reused stale state. Ensure stale references are dropped..."
I can force the agent to pass the test suite before it considers itself done. I can reject edits of such test cases to partially mitigate reward hacking. etc etc
The CAPA is a learning that sits outside of the mechanism of verification; it is a record of problem:root_cause:preventative_action. I see it as the instruction that would be required to generate the test case to prevent the next occurrence of a class of failures.
In a real-world process, for example, there is usually a QA lead that is verifying that the process is followed by looking at the paperwork and evidence.
How would prevent removing the test case and the code that it tests? That's a legitimate thing to do when you're modifying a codebase. My intuition is "corrective and preventative actions (CAPA)" is a level or two up from test cases.
One approach, for example, might be to have the a standalone code reviewer agent that is solely responsible for interfacing with the CAPA system (e.g. via a tool, via MCP) and acts as a back stop. When it finds a new type of CAPA, it stores it (and the backend indexes it with enough metadata to support broad types of retrieval). When it reviews a piece of code, it finds past CAPAs. By file locality. By business domain in the application. By keywords.
Same tool and repository available to both building agents and review agents, but use the review agent as a dedicated back stop as part of the verification process.
I think a point we are grappling with is, what necessitates human intervention, like philosophically. Is it accountability? I was thinking about this in terms of code review and it's not like we would fire someone if they broke prod, so at what point do we need someone accountable. Maybe it's for approvals for certain classes of risk (like those associated with actual harm to life). It's also not lost on me that many human systems lack accountability.
I once worked at a very large company, and one of my coworkers was like "eh, I'm not going to do any of this work, because I don't have to", and that was a clear indicator that the problem wasn't (just) with the system, but with him individually.
It will be interesting to see how the notion of 'responsibility' carries over as agents handle higher and higher levels of abstraction. Agents built using the popular frameworks of today are somewhere in between code, formerly written by engineers, and an actor (like an engineer). When an agent wrote the framework to audit and monitor ops agents who are monitoring the performance and reliability of the code that coding agents wrote, who is responsible for ultimately ensuring it doesn't happen again? Should the findings just be added to whatever RAG and a few prompts/hooks/skills changed by another agent? When does a human follow up and to what degree?
This is the sort of thing that makes me believe that software engineers will never truly go away; at the very least, they serve the very useful function of being ultimately responsible for something going wrong.
I personally use it for the agents I use, so why did it fail, what was the root cause, what can we do to prevent it again.
I think correction actions still require the ability to execute them, which (in all the cases I've had) would require more capable models.
Long term, I think you're probably correct.
Write a program! If it has a bug, debug it and update the program. Use an LLM to debug it if you must. If you're doing something repeatable, then use a traditional automation scheme.
If an LLM has to handle a subsection of the program, so be it. Just wrap that part with clean inputs and outputs.
Why does it feel like we've forgotten how to automate things?
> Why does it feel like we've forgotten how to automate things?
Would love to be proven wrong here, feels like I've just seen the same basic one-level kart racer in like 3 model announcements. Not even sure if they were one-shotted, I certainly would not describe them as "fun"
You need to playtest the hell out of games. Play a bit, iterate, play a bit more, iterate, etc. You will get way better results if you start small and don’t try to one-shot it. Start small, vertical slice, playtest, and go fro there.
It's definitely not fun if you consider building an actual community around it and have a long term plan for it.
(If that doesn't load try cube-sweeper.onrender.com some isps block it)
I did make a bunch of changes after the initial "one shot" but the bulk of the game was done after a single prompt
It's a working pile of trash that has quadratic growth in the remote postgres database resulting in 10GB stored in a few days for what should have been like 5 MB of data.
It's not a skill issue, I know how to get good results but I figured maybe I don't need to be as cautious as I am all the time.
Not that I don't believe we can get there eventually.
> resulting in 10GB stored in a few days for what should have been like 5 MB of data
We wrote this post as part of a launch, which you can check out here: https://x.com/danlovesproofs/status/2095182189499711759
Anyway, here's mine, still wip:
Why? Because LLMs are always going to be dumb when they're trained at scale. Their ability to speak software diverges from their friendly user input layer. A harness won't overcome that, but an LLM saddle ontop of a larger model would provide the type of feedback loops you'd want to look into.
I don't think you'll find two deterministic systems will produce much.
I'm working towards both in my homelab to see which works better with little qwen
Your thoughts on the matter seem clearer than mine.
I’m kind of relieved to see so many stages; that’s something I’ve grappled with conceptually. The volume of (sometimes optional) stages has made it hard to resist the temptation to hack something together with a GHA workflow.
I salute your choice to support distributed; that’s a requirement in my book.
I look forward to trying it out.
- a workflow is a series of steps
- a step is one or more tasks that can be done in parallel
- a step is complete when all of its tasks are complete
- a task can be a unit of work, or a pointer to another workflow
But then there's the cases where the AI can't actually drive. What if you tell the agent to invent AGI? Or time travel? Where does it drive to? Does it tell you that it can't? How does it know it can't just drive here? How you you know? At the end of the day these are not wish granting machines, so someone connected to reality is still going to have to make decisions. And that person is going to be the one held liable for whatever the AI does so would they want a self-driving codebase in the first place?
Any attempt to describe software by "specification" leaves enough ambiguity that the agent will do deranged stuff like add a ton of code to satisfy error cases that can't happen. Or, like you imply, it will confidently create a bunch of nonsense to "solve" a problem in a way that is not really possible.
One is to take the human out the OODA loop for cyber defense. Servlet libs (for example) are going to become fluid, self-modifying things w/ contracts that operate much differenlty from how they do today. The engineering practice around these things will need to change.
The second is that UX will be self-modifying. Just like how pi can modify itself, I can see this being a general practice for user-facing applications. Perhaps a text-box in which users can describe tweaks in can request changes to how the application functions. Engineering an application will focus on modeling the non-negotiables of how an application works, and providing the correct primitives for user-driven LLMs to modify software on the fly, as well as track and rollback changes. There'll also be funsies around how to ensure that ads get delivered regardless of the user trying to get rid of them.
This smells like worm food. Those contracts had better be airtight.
The point of the example is that with the first choice you have 5% legitimate customers and 5% malicious "customers" and by letting all of them in aka all 10%, you get maximum customer satisfaction including satisfaction of the malicious customers.
Alternatively you reject the malicious customers and as collateral damage accept that some customers do not get their request approved.
You're also confusing "normal operation" with what a user request is. A user request is a request to change something, so by definition it is not the continuation of the existing system. You're thinking that a user request is a HTTP request which is not what's being discussed.
It also is de-facto the case today when logging in as per usual sometimes requires a captcha or additional verification.
As a random sample of one, I looked at one of the bugs this reported on Tailscale (first thing on the homepage) [0], and the pull request ends with "Apologies for the lack of due diligence here. I'll go ahead and close this out."
I feel that way.
But I'm now able to actually create and release things. Before, I used to obsess over code architecture, optimizing every single line. I could and can craft beautiful code. And then I always burned out and lost interest (gemini). Now I don't touch code at all and have been actually creating and releasing the ideas I've always had in mind. Then I realized that's what I always wanted. The code was always a means to an end. But that's me.
AI have a lot of advantages over humans like speed, controllability, good defaults (e.g. Claude will write helpful comments unprompted which is better than most engineers), and the ability to implement best practices like comprehensive testing with much lower cost than a human team. Using AI in my personal projects, I am able to write software at a much lower bug rate than manually, because now I can make an automated test suite whereas previously I had no time to do so. However, AI also has downsides like rigidity, limited context window, and lack of creativity by default which needs to be resolved with harnessing (e.g. different agents with separate, well-compressed contexts) or worked around with human review layers.
I suspect the best core pattern for coding automation will be to create an engineering manager agent layer that can flexibly apply a set of engineering practices to the concrete problems at hand. It would handle all coding agent pings before it reaches a human, with prompting so that it reliably considers the big picture when making decisions. Very rarely, it would escalate an issue or plan review directly to humans when it is truly blocking, but most likely it would organize the information it obtains from the lower layer coding agents and surface them as reports for humans to review.