> Purely AI written systems will scale to a point of complexity that no human can ever understand and the defect close rate will taper down and the token burn per defect rate scale up and eventually AI changes will cause on average more defects than they close and the whole system will be unstable.

Wow, it’s true, AI really is set to match human performance on large, complex software systems! ;)

A non-technical friend of mine has just won some hospital contracts after vibecoding w/ Claude an inventory management solution for them. They gave him access to IT dept servers and he called me extremely lost on how to deploy (cant connect Claude to them) and also frustrated because the app has some sort of interesting data/state issues.

Reminds me of the quote in the original Westworld movie:

“ These are highly complicated pieces of equipment… almost as complicated as living organisms.

In some cases, they’ve been designed by other computers.

We don’t know exactly how they work.”

Now how did that work out ;-)

Heh. Got a customer recently around this. Entire infrastructure and CI/CD vibecoded. They half implemented Kubernetes in Github Actions that were several thousand lines long and impossible to understand.

I think the problem will get worst. I dislike the marketing around AI, but I do think it is a useful tool to help those who have experience move faster. If you are not an expert, AI seems to create a complex solution to whatever it is you were trying to do.

This might not pan out to be the glorious victory of human craft as you’re imagining it to be.

Here’s a slightly different future - these AI rescue consultants are bots too, just trained for this purpose.

Plausible?

I have already experienced claude 4.7 handle pretty complex refactors without issues. Scale and correctness aren’t even 1% of the issue it was last year. You just have to get the high level design right, or explicitly ask it critique your design before building it.

> Purely AI written systems will scale to a point of complexity that no human can ever understand

I think it will be needless verbose complexity.

I kind of imagine someone having an unlimited budget of free amazon stuff shipped to their house.

In theory, they are living a prosperous life of plenty.

In reality, they will be drowning in something that isn't prosperity.

"Purely AI written systems will scale to a point of complexity"

You have not seen the spreadsheets that accounts run the firm on.

Bloody kids!

I've already done a handful of these gigs for early vibecoded products that had collapsed in on themselves. The scope of work was to stabilize the product and only make existing features work.

The issues have all been structural, not local. It's easier to treat it like a rewrite using the original as a super detailed product spec. Working on the existing codebase works, but you have to aggressively modularize everything anyway to untangle it rather than attack it from the top down.

All of these projects have gone well, but I haven't run into a case where a feature they thought was implemented isn't possible. That will happen eventually.

It's honestly good, quick work as a contractor. But I do hope they invest in building expertise from that point rather than treating it like a stable base to continue vibecoding on.

But it's so easy now to redo it all ground up, and if models improve, do it better next time.

I exaggerate only a little.

> reach a stable set of design principles

Are you sure about this? Yes, there is a stable set, but they are used in all of the wrong places, particularly in places where they don't belong because juniors and now AIs can recite them and want to use them everywhere. That's not even discussing whether the stable set itself is correct or not - it's dubious at this point.

What you're describing really isn't a new problem for organizations. Historically it's been a team of humans not using AI who gets over their skis and they have to have other more capable humans (also not using AI) to bail them out.

Those design principles it will take us 20 years to learn are just the principles for writing good maintainable, debug-able, understandable code today. Will just take 20 years to figure out they still apply when AI writes the code, too.

As the models keep improving, wouldn’t you be able to task a newer AI to “clean up this mess”?

> Purely AI written systems will scale to a point of complexity that no human can ever understand

In their current forms, it's unlikely for a product that actually needs to work.

It's not getting that complex and working with current LLMs.

The complexity you would come to the rescue to solve, would that be from AI or from the style of programming you let the AI have? I mean, you have very different problems if you use functional style vs object-oriented. It is up to the programmer to realize they want a functional style and request that from the AI, as much as possible. Even AI cannot imagine every state transition, unless it is so smart that it should be the one telling you what to do.

Interesting perspective. Fundamentally at conflict with the data, science, and 20+ year trends of AI coding systems - to the point of dogmatism. But interesting from a sociological point of view.

I'm sure AI capabilities will plateau any moment now..

> I think AI rescue consulting is going to be come a significant mode of high value consulting

I thought the same when I saw development outsourced to Indians that struggled to write a for loop.

I was wrong.

It turns out that customers will keep doubling down on mistakes until they’re out of funds, and then they’ll hire the cheapest consultants they can find to fix the mess with whatever spare change they can find under the couch cushions.

Source: being called in with a one week time budget to fix a mess built up over years and millions of dollars.

is this true because training companies have not been training AI for both performance and brevity (or some other metric like that)? If this becomes a much more serious issue surely they would adjust the training processes

Financial auditing with pre-AI technical chops will be uniquely niche-valuable, too :)

This is def true but I also wonder if AI models and context sizes and capabilities will scale to keep up and eventually be able to untangle the mess.

Have you watched Jurassic Park? That story is not about Dinos.

AI janitors

> Somewhere in the future, the new software engineering will be primarily about principles to avoid this in the first...

It's really nowhere near as complicated as making distributed systems reliable. It's really quite simple: read a fucking book.

Well, actually read a lot of books. And write a lot of software. And read a lot of software. And do your goddamn job, engineer. Be honest about what you know, what you know you don't know, and what you urgently need to find out next.

There is no magic. Hard work is hard. If you don't like it get the fuck out of this profession and find a different one to ruin.

We all need to get a hell of a lot more hostile and unwelcoming towards these lazy assholes.

Correct. I use AI a ton and I'm having more fun every day than I ever did before thanks to it (on average, highs are higher, lows are lower). Your characterization is all very accurate. Thank you.

Here's some other topics I've written on it:

- https://mitchellh.com/writing/my-ai-adoption-journey

- https://mitchellh.com/writing/building-block-economy

- https://mitchellh.com/writing/simdutf-no-libcxx (complex change thanks to AI, shows how I approach it rationally)

The way I put this to myself is that AI gives “correct correct answers and incorrect correct answers”.

They almost always generate logically correct text, but sometimes that text has a set of incorrect implicit assumptions and decisions that may not be valid for the use case.

Generating a correct correct solution requires proper definition of the problem, which is arguably more challenging than creating the solution.

when you outsource thinking to AI, you get that magical speed up. the agent is making decisions for you, so things move at agent speed. it often makes decisions without telling you, and the final "here's the plan" output often requires you to understand the problem at great depth, which requires return to human speed, so you skim and just approve.

the trick is to be mindful, aware, and deliberate about what decisions are being outsourced. this requires slowing down, losing that absurd 10x vibe coding gain. in exchange, youre more "in-the-loop" and accumulate less cognitive debt.

find ways to let the agent make the boring decisions, like how to loop over some array, or how to adapt the output of one call into the input of another.

make the real decisions ahead of time. encode them into specs. define boundaries, apis, key data structures. identify systems and responsibilities. explicitly enumerate error handling. set hard constraints around security and PII.

tell the agent to halt on ambiguity.

a good engineer will get a 2x or 3x speedup without the downsides.

I wonder how different this is from having companies let Fortune or Inc magazine do their thinking for them.

Or random consultants.

Is "AI said it was a good idea" and worse than "we were following industry trends"?

Several people I know have already gone through phases like this. When you're doing it alone there is a moderating factor when their friends and family start calling them out on their behavior or weird things they say.

I can't imagine how bad it would be if your employer started doing this from the leadership. You'd be pressured to get on board or fear getting fired. Nobody would be trying to moderate your thinking except your coworkers who disagree with it, but those people are going to leave or be fired. If you want to keep your job, you have to play along.

> if you just prompt the AI and believe what it tell you then you have AI psychosis

This is the right definition. LLM outputs have undefined truth value. They’re mechanized Frankfurtian Bullshiters. Which can be valuable! If you have the tools or taste to filter the things that happen to be true from the rest of the dross.

However! We need a nicer word for it. Suggesting someone has “AI psychosis” feels a bit too impolitic.

Maybe we reclaim “toked out” from our misspent youths?

e.g. “This piece feels a little toked out. Let’s verify a few of Claude’s claims”

He uses AI himself, so I agree he doesn't see AI use as black/white.

Hard agree about ideas, thinking, advice. AI's sycophancy is a huge subtle problem. I've tried my best to create a system prompt to guard against this w/ Opus 4.7. It doesn't adhere to it 100% of the time and the longer the conversation goes, the worse the sycophancy gets (because the system instructions become weaker and weaker). I have to actively look for and guard against sycophancy whenever I chat w/ Opus 4.7.

> if you just prompt the AI and believe what it tell you then you have AI psychosis. You see this a lot with financial people and VC on twitter

I'm seeing it with lawyers, too. Like, about law. (Just not in their subject matter.) To the point that I had a lawyer using Perplexity to disagree with actual legal advice I got from a subject-matter expert.

I digress; this article actually has helped identify useful knowledge gaps around topics I have researched. https://drensin.medium.com/elephants-goldfish-and-the-new-go...

While you have to think about things objectively no matter what, when I start researching topics like physics, using AI as suggested in that article has proven very useful.

I didn’t think just offloading your thinking to AI was AI psychosis.

To me AI psychosis is the handful of friends I’ve had who have done things like have a full on mourning session when a model updates because they lost a friend/lover, the one guy who won’t speak to his family directly but has them talk to ChatGPT first and then has ChatGPT generate his response, or the two who are confident that they have discovered that physics and mathematics are incorrect and have discovered the truth of reality through their conversations with the models.

But language is a shared technology so maybe the term is being used for less egregious behavior than I was using it for.

I agree with you, except it isn't even good at writing code. Almost every time that you get an LLM to write a bunch of code for you, it has mistakes in it. The logic isn't right, the API calls aren't right, the syntax isn't right (!). That problem hasn't yet been fixed and it looks as though it never will be. That means that every line of code it generates, you have to review, because even if 95% of the code is correct, you need to find the 5% which isn't. But if you have to do that, it becomes slower than just writing the code yourself. As people have pointed out over and over again: typing in the code was never the part that took time. So I don't agree that LLMs are really useful for writing code.

> companies and people outsourcing their decision making and thinking to AI

It's so interesting how easy it is to steer the LLM's based on context to arriving at whatever conclusion you engineer out of it. They really are like improv actors, and the first rule of improv is "yes, and".

So part of the psychosis is when these people unknowingly steer their LLM into their own conclusions and biases, and then they get magnified and solidified. It's gonna end in disaster.

Yeah: If the tools aren't good enough and fast enough to fix the bugs before release, what makes anyone think they'll be able to so easily catch up afterwards?

Maybe they're assuming that doubling the code-base/features is more beneficial versus the damage from doubling the number of bugs... Well, at least for this quarter's news to investors...

Which is super fun as a user because every day something doesn’t work and it’s a different something than yesterday.

I was talking with a friend in the early days of AI boom. I argued that over-reliance in AI will create all kinds of catastrophes.

The answer I got is "It's game theory. Someone will do it, and you'll be forced to do it, too. It can't be that bad".

I mean, yes, logic is useful, but ignorance of risks? Assuming that moving blazingly fast and pulverizing things will result in good eventually?

This AI thing is not progressing well. I don't like this.

Yeah how do they know the fix doesn't have a bug and it will just keep deploying mire crap. What is the feedback loop, the customer?

If they’re so fast why not fix the bugs real quick before shipping

My prediction is that in the next year, we’ll start to see some dismantling of code review at some companies. It might take the form of “AI-only review,” or something similar, but many companies are getting frustrated with developers saying “no” to immediately merging slop they can barely understand.

the reality is my business continues to operate at higher efficiency, even with the bugs.

i don't think it's 'our side' that has the psychosis.

Literally the plot of Battlestar Galactica! Life imitates art indeed...

yes, I was never so happy to work in Germany. People used to joke about the proverbial fax machine still being a thing but I've never been so glad to work in a culture where this mania doesn't exist. Reading HN is like entering Alice's Wonderland of token maxxers and AI psychotics. Genuinely don't know a single person here who is forced to work like this.

Spoiler: it's not

Cynic!

> I use AI coding tools every day, but AI tools have no concept of the future. The selfish thinking that an engineer has when they think "If this breaks in prod, I won't be able to fix it. And they'll page me at 3AM" we've relied on to build stable systems.

Maybe it's just my prompt or something but my coding agent (Opus 4.7 based) says things like "this is the kind of thing that will blow up at 2am six months from now" all the time.

Add this the real possibility that significant part of reports that get filed might be AI generated or rewritten. With high possibility of being misreported because of that. Or have incorrect parts... So attack on multiple sides.

And we do not get even get into potential adversarial tactics. If you have no morals what is better than using agents to flood your competitor with fake bug reports.

Just let AI report the bugs. Problem solved!

I agree, and I'd like to point out that this problem isn't unique to AI driven projects. I think much, if not all, of what Mitchell has been observing can readily happen without AI in the mix.