Posted by bucket2015 2 days ago
However, I just don't think that's realistic. It's asking an author to suddenly become an editor. It's asking somebody who writes code to now read and debug others code.
It can actually be harder to find the the bug in a tricky piece of code than it can be to write your own correct code from scratch. I see AI introduce all sorts of bugs all the time in my personal projects that I would never introduce, and would never think to test for, especially around anything graphical.
This has been a big part of the job for anyone on a team for at least 20 years. I do agree that it’s the hardest and worst part of the job, and has now become the majority of the job for anyone who isn’t vibe coding. So, that sucks.
The largest problem these days is the volume of code developers are expected to review. The volume went up significantly.
This is the best and worst thing about LLM coding agents. They trust comments way too implicitly. And then the errors just keep compounding.
Or a temporary hack that becomes "load-bearing" because the agent doesn't figure out that it's supposed to be a temporary testing shim - instead it keeps building on it until it basically duplicates what it's mocking.
By far the biggest problem 90% of developers have with AI is that they should be turning off comments, as it's clear that the training data they have is no good for developing a theory of mind for an engineer who has to read them.
I've turned them off and add them myself at review time and am quite happy.
Anybody who has reviewed pull requests can tell you that sooner or later you approve a PR after many rounds of changes because it's finally "good enough".
Fighting with a robot to just do the damned thing is less fraught because they don't get offended by critiques but it takes more round trips to get them pointed in the direction you want.
That's exactly right. Open source projects are currently drowning under LLM generated PRs, where those who used to write code are simply punting that work to AI, but still expecting others to review it. It's not okay to expect such a free lunch. If you moved the labour of writing code one step away, then you are yourself the first line of defence now, so you better start reviewing code that you claim to be yours.
I expect the same from colleagues, I'm not interested in treating them as a middle man between me and Claude.
Writing code has always involved reading and debugging your own code, at an absolute minimum, even if you did everything solo. In any remotely serious collaborative effort, it also involved code review and collaborative debugging; people use issue trackers and assign themselves and each other "tickets", which often involve fixing issues that are ultimately caused by someone else's code.
> It can actually be harder to find the the bug in a tricky piece of code than it can be to write your own correct code from scratch.
Part of the point is to reject tricky code exactly because it is tricky (as this is rarely actually necessary).
This is referred to in the need for E2E testing and E2E testing not being a substitute.
Code review is definitely the biggest challenge of AI-driven development IMO. I still have not found good processes that work in my org, but for my personal work I independently reached the author's conclusions a while ago and am very satisfied with the results.
I think that’s right, and what is needed. It still gives a significant speed up for coding, while still keeping the output human maintainable.
There is the idea that the agent will just produce binary code directly at some point. I don’t know if it ever comes to that but for now I’m in the ‘I’ve become an editor’ camp.
This way I don't need to scrutinize every detail, I just look over the big picture. I also care a lot more about the big picture - architecture and data flow etc. Basically if you view your codebase as a tree I care much more about the trunk and the big branches than I do about the smaller branches and particularly the leaves. So the details of some little leaf function somewhere are fairly insignificant, it's trivial to change at any time. As long as it works and isn't unreasonably slow it's fine.
Working this way I can get things done in minutes or hours that would previously take days or even weeks.
I'll bet you know it because you wrote and/or worked on it manually, likely over a period of years. The odds of you knowing a slop codebase that well, or even particularly at all, are much lower.
And prompted it can extract the black boxes if you tell it what the boxes are or what to look for.
Same for cleaning up tech debt after organic development, it's suggestions on how to simplify and modularize are good, but you need to prompt.
Given that the prompts are quite generic, "look for technical debt, suggest simpler architectures, what could be extracted in a separate module", it won't be long till it will do it on it's own.
It’s asking an author to suddenly become an editor if they decide to use the robot for a task.
Certain workplaces are demanding this - but not all.
Many still just want working commits without tech debt.
In fact, private and public teams alike are backed up at the PR review stage, so, lots of sane places wouldn’t mind individual contributors using the robot less - especially if its use increases the complexity of reviewing the task.
Speed isn’t the only variable to optimize for!
I think the bigger issue (like many things in software engineering) is a management issue. Once upon a time, I could take a look at the final output of a project and if it looked like a Ferrari on the outside, I could have some confidence that there was a good engine under the hood. OF COURSE THIS WASNT ALWAYS TRUE, but something that looked good, or was performant, or whatever, was a decent proxy for the code underneath being good. And with a smart human, there were ancillary things. Having spent 20 hours coding something, they probably thought through the edge cases that their manager, or product team hadn’t considered.
With AI, everyone’s output looks like a Ferrari, so it is hard to know what the internals are like.
A lot of people will probably look at this and say “well you need better management”, but better management has always been elusive in software engineering. Furthermore, reviewing AI generated code is soul crushing work and I don’t know who wants to do it.
In my guesstimate the number of good engineering managers out there is actually very very small and in practice, the best managers that I’ve seen are the ones who don’t think they are good managers, so they just set a very high hiring bar and hire people who don’t need much management.
Based on my experience, this is a significant issue with AI generated code. You wouldn't expect a real Ferrari supercar to have random internal mechanical components that are, for no reason, completely inappropriate for a high-speed car design. With AI generated code, such inappropriate components can appear randomly at any point in the implementation stack. And very often, they are deeply buried under non-trivial algorithms and are thus not easy to spot.
I’m doubtful that the author’s recommendation always work, but I do some similar things and they do seem to help.
I'd like to think the time and practice I've put into software engineering has made me better at it. If that's not true, then there's no reason to prefer senior or principal engineers with years of experience over newcomers.
Ultimately, I would take LLM code + high-quality multi-strategy testing over human code with little to no tests. And some would say "well that's a false dichotomy". I disagree, before LLMs engineers didn't have the time or incentives to aggressively test. The tests either would not exist, or would be shitty unit tests intended to get an arbitrary coverage percentage. Now, we can write high-quality tests, differential testing, fuzzing, and more, in much less time.
Another spectrum that I've found useful to explore is the scope of what I ask the coding agent to do in one turn. I see some people trying to do one massive prompt that the coding agent works on for a day or more. I find a large boost in overall quality if I do 10-20 prompts per day (not counting the prompts where I'm just trying to understand things). It's still much less of my time than hand-coding, but the resulting architecture looks like my own. The quality of the overall system is great. There are certainly issues here and there in the code, but it's always that way once a project gets large enough. Now it's easier to address any particular issue throughout the code base in one go.
The larger issue of good code isn't the actual individual lines, it's the overall architecture. And that's what I'm going to be reviewing first is, is this a good approach? Then the interfaces to other code is this a good interface. Get those two right and we can go back for the details. In a lot of cases, the LLM is plenty good at those details.
In some cases, an LLM is better than what I could do. Well, I suppose I can trace down all the locks in all the different special cases, and I have done that, but that was a huge amount of effort that I really don't want to repeat.
Note that I'm talking about recent models. If you're asking about the models of just one year ago, I would give a very different answer about the type of code an LLM produces.
Sorry, but you are 100% wrong.
I have 30+ years of professional development experience working on complex, very large-scale C++ code used by companies around the world.
I care deeply about code quality and always have. More than any other developer I've worked with in my 30+ year career. And I'm now using Claude Code to push the quality bar much higher.
But you have to learn how to use it properly. It's a tool. Quality doesn't happen automatically.
It's a big mistake, and frankly quite arrogant, to assume that because it doesn't work for you, it can't work for anyone else. Or that the rest of us must either be lying or incompetent.
However, if you’re willing to share how you’re prompting the AI and some examples of where you think the results are poor, we might be able to help identify what’s causing the difference. I’d be genuinely interested in understanding why we’re getting such different results.
Rather than assuming one of us must be wrong, it would be more useful to compare approaches and see what we can learn from each other.
You've missed the point. Nobody doubts writing code well or badly is indeed a skill issue.
The question is that "once you account for all of the things you need to do to make the code very high quality, did vibe coding actually provide any real value?"
I'm certain there are guardrails that help bolster vibe coding but I'm equally certain that when ive prompted something important I usually have to redo it enough times that just writing it manually myself usually would have been quicker.
Then I watch other people who code who dump on that opinion and I see total slop. They just can't tell the difference.
That seems to be the primary difference I’ve found between people who embrace gen code and those who dont
The ones who dont, seem to like the physical act of typing, and that tends to cluster with people who write software all day
Mechanically writing boilerplate is not enjoyable, and unfortunately in some languages and domains most of the coding is writing boilerplate. Machines can help with that no problem.
What is presumably enjoyable to most programmers is writing the parts where the actual magic happens. The translation of informal ideas into formal representation has beauty, like mathematics has beauty. Designing and implementing structures of code and data that are as simple as possible, but not simpler, is rewarded with a feeling of artisanal satisfaction and pride. Few things in life are as satisfactory as figuring out an elegant solution to a challenging problem.
None of the above are necessarily bound to the actual typing of words and symbols. AIs can help with all of them, and act as a genuine force multiplier. I would describe that as "responsible use of AI". Unfortunately, it seems that incentives are often against such use.
if you find the ratio between typing and thinking to be very high then you're probably producing a lot of slop.
This is a common theme I find when I hear about people's AI coding success stories. Where they say "its good at X" where X might be "backfilling unit tests" or "writing boilerplate" I usually think "if you find you need to do X a lot youre definitely doing programming wrong.
Ive actually yet to hear an X applied to production code that doesnt make me think that.
This is one of those things we value in theory in engineering but not in practice. Reducing code as an artifact might mean coming up with clever ways or compressing data, like making code that generalizes and abstracts. This is fine if you're experienced and clever. But a lot of organizations don't have that many clever or experienced engineers and those tools cause more harm in the hands of those people. Hence compromises must be reached and verbosity is valued because it is explicit.
I used to believe otherwise but then I worked in larger orgs with a lot of mediocre people who still provided value but needed to be given the means to add value.
That wasn't the case with 00s agile / Uncle Bob stuff since proving that any of it was helpful was impossible - you just had to believe (and if you didn't believe there was something wrong with you!).
Though arguably more of a process and judgement issue than skill.
What makes LLM-generated code a bit special there is that misjudging how to deal with it seems to be what most people do. So the default is broken.
Whereas in prior iterations of "skill issue", the default was working.
1. they don't care
2. the rest of the team doesn't care
3. the powers that be actively discourage it because velocity.
LLMs let you move faster.
But it's not as if introducing them is the only reason your codebase isn't high quality.
Every study I’ve seen correlates the use of AI with large increases in the number of bugs. Look at Amazon dialing back AI after massive outages. Microsoft patch Tuesday releases are bricking computers (they even managed to break notepad somehow). The rash of Facebook bugs also coincided with their move to AI. Leaks from Google have engineers saying AI either doesn’t save any time because it takes so to remote stuff or it causes breakages if they speed up.
These companies can afford to get the best devs. They have access to essentially unlimited token budgets. They have STILL fallen off a cliff in quality.
What more proof could there be that this isn’t sustainable?
Left to its own devices, that system will result in AI autophagy (model collapse) and iterative degradation.
The only area with serious uncertainty is how humans interact, but we now have research showing humans suffer cognitive issues very quickly using AI (some studies indicate effects happen in as little as 10 minutes) with cognitive surrender being a particularly big issue.
In a lot of systems, the only new data seems to be a few brainstorming sentences (you can read slop as entropy decaying things). The AI slops that into requirements. That slop feeds into an agent which generates a bunch of “reasoning” slop, maybe compacts everything (more slop), and spins up agents that get handed slop. They then open files with who knows how many generations of slop (maybe never even touched by a human) and write out a bunch more slop (it’s ironic that humans get better the more they edit a file, but AI gets worse). That slop gets “tested” by another agent reading all the other slop and maybe all this recurses a few generations.
At the end of this AI equivalent to “the human centipede”, you get a developer who’s handed 10x or maybe even 100x more code than their brain would possible process. They are suffering complete cognitive surrender (not to mention often reaching mental and maybe physical collapse from the workload and stress). They don’t understand the system and rubber stamp it so they can move on to the next 50 PRs of the day.
From start to finish, it’s 100% entropy outside a handful of lines worth of human input.
Many people predicted bugs and even discussed entropy issues before AI coding was popular. The buggy mess timing aligns not only with AI adoption, but happens to each company ramping up as they ramp up AI usage.
This is like seeing Einstein’s predictions happen, but arguing he can’t prove correlation/causation. What evidence would you actually accept that is feasible to study?
I think if you're on a team that cares about quality, LLMs can help you write quality code faster.
If you're on a team that's mindful about technical debt, you can have make practical trade-offs for velocity now at the expense of paying off technical debt later.
And if you're on a team that's unable to care about code quality ("I gotta merge this code now!"), then you can write mountains more code than you can understand.
You don't think crash will happen because XYZ. You _wish_ for the crash because you are hateful of progress that you are not part of.
Microsoft 2000
> You _wish_ for the crash because you are hateful of progress that you are not part of.
Facebook 2008
> You _wish_ for the crash because you are hateful of progress that you are not part of.
Cryptobros 2013
> You _wish_ for the crash because you are hateful of progress that you are not part of.
Altman/Dario/Musk 2020-onwards.
There might be a trend here...
Facebook has been shown to derange young minds and has been a nonstop firehose of disinformation into global public discourse.
Crypto moved an insane amount of wealth from the poor to the rich through and uncountable number of scams and empty promises.
These? These are what you call progress? Yes, the created wealth for a few individuals, but progress? No, my friend.
Really? That is quite a claim. You are talking about one of the worlds most successful software companies. What fact based research or large scale survey do you base that on?
But Facebook is extremely successful. Some crypto companies are very successful. I don’t hold any of these up as exemplars of the progress of humanity.
EDIT: My response to the GP, whose claim was that the only reason for the haters is that in each case they wanted the business to fail because they weren’t part of it, should been, no, actually there were at the time other and valid reasons detractors of those companies thought the way they did, and the same is true this time.
Over the last year, LLM coding agents gotten pretty good. It's no longer "if your results suck, you gotta try the latest and greatest model". You can get capable results on a wide variety of tasks, with a wide variety of models, used in a wide variety of ways.
I think where things get dicey is being able to write in any language. I write and review code in many languages and frameworks I'm not fluent in, so it's hard for me to distinguish between working code and great code. I can spot when the fundamental logic is wrong, but when it comes to "best fit" choices I'm clueless.
The thing is, if you follow SWE best practices indiscriminately, then you ll have a shit code base in no time.
There is no silver bullet, and no replacement for experience and mindfulness.
At work, with a team and code reviews, the 50%-100% figure seems much more likely.
This can probably move towards the more spectacular productivity gains as the AI's output becomes more reliable, people realize this, and less time is spend on code review and cleaning up the output.
This is something I have been trying to get right as well. I've attempted to use lots of linting and things like strong typing, duplicate checks, cyclomatic complexity, and robust tests. However, I still happen to find issues, which requires me to look at the code (at least at a high level)
For example, I can say "Don't repeat yourself, and don't re-write helper functions" and I will even have a duplicate linter check, but inevitably the LLM will always want to re-write a similar yet slightly different helper function. Like it will always want to re-write something small like a trim() or a toString() function in every file.
But if I add a separate post-implementation pass to "find and fix X" by the agent, it'll usually find and fix the issues.
So I've started doing it for everything from naming conventions to duplicate code to other problems. It does cost more tokens, but now I get less frustrated at having to fix basic issues in the PRs.
I have better luck telling it positive things rather than lots of "never do X" style things.
"Never re-write a helper function" vs "Always search for helper functions before writing one" the "never... " one doesn't tell the LLM what to do, so it would have to make the logical leap from not re-writing to knowing that it should search. While it's a minor leap to make in isolation, I suppose stacking many negative rules in an AGENTS.md would assume that every time it will always make that logical conclusion on what to do.
It's worth reflecting on why these things are important to you and whether they remain important in an agent-developed codebase.
When implementing new features or making large refactoring changes; I use the superpowers:brainstorming skill. That has consistent process which has worked really well. I alway review the code before merging, but most of the time there are few issues to correct.
I don't do 95% coverage, but I have increased it from 65% to about +80% and that is sufficient.
Thanks for adding such a thoughtful and level-headed comment to the discussion.
Kindly share the metrics by which you evaluate the changes.
It's a simple question that seems to make AI cheerleaders really mad.
"How are you measuring improvement."
I use AI for coding every day and see it fail all the time, I'm a seasoned developer and early tech adopter just like everyone else on HN. I'm still very skeptical because of how often these systems just miss. It's gambler's ruin on a very large scale, we remember the hits and forget the misses.
I don't expect AI to be perfect. Are people perfect? I leave room for corrections, but at this point they are usually minor and acceptable.
If this is true, then you are not saving a lot of time. Because most of the time is spent evaluating various options and ways to implement the functionality. Even when you are reviewing, you ll have to do that. (With LLMs, this is even more feasible, because now you can actually implement some of the variants, and evaluate them).
But on the other side, you are saving from typing the code. So if you are really reviewing everything, then you are not saving much time. The alternative is that you settle for some local maximum during each review, that in long term won't necessarly translate to a globlal maximum or even a global "good enough" position...
This is not a binary thing. It could be useful at the same time detrimental in some manner. Look at smartphones. I am just raising the possibilities if one use LLMs indiscriminately to generate code.
They can LLM themselves to complete lock in for all I care...
No one said they are not...look at my comment above in this thread!
Why are you so mad lol...
I’ve had a long discussion with a coworker on a long drive.
What we came to realize is the difference in our attitude towards writing code.
I approach it as craft. Even when I’m doing 100% of my coding with an agent these days. I still care about the result to be of high quality and maintainability. I still use my system design knowledge to guide the agent to produce scalable systems.
He treats it like just a job. If it’s good enough he ships. The edge cases and bugs don’t matter. Can be fixed later.
But in my mind that’s a fallacy. We all know things don’t get fixed later unless they are obvious defects and users complain.
Instead we get slow degradation of overall quality. All those small issues compound overtime to create a brittle systems that is difficult to debug and maintain.
My mental model of software engineering is like this. Each commit/PR is a small LEGO block. If you make them well they’ll snap well and create a stable structure that can withstand forces. If every LEGO block you make is just slightly off here and there. Your structure becomes unstable and will always have faults and will always have failures under unpredictable environmental pressures.
I'm happily vibing my own toy projects, but would prefer if the tech in hospitals is not vibe coded.
And I don't think it's plausible that the gap between those two is "well you just need to use it right".
But this has always been the gap between effective software engineering and garbage. When humans write software we put a large amount of effort in having best practices, hiring seniors with a track record, and enforcing process that empirically shows good results in reliability.
This is the same in AI. You need to have thorough code reviews by humans and agents, do a lot of manual QA, understand the tradeoffs when codebases grow, keep good documentation, keep bad comments out or anything that wastes the agents' context windows, etc.
The reality is that most people who produce mediocre code are mediocre users of AI, except that now they're empowered to produce crap 10 times faster and are too ignorant to distinguish between productivity and accelerated crap production.
If the compiler that I write produces lousy code, I get bugs that I fix until it doesn’t.
And that is the most annoying thing about this revolution. It’s obviously powerful and transformative and I use in my job all the time.
But many, perhaps even most, purveyors seem intent on blaming their users when they have issues, rather than fixing their own bugs.
General model improvement is going a long way here, but basic things like “ensure you use good style and programming practices” really shouldn’t be a thing users need to put in any .md file.
AIs are stochastic/probabilistic machines. Their big potential is in how they take malformed, incomplete, ambiguous inputs and come up with valuable and usable solutions.
Good defaults are expected in pretty much every other tool.
And “You just have to set it up carefully and properly” is pretty much saying that the defaults are never good enough.
They are already present in the harness.
In my opinion there is all kind of worthless advice going around, including skills or prompts, where the authors have never benchmarked them against clean runs.
That said, when you are dissatisfied with specific aspects, it can be beneficial to request them as a separate review stage.
Its lack of “quality” (always invoked in a metaphysical sense) isn’t a problem for most of its uses. It can automate, research, build boilerplate, and test way faster than a human.