Posted by dimonomid 14 hours ago
Is it a fact though? Maybe my projects aren't ambitious enough, but about 6 months ago I stopped hitting the point where AI can't maintain what it has written. It's probably unmaintainable by humans, but that might be an increasingly irrelevant quality.
> The proficient developers, the experts, rely on their intuition built with sweat and tears, working long hours trying to debug and fix production issues, swearing to never again be so foolish as to repeat past mistakes. It’s the kind of intuition that can’t really be made into a list of rigid rules, because everything is context-dependent. Experts are incompatible with the same rules and recipes that make beginners more productive. Experts don’t follow the rules, they make the rules.
.... because I have found the same thing - that there's nobody more zealous about some paradigm than those who are recently converted to it and who haven't come to find that everything has its trade-offs. Design is always about evaluating the trade-offs and seeing which ones most suit the given situation.
We're a few years into a new technology that is still improving. This is a point-in-time critique.
My belief is that paying off tech debt requires a better model than creating it. At the same time there are people who will create tech debt no matter the tool.
Should the models stop improving, the debt will pile up.
Alternatively stated: codebases will expand to the limit of an organization's ability to manage them, so the equilibrium will remain at the point of near, but not total, incomprehensibility.
It was mostly 256k, then it went to 1M and now it has stalled there.
Meanwhile, the people who work on actual products that matter have a valid criticism that can’t be dismissed with “some humans don’t even manage to do that.”
Probably a vast majority of software written never actually gets interacted with by a user. A vast majority of VC-funded software probably never gets interacted with by a user.
Feels pretty great to work on a product that actually has a user, and that user isn’t actually the product (and that user isn’t being manipulated or exploited). Tiny, tiny minority of the software industry I work in.
If you don’t think about it as a per-software criteria, but per-feature criteria, the vast majority of software features developed are probably completely ignored or even untouched compared to the “critical few”.
Very, very long tail distribution. The most executed 100 lines of code (maybe some Java class init snippet, or some Linux kernel snippet) is probably executed more per day than the per-day bottom 80% of all code combined. Maybe the bottom 99.9% of all code combined.
[1] There is probably a more fitting word, I am just reusing incompetence here, but that is not a really fitting description, I think. I would maybe say carelessness or something like that, but a single word is not going to capture the issue accurately.
But the point OP was trying to make is that because LLM harnesses output code that is at least better than the worst, say, 20% of developers then we should be fine with it. Meanwhile, I’ve worked in shops where you have to be way above the worst 20% in order to keep your job or even be hired in the first place. And it isn’t some crazy lose-sleep-over-it, stressful requirement. It’s just that literally if you’re not good enough your work will be identified as a liability and you’ll be let go. It’s only happened to 3 people fwiw.
LLMs commit crap, and read the "pattern" back, and consider it as gospel and repeat it all across the code base...
Generally though, you are also investing your time into leveling up junior engineers to take over responsibilities from you. I just never really see that happening with AI. Even as it gets "better" technically, there's no real growth pattern to its work and it doesn't understand ownership or responsibility.
But if scaling isn't a problem, then sure just write it yourself.
It doesn’t take much effort to setup cross-agent reviews and automatic reviews for slop and accretion, while directing design decision questions back to the human to consider. I have had a considerable increase in throughput of code that I designed and made the important decisions about, and that I’m pleased with the quality of, although as always in these discussions, someone will be a long shortly to tell me that that implies I must be a terrible engineer.
For starters, LLM’s need to stop being our friends. But that won’t happen because the dopamine loop is baked in on purpose.
Mostly stalled now...
If the models are as capable in a years time as they are today you could say they have stalled.
And the observation that it's happening slower than it was means, not that it's stalled, but at least that it's not exponential anymore. The exponential phase is already over. That has implications for where we think the plateau is likely to be.
it just doesn't happend, too much time pressure, and low-hanging fruits must be had, money must be made, and so it goes... that begs the question, if most enterprise software and/or game projects are an unmaintainable mess, yet they still make money and "work" somehow, whats the value in "beautiful" projects/architectures/code over time, if it provably works either way?
They are often too verbose, or miss capturing an important concept or purpose, or add bullets for parts of the changes that no one cares about.
Most code in general is bad by some metric, and likely many metrics. Let’s not pretend that closed-source, in-house code is better.
> There is no fitness function you can define for maintainable code, at least not one that we can discern, otherwise it would’ve been baked into our linters.
This is emotionally appealing because it tells me that my judgement is irreplaceable. But it’s also essentially an appeal to magic. A metric than cannot be defined is either not real or is a matter of taste. And maintainability cannot be a question of taste because it makes concrete claims about the software, not just human perception of it.
All this is to say that we have yet to establish that “vibe-coded projects devolve over time into an unmaintainable mess” is a fact, or that it’s more true of vibe coded projects than hand coded projects. We are still learning how to build the right guard rails on vibe coded software because previously we relied a lot on humans reasoning over the code and saying “this looks about right” which frankly is not a strong engineering practice to start with.
You, as an experienced engineer, are doing a lot of hand holding and review of LLM-generated output, maybe even(?) using it as purely a check on your own work. There are others, though, that are essentially outsourcing the entire process to a basic, underspecified chat prompt.
https://github.com/ityonemo/bpa
For my projects that I use day to day in prod there is considerably more hand-holding and code review.
I suppose it'll be a few years before we see the true volume of technical debt catch up with the worst offenders, but even then, without the original LLM conversations associated with it, it'll be difficult to assess that in a structured way. Mind you, that doesn't even address the ever-improving models.