Top
Best
New

Posted by wek 2 hours ago

Speed at the cost of quality: Study of use of Cursor AI in open source projects(arxiv.org)
52 points | 23 comments
duendefm 1 minute ago|
AI is not perfect sure, one has to know how to use it. But this study is already flawed since models improved a lot since the beginning of 2026.
matt_heimer 1 hour ago||
Yes, it's not surprising that warnings and complexity increased at a higher rate when paired with increased velocity. Increased velocity == increased lines of code.

Does the study normalize velocity between the groups by adjusting the timeframes so that we could tell if complexity and warnings increased at a greater rate per line of code added in the AI group?

I suspect it would, since I've had to simplify AI generated code on several occasions but right now the study just seems to say that the larger a code base grows the more complex it gets which is obvious.

AstroBen 1 hour ago||
"Notably, increases in codebase size are a major determinant of increases in static analysis warnings and code complexity, and absorb most variance in the two outcome variables. However, even with strong controls for codebase size dynamics, the adoption of Cursor still has a significant effect on code complexity, leading to a 9% baseline increase on average compared to projects in similar dynamics but not using Cursor."
ex-aws-dude 1 hour ago||
That was my thought as well, because obviously complexity increases when a project grows regardless of AI
bensyverson 1 hour ago|||
Yeah, I have a more complex project I'm working on with Claude, but it's not that Claude is making it more complex; it's just that it's so complex I wouldn't attempt it without Claude.
rfw300 2 hours ago||
Super interesting study. One curious thing I've noticed is that coding agents tend to increase the code complexity of a project, but simultaneously massively reduce the cost of that code complexity.

If a module becomes unsustainably complex, I can ask Claude questions about it, have it write tests and scripts that empirically demonstrate the code's behavior, and worse comes to worst, rip out that code entirely and replace it with something better in a fraction of the time it used to take.

That's not to say complexity isn't bad anymore—the paper's findings on diminishing returns on velocity seem well-grounded and plausible. But while the newest (post-Nov. 2025) models often make inadvisable design decisions, they rarely do things that are outright wrong or hallucinated anymore. That makes them much more useful for cleaning up old messes.

AlexandrB 1 minute ago||
> Super interesting study. One curious thing I've noticed is that coding agents tend to increase the code complexity of a project, but simultaneously massively reduce the cost of that code complexity.

This is the same pattern I observed with IDEs. Autocomplete and being able to jump to a definition means spaghetti code can be successfully navigated so there's no "natural" barrier to writing spaghetti code.

joshribakoff 1 hour ago|||
Bad code has real world consequences. Its not limited to having to rewrite it. The cost might also include sanctions, lost users, attrition, and other negative consequences you don’t just measure in dev hours
SR2Z 1 hour ago|||
Right, but that cost is also incurred by human-written code that happens to have bugs.

In theory experienced humans introduce less bugs. That sounds reasonable and believable, but anyone who's ever been paid to write software knows that finding reliable humans is not an easy task unless you're at a large established company.

MeetingsBrowser 1 hour ago|||
The question then becomes, can LLMs generate code close to the same quality as professionals.

In my experience, they are not even close.

mathgeek 31 minutes ago||
We should qualify that kind of statement, as it’s valuable to define just what percentile of “professional developers” the quality falls into. It will likely never replace p90 developers for example, but it’s better than somewhere between there and p10. Arbitrary numbers for examples.
verdverm 1 hour ago|||
There was a recent study posted here that showed AI introduces regressions at an alarming rate, all but one above 50%, which indicates they spend a lot of time fixing their own mistakes. You've probably seen them doing this kind of thing, making one change that breaks another, going and adjusting that thing, not realizing that's making things worse.
MeetingsBrowser 1 hour ago|||
This only helps if you notice the code is bad. Especially in overlay complex code, you have to really be paying attention to notice when a subtle invariant is broken, edge case missed, etc.

Its the same reason a junior + senior engineer is about as fast as a senior + 100 junior engineers. The senior's review time becomes the bottleneck and does not scale.

And even with the latest models and tooling, the quality of the code is below what I expect from a junior. But you sure can get it fast.

i_love_retros 14 minutes ago||
> have it write tests

Just make sure it hasn't mocked so many things that nothing is actually being tested. Which I've witnessed.

mentalgear 48 minutes ago||
> We find that the adoption of Cursor leads to a statistically significant, large, but transient increase in project-level development velocity, along with a substantial and persistent increase in static analysis warnings and code complexity. Further panel generalized-method-of-moments estimation reveals that increases in static analysis warnings and code complexity are major factors driving long-term velocity slowdown. Our study identifies quality assurance as a major bottleneck for early Cursor adopters and calls for it to be a first-class citizen in the design of agentic AI coding tools and AI-driven workflows.

So overall seems like the pros and cons of "AI vibe coding" just cancel themselves out.

dalemhurley 25 minutes ago||
I think the issue is people AI assisted code, test then commit.

Traditional software dev would be build, test, refactor, commit.

Even the Clean Coder recommends starting with messy code then tidying it up.

We just need to apply traditional methods to AI assisted coding.

AstroBen 1 hour ago||
They're measuring development speed through lines of code. To show that's true they'd need to first show that AI and humans use the same number of lines to solve the same problem. That hasn't been my experience at all. AI is incredibly verbose.

Then there's the question of if LoC is a reliable proxy for velocity at all? The common belief amongst developers is that it's not.

PeterStuer 1 hour ago||
Interesting from an historical perspective. But data from 4/2025? Might as well have been last century.
happycube 1 hour ago|
I think the gist of it still applies to even Claude Code w/Opus 4.6.

It's basically outsourcing to mediocre programmers - albeit very fast ones with near-infinite patience and little to no ego.

Miraste 1 hour ago|||
It doesn't map well to a mediocre human programmer, I think. It operates in a much more jagged world between superhuman, and inhuman stupidity.
chris_money202 44 minutes ago|
Now someone do a research study where a summary of this research paper is in the AGENTS.md and let’s see if the overall outcomes are better
More comments...