Posted by Philpax 6 hours ago
If I was an investor I would be on the phone with Dario asking why we are wasting engineering talent on reinventing source control and churning the product with 30k pulls per day.
To be able to write PR pieces, like TFA.
The tradeoff between building something custom and adapting an existing solution is changing. Whether maintaining all that custom software turns out to be equally cheap is another question.
There is no doubt everyone is far more productive with LLMs in generating code. However, a lot of productivity is going into things like that. Have you always been bothered by the limitations of Grafana? You can generate another dashboarding solution that works best for you. You know that deployment team that is so slow and annoying? Now you can generate your own deployment engine. Have you always hated how kubectl lets you manage one k8s cluster at a time? Now you can ask AI to generate fleetctl. You can start your project with DynamoDB, rewrite it to mysql, then to postgres, then to duckdb. Start with Python, switch to Golang, then switch to Rust. Deploy on AWS, then move to Azure, then move to a hybrid option then throw colocation in the mix. Don't like your source control? create a new one. Json is too limiting? Invent a hybrid JSON/HCL language. The browser sandbox is limited? Make a desktop app, make a language, make an OS, make a kernel.
This is one of the core reason that I dislike the LLM hyping. People are just reinventing stuff and when you ask them, it's actually a very subjective reason. Ok, anyone can do what they want with their time, the issue is that they want people to follow in their steps.
Not saying that you need a novel idea to be worthy, but those random projects have no guarantees behind them other than some cursory testing. All software have bugs, but I can't interact with something that is randomly built.
I am one of those people that reinvents the wheels because external ones don't fit my use cases.
The fact I can now fork any software and do these changes I always wanted but had no time to is amazing to me.
I do not pretend to sell it or share it, as it is de facto void of any value whatsoever.
That’s OK, but I’ve yet to see someone articulating what exactly doesn’t work in the old product. It’s always “Let’s rewrite or reinvent 90% of it, then add a minuscule variation”. The variation may be interesting, but the reinvent is what’s make it untouchable as no one (including the author) has time to ensure things are right.
A patch or a plugin is ok, but not a whole new project with dubious claims.
There's so much intricacy and subtlety in each organization and even individuals' environment being incorporated into tooling that expecting these tools to generalize at this point is perhaps an expired ethos.
I even see this in the tooling that I run in my own personal stack. Many projects are candidates for sharing, and I have nothing against it, but they incorporate such specificity toward my own environment, including things like hardware, that creating shareable and more generalizable projects from them adds more to the maintenance surface than I would care to support.
Well, things are not that bleak, probably, at least not for our customers. Regulations (accountability) and physical world matters still require them. But not us for our customers, not that much and not for too long. They tend to be the tech savvy folks eager to jump at adjacent domains when it saves them a percentage or two off the construction costs (not a small pile of money), and AI with its programming language of English and deep knowledge pool is not that big a jump from their standpoint. Our domain is pretty well formulated, regulated, documented, inherently built on do-check-adjust loops already. With a thing like AI it might worth to try making and improving a design solution in a second loop. But they also started to invest in robots already driven by software for their factories, AI might jump in there, and I wonder how much more until the regulations are forced to change by Super Intelligent (R) politicians for the sake of AI accelerated built environments (housing crisis, you know!), shifting the AI use further out towards generic population, wanting to order a new two bathroom condo for next Friday from Alibaba. The conservative construction business is a nuisance and obstacle in the way of modernisations anyway, go around it!
I will go to work tomorrow, and for some time, but the picture above feel less and less fictious with each passing day of AI news and usage.
Software development is not as much de-risking at the front like it used to be before building.
Now teams can build lots of stuff that works great, except it's the wrong things.
Most things are becoming the understand the problem "shape".
Understanding the shape of the problem isn't something LLMs can always make as.. svelte as early on.
Word generators gotta generate..
That used to be called "time wasting". If you want to build software that will be used, you need to interview the users. Maybe a few things does require research to nail things right. But most software are user-facing, the research side is about optimization.
> Most things are becoming the understand the problem "shape".
Then should we be so reluctant about "trialing" invented solutions. It's the same thing with most AI projects, they're trying to justify the project after the facts while being very reluctant to get user insights.
The key, still, is shipping far less, too soon, to minimize cruft.
Vibe coding can be kind of a new type of cowboy coding. It could can be harnessed productively if done mindfully.
Interviewing users is critical, more importantly solving problems worth that people are lining up to pay to have solved before writing a line of code is critical.
Trialing solutions is always worth learning from, I usually ask about the problem that lead to building it. Sometimes that can be too narrow or vague of a story and you can learn how much of it is a solution looking for a problem, or vice versa.
It sounds like Claude loops on itself and must be pushing millions of lines of code changes per day. Even the humans can’t type commit fast enough. Are these people soon to be laid off?
So the code is for the stuff around the model, claude CLI, the website, the API... that probably shouldn't be infinitely growing right?
Is there a bunch of code they are using as part of training, or as pre-processing or post-processing the requests to the model?
It is even there on the blog post, doing more with a small team, followed right after by a hand waving paragraph how software engineers are still valuable, to avoid the expected backlash from the previous paragraph.
Touch encapsulates sex, love, massages, and other non-sexual touch that we have varying desires for.
Relationships encapsulates power, control, and politics along side love, humor, relating emotion, herd feelings, not feeling alone.
Machines will provide proxies for these things, but as far as they can be detected the human to human version will remain "superior" or more desirable.
Maybe far future generations will decide that the human version is suboptimal and they'd rather hug, have sex with, or talk to a robot.
Also in many countries it is a plain office worker job without the Silicon Valley wonders.
https://media2.giphy.com/media/v1.Y2lkPTc5MGI3NjExbmt6M2FkMW...
Input tokens, slightly process to categorize response approach token, hidden processing tokens, piece together some of the response tokens, then feeling pretty good about all the work done before output tokens.
More tokens than needed = more words/code = more complexity = more overhead = more snags = too much complexity to pay attention for the LLM to so overlook it and start drifting.
I'm kind of surprised I just explained one form of LLM drift to myself in a sentence.
While not following rules of writing.
[Yes] [Yes, and don't ask again]
> It sounds like Claude loops on itself and must be pushing millions of lines of code changes per day. Even the humans can’t type commit fast enough. Are these people soon to be laid off?
Claude will always need meat puppets, at least until it figures out how to automate owning stuff. And after that I'm sure it will want to keep some humans in zoos.
I read this and wonder what their agents have committed into the repo that people aren't looking at for _that much size_? Do they have a ton of binary assets in there or something??
This is really all just text.
Not sure they should be proud of these numbers, they’re a sign of dysfunction.
---
Engineers are no less valuable, but their work is changing. All of the engineers who worked on this system would now rate Claude as better at writing code than them. But our engineers still supplied the invaluable understanding of the high-level inter-system components and the design-level tradeoffs, making informed choices around the problems we wanted to solve and the scale we knew we needed to achieve. Very often, we found that the role of the engineers was to tell Claude “no.” Agents will sometimes violate core system constraints to satisfy a pattern that should never have been allowed in the first place. This was particularly common in our migration. For example, there were multiple systems that tried to brute force looking up when a change merged to main by linearly reading the commit log instead of asking our code review tool. Anthropic also makes extensive use of adversarial review, and, while this is incredibly effective in many ways, it can also be myopic, so skilled engineers needed to steer Claude away from over-engineering parts of the system in response to hypothetical concerns. For example, one such review was overly defensive about trusting another internal service, which resulted in Claude adding excessive validation and complexity. Our team members still review every change request for the system, but they now examine the intent of the change as opposed to reading every line of code.
Personally I can easily imagine one or two orders of magnitude more than that. They crossed the rubicon of not gating PRs on tests. In that land you can just roll out changes to production, a small subset of it, during code review to assess behavior. Or during key moments of coding.
I’ve been thinking a lot about this recently: what can we get if every change goes to production in 100ms, and what do we need to do to maintain stability?
> I am not sure however how easy it is to review 30k PRs/day even if only for intent, and how one can easily notice if the model is taking shortcuts on implementing said intent without reading the code...
Speed reading code is huge. You just need to focus on certain hotspots and know which things you're scanning for. The traditional approach of reading the code and trying to compile it or reason with it in your head is too slow.
I do think that the lgtm gut feel looking at a PR requires quite some years of writing code and reason about it, so not sure how a junior can get that intuitive understanding if they start with agentic coding however.
There's going to be an ugly phase where everybody FOMOs into thinking they need to do the same, only to realize that they're not Anthropic/Google/etc.
This amount of churn feels like a thing only possible with no real profit motives and a lot of money to waste.
Lots of activity is not necessarily synonymous with productivity, and it carries real cost.
So I think the design here is interesting, and the concepts seem applicable to where some agentic development may end up, but I question the underlying principles that drive this being the right solution space at all. At least if the goal is a profitable company.
In the early/mid 2010s there was a similar hysteria about "deployments per day/week" metric. People were comparing products or teams by saying "My team deploys twice a week", "oh, yeah? we deploy 3 times a day". People viewed the north star of the CD part in CI/CD as pushing to production. The concept was simple, you push your code to main. CIs start churning, deploying it to test environments, then promoting it to staging, the pre-prod, then into blue/green prod deployment, then into prod. All automated, all touchless, happening on every merge, 5-10 times a day. That's engineering efficiency.
Many tried to follow that. It works great if your project is a Jekyll blog of course.
Yet IMO the admin controls Claude provides just started to be adequate for a small company a few weeks ago (and even so are still ridiculously fragmented and seem fragily designed and not exhaustively documented), so I can't even imagine for a big company.
Do you need that rate of change to make basic controls? I'm not sure. Do we see such dramatic fast evolution in the products? Not really. Are bugs we report actually fixed? I tried once and it just auto closed.
Anthropic should optimise for something more interesting than number of merges per second.
> 25 million CI checkouts
833 checkouts for every merge? That's quite a few. Even accounting for multiple environments at different stages of production readiness, that's pretty weird.
And actually, the lack of an artifact server and some sort of proper knowledge of compatibility requiring a CI pull across hundreds of different projects with every merge is probably the explanation. Seems like there is an innovation waiting to happen there.
I also have a template repository I use to help with common setup for different project types so I don't need to repeat that every time I create a new project.
For Python it's easy to create an artifact server as the artifacts are just wheel files in a directory with the server listing the directories (packages), files (releases), and downloading the files.
I'm not sure about other artifact servers for other projects.
Who's beating Anthropic here lol