Posted by tosh 3 hours ago
Prompts like "move the code relating to SQL query analysis into a new file", "look for opportunities to use pytest parametrize to remove duplication in that test", "rename method X to Y".
Early indications are that this is helping a lot with the problem where it's easy to churn out thousands of lines of code and not really have it stick in my head, even if I review every line of it.
Reviewing code and actively refactoring it is less tedious and more mentally engaging than reviewing code without changes.
If this was a human collaborator I'd be worried that I'm just creating busywork for them, but I don't care about busywork for LLMs!
The goal is to produce code that I understand and that I can remember just well enough that I get an updated mental model to help me productively make future decisions about the codebase.
There’s a lot of overlap there with the sorts of things traditional automated refactoring tools can do approximately instantly, locally, and for free.
1. Find the code you want to change
2. Run the tests to confirm that test coverage is good for the starting point
3. Track down everywhere else that might call or interact with that code
4. Update the tests (red/green TDD)
5. Alter the code
6. Update the things that call the code
7. Run the tests again
8. Apply linters/formatters
9. Address any feedback from linters
10. Check to see if any documentation needs updating and do that
11. Land a commit with a descriptive commit message
I can get all of that done with a coding agent with a single sentence prompt - especially if it's already in a session where it knows that I do "red/green TDD".
... and then I can work on something else while the agent is churning through those steps.
I guess the difference may be in people's mode of AI working: Do you primarily develop in your IDE or a bunch of terminals running vim, and occasionally fire up claude to do more complex things? Or do you primarily develop in a long-lasting claude terminal, and occasionally tab over to the IDE to watch/codereview? In other words: What dev tool is on your primary monitor and what's on your secondary monitor? It's getting hard for developers in one camp to discuss coding and see eye-to-eye with developers from the other camp.
There are a lot of small refactorings that I wouldn't consider to be worth 15 minutes of my time, so I wouldn't do them.
Outsourcing those to an agent means I don't have to make that tradeoff, which means I can get better quality code.
But yes, for a lot of my work I'm now a Claude Code / Codex first developer. I run Zed so I can navigate the code and occasionally make small edits.
Getting them to run ast-grep is really fun, especially when it saves me from having to memorize that syntax myself.
Mature workflows for those kinds of tasks have been mostly ubiquitous across professional-grade engineering tools like those from JetBrains or Visual Studio itself for longee than many people here have even been working in the trade.
It's clearly not the case for simonw, but much of what many people task AI tools to do foe them are only a novelty for the "VS Code"-type users who stubbornly refused to explore more professional-grade paid tools in the past.
Yet for many tasks, those mature paid tools provided reliable and efficient features that make the AI approach look like an expensive, slow, and dangerously nondeterministic regression.
I've never liked the larger IDEs - VS Code only won me over because it was indistinguishable from a lighter text editor at first, and the IDE tools then emerged slowly as I used it.
It’s almost like a buffer space would be useful for code.
I’ve been using tuicr for agent code reviews and have been enjoying that. I think I’ll try your idea as part of my workflow.
This is technically true, but lets not act like we haven't seen immense improvement of both models are harnesses for these models in the past years. They may not be learning, but they are getting better
As a recent example, I recently had to abandon the multiple LLM reviewer/verifier model I was using because zig 0.16 was released with major changes.
I actually reverted back to full self hosted because the foundation models we’re trying too hard to revert to the older versions of the language.
It is going to be a balancing act and there is fundamentally no way for LLMs to get around this.
We will have to develop methods to do so, most likely by focusing agents on problems that are more static.
I don't go along with their mitigations though.
In programming we have one tool for this: abstraction. Decomposition, pattern recognition, even data structures and algorithms are all down stream of abstraction. Collectively, we've never truly mastered abstraction, but it's what we have and we collectively wield it well enough that it's usually somewhat effective.
We are in dire need of a better abstraction.
And I'm not sure how this relates to TFA's point. Are you saying we collectively need to get better at abstraction so that LLMs get better at abstraction (either by training, or our prompting), so that their code is easier to read?
> Are you saying we collectively need to get better at abstraction so that LLMs get better at abstraction (either by training, or our prompting), so that their code is easier to read?
No - our current abstraction for coding agents is a loop where we express some freeform specification of a goal, then a sub loop kicks off where an llm takes a stab at what good looks like for the next step (make an edit, search for info, run a command to cause some side effect etc etc), it iterates in this loop and when it's finished its sub loop, it declares end of turn and the loop returns to the user for steering input.
That inner agent loop can make it quite hard to stay in control.
What if instead of only these low level free form prompts we additionally had some higher level primitives to work with?
We have chatbots in a sidebar that will just generate code for you or, more helpfully, answer your questions. We also have inline LLM code completion, which I've turned off completely because they're incredibly noisy.
What I want is something between those. My ideal use of LLMs while coding would be, i start writing a function and need to act on some data. I don't know what method to use, maybe I'm in an unfamiliar language/framework and don't know what my options are. I want the AI to explain what methods I can call to do X in this specific place, no more, no less. It would need to know what outcome I want, which would be hard to do without jumping out of the code and typing into the chat, but I basically want it to function like Intellisense on steroids. Something that doesn't break my focus.
Current LLMs are anti-flow. For me, that's poison.
I understand the rationale behind this, but can't help feeling that this is a downward spiral. The software industry has always been a hard place to build and sustain a career because of the pace of change. With these tools, the pressure to increase output is going to grow, jobs are going to be axed - so software devs need to work harder to stay relevant. Weren't these tools supposed to make our lives easier?!
But of course AI is also making union/worker pressure matter even less, since it's function is to cheapen the cost/leverage of workers.
So the only solution is fighting that at the political/legal/social level. Which I ain't see happening anytime soon.
Maybe, but to eat it you need to kill it first or it will stomp you. And this can only happen all at once :)
If there's this huge productivity boost what is it being spend on? I know, many have been laid off, but that's not universally true. So we have a productivity boost that doesn't really deliver anything and overall quality a lot of products/code/writing/communication is going down, yet we spend an ungodly amount of money on datacenters... for what, just spinning the wheels?
In a late stage capitalism market economy, their only actual requirements were to make profit for the shareholders and VCs.
If that means making our lives harder, firing most of us, making us stupider, being addictive, being used for surveillance to sell us shit or control us, or even being used to kill people, all of those are fine, if they fulfil that requirement.
I like to do the opposite, asking the LLM to give me relevant follow-up documentation, like the actually docs, where I can read and understand things myself. Data structures, techniques, etc. I still like to read that from the authors, much easier and trustworthy to grasp.
I think this is how we should be reading code as well.
First understand the top level. Then the next level of detail and so on. I treat my understanding as graph of interconnected black boxes. If I don't understand a particular black box or a node in the graph. I click expand on it, grok the details and then collapse the node. Here's the grokking details of a particular sub-node also follows the same structure as understanding the root node. You don't need to understand everything from the get-go, expand your understanding on the need-to-know basis.
I'm really trying to do this too. The problem is it's *so easy* to let your standards slip, even for just a moment, and that piece of code suddenly becomes foreign.
I find more mental energy is spent on restraint than execution these days.
Second this. This is why zuckerberg is dying to spend as much as he can to make meta an AI company.