Posted by yedhukrishnan 5 hours ago
Doesn't the original conversation at least retain the context about why the change was made? A different LLM literally has no way to tell why you made this change besides guessing from the codebase and git history.
I believe they can do this due to having millions of public pull requests and github issues in their training data.
* business objectives,
* the result of experimentation,
* the result of an offline conversation
etc.
This cannot be captured in code alone. This is why we write commit messages.
I did this for months without anyone noticing, and eventually my manager schedules a very awkward meeting asking me why I wrote saying “cfquery fucking blows sometimes”. I had to sheepishly explain that I thought it was funny and then I stopped doing that and my commits became much more utilitarian and much less fun.
Happy to read good reasoning why it's fucking blow-up.
So I've been instructing Claude to commit like Jeff King.
At first, Claude would mainly just cosplay Peff. Emulate the prose and not the process. Over the last few months I've been iterating on it and now Claude writes vastly better commit message than by default.
Initially, Claude would produce A LOT of plausible sounding reasons the LLM "thought" made sense. Instruct an LLM to give reason and it'll give you reasons -- whether they are real or not. After trying to instruct it not to lie, make shit up etc (which did not work) I instead started forcing it to articulate the source of the rationales. Especially which claims where unsubstantiated, and this seems to have helped a lot.
Then I instructed it to do some thorough investigation before it commits.
Start by writing a brief that gathers different "evidence" that underpins a change. The diff itself. The surrounding context. A bit short git log. A blame on the touched lines to see what previous commits touched this code and for what reasons.
Once it's done the agent has to tag each claim according to a category. I.e. what claims are attributed to the change itself (the diff), the inciting incident (gathered from session or if missing, by follow-up questions), what's inferred by the model (unsubstantiated claims.)
Only after this supersize is it tasked with writing a commit message given this brief. Or to ask follow-up questions if there's only unsubstantiated claims or gaps in the brief. Furthermore, it is tasked with writing a note to detail assumptions it has made and, or other relevant bits of information that are not commit message worthy, but possibly still interested in noting down. Decisions made. Options not taken. Possible rationales for the change that didn't make the cut.
All of this tends to make pretty good commit messages. Not perfect, but a good starting point.
Right now my biggest challenge is finding instructions to write the Goldilocks message. Not too brief and not too long. Instruct it to be clear and concise and relevant information gets left out. Say nothing and get a Dostoevsky novel. At least when it writes too long messages it's easy enough to go in afterwards with a `git history reword` and take out the axe.
One of the biggest upsides has been, just as when you read a human that writes commit messages like this, is spotting misunderstandings. Several times I've spotted gaps in the reasoning of the message that doesn't match reality, and caught mistakes. A bit like when you use plan mode.
The journaling of any iterative process requires clear notes that answer "why?" for each step. This is what will guide future maintenance.
Writing code faster than you can digest and explain it is at odds with this. You will incur runaway technical debt. This was already a problem long before the LLM era.
It is nice that more people are finally realizing this, but I'm still waiting for when we start speaking in generalities again and get over all the hype. Nothing ages writing faster than bringing up the specific tools.
I'll ask AI to write integration between 3 services, and it'll write the class names from the service A in comments in service B and C if I'm not careful.
No, we are not. Sure, there is a lot of slop-coding/vibe-coding going on, but not much of it in serious code. In my experience and to my knowledge.
Of course, I encounter the opposite problem with humans: They often don't bother to write proper commit messages; and many tend to squash them in favor of giant single-commits which just say "Implemented feature #123".