Posted by Labo333 22 hours ago
Is it possible to expand this analysis beyond words to other Claude ticks? Contrastive framings, sentence length, caveating, for instance.
A prototype I did tried to detect some grammatical constructions, eg "it's not ..., it's ...", but I am not sure how to systematize that.
Also just a disclaimer: I am NOT tracking Claude tics, I am merely finding that a particular cluster of vocabulary increases. Tracking Claude requires labelled data IMO. I tried using model release dates in a structural model to constraint the clusters but the result was not compelling, so I ended up simplifying the model a lot!
So, yes, it's amusing to see clear Claude-isms like "load-bearing", "outright", and "genuine" in a [very nice] bit of analysis like this. And there's a (maybe negative? or not?) argument to be made about the world being filled with more Claude-isms or LLM-isms in general.
But I think the data say a second thing which is just as interesting and an absolute positive for the typical source code base. Look at the clusters that shrank significantly. Most of what you'll see in there is just incomprehensible...not even English. Cluster 4 has, after "pullrequest", a bunch of seeming usernames in the top tier. Cluster 6 seems to have names of repositories or tags in the top tier. Cluster 9 has branch names in it.
Meanwhile, keep going through cluster 1 and you'll see words I don't consider Claude-isms that really, really grow in usage. Words like "died", "nothing", "worse", "ever" all have well over 10x growth. This tells me something else. That the average commit log was BARELY ENGLISH. And then the LLMs came along and made commit logs that were ACTUALLY ENGLISH.
I count this as a good thing. I don't know the cross-section of repos chosen for this analysis, and I get it...some repos are garbage/throwaway, some commits come from automated processes that generate uninteresting commit logs, etc. But I've been benefitting from my work team's actually explanatory commit logs when doing code/bug archeology for decades, when doing PR review for the last decade, and I've even seen LLMs benefit from it in the last year (granted, not as often). A large part of professional software development is communication, and while the most important communication is via the code/comments, the commit logs are not unimportant. So, if this is making the average GitHub PR better (arguably more professional) by including actual English descriptions of code changes in commit logs...well, that's a genuinely load-bearing concept for me. :)