Posted by ianberdin 8 hours ago
This space can be increasingly avoided by becoming, and remaining, efficient and effective with prompts.
Regardless, it is cool to be able to contextualize the actual spend in terms of physical energy utilization. It even has a little co2 number (though again, kind of a "trust me bro" metric).
The main observation when you compare to token prices our input/cached energy is much lower than the equivalent token prices while the output energy is generally higher. One of the reasons this is a bit cheaper, especially for agentic systems, is that in an fully built/cached session your input & cached to output ratio is really massive input to very small output (multiple orders of magnitude usually). The other thing we do is we actively work to optimize the system around tokens/joule with the goal of making the most energy-optimal system.
Happy to answer any other questions.
How we drive AI will cause mileage variances, but over time the improved practices can measurably change.
I find my brain disengages once I suspect something of being written by an LLM. If the author didn't put much effort into writing it, should I expect them to have put much effort into fact-checking it?
Edit: this specific title has been deleted from the article. That was not my point! Please put in more effort into writing things that you want others to read! Rather than putting in low effort but being better at hiding it.
"Authoritative: it is the same count Anthropic bills against."
"This reframes a headline that looked like good news."
It’s also using a bazillion words to make a point that could be summed up in a single paragraph: there’s a huge variance in the number of tokens required to encode the same content, with code leading the charts.
To be fair, most of this was already known, and Anthropic communicated very clearly about the different tokenizer they started using.
Their compute is also mostly 1:1 correlated to the number of tokens, so I don’t believe in the conspiracy that this is just to inflate prices.
It’s a shame because it’s making an excellent point! It just takes so long to get to the point that the reader loses the will to live.
Yes, I could probably ask an LLM to summarise it for me. No, I’m not going to. I would prefer the author just take care of that for me.
I could live with ai content if it was short and to the point. But it's always so lengthy. Hope that will change.
A tl;dr section at the top and then the long read from ai could also be OK if they marked it.
My general vibe hearing about AI.
The nudge to think about both "tokenization as variable" as well as actual tokens consumed per task is still good.
The issue is that it’s not just code - they suck at writing. Really bad. Unreadable, incoherent, messy.
Humans are also bad at judging the quality of things they themselves aren’t very good at. So a senior swe sees what claude spits out and says “This is trash.” And spends x amount of time getting it to not be trash. And Jr dev thinks “this is magic!” And pushes it to a PR.
So my theory is the people “writing” this AI slop think its great! But actually just aren’t very good at writing copy and don’t have the skill to recognize it and prompt their way out of it.
Or they don’t care. That’s an option as well.
PS for anyone reading, next time AI does something that you aren’t super familiar with that looks pretty good… maybe find an expert to review it.
It is actually a big result of work, a lot of research and attempts. And to just say that "oh, this is AI-slop," I consider unfair, but that is your choice.
There is a difference: - There are people who do, - And there are those who criticize.
LLM speak is like the new corporate speak. Enterprise writing is fulll of fluff and nothings and they all read the same. That sameness is what most readers here are sick of.
(Your comment here that I replied to is also written by AI which is even more sad :| )
Weird shield to hide behind, considering there are also people who can do both.
One way or another, I want to note that yes, this text was made in collaboration with AI. My English is non-native. It helps me translate, helps me structure better. Yes, there is a downside, it can bloat the text with unnecessary words. But that, unfortunately, is the price.
But the key thing is that I tried very hard to share my many years of experience, or rather a part of it, which I acquired, with all of you. And I am very glad that this information turned out to be useful to you.
The key here is: * The information that is written in the article. * Not how it is written, but what I was trying to convey to you.
Thank you very much for reading and responding.
Chattiness remains an open issue for some of the SoTA open weights & (to a lesser extent) Claude.