Tokens too cheap to meter

Posted by teoruiz 13 hours ago

203 points | 163 commentspage 2

mrinterweb 3 hours ago|

If inference continues toward the trend of becoming a commodity, and inference hardware efficiency is doubling every two years, that will likely mean that the price of inference will continue to fall so long as the market is competitive. I don't see how companies investing billions in inference hardware today will see profit in the lifespan of the hardware.

rwolf 9 hours ago||

for the first chart (sourced from https://epoch.ai/data/machine-learning-hardware?view=graph&y...), what is the audience supposed to think about that trend line? there's a step after you slap a regression on some points where you evaluate whether there's a real trend or noise, right? i don't see that either in the article or the linked source.

vb-8448 6 hours ago||

I agree with OP that we will continue to see improvements, but there are also some serious bottlenecks ahead of us:

- Energy is not infinite, neither energy efficiency is. - Datacentres neither. - Benchmarks are an abstraction of real world problems!

On top, there is an overall "economic" aspect that most of the people miss: every change carries a certain degree of risk (lose money, reputation, customers, death of people, ecc) that very few want to take and a lot of changes(e.g. rewrite some piece of SW in another Lang) don't produce a positive economic impact.

codedokode 6 hours ago||

> We are likely to see LLMs integrated into every part of computing as infrastructure, not just as a product, in the next year or two.

Of course, for collecting better telemetry using local AI for analyzing video from camera and audio from a microphone.

eltokh7 6 hours ago||

jgrep is not meant to replace grep. It’s supposed to extend it to new use cases. Keyword search will always have a place!

bigbuppo 5 hours ago||

What should I do with this old Juicero that's taking up space on my kitchen counter?

mwkaufma 3 hours ago||

Magical thinking intensifies.

hermitcrab 9 hours ago|

It is hard to see how the environmental side effects of this aren't going to be somewhere between bad and disastrous.

cestith 4 hours ago||

If we're all using distilled open-weight models in ASICs in our own systems the energy cost will come way down. The question is when that becomes a reasonable solution for a broad set of use cases.

empath75 8 hours ago|||

I think this is a case where just drawing a "line goes up" extrapolation is incredibly misleading because there is _tremendous_ economic pressure to get costs down, and costs are very tightly tied to energy use. All of these systems are incredibly inefficient right now and have a lot of room to go down in energy use. I'd guess that the absolute _floor_ is burning model weights directly to silicon and that's like a 90+% reduction in energy use.

xienze 9 hours ago||

No it'll be fine as long as you do your part and not drive a car, or have AC, or eat meat, or have children, or live in detached housing, or...

More comments...