You seem to imply that its outputs aren't found by people to be useful, which isn't true.

A badly ridden horse mostly produces manure. A well-ridden one gets you somewhere

Yes. There are leaderboards or evals or something.

Yeah, well... not really.

I used to tell my Into-to-Programming-in-C course students, 20 years ago, that they could in principle skip one or two of the homework assignments; and that some students even manage to outsmart us and submit copied work as homework, but - they would just not become able to program if they don't do their homework themselves. "If you want to be able to write software code you have to exercise writing code. It's just that simple and there's no getting around it."

Of course not every discipline is the same. But I can also tell you that if you want to know, say, history - you have to memorize accounts and aspects and highlights of historical periods and processes, and recount them yourself, and check that you got things right. If "the AI" does this for you, then maybe it knows history but you don't.

And that is the point of homework (if it's voluntary of course).

I hear the cool companies offer free ponyium to their employees. Apparently, it works wonders for morale

Through many attempts to make ingesting the ponyium more bearable, I’ve found that taking it with more intense flavors (wintergreen mint, hoppy hops, crushed soul, dark roast coffee, etc) improves its comestabilty. Can’t let it pile up. We’ve always eaten ponyium right, and we all like it, right, guys, folks?

you forgot to write pAIn and it reminded me of this: https://youtube.com/watch?v=nt9mRDa0nrc

>LLMs do not have that at all so the analogy fails.

I somewhat disagree with this. AI doesn't have to worry about any kind of physical danger to itself, so it's not going to have any evolutionary function around that. If the linked Reddit thread is to be believed AI does have awareness of information hazards and attempts to rationalize around them.

https://old.reddit.com/r/singularity/comments/1qjx26b/gemini...

>Horses can survive on their own.

Eh, this is getting pretty close to a type of binary thinking that breaks down under scrutiny. If, for example, we take any kind of selectively bred animal that requires human care for it's continued survival, does this somehow make said animal "improved automation"?