Top
Best
New

Posted by hruvhwe 9 hours ago

Models Are Getting Dumber on Purpose(w4g1.dev)
281 points | 158 commentspage 2
themgt 7 hours ago|
You can look at the benchmark and the GPT-5 failures like answering "April 22, 2019" instead of the correct "Oct 23, 2018" for the question:

What day, month, and year was Carrie Underwood's album "Cry Pretty" certified Gold by the RIAA?

If your idea of the smartest person in the world is the guy who always wins tuesday night pub trivia, this blog post is for you. It also gets it's foundational factual claim wrong (as seen via epoch.ai). Very on brand.

https://epoch.ai/benchmarks/simple-qa-verified?view=graph&ta...

https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?lo...

Groxx 7 hours ago|
Not sure what you're trying to claim here tbh - Oct 23, 2018 is what the answer checker is looking for, it correctly judged "April 22, 2019." as incorrect: https://logs.epoch.ai/inspect-viewer/c79c08da/viewer.html?lo... (row 10, I don't see a way to link directly) and the correct answer matches the Wikipedia article's claim: https://en.wikipedia.org/wiki/Cry_Pretty and the RIAA's site it uses as a citation: https://www.riaa.com/gold-platinum/?reload=1786913260239&tab...
Gecko4072 8 hours ago||
With all the focus on coding and agentic use, I wonder if the rest of the world will notice or care? Most AI use is not for coding or harnesses contrary to what this website thinks. Maybe this is how American companies stay winning.

For example I prefer Kimi K2.6 1T parameter to Flash V4 0731 230B parameter, even if it is less intelligent.

kennywinker 8 hours ago||
What is the rest of the world using LLMs for? Agentic stuff seems pretty universal. Obviously coding stuff is only interesting to people who want to code, but automating complex digital tasks seems useful in all kinds of contexts.
Espressosaurus 8 hours ago|||
I’m working at a company where everyone is using LLMs for everything and I’m not aware of anybody using anything truly genetic. It’s all human in the loop babysitting the tool.
dghlsakjg 7 hours ago||
What tool?

Even basic clients are now harnesses. A lot of chat interfaces are using memory systems, web search and other stuff under the hood.

Not as agentic as openclaw, but not a straight closed conversation either.

Gecko4072 8 hours ago|||
Learning new things for example, general curiosities, text processing
ismael_rr 8 hours ago||
I agree that most AI use in terms of users may not be for coding or agentic use (everyday people are asking chatgpt for something or looking at google ai summary), but with respect to AI usage, I speculate that the vast majority of usage is coding and agentic because they're super token hungry.

In terms of the value proposition of AI replacing knowledge workers, all value is in coding agents (coding agents as general agents).

gagan2020 8 hours ago||
Intelligence vs Knowledge

LLMs work is being intelligent not having knowledge of everything is ok. But, they have to be intelligent enough (with some degree of knowledge) that where to find the information (search tools or any other tools for that matter)

hajile 8 hours ago||
Not too long ago, the claim was that having multiple Phds worth of knowledge was the real value of AI, but now that's gone and it's suddenly a good thing.

That's a sales tactic -- not a logical position.

lowsong 7 hours ago||
LLMs are not intelligent. Please don't anthropomorphize.
kennywinker 6 hours ago|||
As the Portal companion cube taught us, we will anthropomorphize whatever the hell we want.
gagan2020 7 hours ago||||
I know they are just next token predictors. However, Our learning also started with that only. They moved from Alphabets to sentences. Intelligence is what AGI stands for and that's still in distant future. But, we need to push them for AGI. That's the ultimate goal of humanity for now. Finger Crossed for AGI in our lifetime.

After that Space colonization will come.

williamcotton 7 hours ago|||
At least later-era Wittgenstein would disagree with you.
RGS1811 7 hours ago||
Reasoning is not separable from the particulars of a specific language game / linguistic practice. I’m sure some of the things the author predicts will happen, but the idea of some sort of abstractly perfect reasoner separated from the semantic content of language gets fundamentally wrong what reasoning is.

Edit: I ran this article through pangram and it is “100% AI generated”. Cool.

xena 7 hours ago||
This article is AI output.
PaulStatezny 7 hours ago|
Which is totally ironic, given that the article is largely about how untrustworthy LLMs can be. (Hallucination.)

Putting readers through this exercise disrespects their time. Even if as a writer you did the work of researching, reasoning, and fact-checking, you shoot yourself in the foot by running it through an LLM because there's no way for the reader to know which thoughts/research are from you. It demolishes the Ethos of the writing; readers feel they must do quality assurance on the reasoning, research, and facts.

andai 6 hours ago||
I wonder if Gemini tops QA due to Google Books in training data? You don't need to pirate LibGen if you started scanning all the books decades ago.

(In retrospect it looks like they were pretty forward thinking!)

mips_avatar 8 hours ago||
I think the biggest problem with the models is they don’t actually have any decent lookups except chunked document embedding search
SubiculumCode 7 hours ago|
In human and development, there is a transition from remembering things verbatim to remembering using semantic concepts. The ideas are getting bigger and more useful. The details are still in the library where the fidelity is better.
More comments...