My take: I don't think recursive self-improvement (RSI) will happen with current LLM architectures, even if they become 1000x "smarter".

RSI requires more than being better at solving problems. An AI needs to decide what is worth improving, come up with genuinely new ideas, and decide which ideas are worth pursuing.

Current LLMs are still weak at two of these: judgement (what matters) and new ideas. They're very good at combining and developing existing ideas, but that's different from independently finding important new directions.

That's also why I don't think most knowledge workers can simply be replaced by LLMs. Many tasks can be automated, but the most important ones can't.

I am getting a stronger and stronger sense that ai REALY is a sex/drug cult, that not long ago would have coalesed around a band or guru, but has somehow formed around the mad concept of conjuring a tame god, this last unspoken, the tame bit, but it's just people hacking everything, and just like all cults it's built on hype, money, and influence, and the eternal scam that if you do as we say, god, unlike the band, will take requests.

Can someone explain why RSI is that critical? At best it is replacing a few 1000 engineers. So what? More than RSI I’m interested in whether there exists a ceiling in intelligence. My intuition says that because we live in a human world with human data, there’s a tendency for AI to be at human scale of intelligence. For example I see context window of 1 million which is suspiciously similar OOM to our own. Why not 1000 million??

the crucial point therefore should be whether our techniques can scale intelligence beyond the scale of humans while being constrained to human level data and techniques.