It's embarrassing to admit, but I tend to trust the research materials LLMs bring more than my colleagues or the programmer friends I once respected.

We put 2 + 2 in our calculator, we get 4. We've spent decades pushing computers as accurate. Making them accurate. We trust the machine and the process to give us the right answers when we put in the right data.

So when we disagree with the computer, we doubt ourselves.

I suppose I also grant it significant power to define me psychologically, and, in doing so, to understand what is happening to me.

> Even in Hackenburg’s recent preprint, Claude spouted numerous inaccuracies and falsehoods

> when Hackenburg ran his competition of coached elite debaters and AI, there was one way he could bring AI down to human levels of persuasiveness: by forcing it to write human-length messages at human writing speed.

> they found that an AI could talk people into conspiracy theories, and that the magnitude of their increase in belief was roughly the same as that of the decrease in belief after talking to a debunking bot.

None of this seems good.

I see not everyone is familiar with how social network debates happen. It also seems they are quite convincing, as politicians are employing AI bots extensively.

The biggest problem with using LLMs is that it prevents you from going _outside_ the distribution. It always flattens. It can be fixed but that's how it works today.

As an example, take something that the world converged on today that is incorrect and ChatGPT will agree with it. In a few years when society changes, chatgpt changes along with it. It doesn't do first principles analysis.

This may be (much) harder to do than generating rhetoric that is convincing to a large majority of people, perhaps so convincing that it can be classified as a superstimulus that bypasses any critical thinking in those that are especially vulnerable. It might not require anything like general intelligence.

If this is so, then it is the path they are going to take, not out of some malicious plan but as a simple matter of statistical probability. It's well known (especially in "AI alignment" circles) that any metric that can be gamed, will be. Falling over really fast vs. learning to walk, etc.

It feels to me like this is happening, and that many of those most exposed to LLM output display a literal inability, like some kind of blind spot, to notice the most blatant errors, and a complete conviction that those things have actual intelligence and even conciousness. And trying to argue them out of it can be like explaining the Monty Hall problem, some are just incapable of getting it.

And many of them have lots of money and power. IMO, that's the real risk, rather than sci-fi scenarios about perfectly simulating someone's brain in order to convince them to let the AI out so it can turn everything into paperclips. To do real damage, it only needs to be persuasive to some powerful people, most of the time. It does not need to have conscious intention, or planning, or even the rudiments of what one might call general intelligence. Just blind brute-force optimization for generating convincing bullshit.

:-D

Imagine this kind of method being deployed at scale to influence elections... no doubt it is already happening...