Just a couple of days ago, after having heard that there's a new Mersenne prime, I had a question pop into my mind: what if it's a twin prime? Is that even possible?

I could try to calculate that on my own, but since it's such a large number, the potential twins are going to take forever to calculate, so I figure I could start with a quick Google search.

And the "AI Overview" feature said it was impossible, because a Mersenne Prime is of the form "2^p - 1" (where p is prime) and thus adding 2 to it will result in "2^p", which is necessarily even.

And then a couple of search results later, there was a post listing Mersenne primes with twins. There are four known, and it's doubtful that there will be more (which is what I expect!) -- but I am astounded by how wrong the AI Overview was on this!

And this was for a mathematical question, where it can be relatively easy to see what went wrong. I would hate to see this kind of mistake show up in health care diagnostics!

Well, I'm willing to make one exception: if the doctor makes the diagnosis first, and then checks it with an AI model, to see if the model catches anything they missed, I'm ok with that. That's probably the "proper" way to use AI in general. If we use AI to replace human competence and knowledge, that is a path that will lead to disaster!

And that's true even if, or worse, especially if AI never actually gets the sentience needed to enslave and/or destroy humanity.

I feel like the solution to the paperwork issue is to improve the requirements for paperwork rather than getting an LLM to write it. You can only ever lose data from what was input to the LLM, so this is a strictly destructive process. Then an "AI-powered clinical support system" sounds positively dystopian.