Patients are guilted into allowing the doctors to use it. I have gotten pushback when asked to have it turned off.

The messaging is that it all stays local. In reality it’s not and when I last looked it was running on Azure OpenAI in Australia.

I spoke to a practice nurse a few days ago to discuss this.

She said she didn’t think patients would care if they knew the data would be shipped off site. She said people’s problems are not that confidential and their heath data is probably online anyway so who cares.

That memo is how you make staff hide things instead of asking for help.

The scarier part though is that LLM-written clinical notes probably look fine. That's the whole problem. I built a system where one AI was scoring another AI's work, and it kept giving high marks because the output read well. I had to make the scorer blind to the original coaching text before it started catching real issues. Now imagine that "reads well, isn't right" failure mode in clinical documentation.

Nobody's re-reading the phrasing until a patient outcome goes wrong.

This is just about not using free/public AI tools.