A family member has cancer and we treat chatgpt as part of the team (our doctor's words). I ingest everything into it, work with it to make a good report. Then at the next visit we review it.

This gives you the best of both worlds. You get peace of mind and the doctor explains why and how the agent was right or wrong.

Twice now we've caught consequential mistakes (wrong pain medication and incorrect notation of the exact mutation that he has). Which have made a difference to his quality of life and treatment path.

Most of what the doctors have said is in line with the agent but when there have been disagreements they've been very reasonable. Sometimes the doctors have gone with the agent's version sometimes they've explained why that's inappropriate.

My dog had been acting off. Wouldn’t eat, was hunched over, looked sad. We took him to a local vet who did an X-ray because they suspected a blockage. They didn’t see one, so they sent us home with standard pain meds.

Randomly, we had a dinner party that night and another vet was there. She heard the story and immediately said, “Go home right now and take your dog to an emergency vet with ultrasound.”

Turns out, at the time, most vets had been trained to use X-rays to look for blockages, but newer evidence showed X-rays were only something like 20% effective compared to ultrasound, which was closer to 95%. (forget percentages but somethign like that)

The ultrasound found an avocado pit stuck in his intestine. He had emergency surgery that night.

That chocolate chunk of an English Lab ended up living until 15, and only needed two more blockage surgeries after that...

I know doctors hate patients reading the internet, and LLMs are going to make that 1000% worse for them. But hopefully over time, we all adapt together and end up better off in the long run.

Instead, it is my experiences with LLMs in a domain that I know very well that makes me skeptical of their performance across the board. I find issues in code review multiple times a day with their output, and they are explicitly and extensively trained on this use-case, unlike with the MRI data. Sometimes I veer into other domains I have decent knowledge about (construction, carpentry, landscaping) and LLMs disappoint me there as well.

I suppose Gell-Mann amnesia is a universal human quirk and not restricted to just the news.

This single sentence provides a huge clue about what’s going on: This person’s medical team is not good. It’s not hard to get an LLM to perform better than a team that is injecting homeopathic botanical formulations and performing procedures that aren’t indicated for the condition.

I think the real takeaway from this article shouldn’t be “ChatGPT is better than doctors”. It’s a story about LLMs identifying that someone was not in good hands.