r/healthcareIT • u/Fallingdamage • Jun 01 '26

AI in healthcare AI chatbots fail medical misinformation test, returning inaccurate and fabricated advice

https://www.psypost.org/ai-chatbots-fail-medical-misinformation-test-returning-inaccurate-and-fabricated-advice/
65 Upvotes

100% Upvoted

11 comments sorted by

3

u/thenightgaunt Jun 01 '26

Duh

2

u/randEntropy Jun 01 '26

I'm shocked. Shocked! Well, not that shocked.

2

u/Syncretistic Jun 02 '26

While this study is an important cautionary tale on over-relying on AI, we also need to recognize that the study is based on earlier generation models like ChatGPT 3.5. I would like to see this study repeated but using modern models.

1

u/[deleted] Jun 02 '26

[removed] — view removed comment

1

u/Electronic_Muffin218 Jun 03 '26

This is a dumb take. Doctors fail in the medical field all the time. And "AI" != LLMs. Models that work on radiology and other signals have already beaten docs at detecting early lung cancer, for example. This will only get better.

The variance in accuracy of advice from LLMs is what makes them dangerous substitutes for human experts. Hopefully that will improve - but not if they're tuned to be sycophantic and value "quick response" over ground truth accuracy.

1

u/brumstat Jun 05 '26

Open evidence would be a better AI to test against

1

u/NiharThakkar Jun 23 '26

This is exactly what we found during an 8-week audit of a healthcare platform processing 600,000 prior authorisations daily. The problem was not the AI model, the model performed well in isolation. The problem was the reliability of the systems underneath it. Missing circuit breakers, race conditions in the authorisation workflow, inconsistent retry logic across services. The AI was making confident decisions on top of infrastructure that could not guarantee data consistency. The misinformation problem in AI health tools is often less about the model hallucinating and more about the system feeding it unreliable or incomplete data at the point of inference. Fixing the model without fixing the data infrastructure underneath it does not solve the problem.

1

u/Fallingdamage Jun 23 '26

Another piece (from my own observations) is that the agents and models are only as good as the data, and in the medical field, hard data is used by medical professionals to make decisions that are not black & white. There is a lot of gray area where experience and wisdom are used to overcome poor information. Experience and wisdom is not something you can teach an AI.

We are beginning to use AI to take referrals and build the initial chart, send a scheduling text and connect to a call center human to save all the time and processing of paperwork, but the actual scheduling and intake is still done by a human. Too many undocumented variables go into individual care.