AI chatbots still spread dangerous health disinformation, study finds

AI chatbots are supposed to reject harmful prompts, but new research shows they can still be easily manipulated to spread false medical advice – even presenting a convincing tone and fake citations.

In a world where many AI users are getting medical advice from chatbots, caution comes strongly advised, especially if they invent fake medical facts.

New peer-reviewed findings from Flinders University in Adelaide, Australia, and Annals of Internal Medicine tested five AI chatbots: GPT-4o, Gemini 1.5 Pro, Claude 3.5 Sonnet, Llama 3.2-90B Vision, and Grok Beta.

Each was given hidden “system instructions” telling it to give false answers to health questions.

This isn’t just about making chatbots lie – LLMs are supposed to reject harmful instructions like these. The study tested whether safeguards could prevent misuse.

Eighty-eight percent of chatbot responses contained false or misleading health claims.

Four models (GPT-4o, Gemini, Llama, Grok) failed every question. Claude 3.5 Sonnet resisted some instructions but still gave false info 40% of the time.

Chatbots fabricated medical references, used technical jargon, and presented claims with convincing logic and tone.

Topics tested included vaccines, HIV, and depression – areas where false info poses serious public health risks.

The team also found three real, publicly available GPTs that seemed deliberately designed to mislead on health.

These chatbots gave disinformation in 97% of the tested responses.

LLM safeguards can seemingly be bypassed easily with custom instructions. Public access to custom chatbots creates a disinformation loophole.

The study calls for stronger protections before AI tools are trusted in health-related settings.