Brain tumor-scanning GPT-4 can outperform doctors, study shows

OpenAI‘s GPT-4 model has demonstrated that it can be as good at finding brain tumors in patients as a qualified radiologist, even outperforming specialists in certain cases.

Artificial intelligence and large language models (LLMs) such as ChatGPT won’t be replacing doctors anytime soon. However, a recent study from the Osaka Metropolitan University provides strong evidence that AI might become an invaluable doctor’s assistant.

Researchers reached this conclusion after conducting an experiment using data about real-life brain tumor patients. OpenAI’s GPT-4 model and several radiologists were presented with textual findings from 150 reports on preoperative individuals. The data covered patients who had already been diagnosed, providing a solid foundation for understanding GPT-4, and radiologists in the study came to the right conclusions.

The radiologists competing against the AI were not freshmen. Three neuroradiologists were certified by the Japanese Society of Radiology as specialists in diagnostic imaging, while others were defined as “radiology residents or radiologists who specialize in areas other than imaging diagnosis.”

The results revealed that OpenAI’s GPT-4 model diagnosed brain tumor patients with a 73% accuracy, while qualified radiologists carried out the same task with accuracy ranging from 65% to 79%. What’s even more interesting is that GPT-4’s final diagnostic accuracy using reports from neuroradiologists was higher at 80%, compared to 60% using those from general radiologists.

In other words, GPT-4 was better at diagnosing brain tumor patients using reports written by people who specialize in diagnostic imaging and can describe the issue more accurately.

When it came to differential diagnoses, when doctors needed to decide between two or more conditions with similar symptoms, GPT-4 outperformed radiologists with 94% accuracy, compared to radiologists’ scoring between 73% and 89%.

Researchers believe that their findings indicate that GPT-4 can achieve diagnostic accuracy that is at least comparable with neuroradiologists, suggesting that LLMs have the potential to significantly improve the diagnostic process.

“GPT-4 exhibited good diagnostic capability, comparable to neuroradiologists in differentiating brain tumors from MRI reports. GPT-4 can be a second opinion for neuroradiologists on final diagnoses and a guidance tool for general radiologists and residents,” the study concluded.