AI doctor will see you now: ChatGPT takes medical exams
A group of researchers had the viral chatbot take the same US medical licensing exams as aspiring doctors. It performed “comfortably within the passing range,” they said.
Image by Shutterstock
A group of researchers had the viral chatbot take the same US medical licensing exams as aspiring doctors. It performed “comfortably within the passing range,” they said.
ChatGPT, a large language model by OpenAI, has demonstrated “a surprising and impressive” result in all three tests that are part of the US Medical Licensing Exam.
“ChatGPT performed at or near the passing threshold for all three exams without any specialized training or reinforcement,” the research paper published by medRxiv said.
The paper was written by researchers mainly from Ansible Health, a Silicon Valley startup that has been researching AI and machine learning tools to improve healthcare. It is currently under peer review, according to Axios.
One caveat to the research was the removal of “indeterminate” answers as ChatGPT is programmed against giving medical advice, and it was limited by a “relatively small input size.”
More from Cybernews
Sainsbury’s installs more facial rec tech in stores: Should shoppers be worried?
"Phantom squatting” uses AI hallucinated domains for cyber attacks
What they’re doing to gamers is a robbery in broad daylight
Irish parliament expands Microsoft use, despite the EU’s efforts to ditch it
Can government AI actually scrub UAP footage from the internet?
The EU and Apple CEO Tim Cook held "constructive" talks after their Siri AI dispute in Europe
Other than that, it demonstrated “moderate accuracy” and “high internal concordance,” researchers said, noting that AI performance was likely to improve further as language models continued to mature.
The chatbot demonstrated the lowest accuracy for the first part of the exam, which its human takers consider the most difficult.
“This result exposes a key vulnerability in pre-trained large language models, such as ChatGPT: AI ability becomes yoked to human ability,” the paper said.
Add to that concerns over AI’s reported susceptibility to misinformation and cybersecurity risks, and independent machine doctors are still years away. But it could be applied as a risk assessment, data analysis, or education tool, among other potential uses.
“ChatGPT… possesses the partial ability to teach medicine by surfacing novel and nonobvious concepts that may not be in learners’ sphere of awareness,” the research said.
Typically, the first part of the Medical Licensing Exam is taken after two years of medical school and requires students to dedicate up to 400 hours of studying to pass. Fourth-year medical students take the second part of the exam, while physicians with up to one year of postgraduate medical education take part three.