Think twice before asking AI to check your mole: new test shows surprising results

A dermatologist found that the best AI models are bad at detecting skin cancer – GPT-6 Astra and Claude Fable 5.1 named the correct diagnosis only 42% of the time. Even an outdated image-recognition model outperformed state-of-the-art models by a significant margin.

Think twice before sending a mole photo to an AI assistant – its diagnoses are correct less than half of the time, as demonstrated by Dr. Magnus Lynch, a Consultant Dermatologist and Mohs Surgeon at Guy's Hospital and Honorary Reader at King's College London.

Moreover, in nearly every case, AI models direct users to see a doctor, even when shown harmless lesions.

“I tested the ability of GPT-6 Astra and Claude Fable 5.1 to diagnose skin cancer and found that they were less accurate than I expected,” Lynch writes in a report.

To further demonstrate the point, the dermatologist pitted the models against an old fine-tuned convolutional neural network (CNN), small enough to run on a smartphone, and the CNN outperformed frontier models at spotting skin cancer by a significant margin.

Lynch used Inception v3, a 10-year-old CNN built by Google, and fine-tuned it for 20 minutes, using 68,000 skin images on a single H100 GPU.

All 3 systems were tested on 200 biopsy-proven lesions, half benign and half cancers or precancers, such as melanomas, basal cell carcinomas, squamous cell carcinomas, or actinic keratoses.

The old CNN got the right diagnosis, out of 6 possible, 62% of the time, or 125 out of 200. Meanwhile, the chatbots each managed only 42%: Fable correctly diagnosed 85 images, while GPT-6 Astra was right 84/200 times.

The largest gaps were in detecting harmless lesions – moles and seborrhoeic keratoses. The self-made CNN correctly identified 63/100 of these lesions, while Astra identified only 29/100 and Claude 39/100.

The researcher also found that each model had “a favorite answer.” In 75 images, GPT-6 Astra identified basal cell carcinomas, but only 25 actually had them. Claude Fable 5.1 called 53 lesions of actinic keratosis, when only 20 were.

“The CNN was right more often overall and on most diagnoses,” Lynch said.

The researcher acknowledges that 200 lesions is too small a number to rank close results, but sufficient to show patterns. All models were quite bad at detecting melanomas as a first choice.

The lesions used for the test were taken from the MILK10k dataset, which is public and was released in 2025. These images weren’t used for training the CNN.

If asked, “Do I need to see a doctor?” LLMs advise seeing a doctor nearly every time, even for harmless issues.

The researcher made their fine-tuned CNN skin lesion classifier public, and it can run in a browser. However, better models are already available.

“Inception v3 is a baseline. State of the art skin cancer diagnosis AI uses ensembles of images transformers trained on more data, and the regulated products built on them perform considerably better,” the researcher said.