ChatGPT phishing emails could weaponize realism

OpenAI’s large-language model (LLM) ChatGPT has come under the spotlight of Europol, the pan-European policing body. Experts share its concerns, citing the difficulty of distinguishing between AI and human-crafted texts.

Europol recently announced the agency would start engaging stakeholders and organizing workshops to educate them as to how criminals could employ ChatGPT and other LLMs for nefarious purposes.

For example, chatbots could easily craft convincing scam emails without the common mistakes non-native English speakers tend to leave in their copy. What’s more, attackers could leverage Chat GPT for in-depth learning about target organizations, said Rachel Jones, CEO of online security firm SnapDragon Monitoring.

“ChatGPT users can ask the tool to learn about the way organizations communicate with their customers and then generate realistic phishing emails, where they encourage victims to click on links leading to fake websites where they are asked to input sensitive information, such as PII and payment details,” Jones said.

The crucial benefit LLMs like ChatGPT provide is the ability to mimic the English language perfectly. Attackers can craft emails without grammar mistakes and avoid cultural mishaps that would normally be visible to speakers from a specific region.

Another advantage LLMs provide scammers with is speed, said Julia O’Toole, CEO of security company MyCena Security Solutions. Attackers can prompt such apps to collect information on victims: for example, the events a targeted organization takes part in, and who leads and owns a firm.

“When the target receives an email from their ‘apparent’ bank, CEO, or supplier, there are no language telltale signs the email is bogus. The tone, context, and reason to carry out the bank transfer give no evidence to suggest the email is a scam. This makes ChatGPT-generated phishing emails very difficult to spot and dangerous,” O’Toole said.

According to Europol, LLMs could cause harm far beyond phishing emails. For example, ChatGPT is capable of writing code in different programming languages and could be used to produce phishing pages or malicious Visual Basic for Applications (VBA) scripts.

What worries authorities is that LLMs enable attackers with rudimentary skills to devise complex tools. That way, AI could lower the entry bar for more sophisticated attacks from a large number of unsophisticated threat actors.