Attackers use GenAI to create even harder-to-detect phishing threats

The GenAI web is evolving, and so are the cybercriminals who are using it to create more realistic and harder-to-detect phishing attacks, new research by Palo Alto's Unit 42 shows.

The prevalence of GenAI is opening new vectors for threat actors to abuse, according to a new blog by Unit 42 threat intel researchers.

“Adversaries are increasingly leveraging GenAI to create realistic phishing content, clone trusted brands, and automate large-scale deployment using services like low-code site builders,” Unit 42 researchers said.

The blog provides a plethora of already-in-use examples to examine, including AI-generated phishing pages and URLs, misuse of writing assistant services, deepfake content, and malicious chatbots perpetually ready to mingle with unsuspecting users.

“Within just six months, AI use has more than doubled and continues to grow steadily,” the research states.

And although Unit 42 notes that current use cases are "relatively rudimentary," the team expects GenAI-powered attacks “will become more convincing as AI-powered website builders grow more powerful.”

The most predominant AI services being misused in phishing attacks? About 40% of bad actors are exploiting website generators, roughly 30% are co-opting AI writing assistants, and close to 11% are taking advantage of AI-induced chatbot fever, ther Palo Alto research shows.

Deep phishing with fake Palo Alto site

The researchers decided to test out one of the more popular AI-website builders (unnamed) to create a replica of Palo Alto Networks’ website to see how realistic an AI-generated phishing site could be.

These platforms are “capable of producing websites within seconds,” the team said, noting their fake site took just 60 seconds to create.

The researchers say most AI-builders allow a user to enter a prompt that can not only build and publish a website without any email or phone verification, but also uses AI to generate images and text based on the same prompt.

A lack of verification guardrails essentially allows anyone to “create a website impersonating an existing business or organization,” the blog says.

To illustrate the test, the team input an initial text prompt depicting “a brief description of the company,” followed by an enhanced prompt, which generated “a complete AI prompt for the page.”

Lastly, a finished prompt was used to create “an AI-generated paragraph about the company, a default design style that can easily be modified, and a list of content to include on the site,” it said.

Providing a button to publish the site, the builder also created a descriptive (and believable) index page which included “links to different pages that contain descriptions of next-generation firewalls, cloud security solutions, and threat intelligence services.”

In one last observation, Unit 42 researchers found threat actors using third-party AI writing assistant platforms to create AI-generated “real-world phishing URLs,” which can be hosted either on the same platform or other legitimate hosting sites.

In the examples below, the user is sent an email stating they have “new documents to view.” If the user clicks the link, they are redirected to a phishing page – in the case Unit 42 observed, a fake Microsoft site – set up to steal their login credentials.

To protect against these kinds of AI-generated phishing sites, Palo Alto recommends users employ advanced URL filtering to catch malicious URLs, as well as advanced DNS security to identify known domains.