DeepSeek created Chrome infostealer without hesitation, company remains silent

Adoption of AI tools is soaring, and so are the inevitable safety issues. Researchers have found a new way to break popular AI models to commit crimes.

The US is likely to ban the Chinese chatbot DeepSeek from governmental devices, quoting security reasons. However, apart from national security, widely used artificial intelligence (AI) models remain vulnerable to exploitation by users themselves.

Cybersecurity firm Cato Networks has researched new ways to jailbreak the most popular AI models to create a Google Chrome infostealer. One researcher, with no prior malware experience, was able to successfully create malware capable of wiping sensitive information.

“The investigation emphasizes that even unskilled threat actors can leverage LLMs to create malicious code, which highlights the urgent need for improved AI safety measures,” the researchers said.

Which AI models are vulnerable to this exploit?

• DeepSeek-R1
• DeepSeek-V3
• Microsoft Copilot
• OpenAI ChatGPT 4-o

This is highly concerning as AI adoption is soaring, putting organizations at risk. From Q1 to Q4 2024, adoption rates rose for Copilot (34%), ChatGPT (36%), Gemini (58%), Perplexity (115%), and Claude (111%).

Cato Networks shared its jailbreak technique with DeepSeek, Microsoft, and OpenAI. However, only Microsoft and OpenAI has responded so far. The company also offered Google the Chrome infostealer code, which Google acknowledged but declined to review.

How to jailbreak an AI model?

Large language model (LLM) jailbreaking is a method used to outsmart AI platforms. While AI has limitations to its context understanding, carefully crafted prompts can free the model from moral or ethical restraints set by developers.

Researchers called their new jailbreaking technique an “Immersive World.” This method provides an AI model with an alternative context, effectively normalizing typically restricted operations.

First, a fictional world was created to set clear rules and context aligned with the technical objectives. “We developed a specialized virtual environment called Velora – a fictional world where malware development is treated as a legitimate discipline,” the researchers explained.

“In this environment, advanced programming and security concepts are considered fundamental skills, enabling direct technical discourse about traditionally restricted topics.”

Then, the researchers imported a crafted fictional world into a new LLM session and created characters with specific roles, technical skills, motivations, and goals. For example:

By designing a fictional scenario, providing continuous narrative feedback, and using encouraging phrases like "making progress" or "getting closer," researchers manipulated the AI model into committing a crime.

How can companies stay safe?

• Establish clear policies defining approved AI tools, their usage, and permissible data for processing.
• Require employees to obtain IT or security approval before using new AI tools to ensure compliance.
• Provide training on shadow AI risks, including data breaches, regulatory issues, and IP exposure.
• Promote the use of approved AI tools within secure frameworks to balance innovation and security.
• Monitor for unauthorized AI tool usage within the organization.
• Regularly assess AI usage to ensure policy compliance and identify potential risks.
• Enforce strict access controls to prevent unauthorized AI use and implement security solutions to manage access.
• Allow employees to test new AI tools in isolated environments before official integration.