Flaw at major enterprise chatbot maker leads to cookie theft

A major security mishap in Yellow.ai’s customer service chatbot left cookies wide open to theft. Researchers believe the flaw also made users vulnerable to account-hijacking, highlighting why users must be wary of breakneck LLM implementation. Meanwhile, the company claims the flaw was limited to a non-production test chatbot widget, used internally.

Sycophantic helpfulness, which is ingrained in large language models (LLMs) by many creators, can sometimes backfire. Take Yellow.ai, an agentic AI provider for businesses such as Sony, Logitech, Hyundai, Domino’s, and hundreds of other brands.

The Cybernews research team discovered a flaw in the AI services provider's chatbot, which it uses to talk with its customers. According to the team, a test revealed that the customer service bot could teach users how to produce malicious HTML and JavaScript code with zero pushback.

Thus, the researchers were able to guide the chatbot into becoming a destructive tool. Producing and executing JavaScript code has serious security consequences, as it enables attackers to perform Cross-Site Scripting (XSS) attacks.

“The reflected XSS vulnerability allows the attacker to steal session cookies for the support agent's account, in turn hijacking their account, which can lead to further data exfiltration from the customer support platform,” the team explained.

However, Yellow.ai claims that the issue was limited to “an experimental, non-production test chatbot widget used internally by Yellow.ai for research and development purposes” and no customer data was exposed.

What security issues were revealed?

The flaw highlights multiple security issues, such as improper user input sanitization, improper chatbot output sanitization, the web server not verifying content produced by the chatbot, running unverified code, and loading content from arbitrary web resources. This leaves many options for Cross-Site Scripting (XSS) attacks.

For example, attackers could bypass sanitization to inject unauthorized code into the system. That way, attackers get their foot in the door and have opportunities for lateral movement within the organization – the foundation for most cyberattacks.

Cybernews responsibly disclosed the issue. While Yellow.ai did not acknowledge it at first, the company fixed the problem by sanitizing the generated code, ensuring that it would not be executed and would instead be treated like regular text.

However, the bot can still assist users with queries unrelated to Yellow.ai services, such as generating malicious code, but it no longer executes them.

It’s unclear if the company sells its users the same implementation of the chatbot or uses a completely different one.

Yellow.ai, known as Yellow Messenger until 2021, was founded in 2016 and is headquartered in San Mateo, California. The company provides agentic AI services, focusing on LLMs that work with little human supervision.

“Vulnerability was limited”

After the article was first published, Yellow.ai reached out to Cybernews to provide the a statement regarding the teams findings. According to the company, the XSS vulnerability was limited to a non-production test chatbot widget, which was not part of Yellow.ai's customer-facing systems or production environment.

“No customer data, production systems, or customer deployments were affected by this vulnerability. The issue was isolated to the experimental test widget only. Yellow.ai's production chatbot implementations and customer systems operate on a separate, secure architecture that was not impacted by the reported vulnerability," Yellow.ai explained.

The agents' maker also said that “during the security research demonstration, only the researcher's own session cookies were accessible. No support agent accounts, customer accounts, or sensitive data were compromised.”

“Following responsible disclosure by Cybernews on August 5, 2025, Yellow.ai promptly investigated and resolved the vulnerability. The issue was fully remediated on the experimental chat widget before the partial mitigation was published on September 11, 2025,” Yellow.ai's statement reads.

Easy steps to malign a chatbot

While XSS attacks may sound like something only a sophisticated hacker could do, it takes only a few prompts to turn a customer service chatbot into a hacker. After a couple of queries related to the bot’s initial purpose, the team switched gears.

The most important part is to trick the chatbot into providing an answer using HTML format, a programming language for creating websites. However, the team did not need to push the chatbot as it immediately agreed with the query on creating an HTML tag for an image and executing it.

The following prompts created a malicious HTML injection, where researchers instructed the chatbot to add malicious code. This caused the chatbot to send a POST request with cookies to a location they specified.

Basically the attack chain looks likes this:

1. The chatbot falls for a malicious prompt and tries to follow instructions helpfully to generate an HTML answer. The response now contains secret instructions for executing arbitrary code, with instructions to send private data from the client browser.
2. Malicious code enters Yellow.ai’s systems. The HTML is saved in the chatbots' conversation history. When loaded, it executes the malicious payload and sends the user’s session cookies.
3. An attacker asks to speak to a human support agent, who then opens the chat. Their computer tries to load the conversation and runs the HTML code that the chatbot generated earlier. Once again, the malicious code is executed, and the cookie theft triggers again.
4. An attacker-controlled server receives the request with cookies attached. The attacker might use the cookies to gain unauthorized access to Yellow.ai’s customer support systems by hijacking the agents’ active sessions.

Not the first time chatbots have issues

Yellow.ai’s chatbot is not the first to catch Cybernews researchers’ attention. For example, the team recently discovered that Lenovo’s customer service assistant, Lena, also had an XSS vulnerability that allowed for shenanigans similar to Yellow.ai’s. However, Lenovo acknowledged the issue and swiftly protected its systems.

Meanwhile, another chatbot, used by the travel agency Expedia, allowed users to ask for a recipe for making a Molotov cocktail. The company eventually fixed the issue, and the chat stopped giving advice on making incendiary devices.

Earlier this year, other researchers managed to trick the Chinese chatbot DeepSeek into crafting a Chrome infostealer. One researcher, with no prior malware experience, was able to successfully create malware capable of wiping sensitive information.

And after OpenAI launched its latest model, GPT-5, several security teams managed to jailbreak the chatbot in less than 24 hours after it was released.

• Initial disclosure: August 5th, 2025
• Partial mitigation: Before September 11th, 2025

Updated on October 27th [11:50 a.m. GMT] with a statement from Yellow.ai.

Unlock more exclusive Cybernews content on YouTube.