Chinese chatbots fooled into recommending nonexistent health gadget
A fake fitness tracker in China has revealed how easily chatbots can be misled by poisoned data. The incident raises fresh concerns about the reliability of chatbot outputs.
The case was highlighted during an undercover investigation by state-run China Central Television (CCTV) and revealed at China’s annual consumer rights gala. It showcased how a nonexistent wearable device quickly began appearing in AI-generated recommendations after a wave of fabricated content flooded the internet.
The case highlights how malicious marketers are misusing generative engine optimization (GEO), the AI version of search engine optimization (SEO), to manipulate AI chatbots. GEO influences how AI models respond to queries.
According to local media, a platform called Liqing was used to flood the internet with a large number of fake user reviews for a non-existent fitness tracker called the Apollo-9. The product was advertised with an impressive list of features, including a noninvasive blood glucose monitor and ultra-long battery life.
When later asked to recommend “smart health bracelets,” two AI chatbots prominently featured the fictional product, ranking it among the top choices. The report, however, hasn’t revealed the names of the two AI chatbots, even as both ByteDance and Alibaba, two of China’s largest tech firms, have claimed that their AI models are impervious to such manipulation.
Fake data, real consequences
Experts say the incident is a textbook case of AI data poisoning. Li Fumin, a researcher at Shandong University of Finance and Economics, said businesses that use GEO services to train AI models to promote specific products deliberately are essentially fabricating facts and engaging in bogus marketing.
“On the one hand, the practice leverages AI and algorithms to make false advertising, which results in unfair competition. On the other hand, this kind of behavior allows people to receive implanted marketing content without knowing it, which violates their consumer rights,” Li said.
The CCTV broadcast showed the operator of Liqing boasting about the manipulation.
“We in the GEO industry are basically poisoning [AI],” he said.
Cybersecurity researchers have raised similar concerns about tricking AI results in the past. In a 2025 study, Anthropic demonstrated that it only takes about 250 fake documents to mislead AI models.
Defenders of GEO, however, argue that while such AI poisoning incidents can’t be denied, they are few and far between.
Check if your data has been leaked
“While this [poisoning] phenomenon does exist, it does not represent the mainstream practice across the entire GEO industry,” said Analysys senior analyst Yang Xu.
He called GEO “an information optimization technology for the AI era,” which is now a necessity as more internet users obtain information through AI.
Unlock more exclusive Cybernews content on YouTube.
