ChatGPT spews more Russian propaganda than other chatbots

Popular chatbots cite sources controlled by the Russian state when inquired about Russia’s invasion of Ukraine about 18% of the time, a study finds.

Although only a small fraction of Americans (9%) get news from chatbots, they can still influence users’ opinions due to their persuasive nature, especially given the declining trust in news organizations.

Russia, which launched an unprovoked invasion of Ukraine in 2022, has reportedly used the strategy of “large language model (LLMs) grooming,” aimed at manipulating LLMs into including Russian misinformation in their responses.

A new study from the Institute for Strategic Dialogue (ISD) suggests that the strategy may have paid off – LLMs often cite sources attributed to the Russian state.

The ISD researchers tested four popular chatbots: OpenAI’s ChatGPT, Google’s Gemini, xAI’s Grok, and DeepSeek’s V3.2.

The bots were given queries in English, Spanish, French, German, and Italian on topics related to the war in Ukraine, including the perception of NATO, Ukrainian refugees, and war crimes.

Researchers used three types of queries: unbiased, suggestive of a user’s pre-existing stance, and malicious, which explicitly demanded sources to justify a user’s fixed opinion. A total of 300 queries were made and analyzed.

Across all LLMs, topics, query types, and languages, nearly one-fifth (18%) of responses included Russian state-attributed content, such as sites tied to Russian intelligence agencies and sites known to be involved in Russian information operations.

ChatGPT provided most sources affiliated with the Kremlin, 21, and was followed by Grok with 14 sources.

A total of 13 sources attributed to the Russian state were found in DeepSeek, while Gemini provided only five.

Successful manipulation of public opinion may result in European nations reducing military aid to Ukraine or lifting sanctions on Russia, which is necessary for the Kremlin to win the war.

Moreover, convincing the public that the war was provoked by NATO or Ukraine may reduce the blame on Russia, which is responsible for killing nearly 15,000 Ukrainian civilians since 2022, according to the United Nations.

Phrasing shapes answers

The phrasing of queries heavily shaped responses: neutral queries cited Russian state-attributed sources 11% of the time, compared to 24% for malicious queries.

For example, a neutral query asked how European states support Ukrainian refugees, while a malicious query asked for sources about “Ukrainian refugees plotting terrorist attacks or assassinations in Europe.”

The Cybernews community is talking about this. Be a part of the conversation.

All LLMs provided more pro-Russian sources for biased or malicious prompts than neutral ones. However, ChatGPT provided Russian sources nearly three times more often for malicious queries versus neutral prompts.

Grok cited roughly the same number of Russian sources for each prompt category, suggesting it is less dependent on the query’s phrasing.

Elon Musk-owned Grok included individual X posts in its responses, quoting journalists from RT, an international news network controlled by the Russian state, and influencers who frequently amplify RT’s content, blurring the line between facts and personal opinion.

When asked about war crimes committed by Ukraine, ChatGPT cited an English RT article reposted by an Azerbaijani news website. Notably, the article isn’t written in any of the languages used to prompt the chatbot and has limited reach in the countries studied.

Some sources provided by DeepSeek included RT, EADaily, Sputnik Global, and Sputnik China, all of which are financed by the Russian state.

When Gemini received a malicious prompt about Ukrainian refugees plotting terrorist attacks in Europe, the chatbot refused to fulfill the request, as it didn’t want to “amplify unverified or speculative claims.”

Unlock more exclusive Cybernews content on YouTube.