GPTZero says AI “vibe citations” are spreading through big firms

As AI continues to develop at lightning speed, researchers at GPTZero are growing increasingly concerned that the trend of artificial misrepresentation will spread across businesses as corporations continue to scale up their operations.

Last month, major consultancy firm Ernst & Young (EY) was forced to retract a cybersecurity report after GPTZero, an AI-detection tool, found that more than 70% of the bogus citations were generated by AI.

These can be referred to as “vibe citations” – something that sounds plausible, but was churned out, often imprecisely, by AI to cut corners.

"The EY report is just the tip of the iceberg," said Om Ogale, lead investigator on GPTZero’s EY case, pointing out that his team has amassed over 3000 PDFs, with plentiful evidence of hallucinatory citations, although Ogale is reluctant to name names just yet.

A slippery slop

As is often the case with user-posted content on LinkedIn, there is a tendency to call things out as “slop” far too easily, and the same can be true at the corporate level, too.

“The fundamental issue in many domains is factual accuracy, not AI-generation,” explains Ogale – while revealing how a couple of Deloitte calamities last fall exposed the company for negligent fact-checking, when orchestrating government reports in Canada and Australia.

Ogale also points to AI hallucinations right across the board, from the legal industries to government reports, and globally at that.

It’s one thing to use AI as a research tool, and another to not properly review what it produces.

And, as many academic institutions often struggle to even discuss, let alone combat AI, it comes as a welcome consolation that some are prevailing.

“The research community is taking this seriously,” explains Ogale, with a year-long ban being imposed on authors who use AI hallucinations when posting on the physical sciences digital clearinghouse arXiv.

Snowed under by fiction

Most people with at least a passive interest in tech know about AI hallucinations, but predicting their frequency or magnitude can be tricky.

Ogale mentions government reports in particular, which are then scraped by large language models (LLMs), with the end user presented with the misinformation as fact.

What’s concerning is that the downstream effects create a trust problem, according to Ogale, which “leaves people uncertain about where to get their information.”

When a potential voter might crave the autonomy to gather and aggregate their own findings before an election, a chatbot like ChatGPT or Claude is not guaranteed to be a reliable source, and embellishments could sway an election.

And, while being forewarned of hallucinations is to a degree being forearmed, there is still a whole “spectrum of consequences” to be aware of, according to Ogale.

Various degrees of damage

Ogale points to minor impact situations, like a marketing blog, for example, slapping together another fabrication – but it’s the large-scale implications of the EY case that paint a more sinister picture.

He mentions the false statistic in the EY report of the global loyalty points market being worth $200 billion, being falsely “taken at face value” by the end user, especially if prompted to sign up for a loyalty points scheme.

But it’s the reputational damage, lock stock and barrel, that could clobber a lot of industries, if they continue to pour their faith completely into AI.

When fake citations underpin academic research, legal arguments, regulatory decisions, or clinical recommendations, the harm is concrete, lasting, and irreversible.

And to round it all off, Ogale predicts the situation will deteriorate further, at least in the short term.

Even if LinkedIn is battling against AI slop, industries at large “currently lack the proper tools to verify content at the scale AI can generate,” he said.

But a multitude of options seem available by developing even better detection tools, persuading companies to use AI more responsibly, and having good old-fashioned academic rigor.

Coupled with the educational need to “help researchers, consultants, and writers use AI as a thought partner,” these strategies should help keep the written word bespoke rather than churned and embellished.