AI chatbots crawl website hundreds of times before sending a single actual visitor

Millions of crawls but just a few clicks – new data shows that AI chatbots are leeching attention away from publishers and site owners, harvesting their content but giving back very little traffic in return.

ChatGPT and other bots of the leading AI developer OpenAI crawl a website 887 times for every single visitors they send, Cloudflare data reveals.

And that’s not the worst ratio among AI companies.

Anthropic, developer of Claude and other popular AI chatbots, leads the pack with the highest crawl-to-refer ratio at nearly 50,000:1. This means that publishers and site owners can expect just 20 visitors for every 1,000,000 times Claude crawls their websites.

“No clickthrough, no eyeballs, and no ad revenue,” Cloudflare writes in a blog post.

“When users turn to these AI platforms with queries that used to go to search engines, they often won't click through to the original source site once an answer is provided — and that assumes that a link to the source is provided at all!”

The company, which handles roughly a fifth of global internet traffic, launched the tool tracking the crawl-to-refer ratios on July 1st, 2025.

Since then, Perplexity sent one user for every 191 crawls it made. Microsoft’s ratio is 43.1 to 1, and Mistral’s is 23:1.

The dominant search company, Google, sends one user for every 4.7 times its crawlers visit the website. However, now it also incorporates AI overviews into classic search platforms.

Meanwhile, DuckDuckGo has a reversed ratio of 0.37 to 1, meaning it sends more visits than times it crawls.

Crawl-to-refer ratios are lower in the news and publications industry, but they still provide little comfort: Anthropic is at 2,500:1, OpenAI is at 152:1, and Perplexity is at 32.7:1.

The data demonstrates that AI companies take content from publishers' sites, overload their infrastructure with endless crawls, and return virtually nothing in traffic.

“Search platforms historically crawled websites with the implicit promise that, as the sites showed up in the results for relevant searches, they would send traffic on to those sites — in turn leading to ad revenue for the publisher,” Cloudflare argues.

However, the emergence of AI models is turning the model on its head.

Most AI bot crawling activity isn’t related to providing content directly to users. 78% of crawls are attributed to “training” purposes. Cloudflare also notes that crawling activity can be aggressive at times, often ignoring directives found in the robots.txt file.

Only 3.5% of crawls are caused by user action, while 17.4% have a “search” purpose.

Among the top five, GPTBot is the most active AI bot, responsible for 24.8% of crawls, followed by ClaudeBot (23.6%) and Meta-ExternalAgent (18.5%).

"AI crawler traffic has become a fact of life for content owners, and the complexity of dealing with it has increased as bots are used for purposes beyond LLM training. Work is underway to allow website publishers to declare how automated systems should use their content,” Cloudflare concludes.