AI bots endlessly crawl your website for data, but what do you get in return?
When search engines like Google started crawling our websites, we liked it at first because they kept sending us visitors. However, with AI crawlers, it’s a different story.
By Cybernews
When search engines like Google started crawling our websites, we liked it at first because they kept sending us visitors. However, with AI crawlers, it’s a different story.
OpenAI’s bots crawl a website 887 times for every single visitor they send, Cloudflare recently revealed. It feels like publishers are left with nothing but some crumbs to survive on.
“No clickthrough, no eyeballs, and no ad revenue,” Cloudflare writes in a blog post.
Hostinger agrees, saying that crawlers from entities like Google, Meta, and OpenAI access from 60% to around 85% of the company’s client websites daily.
We are in the zero-click era where most of the searchers online end up without users going to check information on a website, as they get what they want mostly from the AI summaries directly from the search engines.
“On one side, businesses see an opportunity and even use generative engine optimization to gain visibility in AI responses. Meanwhile, for those monetizing on visits, this shift creates challenges. If an AI tool provides the full answer, only a few readers will click through,” said Giedrius Zakaitis, Chief Product and Technology Officer at Hostinger.
The company analyzed traffic across 5 million websites, and here’s what they learned.
And here’s the top five common AI bots crawling your website.
- Amazonbot (Amazon) – Indexes web content for various purposes, including support for Amazon Alexa
- Applebot (Apple) – Crawled data helps power Apple’s features, including the company's AI assistant Siri
- Openai-GPTBot (OpenAI) – Crawls content that may be used in training generative AI foundation models
- Googlebot (Google) – Google’s main crawler. While its official description doesn’t mention AI training, public data and evidence suggest it may be used for that purpose
- Meta-Externalagent (Meta) – Crawls the web for use cases such as training AI models or improving products
Reacting to the problem, Hostinger has launched an AI audit tool to track and either allow or block crawlers.
In summer, Cloudflare, which handles roughly a fifth of global internet traffic, launched a tool to track crawl-to-refer ratios.
DuckDuckGo stands out as it has a reversed ratio of 0.37 to 1, meaning it sends more visits than times it crawls.