What AI Bots Actually Do On Your Site, And What They Are Looking For

Over three months, AI crawlers made 48 million requests on one of our clients’ marketplace sites. That’s twice as many as search engines like Google, Bing, Yandex, Apple, Baidu, and DuckDuckGo combined. It means AI bots crawl your site twice as actively as traditional search bots. 

You might think that AI bots explore your content more than Google. However, it is a mistake. Based on our data, on one e-commerce website, AI bots’ requests didn’t reach the number of requests made by Googlebot, whereas on a news site, there were zero AI requests. Therefore, it is wrong to analyze bot behavior using average numbers.

In this article, you will learn about the three types of AI bots, how they behave, what they explore, and compare them with Googlebot. Our research is based on half a billion true requests from tens of websites over the last three months.

Types of AI Bots and Their Key Tasks

Not all AI bots are the same. To understand AI bot traffic, you should distinguish the three types of bots. 

1. Training bots, including GPTBot, ClaudeBot, Meta’s crawler, CCBot, and Bytespider, collect content to train large language models (LLMs). They need text rather than the page it is located on. They rarely come back.
2. Search bots, including OAI-SearchBot from OpenAI and PerplexityBot, are a new generation of bots that explore websites to answer user queries in real time. Their logic is closer to Googlebot’s, as they pay attention to content freshness and can revisit your pages repeatedly.
3. User bots, including ChatGPT-User, Perplexity-User, and Claude-User, belong to the newest category of bots. They fetch content the moment AI agents need it. 

Understanding the difference between bots is critical, as AI bot traffic volume doesn’t mean you benefit from it. The same number of requests from a training bot and a search bot is quite a different scenario.

What Content Are AI Bots Actually Collecting? 

Training bots can take everything they see. For example, on one website, we noticed that these bots downloaded a privacy policy page with regulatory text 170,000 times each. The same page had different parameters and local prefixes. So, the goal of training bots is to gather as much content as possible for the training dataset. Since storing data is cheap, these bots don’t care about filtering and cleansing data while crawling. They are vacuum cleaners.

AI search bots browse category and collection pages created to rank and convert. Those are exactly the pages Google crawlers. 

User bots fetch whatever is actually right now. When an LLM needs additional content to answer the user’s question, it fetches a specific page and uses its content in the answer. Because of the high demand for specific content, pages can be fetched thousands of times.

Understanding which pages different bots crawl helps explain how your content is found, shown, and ultimately used in AI-generated answers.

The Way AI Bots Crawl Reveals Their Purpose

By analyzing the number of requests, you can understand how busy a bot was. At the same time, the number of unique URLs it visited tells you what it was doing.

On a fashion retail website we managed, Googlebot visited around 309,000 unique URLs but returned to each about 17 times over the quarter. It already knew the site, but checked whether anything had changed.

Training bots do the opposite. For example, ClaudeBot visited 2.29 million unique URLs on that same site (7 times more than Google), but each page only once. GPTBot hit 3.85 million pages, which was 169 times more than Googlebot, again visiting each page just once.

So does 169x mean GPTBot was more thorough? No, as most of those links were duplicate pages, internal search results, and privacy policies – content Google skips on purpose. Thus, more links visited don’t mean more useful content collected.

AI search bots are the exception. Their crawl pattern is actually closer to Google’s. They revisit pages, focus on a smaller set of URLs, and prioritize content that’s meant to be found.

Do AI Bots Follow Googlebot? What’s the Difference between Them?

The biggest finding from this research, the pages that appear in AI answers are mostly the same pages Google already knows about.

On a fashion retail site, 88% of the URLs crawled by OpenAI and Perplexity bots were also crawled by Googlebot. Training bots overlapped with Google with only 22% of links. AI search bots are working from Google’s list. But the clearest evidence comes from user bots – the ones that fetch pages on demand, the moment someone asks an AI a question.

User Bots Move with Google

We tracked ChatGPT-User activity day by day and compared it to both Googlebot and GPTBot. The correlation with Googlebot reached 0.79 on one site, while with GPTBot, it was just 0.14. 

Googlebot itself correlates with impressions and traffic from the SERP; Google often re-crawls indexed URLs. They are in the SERP and receiving traffic.  This explains the connection between the ChatGPT-User and the Googlebot.

User Bots Visit the Same Pages, Google Visits

On a content site, 98% of URLs fetched by ChatGPT-User had also been crawled by Googlebot. It’s how the system works. When ChatGPT looks for a source, it pulls pages that rank. And rankings come from Google and Bing.

Google Gets There First

When ChatGPT-User fetched a page, Googlebot had already visited it in 57–69% of cases across most sites, often days or weeks earlier. GPTBot was only 22–39% of the time at first.

Put it all together, being visible in AI answers starts with being visible to Google. The assistant reads from Google’s results, so the pages that rank are the pages that appear in AI responses.

On the fashion site, we could even see the direct fingerprint. The URLs that ChatGPT-User fetched still carried Google’s own srsltid tracking parameter – the tag Google attaches to result links. The page didn’t just rank on Google. It came straight from a Google search result.

Why Technical Issues Affect AI Bots More Than Google

Low AI visibility isn’t always a content problem. Often, the issue is the technical state of your site and how bots interact with it.

Fake Bots Are More Common Than You’d Think

Not every request that calls itself GPTBot or Googlebot actually is one. Scrapers create fake user agents to look like legitimate crawlers.

7.7% of requests claiming to be GPTBot couldn’t be matched to OpenAI’s official IP addresses. Googlebot and Bingbot sit at around 0.1%. ChatGPT-User, OpenAI’s search bot, and PerplexityBot were cleaner – 1-2%.

This matters for analysis. Without IP verification, your traffic data is telling you a story that isn’t entirely true.

Why AI Bots Hit So Many 404s

Training bots run into 404 errors at 2-16 times Googlebot’s rate. On a content site, 52% of GPTBot’s requests came back with a 404. On a fashion retail site, 28%. Googlebot on those same sites was between 0.5% and 3%. User bots, which fetch specific live pages on demand, had the fewest 404s – under 1%.

So where do all those dead ends come from? We pulled the actual 404 URLs and found two causes.

1. On content sites, the link graph is polluted. On one site, more than half of GPTBot’s 404s led to foreign-language spam slugs. These URLs never existed on the actual site. Google filters this kind of garbage out before crawling it. GPTBot pulled these links from an unfiltered web-scale link graph and dutifully went to fetch them.
2. On e-commerce sites, nearly half of GPTBot’s 404s weren’t broken URLs – they were unevaluated JavaScript template strings, raw ${…} placeholder syntax scraped from scripts and treated as links. The bot fetched code instead of actual pages. Googlebot renders and executes JavaScript, so it resolves these into real product URLs and rarely hits this issue.

Training bots are inefficiently crawling more URLs than Google, burning through resources on spam links and unrendered JavaScript while generating extra load on your servers. More requests don’t mean better results.

Conclusion: The Real Drivers of AI Visibility: Google and Technical Accessibility

Half a billion requests across dozens of websites showed us that not all AI bots work the same way. Treating them the same way is wrong. Training bots inspect lots of pages, but most of what they collect is spam slugs, duplicate pages, and privacy policies. Search bots behave close to Googlebot: they revisit pages and focus on high-value content built to rank. User bots fetch pages when someone asks an AI a question, and this type follows Google almost step by step.

The last point is the core finding of our research. Content that appears in AI answers is usually the pages Google already knows about.

Technical health should be your AI visibility strategy. A clean, fast, properly rendered site wastes far less of any bot’s crawl budget, and what they do fetch is what actually appears in answers.

Three things that matter most for AI visibility:

• Google visibility comes first. If Google doesn’t know your page, AI probably won’t show it either. Optimizing for Google and optimizing for AI are not separate strategies.
• Technical issues affect AI visibility. Training bots can’t move forward on your site because they follow every link they find, including ones Google long ago filtered out. On e-commerce sites, nearly half of the content was unrendered, JavaScript-heavy pages that the bots can’t process appropriately. While Googlebot renders JavaScript and resolves it, most AI bots don’t.
• Volume doesn’t mean better visibility. GPTBot can crawl twice more URLs than Googlebot on the same site and still has almost no correlation with what ChatGPT really shows to users. Volume doesn’t mean better AI visibility.

This is exactly the problem EdgeComet was built for. It sits between your site and every bot that explores it, and makes sure each one gets a fast, complete, and rendered page. Caching is at the core, so bots are served in milliseconds, and the high-demand pages that user bots inspect are already prepared.