Search bots, AI crawlers and AI user agents explained

Bot types in brief

A bot, also called a crawler or spider, is a program that requests web pages on its own and names itself in the user agent string of each request. The types differ by purpose:

• Search engine crawlers such as Googlebot and Bingbot fetch pages to index them for search results.
• AI training crawlers such as GPTBot and ClaudeBot collect pages that may be used to train AI models.
• AI search crawlers such as OAI-SearchBot and PerplexityBot build the search index an AI assistant draws on.
• AI user agents, also called AI fetchers or user-triggered fetchers, such as ChatGPT-User and Claude-User, fetch a page at the moment a person asks an assistant something.

EdgeComet recognizes 122 bots from 153 user-agent patterns, puts each one in a bot category, and splits the main AI bots by these three purposes.

Why training, search and user bot requests are counted separately

Each type of bot request means something different for your site, so one combined number hides what happened:

• A search engine crawl is how a page gets into search results. Googlebot has to fetch a page before Google can index its content.
• A training crawl collects pages that may be used to train a model. No person asked for the page at that moment.
• An AI search crawl keeps an assistant's search index current.
• A user fetch happens because a person asked an assistant something. It is the only one of these four that a person's question triggers.

Take a month in which GPTBot fetched 10,000 pages and ChatGPT-User fetched 50. Added up, that is 10,050 AI requests, a total that moves almost only with the training crawl. If the training crawl doubles next month, the total nearly doubles even when the user fetches stay at 50.

Kept apart, the numbers say two things: OpenAI collected 10,000 pages that may be used for training, and 50 times a person's question in ChatGPT led it to one of your pages.

OpenAI, Anthropic and Perplexity document a separate user agent for each job, and EdgeComet's AI reports likewise show the three purposes side by side instead of one AI traffic figure.

How EdgeComet classifies bots

EdgeComet identifies the bot behind every request as it records it: it matches the user agent string against its catalog, stores the bot's name and category with the request, and checks the IP address against the IP ranges the bot's operator publishes (see bot verification). The main AI bots also get a purpose:

Purpose	OpenAI	Anthropic	Perplexity	Other bots in the group
Training	GPTBot	ClaudeBot	None	CCBot (Common Crawl), meta-externalagent (Meta), Bytespider, Amazonbot (Amazon), KimiBot (Moonshot AI), MistralAI-Training (Mistral AI), Reflectionbot (Reflection AI)
Search	OAI-SearchBot	Claude-SearchBot	PerplexityBot	Kimi-SearchBot, MistralAI-Index, KeenableBot, ExaSearchBot, YouBot (You.com)
User	ChatGPT-User	Claude-User, Claude Code	Perplexity-User	Kimi-User, MistralAI-User, Keenable-User, ExaUser

A server log gives you the user agent string and leaves the rest to you: keeping the patterns current, deciding which bots are AI crawlers and which of those collect training data, and checking IP addresses. In EdgeComet, each request already carries the bot's name, category and verification result. See How Log Analyzer differs from server log file analysis.

Bot categories

A bot category is the kind of bot a request came from. EdgeComet puts every recognized bot in one of six categories, and the Bot filter in Data Explorer > Bot log lists bots under these names:

Category	What it holds	Examples
Search Bots	Search engine crawlers and indexing bots	Googlebot, Bingbot, Applebot, YandexBot, Baiduspider
AI Bots	AI crawlers and the fetchers of AI assistants	GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, CCBot, meta-externalagent
SEO Tools	Crawlers of SEO analysis tools	AhrefsBot, SemrushBot, Screaming Frog
Social Bots	Link preview fetchers of social and chat apps	facebookexternalhit, LinkedInBot, Slackbot-LinkExpanding, WhatsApp, iMessage
Monitoring Bots	Uptime, cookie and site monitoring crawlers	UptimeRobot, Better Stack, Cookiebot
Other Bots	Named bots with no other category, and unrecognized bots	ShapBot, Meta Ads, Slackbot, unrecognized bots

Social Bots holds only the fetchers that build link previews, and Log Analyzer > Social Bots reports on them. Other bots of the same companies sit elsewhere:

• Amazonbot and Bytespider are training bots in AI Bots.
• Slackbot, Slack's plain robot, is in Other Bots. Slack calls it its "default, does everything, kitchen sink robot", while Slackbot-LinkExpanding "responds to links that Slack users post into their channels".
• Meta Ads and Taboolabot are in Other Bots. Meta says Meta-ExternalAds crawls "for use cases such as improving advertising and other business-related products and services".

The Bot Breakdown tab of Log Analyzer > Overview names three of them differently: Search Engines, Social Media and Monitoring. It adds a Fake Bots row for fake bots, and its AI Bots row counts every bot in the AI Bots category together, whatever its purpose.

When your CDN routes it to EdgeComet, a user agent that matches no known pattern but contains bot, crawler, spider or scraper is recorded as an unrecognized bot, listed as Other under Other Bots. A client with an ordinary browser user agent is not recorded as a bot, so bots that hide their identity are not in bot data.

Training bots

A training bot, also called an AI training crawler, collects pages that may be used to train AI models. OpenAI, for example, says GPTBot "is used to crawl content that may be used in training our generative AI foundation models."

EdgeComet counts nine training bots: GPTBot, ClaudeBot, CCBot (shown as Common Crawl), meta-externalagent (Meta Agent), Bytespider (Bytespider (TikTok)), Amazonbot, KimiBot, MistralAI-Training (Mistral Training) and Reflectionbot. They appear on the Training Bots card of the AI Crawlers tab in Log Analyzer > AI Bots. Amazon says Amazonbot is used to improve its products and services and "may be used to train Amazon AI models".

AI search bots

An AI search bot, also called an AI search crawler, crawls pages for an AI assistant's search index, as a search engine crawler does for a search engine. Perplexity says PerplexityBot "is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models."

EdgeComet counts eight AI search bots on the AI Search Bots card of the AI Crawlers tab: OAI-SearchBot (shown as OpenAI Search), Claude-SearchBot (Claude Search), PerplexityBot, Kimi-SearchBot (Kimi Search), MistralAI-Index (Mistral Index), KeenableBot, ExaSearchBot (Exa Search) and YouBot. They are not part of the Search Bots category, which holds search engine crawlers.

AI user bots

An AI user bot, also called an AI user agent or AI fetcher, fetches a page because a person asked an AI assistant something, not on a crawl schedule. OpenAI says that "when users ask ChatGPT or a CustomGPT a question, it may visit a web page with a ChatGPT-User agent."

EdgeComet counts eight AI user bots on their own tab, AI Chatbots: ChatGPT-User (shown as ChatGPT User), Claude-User (Claude User), Claude Code (a Claude-User user agent that names claude-code), Perplexity-User (Perplexity User), Kimi-User (Kimi User), MistralAI-User (Mistral User), Keenable-User (Keenable User) and ExaUser (Exa User). Each fetch is recorded with its URL and status code, which shows the pages people's questions led an assistant to.

See one AI provider's three purposes side by side

The AI Attention Map tab in Log Analyzer > AI Bots shows the training, search and user requests of one provider at a time, ChatGPT, Claude or Perplexity, as separate bands split by page type. See the AI Attention Map for how to read it.

AI bots with no purpose group

Some AI bots are in no purpose group, for example OAI-AdsBot, Google-Agent, Google-CloudVertexBot, DuckAssistBot, meta-webindexer and meta-externalfetcher. They count in the AI Bots category and in no purpose total. AI crawler user agents lists every AI bot EdgeComet recognizes.

Which robots.txt tokens control each AI bot

Most AI bots have their own robots.txt token, the name you write after User-agent:. Two tokens, Google-Extended and Applebot-Extended, belong to no crawler: they tell Google and Apple how they may use pages their search crawlers fetched. Checked against each operator's documentation on October 14, 2026, and on October 18, 2026 for Moonshot AI, Mistral AI, Reflection AI, Keenable and Exa:

Token	Operator	What disallowing it does, per the operator	Fetches pages	Name in EdgeComet
GPTBot	OpenAI	Signals that content should not be used to train OpenAI's foundation models	Yes	GPTBot
OAI-SearchBot	OpenAI	Keeps the site out of ChatGPT search results, though it can still appear as a navigational link	Yes	OpenAI Search
ClaudeBot	Anthropic	Signals that future content should be left out of Anthropic's training datasets	Yes	ClaudeBot
Claude-SearchBot	Anthropic	Stops Anthropic from indexing the site for search	Yes	Claude Search
Claude-User	Anthropic	Stops Claude from fetching the site's pages in response to a user's question	Yes	Claude User
PerplexityBot	Perplexity	Opts out of the crawler Perplexity uses to surface and link sites in its search results. Perplexity recommends allowing it for the site to appear there	Yes	PerplexityBot
Google-Extended	Google	Stops Google from using content it crawls to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google Search is not affected	No, Google's existing crawlers do the fetching	None
Google-CloudVertexBot	Google	Controls crawls that site owners request to build Vertex AI Agents	Yes	Google Vertex AI
Applebot-Extended	Apple	Opts content out of training Apple's foundation models. Search results are not affected	No, Applebot does the fetching	None
CCBot	Common Crawl	Stops Common Crawl from crawling the site for its open repository of web crawl data	Yes	Common Crawl
meta-externalagent	Meta	Controls Meta's crawler for uses such as training foundation AI models	Yes	Meta Agent
meta-webindexer	Meta	Controls the crawler that improves Meta AI search results	Yes	Meta Web Indexer
KimiBot	Moonshot AI	Controls the crawler that collects content potentially used to train Kimi's foundation models	Yes	KimiBot
Kimi-SearchBot	Moonshot AI	Controls the crawler that builds the search index behind Kimi's search features	Yes	Kimi Search
MistralAI-Training	Mistral AI	Controls the crawler that builds datasets for training Mistral's generative AI models	Yes	Mistral Training
Reflectionbot	Reflection AI	Controls the crawler that visits public pages to build and improve Reflection AI's models. Changes are usually picked up within about a day	Yes	Reflectionbot
KeenableBot	Keenable	Stops Keenable from indexing the site, which removes it from the answers Keenable's index gives AI agents and applications	Yes	KeenableBot
ExaSearchBot	Exa	Controls Exa's crawler for search and retrieval	Yes	Exa Search

The tokens are independent. In OpenAI's words, "a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI's generative AI foundation models." OpenAI, Perplexity and Meta say a change can take up to about 24 hours to take effect.

EdgeComet does not block bot requests based on robots.txt. Obeying it is up to each bot, and EdgeComet records the requests bots make, so a bot that fetches a disallowed path still shows up in your bot data.

Do AI user agents follow robots.txt?

Not always. Six operators say their user-triggered fetchers may not follow it, because a person asked for the page:

• OpenAI on ChatGPT-User: "Because these actions are initiated by a user, robots.txt rules may not apply."
• Perplexity on Perplexity-User: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules."
• Meta on meta-externalfetcher: "this crawler may bypass robots.txt rules."
• Google on its user-triggered fetchers, such as Google-Agent: "Because the fetch was requested by a user, these fetchers generally ignore robots.txt rules."
• Moonshot AI on Kimi-User: "robots.txt rules may not directly apply".
• Exa on ExaUser: "Because fetches are initiated by a user, robots.txt rules may not apply."

Anthropic treats Claude-User as a robots.txt control: "Disabling Claude-User on your site prevents our system from retrieving your content in response to a user query, which may reduce your site's visibility for user-directed web search." Keenable says it follows the Robots Exclusion Protocol for both KeenableBot and Keenable-User. For search opt-outs, OpenAI points to OAI-SearchBot instead.

Common questions

Why is Google-Extended not in my bot data?

Because nothing fetches pages under that name. Google says Google-Extended "doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity." Your data shows Googlebot's requests, and the Google-Extended line decides what Google may do with those pages.

Applebot-Extended works the same way: Apple says it "does not crawl webpages." See Why you won't find Google-Extended in your logs.

Do AI bot requests count toward EdgeComet usage?

No. EdgeComet usage counts Googlebot requests. All other bots, including Bingbot and every AI bot, are free.

What happens when a new AI crawler appears?

Until EdgeComet adds it to its catalog, a new crawler that your CDN routes to EdgeComet and whose user agent contains bot, crawler, spider or scraper is recorded as an unrecognized bot. A crawler with none of those words is not recorded as a bot.

To find one, open Data Explorer > Bot log, set the Bot filter (not Bot Type) to Other, and open each row's user agent from its Bot cell. AI crawler user agents lists the AI bots EdgeComet names today.

Related

• AI crawler user agents
• Search engine crawler user agents
• Track GPTBot, ClaudeBot and other AI bot activity
• How to verify Googlebot and AI bot IP addresses
• Why you won't find Google-Extended in your logs