AI bots EdgeComet identifies

EdgeComet identifies AI bots by their user agent string and groups them by what the bot is for, so a request from ChatGPT-User is counted differently from a request from GPTBot.

Bot requests reach EdgeComet because your CDN or proxy routes them there, and each one is recorded as it is served. See Routing bot traffic for the setup. For Googlebot, Bingbot and the other search engine crawlers, see Search engine bots EdgeComet identifies.

Which AI crawlers EdgeComet recognizes

Every token in the first column is the string EdgeComet matches inside the user agent. Search your own records for the same string and you will find the same requests.

User agent token	Operator	Purpose	IP checked	What a hit means
GPTBot	OpenAI	Training	Yes	OpenAI collected the page to train its models.
OAI-SearchBot	OpenAI	Search	Yes	ChatGPT's search index read the page so it can surface it.
ChatGPT-User	OpenAI	User	Yes	Someone in ChatGPT asked something that sent it to your page.
OAI-AdsBot	OpenAI	Not grouped	Yes	OpenAI checked the page because it was submitted as an ad in ChatGPT.
ClaudeBot	Anthropic	Training	Yes	Anthropic collected the page to train its models.
Claude-SearchBot	Anthropic	Search	Yes	Claude's search index read the page so it can surface it.
Claude-User	Anthropic	User	Yes	Someone asked Claude something that sent it to your page.
PerplexityBot	Perplexity	Search	Yes	Perplexity's index read the page so it can link to it.
Perplexity-User	Perplexity	User	Yes	Someone asked Perplexity something that sent it to your page.
Google-CloudVertexBot	Google	Training	No	A Vertex AI agent, built by a site owner, crawled the page.
Google-Agent	Google	Training	Yes	An agent running on Google infrastructure visited the page for a user.
Meta-ExternalAgent	Meta	Training	No	Meta collected the page to train its foundation models.
Meta-WebIndexer	Meta	Search	No	Meta AI's search index read the page so it can cite it.
Meta-ExternalFetcher	Meta	User	No	Someone's request inside a Meta AI product sent it to your page.
CCBot	Common Crawl	Training	Yes	Your page entered the public Common Crawl archive that many models train on.
Bytespider	ByteDance	Training	No	ByteDance collected the page to train its models.
DuckAssistBot	DuckDuckGo	Not grouped	Yes	DuckDuckGo read the page for an AI-assisted answer that cites its sources.
AzureAI-SearchBot	Not declared	Not grouped	No	An AI search index read the page.
LinkupBot	Linkup	Not grouped	No	Linkup's AI search index read the page.
Timpibot	Timpi	Not grouped	No	Timpi collected the page for its search index.
ClueWeb-Crawler	Carnegie Mellon University	Not grouped	No	A research crawl collected the page for the ClueWeb datasets.

Four details the table cannot carry:

• IP checked says whether EdgeComet holds a published range list for that bot. A No means there is nothing to check the address against, not that the bot is suspect. See Verifying that an AI bot is genuine.
• Meta sends its tokens in lower case in the user agent, as meta-externalagent/1.1. EdgeComet matches them either way, so grep for either case in your own records.
• AzureAI-SearchBot declares no operator. Its user agent carries no URL and no contact address, and its operator publishes no crawler documentation. EdgeComet records the token and does not attribute it.
• Bytespider is recorded with bot type Social, not AI. It counts in the AI training group, but the AI Bots type filter does not include it. Filter on the bot itself to isolate it.

Google-Extended and Applebot-Extended are missing from this table on purpose. Neither one fetches pages. See Bots you will not find in your records.

How AI bots are grouped by purpose

EdgeComet puts each AI bot in one of three purpose groups, because a training crawl, an index refresh, and a live fetch on behalf of a person mean different things for your site.

Purpose	What it tells you	Bots
Training	Your content is being collected to train a model.	GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent, Google-CloudVertexBot, Google-Agent
Search	An AI search index is refreshing your pages.	OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer
User	A person asked an assistant a question and the assistant read your page.	ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher

Training. A training crawl says your content is being collected into a training set. It tends to be broad: these bots reach for pages that search crawlers ignore, so the pages they take are often not the pages you rank with.

Search. A search crawl says an AI answer engine is keeping its index of your site current. These bots behave like a search engine crawler, and the pages they revisit are the pages that engine considers worth keeping fresh.

User. A user fetch says a real person asked an assistant a question and the assistant went and read that page to answer it. It is the closest signal you get to demand: a proxy for an AI impression, recorded with the URL, the status code, and the page state the bot received. It is not a citation. EdgeComet observes the fetch, never the answer, so it cannot tell you whether the assistant quoted you or what it said.

Two honest limits:

• Some AI bots sit in no purpose group. The bots marked Not grouped in the table above count in your AI bot totals but do not appear in the three-way split.
• Google-Extended and Applebot-Extended also carry the training group, because the grouping has nowhere else to put them. They never fetch anything, so they contribute nothing to it.

What you can see per bot and per URL

Every bot request is recorded as one row, and the same requests can be read grouped by URL, so you can answer both "how often does PerplexityBot come" and "which pages did it take".

Mode	What one row is	Use it to answer
requests	One bot request	When a bot came, what it asked for, and what it got
unique_pages	One URL, with counts and averages for the period	Which pages a bot takes, and how often
comparison	One URL, the period against a comparison period, split into new, returning and lost	Which pages a bot started taking, kept taking, or stopped taking

Filter any mode by bot, so each answer can be narrowed to a single AI crawler. These are the fields the questions in this section run on:

Question	Field
Which AI bot was this?	bot_name
Was it an AI bot, a search bot, or a fake?	bot_type
Did the bot get a working page?	status_code
Was the page indexable when the bot got it?	index_status
Was the request worth the crawl budget?	allocation
How long did the bot wait for its answer?	serve_time
How large was the page it received?	page_size
Did the IP check pass?	ip_validated

The Data Explorer column and filter reference lists every field, its type, and the modes it runs on.

Verifying that an AI bot is genuine

EdgeComet checks each request's IP address against the ranges the bot's operator publishes, and records one of four outcomes.

Result	What it means	What to do about it
Verified	The request came from an address inside the operator's published ranges.	Nothing. The bot is who it says it is.
Fake	The request claimed a bot whose operator publishes ranges, and its address was outside all of them.	Treat the traffic as a spoof. Fake requests are reported separately, and a rule can alert you when they spike.
Not validated	EdgeComet holds no published range list for that bot, so there was nothing to check the address against.	Read it as unknown, not as suspicious. The operators of several AI bots publish no ranges at all.
Error	The address could not be read or checked.	Rare. Nothing to act on unless the count is large.

The IP checked column in the table above tells you which outcome a bot can reach. A bot marked Yes records as verified or fake. A bot marked No always records as not validated, whatever its address.

Verification works on the address, not on the name in the user agent, so matching the token alone proves nothing: anyone can send any user agent they like.

Bots you will not find in your records

Some AI bot names appear in robots.txt guidance but never in a server record, because they are not crawlers.

Google-Extended is an opt-out token. You put it in robots.txt to say whether Google may use the content it already crawls to train Gemini, and Google publishes no separate crawler behind it. Your records will show Googlebot fetching the page and no Google-Extended request at all. For the detail, see Why you will not find Google-Extended in your logs.

Applebot-Extended is the same shape, and it carries a sharper consequence. Apple's own documentation states that it does not crawl webpages, so any request arriving with an Applebot-Extended user agent is spoofed. That is exactly why EdgeComet checks it against Apple's published ranges: the check is what marks those hits as fake rather than letting them pass as an Apple crawl.

Both tokens carry the AI bot type and sit in the training purpose group, because the grouping has nowhere else to put them, not because they fetch anything. Neither appears as a bot you can select in the dashboard.

A bot that hides is not recognized as a bot. EdgeComet reads the user agent it is given. A user agent that matches no known pattern but contains bot, crawler, spider, or scraper is counted under Other Bots, with no name and no purpose group. A user agent that contains none of those words and matches nothing is not classified as a bot at all. Your bot data therefore covers crawlers that identify themselves, and says nothing about traffic that does not.

Getting this data out

Bot data leaves EdgeComet three ways, and all three read the same records:

• Dashboard export. Data Explorer exports any query as a CSV or Google Sheets file.
• API. The Data Explorer API returns the raw request log, the same requests aggregated per page, and Search Console data, as JSON or CSV.
• MCP. Connect an AI assistant or any other MCP client to the MCP server and query the same data in plain language.

Frequently asked questions

Which AI crawlers does EdgeComet track?

Every bot in the table above, identified by its user agent token and reported individually: the crawlers of OpenAI, Anthropic, Perplexity, Meta and Google, plus Common Crawl, Bytespider, and the smaller AI search crawlers. The list grows with releases, so treat the dashboard's bot filter as the current version.

How do I tell a training crawl from a live fetch on behalf of a user?

By the bot, not by the request. GPTBot, ClaudeBot, CCBot, Bytespider, and Meta-ExternalAgent collect content for training. ChatGPT-User, Claude-User, Perplexity-User, and Meta-ExternalFetcher fetch a page because a person asked their assistant a question. Filter on the bot, or on the purpose group, to separate them.

Can I see which pages GPTBot requested?

Yes. Filter bot requests to GPTBot and read them in unique_pages mode, which returns one row per URL with the request count and per-page averages. The comparison mode shows which pages it started taking, kept taking, or stopped taking against an earlier period.

How do I know a request claiming to be PerplexityBot really is?

Check the ip_validated field on the request. Perplexity publishes IP ranges for PerplexityBot, so EdgeComet checks every such request against them and records it as verified or fake. A request carrying the token from an address outside those ranges is a spoof.

Why is Google-Extended not in my bot data?

Because nothing fetches with it. Google-Extended is a robots.txt opt-out token that controls whether Google may train on content it already crawls, not a crawler that requests pages. The crawl you see in your records is Googlebot.

What happens when a new AI crawler appears?

Until EdgeComet adds it, its requests are counted under Other Bots if its user agent contains bot, crawler, spider, or scraper. To find one early, filter to Other Bots and read the user_agent field, which holds the full string the bot sent. New AI bots are added to the dictionary as they appear.

Do AI bot requests count toward my usage?

No. Usage counts Googlebot requests. All other bots, including Bingbot and every AI bot on this page, are free.

For what is recorded per request, how long it is kept, and how to alert on it, see Bot data: collection, limits, retention and export.