Search engine bots EdgeComet identifies
EdgeComet identifies search engine bots by their user agent string and reports each one separately, so Googlebot fetching a page to index it is counted apart from AdsBot-Google checking the same page as an ad landing page.
Bot requests reach EdgeComet because your CDN or proxy routes them there, and each one is recorded as it is served. See Routing bot traffic for the setup. For the AI crawlers, see AI bots EdgeComet identifies.
Which search bots EdgeComet recognizes
The token in the first column is the string EdgeComet matches inside the user agent, and the second column is the name the request is reported under in the dashboard and in bot_name.
Google
| User agent token | Reported as | What it crawls for |
|---|---|---|
Googlebot | Googlebot Desktop, Googlebot Mobile | Google's search index, Images, Video, News and Discover |
Googlebot-Image | Googlebot Image | Google Images, and image, logo and favicon features in Search |
Googlebot-News | Googlebot News | Google News |
Googlebot-Video | Googlebot Video | Video features in Search and products that depend on video |
Storebot-Google | Google StoreBot | Google Shopping surfaces |
Google-InspectionTool | Google Inspection Tool | URL Inspection in Search Console and the Rich Result Test |
GoogleOther | GoogleOther | One-off fetches by Google product teams |
AdsBot-Google | Google AdsBot | Google Ads landing page quality, desktop |
AdsBot-Google-Mobile | Google AdsBot | Google Ads landing page quality, mobile |
Mediapartners-Google | Google AdSense | AdSense contextual ad matching |
APIs-Google | Google APIs | Delivery of push notifications from Google APIs |
GeminiiOS | Google Gemini | A person opening one of your links from the Gemini iOS app |
Three notes on the Google rows:
Googlebotarrives as two bots. EdgeComet separates the desktop and the mobile user agent, because the two can receive different pages. Filter on either, or on both, depending on the question.GoogleOther-ImageandGoogleOther-Videoare matched too, and both are reported as GoogleOther. Filter on the URL if you need to tell them apart.GeminiiOSis not a crawler and Google does not document it. It is the in-app browser of the Gemini iOS app, so a hit means a person tapped through to your page from an answer rather than a bot indexing it.
Microsoft Bing
| User agent token | Reported as | What it crawls for |
|---|---|---|
bingbot | Bingbot Desktop, Bingbot Mobile | Bing's search index and the answers Microsoft's AI systems generate |
adidxbot | Bing AdIdx, Bing AdIdx Mobile | Microsoft Advertising landing page checks |
BingPreview | Bing Preview | Page snapshots for Bing |
msnbot | MSNBot | Bing's retired predecessor crawler, and msnbot-media |
Apple
| User agent token | Reported as | What it crawls for |
|---|---|---|
Applebot | Applebot Desktop, Applebot Mobile, Applebot Other | Siri and Spotlight Suggestions, and Apple's search features |
iTMS | Apple Podcasts | Apple Podcasts content, not general search |
Applebot splits three ways: the Macintosh user agent is reported as desktop, the iPhone, iPad and iPod user agents as mobile, and anything else carrying the token, chiefly the plain (compatible; Applebot/0.1; +url) form, as other. See Verifying that a search bot is genuine before you read much into the last two.
Applebot-Extended is missing from this table on purpose. It never fetches anything. See Bots you will not find in your records.
Yandex
| User agent token | Reported as | What it crawls for |
|---|---|---|
YandexBot | Yandexbot Desktop, Yandexbot Mobile | Yandex's main search index |
YandexImages | Yandex Images | Yandex image search |
YandexVideo | Yandex Video | Yandex video search, and YandexVideoParser |
YandexMedia | Yandex Media | Multimedia data |
YandexNews | Yandex News | Yandex News |
YandexBlogs | Yandex Blogs | Post comments for blog search |
YandexMetrika | Yandex Metrika | Yandex Metrika analytics |
YandexDirect | Yandex Direct | Ad content matching, and YaDirectFetcher |
YandexAdNet | Yandex AdNet | Yandex advertising network |
YandexAccessibilityBot | Yandex Accessibility | Checking that pages are reachable for users |
Other search engines
| User agent token | Reported as | What it crawls for |
|---|---|---|
Baiduspider | Baiduspider | Baidu's main search index |
Baiduspider-image | Baidu Image | Baidu image search |
Baiduspider-video | Baidu Video | Baidu video search |
Baiduspider-news | Baidu News | Baidu news search |
Baiduspider-favo | Baidu Bookmark | Baidu bookmarks |
Baiduspider-ads | Baidu Business | Baidu business search |
Baiduspider-cpro | Baidu Union | Baidu's ad network |
Yahoo! Slurp | Yahoo Slurp | Yahoo search |
Sogou | Sogou | Sogou search |
PetalBot | PetalBot | Petal Search, operated by Huawei |
SeznamBot | SeznamBot | Seznam search, the Czech search engine |
Verifying that a search bot is genuine
EdgeComet checks a request's IP address against the ranges the search engine publishes, and records one of four outcomes: verified, fake, not validated, or error. The outcomes and what to do about each are described on the AI bots page; they work the same way for search bots.
What differs is coverage, because not every search engine publishes ranges:
| Search engine | IP checked | Why |
|---|---|---|
| Yes | Google publishes its crawler, fetcher and agent ranges | |
| Microsoft Bing | Yes | Bing publishes its crawler ranges |
| Apple | Yes | Apple publishes Applebot's ranges |
| Yandex | No | Yandex states that its addresses change frequently and does not disclose the list |
| Baidu, Yahoo, Sogou, PetalBot, Seznam | No | These operators publish no ranges to check against |
A request from a search engine in the bottom two rows always records as not validated, whatever its address. Read that as unknown rather than suspicious.
Spoofing concentrates where a name is worth borrowing. Googlebot is the most impersonated user agent on the web, and because Google publishes its ranges, EdgeComet reports a Googlebot request from outside them as a fake bot rather than letting it inflate your crawl numbers. Applebot is worth watching for the same reason: most of the Applebot user agents that reach a site are not Apple.
How search bot requests affect your usage
Usage counts Googlebot requests. Every other bot is free, including Bingbot, every other search engine on this page, and every AI bot. A crawl spike from Yandex or Baidu changes what your reports show, not what you pay.
See pricing for plan volumes.
Reading the data
Search bot requests are recorded, filtered and exported exactly like AI bot requests. Rather than repeat it here:
- Per bot and per URL. The three data grains, and the field that answers each question, are on the AI bots page.
- Every field. The Data Explorer column and filter reference lists each field, its type, and the modes it runs on.
- Getting it out. Dashboard export, the Data Explorer API, and the MCP server.
Frequently asked questions
Which search engine crawlers does EdgeComet track?
Every bot in the tables above, identified by its user agent token and reported individually: Google's crawlers and fetchers, Bing's, Applebot, Yandex's family, Baidu's family, Yahoo, Sogou, PetalBot and SeznamBot. The list grows with releases, so treat the dashboard's bot filter as the current version.
How do I separate Googlebot desktop from Googlebot mobile?
Filter on the bot. EdgeComet matches the two user agents separately and reports them as Googlebot Desktop and Googlebot Mobile, so you can compare what each one received for the same URL.
How does EdgeComet verify Googlebot?
It checks the request's IP address against the ranges Google publishes for its crawlers, fetchers and agents. A request carrying a Googlebot user agent from an address outside those ranges is reported as a fake bot.
Why are Yandex requests always "not validated"?
Because Yandex publishes no IP ranges. Its documentation states that its bots use addresses that change frequently and that the list is not disclosed, so there is nothing for EdgeComet to check a Yandex request against.
Do search bot requests count toward my usage?
Googlebot requests do. Every other search engine bot is free, as is every AI bot.
A bot in my logs is not in these tables. Where is it?
It may be an AI crawler, an SEO tool, a social crawler or a monitoring service, which are recorded under their own bot types. If its user agent matches nothing EdgeComet knows but contains bot, crawler, spider or scraper, it is counted under Other Bots, where you can read the full string in the user_agent field.
For what is recorded per request, how long it is kept, and how to alert on it, see Bot data: collection, limits, retention and export.