Bot data: collection, limits, retention and export
EdgeComet does not import or parse log files. Your CDN or proxy routes bot requests through EdgeComet, and each request is recorded as it is served, together with the page state the bot received. Log Analyzer reports, alerts, and Evergreen Crawl all read from this data.
Supported platforms and log formats
Bot traffic reaches EdgeComet through a routing rule at your CDN or proxy. The dashboard's Configuration - Integration page provides the setup for each platform.
| Platform | Setup |
|---|---|
| Cloudflare Workers | Guided automatic setup: connect your Cloudflare account and the wizard installs the routing worker |
| Fastly VCL | VCL configuration provided on the integration page |
| AWS CloudFront | Lambda@Edge functions provided on the integration page |
| nginx | Configuration provided on the integration page |
| Other CDNs and proxies, such as HAProxy | A custom integration guide for any layer that can route requests by user agent |
See Routing bot traffic for the setup steps.
Log formats
EdgeComet requires no log format. It records requests itself, so the web server behind your CDN or proxy (Apache, IIS, nginx, or any other) and the format of its logs do not matter.
Requirements and access
To start collecting bot data, you need:
- Access to change routing rules at your CDN or proxy.
- Permission to allow EdgeComet's requests through your firewall or bot protection. EdgeComet fetches pages from your origin to render and cache them.
- Optional: a Google account with access to your site's Search Console property, to connect Search Console.
Setup requires no changes to your application code. Only bot requests are routed through EdgeComet; human visitors never pass through it.
What is recorded per request
EdgeComet records every bot request it serves as one row.
| Group | Fields |
|---|---|
| Request | URL, timestamp, event type (served from cache, rendered, or fetched directly), status code, redirect target, page size, cache age |
| Bot | Bot name, bot type, user agent, IP address, IP verification result |
| Performance | Serve time, render time, network requests and bytes, JavaScript errors and warnings, failed and blocked requests, third-party requests and bytes |
| Page state | Title, meta description, canonical URL, meta robots, H1, H2 and H3 headings, word count, hreflang, structured data types, internal, external and nofollow link counts, image and alt text counts, indexability status |
For a request served from cache, the page state comes from the render that produced the cached copy.
The Data Explorer column and filter reference lists these fields with their API keys. The redirect target is available through the MCP server.
Bot identification
EdgeComet identifies bots by user agent and verifies them by IP address.
- Recognized bots. Search engine and AI bots, such as Googlebot, Google's ads crawlers, Bingbot, and the bots of ChatGPT, Claude, and Perplexity, plus SEO and marketing tools, social media crawlers, monitoring services, and scrapers that identify themselves as bots.
- Verification. Requests from bots whose operators publish IP ranges are checked against those published ranges. Each request gets one result: verified, fake, not validated (the operator publishes no ranges), or error.
- Fake bots. A request that claims a verifiable bot's user agent from an IP address outside that bot's published ranges is reported separately as a fake bot.
AI bots are grouped by purpose:
| Purpose | What the bot does | Examples |
|---|---|---|
| Training | Crawls pages to build model training data | GPTBot, ClaudeBot, CCBot |
| Search | Builds and refreshes an AI search index | OAI-SearchBot, Claude-SearchBot, PerplexityBot |
| User | Fetches a page because a person asked an assistant a question | ChatGPT-User, Claude-User, Perplexity-User |
Retention
EdgeComet keeps your bot data for as long as your subscription is active.
Historical data
Bot data collection starts when routing is active. EdgeComet reports on the requests it serves, so bot-side reports have no data from before integration, and EdgeComet does not import historical log files.
Search Console data covers earlier dates. The first sync after you connect Search Console backfills the previous 180 days of performance data.
Export, API and MCP
You can take bot data out of EdgeComet three ways: export it from the dashboard, read it through the API, or query it through MCP.
- Dashboard export. Data Explorer exports any query as a CSV file or a Google Sheets file. Report numbers link into Data Explorer with the filters already applied.
- API. The Data Explorer API returns the raw bot request log (one row per request), the same requests aggregated per page, and Search Console pages and keywords, as JSON or CSV.
- MCP. Connect an AI assistant such as Claude, or any other MCP client, to the MCP server and query the same data in plain language. Its
exporttool returns large results as a CSV file.
Alerts on bot data
Alert rules read bot requests as EdgeComet records them and evaluate every 5 minutes, so a change in how bots crawl your site surfaces within minutes.
Each rule combines three parts:
- Metric. What the rule measures, such as the number of bot requests, 4xx or 5xx responses, fake bot requests, render time, or cache hit rate.
- Condition. A fixed threshold, a percentage change against the same hour yesterday, the same hour last week, or the 7-day average, or a deviation from a learned baseline of normal traffic for each hour of the week. A condition watches for an increase, a decrease, or either.
- Scope filters. The Data Explorer filters, such as bot, bot type, URL, status code, indexability, crawl budget allocation, and page fields like meta robots or canonical URL. A rule without filters watches all bot traffic.
Pre-built templates such as Crawl Traffic Drop and Fake Bot Spike activate in one click. SEO change rules such as Page Went Noindex and Canonical URL Changed fire when the same change appears on a set number of URLs.
Alerts are delivered to the dashboard, email, Slack, or webhooks. See SEO alerts for the full template list.
Usage and data volume
- Usage basis. Usage counts Googlebot requests. All other bots, including Bingbot and AI bots, are free.
- No log shipping. Nothing is streamed from your servers: there are no log shipments, storage buckets, or export jobs. Bot requests are answered from EdgeComet's cache, and your origin builds a page only when the cache refreshes it.
See pricing for plan volumes.
Log files compared with request-path capture
Log file analysis works on files that a web server or CDN writes and that you upload or export on a schedule. EdgeComet records each request in the request path instead.
| Log file upload or scheduled export | Request-path capture in EdgeComet | |
|---|---|---|
| Data age when you look | Behind by the upload or export interval | Recorded as each request is served |
| Setup | Export jobs, storage, and format mapping | One routing rule at your CDN or proxy |
| Log formats | Must match what the parser expects | None: requests are recorded directly |
| What each request shows | URL, status code, user agent, IP address, timing | The same, plus the page state the bot received |
| Crawl drop alerts | After the next import | Rules evaluated every 5 minutes |
| Site audit | A separate crawler run | Evergreen Crawl, built from the same bot requests |
| History before setup | Old log files can be imported | Starts at integration; Search Console covers earlier dates |
Because records arrive continuously, per-URL change tracking, alert rules that run every 5 minutes, and Evergreen Crawl all work from the same data.
Frequently asked questions
Can I upload log files?
No. EdgeComet does not accept log file uploads. It records each bot request as your CDN or proxy routes it through EdgeComet, so the data needs no upload, export, or parsing.
Can I import historical logs?
No. Bot data starts when routing is active, and EdgeComet does not import older log files. Search Console data covers earlier dates: the first sync backfills the previous 180 days of performance data.
Which log formats are supported?
EdgeComet needs no log format. It records requests directly, so it works the same whether your origin runs Apache, IIS, nginx, or any other web server.
How long is bot data kept?
EdgeComet keeps your bot data for as long as your subscription is active.
Can I export raw bot data, or access it through an API or MCP?
Yes. Data Explorer exports query results as a CSV or Google Sheets file, the Data Explorer API returns the raw request log one row per request, and the MCP server lets an AI assistant query the same data.
Does EdgeComet find pages that bots never visit?
Only partly. EdgeComet does not crawl, so a page appears once a bot requests it or a fetched page links to it. To find pages outside both, compare bot data with your XML sitemap.
How does EdgeComet verify Googlebot?
EdgeComet checks each request's IP address against the IP ranges Google publishes for its crawlers. A request with a Googlebot user agent from an address outside those ranges is reported as a fake bot.