EdgeComet data freshness: how often each source updates and how long data is kept
EdgeComet records each bot request within seconds of serving it. Every other source updates on its own schedule, from every 5 minutes for page changes to one day at a time for Search Console.
This page lists, for each source, how often it updates, how far behind it runs, how much history you get on day one, and how long it is kept. For how EdgeComet collects each source, see How EdgeComet collects data.
Bot data is kept for as long as your subscription is active.
What this covers
Two tables cover every source EdgeComet reads and every analysis it builds from them. Each row links to a section with the details.
Sources EdgeComet reads
| Source | How often it updates | How far behind it runs | History when you start |
|---|---|---|---|
| Bot requests | As each request is served | Seconds. Charts for longer periods refresh a few times a day | From the moment your routing is active |
| Search Console | One new day every day | About 3 days | The 360 days before you connect, 180 during a trial |
| Sitemaps | Every 12 hours by default, from 30 minutes to 7 days | One run interval | From the first fetch |
| Merchant Center feed | Every hour by default, as often as every 5 minutes | One fetch interval | From the first fetch |
| Keyword and search data | Each time you or Ask request it | None: read live | Not stored |
| Verified bot IP ranges | Every 24 hours | Up to a day | Not applicable |
Analyses EdgeComet builds
| Analysis | How often it updates | First result |
|---|---|---|
| Page changes | Every 5 minutes | After the second fetch of a page |
| Site releases | Every hour | Once the site has enough change history |
| Evergreen Crawl, Current | Every day, over the last 30 days | 3 days after the first verified bot visit |
| Evergreen Crawl, Monthly | In the first days of each month | In the first days of the next month |
| Action Board | Every 14 days | Usually 3 to 4 days after the first verified bot visit |
| Alerts | Every 5 minutes to every day, by rule | At the rule's first check |
| Fan-out queries | With each Search Console day, labels the next day | With the first imported days |
How fresh the cached pages bots receive are is a separate question, set by your pre-cache rules. See How soon bots get a change.
How to read it
Each column answers one question:
- How often it updates: how often EdgeComet reads the source or rebuilds the analysis.
- How far behind it runs: the age of the newest data in reports, right after an update.
- History when you start and First result: what you see on the first day. EdgeComet cannot fetch bot data from before your install, because it records requests as it serves them.
Bot requests
EdgeComet records each bot request as it serves it, and the request appears in Data Explorer > Bot log within seconds.
- Reports: the Last 24h view refreshes within minutes. Charts for 7 days or longer refresh a few times a day, which is why a request you just made may not show in a 30-day chart yet.
- History: bot data starts when your routing is active. Bot reports open at the first verified bot visit, usually within 24 hours of the install. Older server logs cannot be imported. See Can I import historical logs into EdgeComet?
- Periods: reports, Data Explorer and the Data Explorer API read any period up to 365 days.
- Kept: for as long as your subscription is active.
Search Console
EdgeComet imports one new day of Search Console data every day, and each day arrives about 3 days after it ends.
- When you connect: EdgeComet imports the last 360 days on a paid plan, or the last 180 days during a trial, newest first. Configuration > Search Console shows the progress as days loaded out of the total, such as "120 of 361 days". Search reports work from the first imported days. When the website owner subscribes after a trial, EdgeComet imports the older days the next night.
- In reports: Search reports end at the last complete day. Search > Keywords and Traffic decay show it as "Data as of" and the date.
- Disconnecting stops the import, and your branded keywords stay saved. See Disconnect.
Google withholds rare queries from keyword data, which keeps keyword totals below page totals. See Anonymized queries.
Sitemaps
EdgeComet fetches your sitemaps on the Run interval of the Sitemap Re-cache rule, every 12 hours by default, and from every 30 minutes to every 7 days.
- Data Explorer > Feeds > Sitemap shows the newest fetch, with "Fetched" and its time, and the number of URLs.
- History starts with the first fetch after you turn on the rule.
- A URL removed from your sitemap leaves the Sitemap tab at the next fetch. Sitemap comparison shows what changed between two fetches. See Sitemaps and product feeds.
To set the interval, see Pre-cache from sitemaps.
Merchant Center feed
EdgeComet fetches your Merchant Center product feed on the Fetch interval of the Feed Re-cache rule, every hour by default and as often as every 5 minutes.
- A product whose price, availability, title, link, sale price or availability date changes in the feed is re-cached at high priority on the next fetch.
- The feed's fetch history covers the last 30 days. See Fetch history.
- History starts with the first fetch after you add the feed.
To set the interval, see Pre-cache from a Merchant Center feed.
Page changes and site releases
EdgeComet compares each new fetch of a page with its last known state every 5 minutes, and a change appears in Changes within minutes of the fetch that carried it.
- Fetches decide when a change is seen. A bot request that EdgeComet renders or fetches, or a pre-cache refresh, carries the page's current state. In the dashboard's words: "A change is detected when a bot fetches a page again, so the first one appears after the second fetch of a page."
- Faster detection: Smart Re-cache re-checks pages bots request as often as they change.
- When checks fall behind, the Changes strip shows "Last update" and the time of the last check.
Site releases
EdgeComet updates site releases every hour, once your site has enough change history. See Site releases.
Evergreen Crawl
The Current crawl covers the last 30 days of bot and pre-cache fetches and is rebuilt every day.
- First crawl: 3 days after the first verified bot visit. See Your first crawl.
- Rebuild by hand: Rebuild now in the Crawls manager, once a day. See Rebuild the Current crawl.
- Monthly crawls: built in the first days of each month from the 30 days before. They build a month-by-month history you can compare.
- Custom crawls: built when you start one. A site keeps up to 5 at a time, and each expires 90 days after you start it unless you keep it. See Crawl kinds.
Action Board
The Action Board analyzes your site every 14 days. The first board arrives once the site has enough data, usually 3 to 4 days after the first verified bot visit and after 7 days at most. See Before the first board.
Alerts and the weekly digest
Each alert rule checks on the schedule of the data it reads:
- Bot requests and renders: every 5 minutes, 15 minutes, hour or day, depending on the rule.
- Page changes: every hour by default.
- Site releases: every hour.
- Search Console: when a complete day arrives, about 3 days after it ends.
- Evergreen Crawl: when a crawl finishes.
See How often each alert checks. The weekly digest goes out every Wednesday at 13:00 UTC and covers the previous Monday to Sunday.
Fan-out queries
Fan-out query counts come from your Search Console data and update as each day arrives. The labels, such as question type, intent and topic, are added the next day. Until then the report counts the keywords as pending classification. See Pending classification.
Keyword and search data
Search volume, difficulty, ranked keywords, live search results and backlink data in Ask and MCP are read live from a keyword data provider each time you or Ask request them. EdgeComet does not store them between requests.
A table of this data attached to an Action Board finding keeps the date it was fetched. See Live keyword and search data.
Verified bot IP ranges and integration health
EdgeComet refreshes the IP ranges that Google, Bing, OpenAI and other operators publish every 24 hours, and checks each bot request against them as it arrives. See IP validation.
Integration health, the check that verified bots keep arriving, is re-checked about every 15 minutes.
How long data is kept
Bot data is kept for as long as your subscription is active. Other records are kept for set periods:
| Record | Kept for |
|---|---|
| Bot requests | As long as your subscription is active |
| Custom Evergreen crawls | 90 days after you start them, unless you keep them |
| Closed alert incidents | 90 days |
| Alert check history | 180 days |
| Smart Re-cache run history | 14 days |
| Other pre-cache rule run history | 90 days |
| Merchant feed fetch history | 30 days |
| Ask live tables | 24 hours, then reopen them to refresh |
| Custom report versions | The 50 most recent |
To delete a website or your account, contact [email protected].
Date windows in the API and MCP
The Data Explorer API reads any period up to 365 days, like the dashboard. MCP tools read up to 90 days per call, 90 by default. The MCP limit applies to each call's window, not to the history EdgeComet keeps.
Changes to this list
- 2026-10-26: First version.