Route bots to EdgeComet with Fastly VCL

On Fastly, five VCL snippets send search engine and AI bot page requests to EdgeComet and every other request through your service as before. When EdgeComet cannot answer, the request restarts and goes to your origin. This is the Fastly setup for prerendering or dynamic rendering, plus EdgeComet's caching and bot analytics.

The snippets leave visitors, static files and Fastly's cache untouched. Only crawler page requests change their backend.

In the dashboard, the guide is under Configuration > Connections > CDN integration > Fastly VCL, filled in with your render key and render host.

How it works

Snippet	Placement	Job
edgecomet_backend	init	Declares EdgeComet's render host as a backend, with its own Host header, TLS with SNI, a 60-second first byte timeout and a health check
edgecomet_recv	recv	Picks crawler GET and HEAD page requests on their first Fastly visit, sets the EdgeComet backend and passes them
edgecomet_pass	pass	Builds the request to EdgeComet on bereq: the encoded URL, your render key and the bot's IP. The client request stays unchanged
edgecomet_fetch	fetch	Restarts the request when EdgeComet's answer has no EC-Source header or a 5xx status
edgecomet_error	error	Restarts the request when EdgeComet times out, cannot be reached or fails its health check

After a restart, the request is the one the bot sent. The routing snippet skips restarted requests, and Fastly sends it to your origin as if EdgeComet were not there.

Crawler requests go from the first Fastly server straight to EdgeComet, without shielding, and Fastly does not cache them. EdgeComet keeps its own cache.

Before you start

• A Fastly VCL service that serves your site, and permission to edit and activate it.
• Your Render Key and Render Host from the dashboard guide. If your guide shows render-de.edgecomet.com, replace every render-us.edgecomet.com in the backend snippet.
• Your Fastly service ID, shown on the service page.

VCL snippets

All five snippets are Regular snippets. Replace YOUR_FASTLY_SERVICE_ID and YOUR_RENDER_KEY.

edgecomet_backend (init)

backend edge_comet {
  .dynamic = true;
  .share_key = "YOUR_FASTLY_SERVICE_ID";
  .host = "render-us.edgecomet.com";
  .port = "443";
  .ssl = true;
  .ssl_cert_hostname = "render-us.edgecomet.com";
  .ssl_sni_hostname = "render-us.edgecomet.com";
  .ssl_check_cert = always;
  .host_header = "render-us.edgecomet.com";
  .always_use_host_header = true;
  .connect_timeout = 10s;
  .first_byte_timeout = 60s;
  .between_bytes_timeout = 15s;
  .probe = {
    .request = "GET /health HTTP/1.1" "Host: render-us.edgecomet.com" "Connection: close";
    .expected_response = 200;
    .interval = 30s;
    .timeout = 5s;
    .window = 5;
    .threshold = 3;
    .initial = 3;
  }
}

edgecomet_recv (recv, priority 100)

# Send crawler page requests to EdgeComet
if (fastly.ff.visits_this_service == 0
    && req.restarts == 0
    && (req.method == "GET" || req.method == "HEAD")
    && !req.http.X-Edge-Render
    && req.url.ext !~ "(?i)^(avif|css|eot|gif|gz|ico|jpeg|jpg|js|json|map|mp3|mp4|ogg|otf|pdf|png|svg|ttf|txt|wasm|wav|webm|webmanifest|webp|woff|woff2|xml|zip)$"
    && req.http.User-Agent ~ "(?i)(bot|crawl|spider|slurp|whatsapp|snapchat|facebookexternalhit|amzn-user|claude-user|perplexity-user|chatgpt-user)") {
  set req.backend = edge_comet;
  return(pass);
}

edgecomet_pass (pass, priority 100)

# Build the EdgeComet request; the client request stays unchanged for the fallback
declare local var.edgecomet_url STRING;
if (req.backend == edge_comet) {
  set var.edgecomet_url = if(fastly_info.edge.is_tls, "https://", "http://") + req.http.Host + req.url;
  set bereq.url = "/render?url=" + urlencode(var.edgecomet_url);
  set bereq.http.X-Render-Key = "YOUR_RENDER_KEY";
  set bereq.http.X-Forwarded-For = client.ip;
  set bereq.http.X-Forwarded-Proto = if(fastly_info.edge.is_tls, "https", "http");
}

edgecomet_fetch (fetch, priority 10)

# Fall back to the origin unless EdgeComet produced the response
if (req.backend == edge_comet && req.restarts == 0
    && (!beresp.http.EC-Source || beresp.status >= 500)) {
  restart;
}

edgecomet_error (error, priority 10)

# Fall back to the origin when EdgeComet is unreachable, times out or is unhealthy
if (req.backend == edge_comet && req.restarts == 0) {
  restart;
}

Why the snippets look this way

• .host_header with .always_use_host_header sends the render host as Host. Fastly forwards your site's Host by default, which the render host refuses with a 403.
• return(pass) right after set req.backend keeps Fastly's default backend assignment, which runs after snippets, from replacing EdgeComet. It also keeps crawler requests out of Fastly's cache.
• fastly.ff.visits_this_service == 0 routes a request only on its first Fastly server. Clients cannot fake this value.
• The pass snippet changes bereq, not req. A restart keeps req with its changes, and an unchanged req reaches your origin cleanly.
• urlencode encodes :, /, ?, = and & in the page URL, which keeps the query string whole.
• Priority 10 on the fetch and error snippets runs the fallback before your own fetch or error snippets, such as a custom error page.
• The probe sends bots to your origin without waiting while EdgeComet fails 3 of its last 5 health checks.

Install in the Fastly control panel

1. In the Fastly control panel, select the service that serves your site.

2. Click Edit configuration and clone the active version.

3. Click VCL, then the VCL snippets tab.

4. If snippets from an earlier EdgeComet guide exist, such as eg_backend_init, eg_router_recv and eg_failopen_error, delete them. Two backends named edge_comet stop the activation.

5. Click Add snippet, or Add a VCL snippet while the service has none: Regular, name edgecomet_backend, placement init. Paste the backend snippet and click Add.

   

6. Add edgecomet_recv: placement Within subroutine, subroutine recv, priority 100.

   

7. Add edgecomet_pass: Within subroutine, pass, priority 100.

8. Add edgecomet_fetch: Within subroutine, fetch, priority 10.

9. Add edgecomet_error: Within subroutine, error, priority 10.

10. Add the firewall exemption if your service blocks or rate-limits requests.

11. From the Activate menu, select Activate on Production. No purge is needed: crawler requests are never cached by Fastly.

12. Run Test connection and Verify install in the dashboard guide.

Lower priority numbers run first. Snippets that block or redirect requests can run before edgecomet_recv. A recv snippet that ends vcl_recv with return(lookup) or return(pass) for page requests must run after it, or bots never reach EdgeComet.

To see where each snippet lands, click the three dots next to it and select Show in generated VCL.

Custom VCL

Snippets also work with a custom VCL file: Fastly renders them inside the #FASTLY macros your file keeps. To paste the code into the file instead:

• the backend at the top level, outside any sub;
• the recv code in sub vcl_recv, right after #FASTLY recv;
• the pass code in sub vcl_pass, after #FASTLY pass;
• the fetch code in sub vcl_fetch, right after #FASTLY fetch, before the first return;
• the error code in sub vcl_error, after #FASTLY error.

Upload it under VCL > Custom VCL, then activate.

What bots get when EdgeComet cannot answer

Case	Bots get
EdgeComet answers with EC-Source and a status below 500, including redirects, 404 and 410	EdgeComet's response
5xx from EdgeComet	Your origin's page
An answer without EC-Source: 401 for a wrong key, 400, a 404 for a wrong path, 502 No healthy backend	Your origin's page
Connection failure, TLS failure or no first byte within 60 seconds	Your origin's page, after the failed attempt
EdgeComet failing its health check	Your origin's page, without waiting

One case cannot fall back: a timeout between bytes after EdgeComet has sent its headers. Fastly then treats the response as complete. Fastly allows three restarts per request, and the fallback uses one.

Let EdgeComet through Fastly security

EdgeComet's own requests to your site pass through your Fastly service with X-Render-Key and X-Edge-Render. The routing snippet ignores them, and your blocking rules still see them.

In VCL, wrap your blocking rules, such as ACLs, geo blocks, user agent blocks and ratelimit.check_rate checks, in a condition on the exact key:

# Skip blocking for EdgeComet's requests
if (req.http.X-Render-Key != "YOUR_RENDER_KEY") {
  # your existing ACL, geo, user-agent or rate-limit checks
}

Do not use return(pass) for the exemption. It ends vcl_recv before Fastly assigns the backend and skips the cache.

For Next-Gen WAF, see Fastly Next-Gen WAF.

Fastly documents no header-based exemption for Edge Rate Limiting policies set up in the control panel, only exclusions by client. EdgeComet publishes no IP list. If such a policy limits EdgeComet's requests, contact [email protected].

Check the integration

# Crawler through Fastly: expect EC-Source and the other EC- headers
curl -s -o /dev/null -D - -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  "https://www.example.com/" | grep -i "^ec-"

# Browser: expect no EC- headers
curl -s -o /dev/null -D - -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36" \
  "https://www.example.com/" | grep -i "^ec-"

# EdgeComet directly, bypassing Fastly: expect EC-Source; 401 means a wrong key, domain or an inactive website
curl -s -o /dev/null -D - -A "Googlebot" -H "X-Render-Key: YOUR_RENDER_KEY" \
  "https://render-us.edgecomet.com/render?url=https%3A%2F%2Fwww.example.com%2F" | grep -i -E "^HTTP|^ec-"

Repeat crawler requests show EC-Source: render_cache and a growing EC-Cache-Age, from EdgeComet's cache. Then run Test connection and Verify install, and see Check that search and AI bots reach EdgeComet.

Update or remove

• After a render key rotation, edit edgecomet_pass and your firewall condition in a cloned version and activate. Created as a Dynamic snippet, edgecomet_pass takes changes without an activation.
• To remove EdgeComet, delete the five snippets in a cloned version and activate.

If it does not work

Bot requests return no EC- headers

The version is not active, a recv snippet with a lower priority ends vcl_recv first, the request is not a crawler page request, or Fastly fell back. Check the order with Show in generated VCL, then test EdgeComet directly with the last command in Check the integration.

The direct test returns 401 Authentication failed: invalid render key or domain mismatch

The key in edgecomet_pass differs from the dashboard key, often after a rotation, or the host is not one of your website's domains, or the website is inactive. Fastly falls back and bots get your origin pages.

A Cloudflare 403 Forbidden page from the render host

The request reached the render host with your site's Host header. Replace an older backend snippet with edgecomet_backend, which overrides Host.

Activation fails with a VCL error

Snippets from an earlier guide still declare edge_comet, a snippet has the wrong placement, or a placeholder is still in place. bereq is not available in vcl_recv.

Slow bot responses

The first byte timeout of 60 seconds must stay above EdgeComet's render timeout, 15 seconds by default. A fallback adds the time of the failed attempt.

Visitors receive EdgeComet's pages

The crawler pattern in edgecomet_recv matches a browser user agent, or other VCL sets req.backend = edge_comet. Test the pattern against common browser user agents.

Requests loop or end in a 503 after restarts

Another snippet removes X-Edge-Render, or your VCL restarts requests too and passes Fastly's limit of three restarts.

Static files bypass the Fastly cache after an earlier install

An older routing snippet passed static files for every visitor. Replace it with edgecomet_recv.

Test connection fails

Your service blocks EdgeComet's request with a 403 or 429. Add the firewall exemption.

Related

• How EdgeComet routes bot traffic
• Let EdgeComet through your firewall
• Fastly for the self-hosted engine, which covers the open-source engine you run yourself