Add EdgeComet to an existing Cloudflare Worker

If a Cloudflare Worker already runs on your site, add EdgeComet to it: one module file and one call at the start of your fetch handler. Bot page requests get EdgeComet's response. Every other request, and any bot request EdgeComet cannot answer, runs your code as before.

Cloudflare runs one Worker per request on a route. The automatic install stops when another Worker has a route on your site's hostname, with We stopped before changing anything. Merging keeps your Worker and adds EdgeComet's routing to it.

When to merge

Merge when your Worker has a route that covers the pages bots crawl, such as example.com/*. When several route patterns match a request, Cloudflare runs the Worker on the most specific one. Two Workers on the same pattern cannot both run.

Other setups need no merge:

• Your Worker covers only paths bots do not need rendered, such as example.com/api/*. Run EdgeComet as its own Worker on example.com/*, and your more specific route keeps its paths. The automatic install stops on any other Worker route on your hostname, whatever its path. Deploy EdgeComet's Worker by hand instead. See Run EdgeComet as its own Worker.
• Your site is a Worker on a Custom Domain. Cloudflare treats a Custom Domain Worker as the origin, and a Worker on a route runs in front of it. EdgeComet's Worker passes visitors on with fetch(request), which reaches your Worker. Try the automatic install first. If a check fails, see Run EdgeComet as its own Worker.
• You would rather not change your code. On the stopped screen, click Schedule a call with an engineer, and EdgeComet's engineers wire the two together with you.

How the merged Worker works

The EdgeComet call runs first and returns either a response for the bot or null. On null, your code runs. The call returns null without contacting EdgeComet for:

1. Methods other than GET and HEAD.
2. Requests with an X-Edge-Render header. EdgeComet adds this header to its own requests to your site, which prevents loops.
3. Paths that end in a static file extension, such as .js, .css or .png.
4. User agents that are not crawlers, such as human visitors.

Everything left goes to EdgeComet as GET https://{render-host}/render?url={encoded page URL} with your render key. The call returns EdgeComet's response when it carries an EC-Source header and a status below 500, and null otherwise. For the full rules, see How EdgeComet routes bot traffic.

EdgeComet renders the page your Worker serves

When EdgeComet needs a fresh copy, it fetches your page through Cloudflare with X-Edge-Render. The call returns null for that request, and your code builds the page as it does for a visitor.

Rewrites, HTMLRewriter changes and response headers your Worker makes reach the page EdgeComet renders. Of your headers, bots get Content-Type, Cache-Control, Expires, Last-Modified, ETag, Location, X-Robots-Tag and Link. See What bots receive.

EdgeComet's requests carry no cookies, a browser user agent and EdgeComet's own IP address. Changes your Worker keys to a session, a cookie, the bot's user agent or the visitor's country follow EdgeComet's request, not the bot's. See What EdgeComet sends to your origin.

Before you start

• Your Worker's source code and a way to deploy it: Wrangler or the Cloudflare dashboard editor.
• Your Render Key and Render Host from the guide in the EdgeComet dashboard: Configuration > Connections > CDN integration, then Legacy: manual worker install guide on the Cloudflare card. Treat the render key like a password. See The render key.
• A firewall rule that lets EdgeComet's requests through. See Let EdgeComet through your firewall.

Add the EdgeComet module

Save this file as edgecomet.js next to your Worker's entry file. If your guide shows render-de.edgecomet.com, use it in RENDER_HOST.

// EdgeComet routing for an existing Cloudflare Worker

const EDGECOMET = {
  RENDER_HOST: "https://render-us.edgecomet.com",
  TIMEOUT: 60000,
};

const CRAWLER_PATTERNS = [
  /bot/i,
  /crawl/i,
  /spider/i,
  /slurp/i,
  /WhatsApp/i,
  /Snapchat/i,
  /facebookexternalhit/i,
  /AMZN-User/i,
  /Claude-User/i,
  /Perplexity-User/i,
  /ChatGPT-User/i,
];

const STATIC_EXTENSIONS = /\.(avif|css|eot|gif|gz|ico|jpeg|jpg|js|json|map|mp3|mp4|ogg|otf|pdf|png|svg|ttf|txt|wasm|wav|webm|webmanifest|webp|woff|woff2|xml|zip)$/i;

// Headers that belong to the incoming connection and are not forwarded
const CONNECTION_HEADERS = new Set([
  "host",
  "connection",
  "keep-alive",
  "proxy-authorization",
  "proxy-authenticate",
  "te",
  "trailer",
  "transfer-encoding",
  "upgrade",
  "content-length",
]);

function isCrawler(userAgent) {
  return CRAWLER_PATTERNS.some((pattern) => pattern.test(userAgent));
}

function buildRenderHeaders(request, url, renderKey) {
  const headers = new Headers();

  for (const [name, value] of request.headers) {
    if (!CONNECTION_HEADERS.has(name.toLowerCase())) {
      headers.append(name, value);
    }
  }

  // Cloudflare rewrites Accept-Encoding before the Worker runs
  const clientAcceptEncoding = request.cf?.clientAcceptEncoding;
  if (clientAcceptEncoding) {
    headers.set("Accept-Encoding", clientAcceptEncoding);
  }

  // EdgeComet headers go last: a forwarded header cannot override them
  const clientIp = request.headers.get("CF-Connecting-IP") || "";
  headers.set("X-Render-Key", renderKey);
  headers.set("X-Forwarded-Proto", url.protocol.replace(":", ""));
  headers.set("X-Real-IP", clientIp);
  headers.set("X-Forwarded-For", clientIp);

  return headers;
}

// Returns EdgeComet's response for a bot page request, or null to run your own code
export async function edgeCometResponse(request, env) {
  const renderKey = env?.EDGECOMET_RENDER_KEY;
  if (!renderKey) return null;

  // Only GET and HEAD are renderable
  if (request.method !== "GET" && request.method !== "HEAD") return null;

  // Loop prevention: EdgeComet's own requests to your site run your code
  if (request.headers.has("X-Edge-Render")) return null;

  const url = new URL(request.url);
  if (STATIC_EXTENSIONS.test(url.pathname)) return null;
  if (!isCrawler(request.headers.get("User-Agent") || "")) return null;

  const renderUrl = `${EDGECOMET.RENDER_HOST}/render?url=${encodeURIComponent(request.url)}`;
  const controller = new AbortController();
  const timeoutId = setTimeout(() => controller.abort(), EDGECOMET.TIMEOUT);

  try {
    const response = await fetch(renderUrl, {
      method: request.method,
      // Return EdgeComet's redirects to the crawler unfollowed
      redirect: "manual",
      headers: buildRenderHeaders(request, url, renderKey),
      signal: controller.signal,
    });

    // Serve only responses EdgeComet produced; gateway errors run your code
    if (!response.headers.has("EC-Source") || response.status >= 500) {
      return null;
    }
    return response;
  } catch (error) {
    return null;
  } finally {
    clearTimeout(timeoutId);
  }
}

The settings:

• RENDER_HOST: your render host with https://.
• TIMEOUT: how long the call waits for EdgeComet, in milliseconds. The default is 60000. Keep it above EdgeComet's render timeout.
• EDGECOMET_RENDER_KEY: your render key, read from a Worker secret. Without it, the call returns null and your Worker runs as before. See Store the render key as a secret.

The module holds the same routing as the Worker the automatic install deploys. If you edit your Worker as one file in the Cloudflare dashboard, paste the module at the top of that file and delete the word export in front of async function edgeCometResponse.

Store the render key as a secret

Keep the key out of your code and your repository: store it as a secret named EDGECOMET_RENDER_KEY.

With Wrangler, run this in your Worker's project folder and paste the key when asked:

npx wrangler secret put EDGECOMET_RENDER_KEY

This command deploys a new version of your Worker at once. Before you add the call, that version runs your code unchanged. For wrangler dev, put EDGECOMET_RENDER_KEY=... in .dev.vars, and keep that file out of git.

In the Cloudflare dashboard, open Workers & Pages, select your Worker, then Settings. Under Variables and Secrets, click Add, select the type Secret, enter EDGECOMET_RENDER_KEY as the Variable name and your key as the Value, then click Deploy.

For a Pages project, select the project, open Settings > Variables and Secrets > Add, enter the name and the key, select Encrypt and click Save.

Call EdgeComet first in your fetch handler

Import the module and call edgeCometResponse before anything else in your handler. Return its response when there is one. Otherwise, let your code continue.

Module Worker

import { edgeCometResponse } from "./edgecomet.js";

export default {
  async fetch(request, env, ctx) {
    const botResponse = await edgeCometResponse(request, env);
    if (botResponse) {
      return botResponse;
    }

    // Your existing code continues here
    return handleRequest(request, env, ctx);
  },
};

Hono

import { Hono } from "hono";
import { edgeCometResponse } from "./edgecomet.js";

const app = new Hono();

// Register before your other middleware and routes
app.use("*", async (c, next) => {
  const botResponse = await edgeCometResponse(c.req.raw, c.env);
  if (botResponse) {
    return botResponse;
  }
  await next();
});

// Your existing middleware and routes follow

export default app;

Pages: functions/_middleware.js

import { edgeCometResponse } from "../edgecomet.js";

export async function onRequest(context) {
  const botResponse = await edgeCometResponse(context.request, context.env);
  if (botResponse) {
    return botResponse;
  }
  return context.next();
}

Service Worker syntax

import { edgeCometResponse } from "./edgecomet.js";

addEventListener("fetch", (event) => {
  event.respondWith(handle(event.request));
});

async function handle(request) {
  // Service Worker syntax exposes secrets as global variables
  const env = {
    EDGECOMET_RENDER_KEY: typeof EDGECOMET_RENDER_KEY === "undefined" ? undefined : EDGECOMET_RENDER_KEY,
  };
  const botResponse = await edgeCometResponse(request, env);
  if (botResponse) {
    return botResponse;
  }

  // Your existing code continues here
  return handleRequest(request);
}

handleRequest stands for your current code. Other handlers your Worker exports, such as scheduled, stay as they are.

• Hono: middleware that returns a Response ends the request there. Register EdgeComet's middleware before your other middleware and routes.
• Pages: keep edgecomet.js outside the functions folder, because every file in that folder becomes a route. If you already have functions/_middleware.js, name the function above edgeComet and export an array that starts with it, such as export const onRequest = [edgeComet, yourMiddleware].
• A Worker a framework builds: check the framework's docs for a custom entry file or middleware, and call EdgeComet first there.

If your Worker already sends bots to another rendering service, remove that code. The EdgeComet call replaces it.

Where to place the call

Place the call first, before code that answers from a cache or changes the request. Code that blocks clients goes before it.

• Before cache lookups that serve HTML. If your Worker serves pages from the Cache API or KV, a call placed after the lookup gives bots the copy cached for visitors. Do not store EdgeComet's responses in your cache either.
• Before request rewrites. Pass the request as it arrived. EdgeComet fetches that URL again through Cloudflare, and your code applies its rewrites to that fetch.
• After code that blocks clients by IP address, country or user agent. EdgeComet fetches your page with its own IP address and a browser user agent, and these rules would not stop the bot.
• Redirects work in either place. Placed after the call, your Worker's redirect reaches the bot through EdgeComet, which passes 3xx responses on with their Location. Placed before, bots get it one round trip sooner.

Headers your code adds to every response, such as security headers, are not on EdgeComet's response unless EdgeComet keeps them from your page. To add them, copy the response first, because a fetched response cannot be changed:

const response = new Response(botResponse.body, botResponse);
response.headers.set("Strict-Transport-Security", "max-age=31536000");
return response;

Workers with static assets

If your Worker serves files with Workers static assets, Cloudflare serves a file that matches the request without running your Worker. Bots that request those pages never reach the EdgeComet call. This covers the prerendered HTML of a static site and the index.html of a single-page app.

Run your Worker first for page paths with run_worker_first, and leave your asset folders out with ! patterns:

wrangler.jsonc

{
  "assets": {
    "directory": "./dist/",
    "binding": "ASSETS",
    "run_worker_first": ["/*", "!/assets/*"]
  }
}

wrangler.toml

[assets]
directory = "./dist/"
binding = "ASSETS"
run_worker_first = ["/*", "!/assets/*"]

Your Worker then runs for those requests and must serve the files itself. End your handler with return env.ASSETS.fetch(request); for requests your code does not answer.

Every request your Worker runs for counts as a Worker request. List each folder that holds only static files as a ! pattern. With an array, Cloudflare also stops treating browser navigation requests differently in single-page app mode.

If your Worker is on a Custom Domain, you can leave the project unchanged: run EdgeComet as its own Worker on a route in front of it. See When to merge.

Pages projects

On Pages, add the call in functions/_middleware.js, shown under Call EdgeComet first in your fetch handler. A middleware file at the root of functions runs in front of your whole site, static files included.

Your project's _routes.json decides which requests run Functions at all. Many frameworks generate it, and it can exclude your pages. Check that include covers the pages bots crawl, such as "/*", and that exclude lists only static folders.

Deploy and check

1. Deploy your Worker with npx wrangler deploy, or click Deploy in the dashboard editor. For Pages, deploy as usual.
2. In the EdgeComet dashboard guide, click Test connection under Step 3: Verify connection. It passes when your firewall lets EdgeComet's request through.
3. Click Verify install under Step 4: Verify install. EdgeComet requests your homepage as Googlebot and as a browser and checks which one EdgeComet answered.
4. Check the headers yourself. A request with a Googlebot user agent gets EC- headers, and a browser request gets none. See Check the headers bots receive.

For the first verified bot visit and integration health, see Check that search and AI bots reach EdgeComet.

Let EdgeComet through your firewall

EdgeComet fetches your pages through Cloudflare, and every page request carries your render key in X-Render-Key. Add a WAF custom rule that skips your security rules for that header. For the expression and the products to skip, see Cloudflare WAF.

Bot Fight Mode cannot be skipped by any rule. See Bot Fight Mode.

What bots get when EdgeComet cannot answer

In a merged Worker, your code runs, and bots get what visitors get. This happens when EdgeComet does not answer within TIMEOUT, the connection fails, the response has no EC-Source header or a 5xx status, or the secret is missing.

A wrong render key is one such case: EdgeComet answers 401 without EC-Source. Responses with EC-Source, such as a redirect or a 404 from a URL rule, reach the bot unchanged.

Run EdgeComet as its own Worker

Run EdgeComet as a separate Worker when your Worker covers only some paths, or when your site is a Worker on a Custom Domain. For every other site without a Worker, the automatic install deploys the same routing for you.

The separate Worker is the module plus a handler that passes every other request on:

import { edgeCometResponse } from "./edgecomet.js";

export default {
  async fetch(request, env) {
    return (await edgeCometResponse(request, env)) ?? fetch(request);
  },
};

To deploy it with Wrangler:

1. Put worker.js and edgecomet.js in a new folder with this configuration:

   name = "edgecomet-router"
   main = "worker.js"
   compatibility_date = "2025-01-01"
   
   routes = [
     { pattern = "example.com/*", zone_name = "example.com" },
     { pattern = "www.example.com/*", zone_name = "example.com" }
   ]

2. Run npx wrangler deploy, then npx wrangler secret put EDGECOMET_RENDER_KEY. Until the secret exists, the Worker passes every request on.

3. In the Cloudflare dashboard, open your domain, then Workers Routes. Edit each route, open Request limit failure mode, select Fail open (proceed) and click Save.

To use the dashboard instead, create a Worker under Compute > Workers & Pages > Create application > Start with Hello World!. Click Edit code, paste the module without export followed by the handler without the import line, and click Deploy. Add the secret, then add the routes in Workers Routes > Add route.

Fail open matters on the Free plan, which allows 100,000 Worker requests a day. After the limit, Cloudflare skips the Worker and sends requests to your origin, instead of returning an error page to visitors and bots.

Keep static files off the Worker

Every request a route matches counts toward your Worker requests, including images, scripts and styles. To save requests, add a route for each folder that holds only static files, with the Worker set to None:

1. In Workers Routes, click Add route.
2. Enter the folder pattern, for example example.com/assets/*.
3. Set Worker to None and click Save. Repeat for each folder.

A route set to None turns off every Worker on that folder. In a merged Worker, add one only for folders where your own code has nothing to do. The automatic install picks these folders for you from a scan of your homepage.

Update or remove EdgeComet

After a render key rotation, set the secret again with npx wrangler secret put EDGECOMET_RENDER_KEY or in Variables and Secrets, and update the firewall rule. The code does not change.

To remove EdgeComet from your Worker, delete the call and the import, deploy, and delete the EDGECOMET_RENDER_KEY secret and the firewall rule. To remove a separate EdgeComet Worker, delete its routes, then the Worker. Bots then get your regular pages.

If it does not work

"We stopped before changing anything"

The automatic install found another Worker route on your site's hostname: Something else is installed in the same place names the Worker and its route pattern. Merge EdgeComet into that Worker as this page describes, or see When to merge for setups that need no merge.

Bot requests return no EC- headers

The call returned null, or the request never reached your Worker:

• Check the secret's name, EDGECOMET_RENDER_KEY, and that the deployed version has it.
• Check that the call runs before cache lookups and before any code that returns early.
• Check that your test user agent contains a crawler token, such as Googlebot.
• On Workers static assets, check run_worker_first. On Pages, check _routes.json. See Workers with static assets and Pages projects.
• A wrong key, or a domain that is not your website's domain, gets 401 Authentication failed: invalid render key or domain mismatch. Copy the key again from the dashboard guide, and confirm the website is Active under System Settings > General.

Bot pages miss changes your Worker makes

EdgeComet's requests carry no cookies, a browser user agent and EdgeComet's IP address. Changes your Worker makes for a session, a cookie, a user agent or a country do not apply to them. See EdgeComet renders the page your Worker serves.

Timeouts

Keep TIMEOUT above EdgeComet's render timeout. Check the dashboard for render failures on the page, and check that Cloudflare can reach your render host.

Requests loop between the Worker and EdgeComet

Keep the X-Edge-Render check in the module, and make sure no Transform Rule or other code removes that header before the call.

Visitors receive EdgeComet's pages

The crawler check matches more than crawlers. Test the patterns in CRAWLER_PATTERNS against common browser user agents, and keep the matching case-insensitive.

Bot Fight Mode blocks EdgeComet

Cloudflare offers no exception for Bot Fight Mode. See Bot Fight Mode.

Related

• Install EdgeComet on Cloudflare automatically
• How EdgeComet routes bot traffic
• Let EdgeComet through your firewall
• Cloudflare Worker for the self-hosted engine, which covers the open-source engine you run yourself