Route bots to EdgeComet with CloudFront and Lambda@Edge

On AWS CloudFront, two Lambda@Edge functions on your existing distribution send search engine and AI bot requests to EdgeComet. Visitors keep getting your site from CloudFront and your origin as before. This is the CloudFront setup for prerendering or dynamic rendering, plus EdgeComet's caching and bot analytics.

The setup changes four things on the cache behaviors that serve your HTML pages: two function associations, a cache policy and an origin group. The origin group sends bots to your origin when EdgeComet cannot answer.

In the dashboard, the guide is under Configuration > Connections > CDN integration > AWS Lambda@Edge, filled in with your render key and render host.

How it works

Piece	Runs	Job
edge-comet-detector, viewer request	On every request, before the cache lookup	Marks crawler GET and HEAD page requests with three X-EC- headers: the full page URL, the bot's user agent and a unique request ID
Cache policy	On the cache lookup	Includes the three headers in the cache key. Visitors never carry them, and their cache key stays as it was. Each bot request gets a unique key, which CloudFront never reuses
edge-comet-route, origin request	On a cache miss	Sends marked requests to https://{render-host}/render?url=... with your render key and the bot's user agent. Leaves every other request on your origin
Origin group	When the origin answers	Retries the request on a copy of your origin when EdgeComet answers 500, 502, 503 or 504, cannot be reached or times out

The design keeps three guarantees:

• Visitors never get a bot copy. A bot response is stored under a cache key no visitor request can have.
• Every bot request reaches EdgeComet, which records it and serves the current copy from its own cache.
• Your render key stays in the origin request function. It never enters a cache key or a viewer-facing header.

CloudFront appends the bot's IP address to X-Forwarded-For on the request to EdgeComet, which EdgeComet uses for bot verification. CloudFront does not allow functions to set X-Forwarded-Proto, and the scheme reaches EdgeComet inside the url parameter.

Before you start

• A CloudFront distribution that already serves your site on its main domain. To move a site behind CloudFront first, set that up and confirm it works for visitors, then return here.
• The cache behaviors that serve your HTML pages, usually Default (*), with Viewer protocol policy set to Redirect HTTP to HTTPS or HTTPS only.
• No viewer request or origin request function on those behaviors. Each event takes one function, and CloudFront Functions and Lambda@Edge cannot share viewer events. If a function is there, merge its code with the EdgeComet function, and run the detector before any URL rewrite.
• AWS permissions for Lambda, IAM and CloudFront, including lambda:GetFunction, lambda:EnableReplication*, cloudfront:UpdateDistribution and, the first time, iam:CreateServiceLinkedRole.
• Your Render Key and Render Host from the dashboard guide.

The detector function

Runtime Node.js 24.x, architecture x86_64, file index.mjs, handler index.handler.

// edge-comet-detector: Lambda@Edge, viewer request
const CRAWLERS = /bot|crawl|spider|slurp|WhatsApp|Snapchat|facebookexternalhit|AMZN-User|Claude-User|Perplexity-User|ChatGPT-User/i;
const STATIC_FILES = /\.(avif|css|eot|gif|gz|ico|jpeg|jpg|js|json|map|mp3|mp4|ogg|otf|pdf|png|svg|ttf|txt|wasm|wav|webm|webmanifest|webp|woff|woff2|xml|zip)$/i;

export const handler = async (event) => {
  const cf = event.Records[0].cf;
  const request = cf.request;
  const headers = request.headers;

  // Only this function sets X-EC-* headers
  for (const name of Object.keys(headers)) {
    if (name.startsWith('x-ec-')) delete headers[name];
  }

  try {
    if (request.method !== 'GET' && request.method !== 'HEAD') return request;

    const userAgent = headers['user-agent'] ? headers['user-agent'][0].value : '';
    if (STATIC_FILES.test(request.uri) || !CRAWLERS.test(userAgent)) return request;

    const query = request.querystring ? '?' + request.querystring : '';
    const url = encodeURIComponent('https://' + headers['host'][0].value + request.uri + query);

    headers['x-ec-url'] = [{ key: 'X-EC-URL', value: url }];
    headers['x-ec-ua'] = [{ key: 'X-EC-UA', value: userAgent }];
    // New value on every request: a crawler response is never cached for reuse
    headers['x-ec-nonce'] = [{ key: 'X-EC-Nonce', value: cf.config.requestId }];
  } catch (err) {
    // This function runs for every visitor: never fail the request
    console.log('edge-comet-detector', err);
  }
  return request;
};

The detector removes any X-EC- header a visitor sends, which stops anyone from steering a request to EdgeComet. It builds the URL with https, which the viewer protocol policy guarantees.

The router function

Same runtime and architecture. Replace YOUR_RENDER_KEY, and use the render host from your guide. Lambda@Edge has no environment variables, which keeps both values in the code.

// edge-comet-route: Lambda@Edge, origin request
const RENDER_HOST = 'render-us.edgecomet.com'; // bare hostname: no https://, no trailing slash
const RENDER_KEY = 'YOUR_RENDER_KEY';

const value = (headers, name) => (headers[name] ? headers[name][0].value : '');

export const handler = async (event) => {
  const request = event.Records[0].cf.request;
  const headers = request.headers;
  const target = value(headers, 'x-ec-url');

  // Visitors, static files and EdgeComet's own fetches go to your origin unchanged
  if (!target) return request;

  // Second run after origin failover: the secondary origin carries X-EC-Failover
  const origin = request.origin.custom || request.origin.s3 || {};
  if (origin.customHeaders && origin.customHeaders['x-ec-failover']) {
    return undoRewrite(request);
  }

  headers['x-ec-host'] = [{ key: 'X-EC-Host', value: value(headers, 'host') }];
  headers['host'] = [{ key: 'Host', value: RENDER_HOST }];
  headers['user-agent'] = [{ key: 'User-Agent', value: value(headers, 'x-ec-ua') }];
  headers['x-render-key'] = [{ key: 'X-Render-Key', value: RENDER_KEY }];
  request.uri = '/render';
  request.querystring = 'url=' + target;
  request.origin = {
    custom: {
      domainName: RENDER_HOST,
      port: 443,
      protocol: 'https',
      path: '',
      sslProtocols: ['TLSv1.2'],
      readTimeout: 60, // above EdgeComet's render timeout (15 s by default)
      responseCompletionTimeout: 60,
      keepaliveTimeout: 5,
      customHeaders: {},
    },
  };
  return request;
};

// If the failover run receives the rewritten request, restore the original one
function undoRewrite(request) {
  const headers = request.headers;
  const host = value(headers, 'x-ec-host');
  if (!host) return request;

  const url = decodeURIComponent(value(headers, 'x-ec-url'));
  const path = url.slice(url.indexOf('/', 'https://'.length));
  const q = path.indexOf('?');
  request.uri = q === -1 ? path : path.slice(0, q);
  request.querystring = q === -1 ? '' : path.slice(q + 1);
  headers['host'] = [{ key: 'Host', value: host }];
  delete headers['x-render-key'];
  return request;
}

With an origin group, CloudFront runs the origin request function again when it fails over. The router recognizes that run by the X-EC-Failover header on the secondary origin and leaves the request on your origin. customHeaders: {} keeps the secret headers of your own origin, if it has any, away from EdgeComet.

AWS documents that responseCompletionTimeout does not support continuous deployment. If your distribution uses continuous deployment, remove that line.

Install on your distribution

1. Allow EdgeComet in AWS WAF, if a web ACL is attached to the distribution. Add an Allow rule for the x-render-key header at the top of the web ACL. See AWS WAF. Then run Test connection in the dashboard guide.

2. Switch the Lambda console to US East (N. Virginia). Lambda@Edge functions must be created there.

3. Create the execution role that both functions share. In IAM, create a role for Lambda with the AWSLambdaBasicExecutionRole policy, which lets the functions write logs, and name it, for example, edge-comet-lambda-edge-role. On the role's Trust relationships tab, edit the trust policy to list both services:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Effect": "Allow",
         "Principal": {
           "Service": ["edgelambda.amazonaws.com", "lambda.amazonaws.com"]
         },
         "Action": "sts:AssumeRole"
       }
     ]
   }

   

   Without edgelambda.amazonaws.com, CloudFront cannot run the functions and answers 503 LambdaExecutionError.

4. Create the detector. Click Create function > Author from scratch, name it edge-comet-detector, select Node.js 24.x and x86_64. Under Change default execution role, select Use another role and pick the role from step 3. Replace index.mjs with the detector code, click Deploy, then Actions > Publish new version. Copy the version ARN, which ends in :1.

5. Create the router the same way, named edge-comet-route, with the router code. Publish a version and copy its ARN.

6. Add the fallback origin. In your distribution, open the Origins tab and click Create origin. Copy every setting of your existing origin. Under Add custom header, add X-EC-Failover with the value 1, and save.

7. Create the origin group. In the Origin groups pane, click Create origin group. Add both origins with your existing origin first, name it, for example edgecomet-failover, and select the failover status codes 500, 502, 503 and 504.

8. Create the cache policy. See Cache policy.

9. Edit the behavior in one save. Open the Behaviors tab, select the behavior and click Edit:

   • Origin and origin groups: the new origin group.
   • Cache key and origin requests > Cache policy: the new policy.
   • Function associations: Viewer request, Lambda@Edge, the detector version ARN. Origin request, Lambda@Edge, the router version ARN. Leave Include body cleared.
   • Click Save changes.

10. Repeat step 9 for every other behavior that serves HTML pages.

11. Wait until the distribution's Status shows a date instead of Deploying, usually several minutes.

12. Check the integration, then run Verify install in the dashboard guide.

Change the origin, the cache policy and the functions in the same save. If the functions go live before the cache policy, CloudFront can store EdgeComet's copy under the key visitors use.

Cache policy

Managed cache policies cannot be edited. Create a custom policy under Policies > Cache > Create cache policy with the settings of your current policy, plus three headers:

Setting	Value
Minimum TTL, Maximum TTL, Default TTL	Same as your current policy. For CachingOptimized: 1, 31536000 and 86400
Headers	Include the following headers: the headers of your current policy, plus X-EC-URL, X-EC-UA and X-EC-Nonce
Query strings, Cookies	Same as your current policy
Compression support	Same as your current policy

Headers in the cache policy are forwarded to the origin request function automatically, and your origin request policy stays as it is. A cache policy holds up to 10 headers.

• Behaviors that do not cache, with CachingDisabled: nothing is stored, and the headers only need to reach the router. The AllViewer origin request policies forward all viewer headers, which should include them; the curl checks in Check the integration confirm it. With any other origin request policy, or none, create a custom one that adds the three headers.
• Behaviors on Legacy cache settings: move them to Cache policy and origin request policy first. With legacy settings, CloudFront forwards headers to the origin without adding them to the cache key, which can serve a bot copy to visitors.

Never put the three headers only in an origin request policy on a behavior that caches, for the same reason.

What bots get when EdgeComet cannot answer

Case	Bots get
EdgeComet answers with EC-Source and a status below 500, including redirects, 404 and 410	EdgeComet's response
EdgeComet answers 500, 502, 503 or 504	Your origin's page, from the fallback origin
EdgeComet cannot be reached	Your origin's page, after CloudFront's connection attempts, up to 30 seconds by default
EdgeComet takes longer than 60 seconds	Your origin's page
401 Authentication failed: invalid render key or domain mismatch	The 401. CloudFront cannot fail over on 401
400 Invalid URL, for example a page URL longer than 2,048 characters	The 400. It is not in the failover codes, because EdgeComet's URL rules can answer with 400 on purpose

CloudFront fails over only on 400, 403, 404, 416, 429, 500, 502, 503 and 504, and only for GET, HEAD and OPTIONS requests.

The 401 case needs care. A wrong render key, a domain your EdgeComet website does not list, or an inactive website sends bots a 401 on CloudFront, where the Cloudflare Worker would fall back.

Run Test connection and Verify install after every key or domain change. Remove the function associations before you set the website to Inactive or rotate the render key.

Visitors' GET and HEAD requests that get 500, 502, 503 or 504 from your origin are now retried once on the same origin, through the origin group.

Loop prevention on CloudFront

CloudFront removes every X-Edge- header, which hides EdgeComet's X-Edge-Render loop marker from Lambda@Edge. On CloudFront, EdgeComet's own requests to your site go to your origin because their user agents are browsers, which the crawler pattern does not match.

Keep the render and bypass user agents of your dimensions set to browser user agents. A user agent with bot in it would send EdgeComet's own fetch back to EdgeComet.

Let EdgeComet through AWS WAF

AWS WAF evaluates requests before the detector runs. Its Bot Control, IP reputation and rate rules can block EdgeComet's requests to your site, which come from hosting providers and run headless Chrome.

Put an Allow rule for x-render-key at the top of the web ACL. See AWS WAF.

Check the integration

# Crawler: expect EC- headers and "x-cache: Miss from cloudfront" on every request
curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://www.example.com/

# Visitor: expect no EC- headers, and x-cache turns to Hit on repeat requests as before
curl -sI -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" https://www.example.com/

# A visitor sending an X-EC- header: expect no EC- headers
curl -sI -H "X-EC-URL: x" -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Chrome/124.0.0.0" https://www.example.com/

Bot requests always show a miss: each one has its own cache key. In Data Explorer > Bot log, your test request shows the Googlebot user agent, not Amazon CloudFront, and your IP address. Then run Test connection and Verify install in the dashboard guide, and see Check that search and AI bots reach EdgeComet.

Update or remove

• Update a function: edit it, click Deploy, then Publish new version, and put the new version ARN in each behavior's function association. Lambda does not move associations to new versions on its own.
• Remove EdgeComet: edit each behavior, set both function associations to No association, and set the origin and cache policy back to the previous ones. Old function versions can be deleted once CloudFront removes their replicas, usually within a few hours.

Lambda@Edge limits

AWS documents these rules for Lambda@Edge functions:

Limit	Value
Region	Created in US East (N. Virginia), runs in every edge location
Version	A numbered version, never $LATEST or an alias
Architecture	x86_64
Runtime	Node.js 24.x or 22.x. Node.js 24 has no callback handlers, which the code here does not use
Scale	Up to 10,000 requests per second per Region for Lambda@Edge
Logs	CloudWatch Logs in the Region where the function ran, group /aws/lambda/us-east-1.{function name}

The detector runs on every request to the behavior, cache hits included. Count it in your Lambda@Edge usage.

If it does not work

503 LambdaExecutionError

The execution role does not trust edgelambda.amazonaws.com, or a function threw an error. Trust both lambda.amazonaws.com and edgelambda.amazonaws.com, use the async handlers on this page, and check the logs.

502 LambdaValidationError

A function added a header CloudFront does not allow, such as X-Forwarded-Proto, X-Real-IP or an X-Edge- header, or set an invalid origin field. Use the code as given, with RENDER_HOST as a bare hostname.

Bot responses have no EC- headers and show a cache hit

The three X-EC- headers are missing from the cache policy, and the bot got the visitor copy. See Cache policy.

Bot responses have no EC- headers and show a cache miss

The functions are not associated with the behavior that serves this path, the distribution is still deploying, the user agent does not match, or the path ends in a static file extension.

Bots get 401

The render key is wrong, the host in the request is not one of your website's domains, or the website is inactive. Add every hostname bots use, such as both www and the root domain, to your EdgeComet website. Fix the key, publish a new router version and update the association.

EdgeComet records User-Agent: Amazon CloudFront

The router did not receive X-EC-UA, because the headers are missing from the policy or an old router version is associated.

Visitors receive EC- headers or rendered HTML

The behavior caches, and the X-EC- headers are only in an origin request policy, or the crawler pattern matches a browser user agent. Put the headers in the cache policy and invalidate the affected paths.

Test connection fails, or the install check is not confirmed

AWS WAF blocks EdgeComet's requests. Add the allow rule. See AWS WAF.

Requests loop after a user agent change

A render or bypass user agent in EdgeComet matches the crawler pattern. See Loop prevention on CloudFront.

Related

• How EdgeComet routes bot traffic
• Let EdgeComet through your firewall
• CloudFront for the self-hosted engine, which covers the open-source engine you run yourself