Appearance

Route search and AI bots to EdgeComet with Nginx ​

Nginx routes bots to EdgeComet with four map blocks and a named location in your existing server block. Crawler page requests go to EdgeComet, and every other request keeps your current Nginx configuration.

When EdgeComet cannot answer, a fallback location serves the bot from your origin.

This is the Nginx setup for prerendering or dynamic rendering, plus EdgeComet's caching and bot analytics. Add it on the layer bots connect to: if a CDN sits in front of Nginx, add the rule at the CDN. See Choose your platform.

In the dashboard, the guide is under Configuration > Connections > CDN integration > Nginx, filled in with your render key and render host.

The Nginx Reverse Proxy guide in the dashboard, with a contents list on the left, the prerequisites, and an EdgeComet Credentials block holding the render key, blurred, the render host and the domain

How it works ​

  • Four map blocks decide whether a request goes to EdgeComet: a crawler user agent, GET or HEAD, not a static file, and no X-Edge-Render header. Nginx evaluates them only when a location reads the result.
  • In location /, a crawler page request returns the internal code 418, which error_page hands to the @edgecomet location. Everything else continues to your origin handling.
  • @edgecomet proxies to https://{render-host}/render?url={encoded URL} over a pool of reused, verified TLS connections.
  • @origin_fallback serves the request from your origin when EdgeComet answers without EC-Source, with a server error, or not at all.

Stock Nginx cannot URL-encode a value. A three-line njs function builds the encoded url parameter. Without it, everything after the first & in a query string is lost, and escaped characters are decoded on the way.

Before you start ​

  • Nginx with ngx_http_proxy_module, which is built in, and the njs module, ngx_http_js_module, version 0.4.0 or later.
  • Access to the Nginx configuration and permission to reload Nginx.
  • Your Render Key and Render Host from the dashboard guide. If your guide shows render-de.edgecomet.com, use it everywhere render-us.edgecomet.com appears.

Install the njs module ​

On Debian 12 or later and Ubuntu 24.04 or later, with the distribution's Nginx:

bash
sudo apt install libnginx-mod-http-js
ls /etc/nginx/modules-enabled/    # expect 50-mod-http-js.conf

The package loads the module for you. Do not add a second load_module line.

With Nginx packages from nginx.org, which RHEL, Rocky, AlmaLinux, Amazon Linux and Ubuntu 22.04 need for njs:

bash
sudo apt install nginx-module-njs    # Debian and Ubuntu with the nginx.org repository
sudo yum install nginx-module-njs    # RHEL family with the nginx.org repository

Then add this line at the top of /etc/nginx/nginx.conf, outside the http block:

nginx
load_module modules/ngx_http_js_module.so;

The njs module package must match your Nginx build: use the distribution's module with the distribution's Nginx, and the nginx.org module with nginx.org Nginx. On OpenResty, see OpenResty.

Configuration ​

Create the njs function and the routing file, then add three parts to your server block.

javascript
// EdgeComet: builds the value of the url query parameter.
// Returns the full original URL (scheme, host, path and query string),
// percent-encoded so that "&", "?", "#", "+" and existing %XX escapes
// reach EdgeComet unchanged.
function render_url(r) {
    return encodeURIComponent('https://' + r.variables.host + r.variables.request_uri);
}

export default { render_url };
nginx
# EdgeComet routing. Included in the http context.

# Full original URL, percent-encoded, for /render?url=
js_import edgecomet from /etc/nginx/njs/edgecomet.js;
js_set $ec_render_url edgecomet.render_url;

# 1. Crawler user agents (case-insensitive substring match)
map $http_user_agent $ec_crawler {
    default 0;
    "~*bot"                 1;
    "~*crawl"               1;
    "~*spider"              1;
    "~*slurp"               1;
    "~*WhatsApp"            1;
    "~*Snapchat"            1;
    "~*facebookexternalhit" 1;
    "~*AMZN-User"           1;
    "~*Claude-User"         1;
    "~*Perplexity-User"     1;
    "~*ChatGPT-User"        1;
}

# 2. Only GET and HEAD requests
map $request_method $ec_method {
    default 0;
    GET     $ec_crawler;
    HEAD    $ec_crawler;
}

# 3. Static files go to the origin
map $uri $ec_page {
    default $ec_method;
    "~*\.(avif|css|eot|gif|gz|ico|jpeg|jpg|js|json|map|mp3|mp4|ogg|otf|pdf|png|svg|ttf|txt|wasm|wav|webm|webmanifest|webp|woff|woff2|xml|zip)$" 0;
}

# 4. Loop prevention: EdgeComet's own fetches carry X-Edge-Render
map $http_x_edge_render $ec_should_render {
    default $ec_page;
    "~."    0;
}

upstream edgecomet_render {
    server render-us.edgecomet.com:443;
    keepalive 16;
}
nginx
server {
    listen 443 ssl;
    server_name example.com;
    # ... your existing ssl_certificate and other server settings ...

    location / {
        # EdgeComet: send crawler page requests to @edgecomet
        recursive_error_pages on;
        error_page 418 = @edgecomet;
        if ($ec_should_render) {
            return 418;
        }

        # Your existing origin handling, unchanged, for example:
        proxy_pass http://127.0.0.1:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }

    location @edgecomet {
        proxy_pass https://edgecomet_render/render?url=$ec_render_url;

        proxy_set_header Host render-us.edgecomet.com;
        proxy_set_header X-Render-Key "YOUR_RENDER_KEY";
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Connection "";
        proxy_http_version 1.1;

        proxy_ssl_server_name on;
        proxy_ssl_name render-us.edgecomet.com;
        proxy_ssl_verify on;
        proxy_ssl_verify_depth 3;
        proxy_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt;

        proxy_connect_timeout 5s;
        proxy_send_timeout 10s;
        proxy_read_timeout 60s;
        proxy_next_upstream error timeout;
        proxy_next_upstream_timeout 10s;

        # Fail open: answers without EC-Source and server errors go to the origin
        proxy_intercept_errors on;
        error_page 400 401 500 502 503 504 520 521 522 523 524 525 526 530 = @origin_fallback;
    }

    location @origin_fallback {
        # Stops a second trip to EdgeComet if the origin handling
        # redirects internally back into location /
        set $ec_should_render 0;

        # Same origin handling as in location /
        proxy_pass http://127.0.0.1:3000;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

Replace YOUR_RENDER_KEY with your render key and example.com with your domain. Keep your own origin handling in location / and repeat it in @origin_fallback.

The njs function builds every URL with https://. Nginx reports http when a load balancer in front of it ends TLS, which would send EdgeComet the wrong scheme. For a site served only over plain HTTP, change it to 'http://'.

If you installed an earlier version of this guide, remove its edgecomet-map.conf file and its @edge_render location, which used the unencoded URL.

On RHEL, Rocky and AlmaLinux, set proxy_ssl_trusted_certificate to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem. On those systems, put the server block in /etc/nginx/conf.d/ as well.

Other origin types ​

Named locations accept try_files. For a PHP front controller or a static single-page app, use one of these fallbacks:

nginx
# PHP front controller (WordPress, Laravel): /index.php is handled by your location ~ \.php$
location @origin_fallback {
    set $ec_should_render 0;
    try_files $uri $uri/ /index.php?$args;
}

# Static single-page app (root inherited from the server block)
location @origin_fallback {
    set $ec_should_render 0;
    try_files $uri $uri/ /index.html;
}

Keep set $ec_should_render 0; in every fallback. Without it, a fallback that redirects internally to /index.html re-enters location / and sends the bot an HTTP 418.

Re-resolve the render host at runtime ​

The upstream block in edgecomet.conf resolves render-us.edgecomet.com when Nginx starts or reloads. If DNS fails at that moment, Nginx refuses to start. On Nginx 1.27.3 or later, this variant re-resolves at runtime and does not block startup:

nginx
upstream edgecomet_render {
    zone edgecomet_render 1m;
    resolver 1.1.1.1 8.8.8.8 valid=300s ipv6=off;
    resolver_timeout 5s;
    server render-us.edgecomet.com:443 resolve;
    keepalive 16;
}

You can point resolver at your own DNS server. Nginx does not read /etc/resolv.conf.

Notes on the configuration ​

  • Host and proxy_ssl_name must be the render host. Without them, the request fails the TLS handshake or gets a 403 in front of EdgeComet.
  • proxy_ssl_verify on checks EdgeComet's certificate. Nginx does not verify upstream certificates by default.
  • The bot's user agent and its other headers pass through by default. No proxy_set_header User-Agent line is needed.
  • X-Forwarded-For appends the bot's address. If Nginx runs behind a load balancer or a CDN other than Cloudflare, turn on ngx_http_realip_module with set_real_ip_from and real_ip_header X-Forwarded-For. Otherwise EdgeComet can record the load balancer's address as the bot's IP.
  • Redirects from EdgeComet reach the bot with their Location unchanged.
  • proxy_cache: if it is on at the server or http level, add proxy_cache off; to @edgecomet. Otherwise Nginx serves its own copy and EdgeComet never sees the request.
  • Other locations: regular expression locations such as location ~ \.php$ win over location /. Add the same three routing lines to every location that serves pages.
  • error_page inheritance: adding error_page 418 to location / hides the server-level error_page lines. Copy those into location /.

Test and reload ​

bash
sudo nginx -t
sudo systemctl reload nginx

What bots get when EdgeComet cannot answer ​

CaseBots get
200, redirects, 403, 404, 410 with EC-SourceEdgeComet's response
401 without EC-Source: wrong key, or a domain that is unknown or inactiveYour origin's page
400 without EC-Source: an invalid URLYour origin's page
Any 5xx, with or without EC-SourceYour origin's page
Connection refused, TLS or SNI failure, DNS failure with the runtime variantYour origin's page
No answer within 60 secondsYour origin's page, after the timeout

Nginx acts on status codes only. It cannot read EC-Source before it chooses, which leaves three gaps:

  • A 403 or 404 without EC-Source reaches the bot. Intercepting them would also replace EdgeComet's own block and 404 rules. A wrong Host header is the usual cause, and the checks in Check the integration catch it.
  • An error after Nginx starts sending the body cannot fall back.
  • A 400, 401 or 5xx that carries EC-Source goes to your origin too. These come from EdgeComet passing on your origin's own answer, or from a URL rule with that status, which bots then do not get.

Intercepted errors close the connection to EdgeComet. With a wrong key, every bot request opens a new TLS connection, which Check the integration also catches.

Let EdgeComet through Nginx rules ​

EdgeComet's own requests to your site carry X-Edge-Render, which sends them to your origin, and X-Render-Key with your key. If Nginx blocks user agents or rate-limits, exempt requests with your exact key. Render keys are 32 lowercase hexadecimal characters, and map string matching ignores case, which is why the example uses an anchored regular expression:

nginx
# 1 when the request carries your render key exactly
map $http_x_render_key $ec_fetch {
    default               0;
    "~^YOUR_RENDER_KEY$"  1;
}

# Example 1: a user agent block list that lets EdgeComet through
map $http_user_agent $ua_blocked {
    default            0;
    "~*HeadlessChrome" 1;
}
map $ec_fetch $block_request {
    default $ua_blocked;
    1       0;
}
# in the server block:  if ($block_request) { return 403; }

# Example 2: rate limiting that does not count EdgeComet fetches
# (requests with an empty key are not counted)
map $ec_fetch $limit_key {
    default $binary_remote_addr;
    1       "";
}
limit_req_zone $limit_key zone=per_ip:10m rate=1r/s;
# in the location:  limit_req zone=per_ip;

A WAF in front of Nginx, such as Cloudflare, ModSecurity or AWS WAF, needs its own exception. See Let EdgeComet through your firewall.

Check the integration ​

bash
# Crawler: expect an EC-Source header
curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/ | grep -i '^ec-'

# Browser: expect no EC- headers
curl -sI -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7)" https://example.com/ | grep -i '^ec-'

# Key and domain, from the Nginx server: expect a status with an ec-source header; 401 means a wrong key or domain
curl -s -o /dev/null -D - -A "Googlebot" -H "X-Render-Key: YOUR_RENDER_KEY" \
  "https://render-us.edgecomet.com/render?url=https%3A%2F%2Fexample.com%2F" | grep -i '^HTTP\|^ec-source'

To see what happened to each request, log the upstream statuses. ustatus=401 : 200 means EdgeComet answered 401 and your origin served the page:

nginx
log_format edgecomet '$remote_addr "$request" $status ua="$http_user_agent" '
                     'upstream=$upstream_addr ustatus=$upstream_status '
                     'connect=$upstream_connect_time ec_source=$upstream_http_ec_source';
access_log /var/log/nginx/edgecomet.log edgecomet;

connect=0.000 on the EdgeComet hop means a reused connection. Then run Test connection and Verify install in the dashboard guide, and see Check that search and AI bots reach EdgeComet.

Update or remove ​

After a render key rotation, update X-Render-Key and the exemption map, then reload. To remove EdgeComet, delete the three routing lines from location / and reload. The other files do nothing without them.

If it does not work ​

Bots get your origin page without EC-Source ​

Check the log for ustatus=401 : 200. The render key is wrong, or the domain is not on your EdgeComet website or is inactive. The direct curl in Check the integration returns 401 Authentication failed: invalid render key or domain mismatch.

unknown directive "js_import" ​

The njs module is not installed or not loaded. See Install the njs module.

host not found in upstream "render-us.edgecomet.com:443" ​

DNS was unavailable when Nginx started. Use the runtime variant on Nginx 1.27.3 or later.

upstream SSL certificate verify error: (20:unable to get local issuer certificate) ​

proxy_ssl_trusted_certificate points to the wrong CA bundle for your distribution. Bots get your origin page meanwhile.

SSL_do_handshake() failed ​

proxy_ssl_server_name on; is missing.

Bots get 403 from EdgeComet's address without EC-Source ​

The Host header is not your render host. Set proxy_set_header Host and proxy_ssl_name to it.

Bots get HTTP 418 ​

error_page 418 = @edgecomet; is missing in the location, or set $ec_should_render 0; is missing in @origin_fallback.

Query parameters are lost ​

The configuration sends the URL unencoded, as older guides did with ?url=https://$host$request_uri. Use the njs function.

Some pages never reach EdgeComet ​

Another location served them, usually a regular expression location. Add the routing lines there too.

OpenResty ​

OpenResty can build the encoded URL with Lua instead of njs, in @edgecomet:

nginx
set_by_lua_block $ec_render_url { return ngx.escape_uri("https://" .. ngx.var.host .. ngx.var.request_uri) }

Remove the js_import and js_set lines when you use it.