Route search and AI bots to EdgeComet with Apache

Apache httpd routes bots to EdgeComet with mod_rewrite and mod_proxy. A rewrite rule proxies crawler page requests to EdgeComet's render endpoint, and every other request keeps your current configuration.

When EdgeComet cannot answer, a fallback rule serves the bot from your origin.

This is the Apache setup for prerendering or dynamic rendering, plus EdgeComet's caching and bot analytics. Add it on the layer bots connect to: if a CDN sits in front of Apache, add the rule at the CDN. See Choose your platform.

The dashboard has no Apache guide. Take your render key and render host from the Generic Integration guide under Configuration > Connections > CDN integration.

How it works

• A rewrite rule proxies a request to EdgeComet when the method is GET or HEAD, the request has no X-Edge-Render header, the path is not a static file and the user agent is a crawler.
• The [B] flag URL-encodes the original URL, taken from the raw request line, into the url parameter. The query string arrives whole.
• A <Proxy> section for the render host holds everything that applies only to EdgeComet: verified TLS, connection reuse, the Host header, your render key and the error handling.
• ProxyErrorOverride with ErrorDocument turns EdgeComet's errors and Apache's own connection errors into an internal request to /__edgecomet_fallback. A fallback rule proxies that request to your origin with the original path, and the bot gets your origin's page and status.

Apache adds the bot's IP address to X-Forwarded-For on proxied requests by itself.

Before you start

• Apache httpd 2.4.47 or later. Version 2.4.30 to 2.4.46 works with one change. See Older Apache versions.
• The modules rewrite, proxy, proxy_http, ssl and headers.
• Access to the virtual host configuration. .htaccess is not supported: <Proxy>, ProxySet, ProxyErrorOverride and the SSLProxy directives are not allowed there, which rules out verified TLS and the fallback.
• Your Render Key and Render Host. If your host is render-de.edgecomet.com, use it in both places render-us.edgecomet.com appears.

Enable the modules

On Debian and Ubuntu:

sudo a2enmod rewrite proxy proxy_http ssl headers

On RHEL, Rocky and AlmaLinux, proxy, proxy_http, rewrite and headers load by default:

sudo dnf install mod_ssl

Configuration

Create one EdgeComet file and include it in each virtual host that serves your site. Pick the fallback for your setup:

• Apache passes pages to an application with ProxyPass, such as Node.js, Python or Java: the fallback proxies to the application.
• Apache serves pages itself, such as static files, PHP or WordPress: the fallback proxies back to the same virtual host over 127.0.0.1.

/etc/apache2/edgecomet.conf (application)

# EdgeComet routing for example.com: crawlers go to EdgeComet, everyone else to your origin.
# Include this file inside each <VirtualHost> that serves the site.
# Needs Apache httpd 2.4.47 or later and mod_rewrite, mod_proxy, mod_proxy_http, mod_ssl, mod_headers.

# 1. Requests to EdgeComet
<Proxy "https://render-us.edgecomet.com/">
    # TLS with certificate, host name and expiry checks
    SSLProxyEngine on
    SSLProxyVerify require
    SSLProxyCheckPeerName on
    SSLProxyCheckPeerExpire on
    SSLProxyCACertificateFile "/etc/ssl/certs/ca-certificates.crt"

    # A named worker reuses connections; timeout must exceed the render time
    ProxySet connectiontimeout=5 timeout=60

    # The Host header must be the render host, even if the vhost preserves Host
    ProxyPreserveHost Off
    RequestHeader set X-Render-Key "YOUR_RENDER_KEY"
    RequestHeader set X-Forwarded-Proto "expr=%{REQUEST_SCHEME}"

    # Fail open: replace these EdgeComet answers with your origin's page.
    # 404, 410, 403 and redirects from EdgeComet still reach the crawler.
    ProxyErrorOverride On 400 401 500 502 503 504 520 521 522 523 524 525 526 527 530
    ErrorDocument 400 /__edgecomet_fallback
    ErrorDocument 401 /__edgecomet_fallback
    ErrorDocument 500 /__edgecomet_fallback
    ErrorDocument 502 /__edgecomet_fallback
    ErrorDocument 503 /__edgecomet_fallback
    ErrorDocument 504 /__edgecomet_fallback
</Proxy>

RewriteEngine On

# 2. Fallback: failed EdgeComet requests land here and go to your origin.
#    Replace http://127.0.0.1:3000 with the address your ProxyPass sends pages to.
RewriteCond %{ENV:REDIRECT_STATUS} !^$
RewriteCond %{THE_REQUEST} "^[A-Z]+ (/[^ ?]*)(?:\?([^ ]*))?"
RewriteRule "^/__edgecomet_fallback$" "http://127.0.0.1:3000%1?%2" [P,NE,L,E=EDGECOMET_FALLBACK:1]
RequestHeader unset X-Render-Key env=EDGECOMET_FALLBACK

# 3. Crawler routing
RewriteCond %{ENV:REDIRECT_STATUS} ^$
RewriteCond %{REQUEST_METHOD} ^(GET|HEAD)$
RewriteCond %{HTTP:X-Edge-Render} ^$
RewriteCond %{REQUEST_URI} !\.(avif|css|eot|gif|gz|ico|jpeg|jpg|js|json|map|mp3|mp4|ogg|otf|pdf|png|svg|ttf|txt|wasm|wav|webm|webmanifest|webp|woff|woff2|xml|zip)$ [NC]
RewriteCond %{HTTP_USER_AGENT} (bot|crawl|spider|slurp|WhatsApp|Snapchat|facebookexternalhit|AMZN-User|Claude-User|Perplexity-User|ChatGPT-User) [NC]
RewriteCond "%{THE_REQUEST} %{REQUEST_SCHEME}://%{HTTP_HOST}" "^[A-Z]+ (/[^ ]*) [^ ]+ ([^ ]+)$"
RewriteRule "^" "https://render-us.edgecomet.com/render?url=%2%1" [B,P,L]

/etc/apache2/edgecomet.conf (Apache serves pages)

# Same file as the application variant, with section 2 replaced by this one.

# 2. Fallback: failed EdgeComet requests land here and are sent back to this
#    virtual host over the loopback address. X-Edge-Render makes rule 3 skip them.
<Proxy "https://127.0.0.1/">
    SSLProxyEngine on
    SSLProxyVerify require
    SSLProxyCheckPeerName on
    SSLProxyCACertificateFile "/etc/ssl/certs/ca-certificates.crt"
    # Sends example.com as Host and SNI, and checks the certificate against it
    ProxyPreserveHost On
    RequestHeader set X-Edge-Render "fallback"
</Proxy>
RewriteCond %{ENV:REDIRECT_STATUS} !^$
RewriteCond %{THE_REQUEST} "^[A-Z]+ (/[^ ?]*)(?:\?([^ ]*))?"
RewriteRule "^/__edgecomet_fallback$" "https://127.0.0.1%1?%2" [P,NE,L,E=EDGECOMET_FALLBACK:1]
RequestHeader unset X-Render-Key env=EDGECOMET_FALLBACK

/etc/apache2/sites-available/example.com.conf

<VirtualHost *:443>
    ServerName example.com

    SSLEngine on
    SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem

    # EdgeComet, before your own RewriteRule lines
    Include /etc/apache2/edgecomet.conf

    # Your application
    ProxyPreserveHost On
    ProxyPass "/" "http://127.0.0.1:3000/"
    ProxyPassReverse "/" "http://127.0.0.1:3000/"
</VirtualHost>

Replace YOUR_RENDER_KEY with your render key, example.com with your domain and http://127.0.0.1:3000 with your application's address.

On RHEL, Rocky and AlmaLinux, save the file as /etc/httpd/edgecomet.conf, not in conf.d/, which Apache includes at server level. Set SSLProxyCACertificateFile to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem, and include the file as Include /etc/httpd/edgecomet.conf.

The loopback fallback needs a virtual host that answers on 127.0.0.1:443, as <VirtualHost *:443> does. Its certificate must chain to the CA bundle in SSLProxyCACertificateFile, which rules out self-signed certificates and Cloudflare Origin CA certificates. Behind a load balancer that ends TLS, with a virtual host on port 80, use http://127.0.0.1 and drop the SSLProxy lines from that block.

Notes on the configuration

• ProxySet creates a named worker for the render host, which reuses connections. Without it, every bot request opens a new TLS connection.
• ProxyPreserveHost Off inside <Proxy> overrides a ProxyPreserveHost On in the virtual host. A preserved Host would also be sent as the TLS server name, which the render host refuses.
• RequestHeader unset X-Render-Key keeps your key off fallback requests to your origin, because an internal redirect copies the request headers.
• The split capture %1?%2 in the fallback rule keeps the ? literal. Apache 2.4.60 and later refuse a rewrite whose first ? comes from an expansion, with AH10508.
• [NE] on the fallback rule sends the raw path, without encoding it twice.
• %{ENV:REDIRECT_STATUS} ^$ stops the routing rule from sending your site's own error documents to EdgeComet.
• The Cloudflare codes 520 to 530 are in the ProxyErrorOverride list, because the render host runs behind Cloudflare. Apache handles them with ErrorDocument 500.
• Every virtual host that serves pages needs the Include: rewrite rules are not inherited.
• TLS ended in front of Apache: set ServerName https://example.com:443, which makes the URL sent to EdgeComet start with https://. Use mod_remoteip to give Apache the bot's address, which it then adds to X-Forwarded-For.

Test and reload

sudo apachectl configtest       # Debian and Ubuntu: apache2ctl configtest
sudo apachectl -k graceful      # or: systemctl reload apache2, systemctl reload httpd

What bots get when EdgeComet cannot answer

Case	Bots get
200, redirects, 403, 404, 410 with EC-Source	EdgeComet's response
401 without EC-Source: wrong key, or a domain that is unknown or inactive	Your origin's page and status
400 or 502 No healthy backend without EC-Source	Your origin's page and status
Any 5xx, with or without EC-Source, and the Cloudflare codes 520 to 530	Your origin's page and status
Connection refused or TLS verification failure	Your origin's page and status
No answer within 60 seconds	Your origin's page, after the timeout

Apache acts on status codes only, which sets these limits:

• A status outside the list reaches the bot, such as a 403, 404 or 429 from Cloudflare in front of the render host, even without EC-Source. EdgeComet's gateway itself never answers with these.
• A failed connection is retried on every bot request. Each attempt costs up to connectiontimeout, 5 seconds.
• A HEAD request that falls back reaches your origin as GET, because error documents are always GET requests. The bot still gets only the headers.

A 400, 401 or 5xx that carries EC-Source goes to your origin too. These come from EdgeComet passing on your origin's own answer, or from a URL rule with that status, which bots then do not get.

Older Apache versions

Apache 2.4.30 to 2.4.46, such as Ubuntu 20.04, has no status list on ProxyErrorOverride. Replace that line with ProxyErrorOverride On. The fallback still works, but EdgeComet's own 404, 410 and 403 answers then show Apache's default error page, with the right status code and without EC- headers.

Let EdgeComet through Apache rules

EdgeComet's own requests carry X-Edge-Render, which sends them past the routing rule to your site, and X-Render-Key with your key. If Apache blocks bots or restricts access, exempt the exact key:

mod_rewrite bot block

# Existing bot block, with an exemption for EdgeComet fetches
RewriteCond %{HTTP:X-Render-Key} !=YOUR_RENDER_KEY
RewriteCond %{HTTP_USER_AGENT} (BadBot|HeadlessChrome) [NC]
RewriteRule "^" "-" [F]

Require allowlist

<Directory /var/www/example.com>
    <RequireAny>
        # Existing allowlist
        Require ip 192.0.2.0/24
        # EdgeComet fetches
        Require expr "%{HTTP:X-Render-Key} == 'YOUR_RENDER_KEY'"
    </RequireAny>
</Directory>

ModSecurity

SecRule REQUEST_HEADERS:X-Render-Key "@streq YOUR_RENDER_KEY" \
    "id:1000100,phase:1,pass,nolog,ctl:ruleEngine=Off"

A WAF in front of Apache needs its own exception. See Let EdgeComet through your firewall.

Check the integration

# Crawler: expect EC-Source
curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/ | grep -i '^ec-source'

# Browser: expect no EC- headers
curl -sI -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36" https://example.com/ | grep -i '^ec-'

# Key and domain, bypassing Apache: 401 means a wrong key, domain or an inactive website
curl -i -H "X-Render-Key: YOUR_RENDER_KEY" -A "Googlebot" \
  "https://render-us.edgecomet.com/render?url=https%3A%2F%2Fexample.com%2F"

To see what each request did, log the EC-Source header and the proxy target:

LogFormat "%h \"%r\" %>s ec_source=\"%{EC-Source}o\" handler=%R target=%f" edgecomet
CustomLog ${APACHE_LOG_DIR}/edgecomet.log edgecomet

A routed request shows target=proxy:https://render-us.edgecomet.com/render, and a fallback shows ec_source="-" with your origin as the target. Then run Test connection and Verify install in the Generic Integration guide, and see Check that search and AI bots reach EdgeComet.

Update or remove

After a render key rotation, update X-Render-Key and your exemptions, then reload. To remove EdgeComet, delete the Include line from each virtual host and reload.

If it does not work

Bots get your origin page without EC-Source

The fallback hides an EdgeComet error, most often a wrong render key. Test the key with the direct curl in Check the integration, and check the error log for AH01114, AH00898 or AH01097. To see EdgeComet's answer, remove its code from the ProxyErrorOverride list and its ErrorDocument line for a moment.

AH01961: failed to enable ssl support

SSLProxyEngine on is missing from the <Proxy> section, or mod_ssl is not loaded.

AH00898: Error during SSL Handshake with remote server

The CA bundle path is wrong for your distribution, or the ca-certificates package is missing.

Bots get 403 and the log shows AH10508

A changed fallback rule puts an expansion before the ?. Keep the split capture %1?%2.

The origin receives double-encoded paths on fallback

NE is missing from the fallback rule's flags.

EdgeComet records URLs without part of the query string

The routing rule lost [B], or it captures %{REQUEST_URI} instead of %{THE_REQUEST}.

URLs sent to EdgeComet start with http://

TLS ends in front of Apache. Set ServerName https://example.com:443.

The rules have no effect

The Include is outside the <VirtualHost>, or in a virtual host that does not serve the request.

Test connection fails

A bot rule in front of or inside Apache blocks EdgeComet's request. Add an exemption.

Related

• How EdgeComet routes bot traffic
• Nginx and HAProxy
• Let EdgeComet through your firewall