Route search and AI bots to EdgeComet with HAProxy

HAProxy routes bots to EdgeComet with four ACLs in your frontend and two internal hops. The first hop sends a crawler request to the second and retries it on your origin when EdgeComet fails.

The second hop rewrites the request for EdgeComet, checks EdgeComet's health and turns every unusable answer into a status the first hop can retry.

This is the HAProxy setup for prerendering or dynamic rendering, plus EdgeComet's caching and bot analytics. Visitors and every request that is not a crawler page request keep your current backend.

The dashboard has no HAProxy guide. Take your render key and render host from the Generic Integration guide under Configuration > Connections > CDN integration.

How it works

• The frontend sends a request to the edgecomet backend when it is a GET or HEAD from a crawler user agent, has no X-Edge-Render header and is not a static file.
• Hop 1, backend edgecomet, forwards the unchanged request to hop 2 over a loopback listener. On a failure it retries the same request on your origin. Its first server tracks EdgeComet's health check: once EdgeComet is down, crawlers go straight to your origin.
• Hop 2, backend ec_gateway, rewrites the request to /render?url={encoded URL}, sets Host and X-Render-Key, and connects to the render host over verified TLS. A response without EC-Source, or with a 5xx status, becomes a 503, which hop 1 retries on your origin.

Two hops are needed because HAProxy retries the request it already rewrote, and its retries see the status before http-response rules change it. In a single backend, a 401 for a wrong key cannot fall back, and a retried request reaches your origin as /render?url=....

Before you start

• HAProxy 2.6 or later, built with OpenSSL, as distribution packages are. HAProxy 2.4.16 or later in the 2.4 branch works with the 2.4 variant. Versions before 2.4 lack url_enc and are not supported.
• Your Render Key and Render Host. If your host is render-de.edgecomet.com, use it in all five places render-us.edgecomet.com appears.
• A free loopback port for the internal listener. The example uses 127.0.0.1:10180.

Distribution	HAProxy	Configuration
Debian 12	2.6.12	Main
Debian 13	3.0	Main
Ubuntu 22.04	2.4.30	2.4 variant
Ubuntu 24.04	2.8	Main
RHEL, AlmaLinux, Rocky 9	2.8	Main
RHEL, AlmaLinux, Rocky 10	3.0	Main

Configuration

The global, defaults, frontend www binds and backend origin stand for your existing configuration. The EdgeComet parts are the four ACLs and use_backend line in the frontend, and the edgecomet, ec_internal and ec_gateway sections.

#---------------------------------------------------------------------
# EdgeComet routing for HAProxy 2.6+
# Placeholders: render-us.edgecomet.com, YOUR_RENDER_KEY, origin 10.0.0.10:8080
#---------------------------------------------------------------------
global
    log /dev/log local0
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    user haproxy
    group haproxy
    daemon

defaults
    mode http
    log global
    option httplog
    timeout connect 5s
    timeout client  30s
    timeout server  30s

frontend www
    bind :80
    bind :443 ssl crt /etc/haproxy/certs/example.com.pem

    # EdgeComet: decide which requests go to EdgeComet
    acl ec_method  method GET HEAD
    acl ec_loop    req.hdr(x-edge-render) -m found
    acl ec_static  path_end -i .avif .css .eot .gif .gz .ico .jpeg .jpg .js .json .map .mp3 .mp4 .ogg .otf .pdf .png .svg .ttf .txt .wasm .wav .webm .webmanifest .webp .woff .woff2 .xml .zip
    acl ec_crawler req.fhdr(user-agent) -m sub -i bot crawl spider slurp whatsapp snapchat facebookexternalhit amzn-user claude-user perplexity-user chatgpt-user

    use_backend edgecomet if ec_method ec_crawler !ec_static !ec_loop
    default_backend origin

backend origin
    option forwardfor
    server app1 10.0.0.10:8080 check

# Hop 1: sends the crawler request to the EdgeComet hop and retries it on
# the origin when EdgeComet fails or answers without EC-Source.
backend edgecomet
    balance first
    retries 1
    option redispatch 1
    retry-on all-retryable-errors
    timeout server 70s

    # One X-Forwarded-For line: the incoming chain plus the crawler's address
    http-request set-header X-Forwarded-For "%[req.fhdr(x-forwarded-for)], %[src]" if { req.fhdr(x-forwarded-for) -m found }
    http-request set-header X-Forwarded-For "%[src]" unless { req.fhdr(x-forwarded-for) -m found }

    # Original scheme, used by hop 2 to build the URL
    http-request set-header X-Forwarded-Proto https if { ssl_fc }
    http-request set-header X-Forwarded-Proto http  if !{ ssl_fc } !{ req.hdr(x-forwarded-proto) -m found }

    server ec_hop          127.0.0.1:10180 track ec_gateway/render
    server origin_fallback 10.0.0.10:8080

# Hop 2: internal listener that rewrites the request for EdgeComet
frontend ec_internal
    bind 127.0.0.1:10180
    default_backend ec_gateway

backend ec_gateway
    timeout connect 3s
    timeout server  60s

    # Reuse TLS connections to EdgeComet across crawler sessions;
    # a request that hits a closed connection is retried once
    http-reuse always
    retries 1
    retry-on conn-failure empty-response

    # Health check on EdgeComet's /health endpoint
    option httpchk
    http-check send meth GET uri /health ver HTTP/1.1 hdr Host render-us.edgecomet.com
    http-check expect status 200

    # /render?url=<URL-encoded full original URL>
    http-request set-var-fmt(txn.ec_url) "%[req.hdr(x-forwarded-proto)]://%[req.hdr(host)]%[pathq]"
    http-request set-uri "/render?url=%[var(txn.ec_url),url_enc]"
    http-request set-header Host render-us.edgecomet.com
    http-request set-header X-Render-Key YOUR_RENDER_KEY

    # Fail open: a response without EC-Source, and every 5xx, becomes a 503
    # that hop 1 retries on the origin
    http-response set-status 503 if !{ res.hdr(ec-source) -m found } || { status ge 500 }

    # ca-file: the system CA bundle. Debian/Ubuntu: /etc/ssl/certs/ca-certificates.crt
    # RHEL/AlmaLinux/Rocky: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
    server render render-us.edgecomet.com:443 ssl verify required ca-file /etc/ssl/certs/ca-certificates.crt sni str(render-us.edgecomet.com) check check-sni render-us.edgecomet.com inter 10s fall 3 rise 2 resolvers default resolve-prefer ipv4 init-addr last,libc,none

Replace 10.0.0.10:8080 in both places with your origin. origin_fallback must reach the same origin as backend origin, with the same ssl and sni options if your origin uses TLS. Replace YOUR_RENDER_KEY with your render key.

Notes on the configuration

• req.fhdr(user-agent) reads the whole user agent. req.hdr splits values at commas, and crawler user agents contain commas, such as KHTML, like Gecko.
• url_enc encodes every character except letters, digits and - . _ ~. The query string arrives whole, and + is not read as a space.
• balance first sends all traffic to ec_hop while it is up, and option redispatch 1 sends the retry to the other server, origin_fallback. A backup server would not work: HAProxy uses it only when no active server is up.
• X-Forwarded-For as one line: option forwardfor adds a second header line, and EdgeComet reads only the first. A single line carries the crawler's address where EdgeComet looks for it.
• ca-file with a full path: @system-ca fails on Debian 12's HAProxy 2.6.12 with the stock ca-base line. The bug is fixed in 2.6.16 and 2.8.4.
• init-addr last,libc,none lets HAProxy start when DNS fails, with the EdgeComet server in maintenance and crawlers on your origin. Without none, HAProxy refuses to start.
• resolve-prefer ipv4 avoids IPv6 on hosts without IPv6 connectivity. The render host has both address types.
• http-reuse always reuses TLS connections to EdgeComet across crawler sessions. With the default, each new crawler connection opens a new TLS connection.
• Timeouts: hop 2 waits 60 seconds, above EdgeComet's 15-second render timeout, and hop 1 waits 70 seconds, which keeps hop 1 from timing out first.
• The loopback listener accepts requests from any local process and sends them to EdgeComet with your key. Use a port nothing else needs.
• Behind a load balancer, %[src] is the load balancer's address. EdgeComet skips private addresses in X-Forwarded-For, but a public load balancer address can be recorded as the bot's IP. Have the load balancer send the PROXY protocol and add accept-proxy to the bind lines, which makes %[src] the bot's address.

HAProxy 2.4 variant

HAProxy 2.4 has no set-var-fmt and no automatic default resolvers section. Change two parts of the main configuration, and use 2.4.16 or later:

# Add before "defaults":
resolvers sysdns
    parse-resolv-conf

# In backend ec_gateway, replace the set-var-fmt and set-uri lines with:
    http-request set-uri "/render?url=%[req.hdr(x-forwarded-proto)]%%3A%%2F%%2F%[req.hdr(host),url_enc]%[pathq,url_enc]"

# In the server render line, replace "resolvers default" with:
    ... resolvers sysdns resolve-prefer ipv4 init-addr last,libc,none

Test and reload

# Check the configuration
sudo haproxy -c -f /etc/haproxy/haproxy.cfg

# Apply without dropping connections
sudo systemctl reload haproxy

The systemd unit checks the configuration before a reload and keeps the running one when the check fails.

What bots get when EdgeComet cannot answer

Case	Bots get	Extra wait
EC-Source with 200, a redirect, 401, 403, 404 or 410	EdgeComet's response	None
401 without EC-Source: wrong key, or a domain that is unknown or inactive	Your origin's page	One round trip to EdgeComet
400, 404 or 502 without EC-Source	Your origin's page	One round trip
Any 5xx from EdgeComet	Your origin's page	One round trip, and your origin is fetched twice
EdgeComet refuses connections, before the health check notices	Your origin's page	About 1 second
EdgeComet down after 3 failed health checks, within about 30 seconds	Your origin's page	None
No answer within 60 seconds	Your origin's page	60 seconds
TLS verification failure, or the render host does not resolve	Your origin's page	None, after the health check

The limits:

• Until the health check marks EdgeComet down, each crawler request pays the failure cost once, then gets your origin's page.
• A connection that breaks mid-body after EdgeComet started a good response is not replayed. The crawler gets a truncated page.
• Requests over one buffer are never retried. HAProxy's buffer is 16 KB by default, and a crawler request with more headers than that gets a 503.

Let EdgeComet through HAProxy rules

EdgeComet's own requests carry X-Edge-Render, which the ec_loop ACL sends to your origin, and X-Render-Key with your key. If HAProxy blocks bots or rate-limits, exempt the exact key. Put the ACL before your blocking rules:

frontend www
    # EdgeComet page fetches carry the render key: exempt them from blocking and rate limits
    acl ec_fetch req.fhdr(x-render-key) -m str YOUR_RENDER_KEY

    # Example bot blocking rule
    http-request deny if { req.fhdr(user-agent) -m sub -i badbot } !ec_fetch

    # Example rate limit: 5 requests per 10 s per IP
    stick-table type ip size 100k expire 30s store http_req_rate(10s)
    http-request track-sc0 src unless ec_fetch
    http-request deny deny_status 429 if { sc_http_req_rate(0) gt 5 } !ec_fetch

-m str is an exact, case-sensitive match. Add !ec_fetch or unless ec_fetch to your own rules.

A WAF in front of HAProxy needs its own exception. See Let EdgeComet through your firewall.

Check the integration

# Crawler: expect EC-Source in the response
curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/ | grep -i '^ec-'

# Browser: expect no EC- headers
curl -sI -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36" https://example.com/ | grep -i '^ec-'

# Health check state of the EdgeComet server: expect UP L7OK
echo "show stat" | sudo socat stdio /run/haproxy/admin.sock | awk -F, '$1=="ec_gateway" && $2=="render" {print $18, $37}'

Check for EC-Source, not for a 200. With the fallback in place, a broken setup still serves your origin's page with a 200.

option httplog shows which way each crawler request went. edgecomet/ec_hop means EdgeComet served it, and edgecomet/origin_fallback with +1 in the retries field means it fell back:

www edgecomet/ec_hop          ... 200 ... "GET /p?a=1&b=2 HTTP/1.1"
www edgecomet/origin_fallback ... 200 ... +1 "GET /x HTTP/1.1"

To log the EC-Source value, add capture response header EC-Source len 32 to frontend www. To send all crawlers to your origin for a test, set ec_gateway/render to maint through the runtime API, and back to ready afterwards.

Then run Test connection and Verify install in the Generic Integration guide, and see Check that search and AI bots reach EdgeComet.

Update or remove

After a render key rotation, update X-Render-Key and the ec_fetch ACL, then reload. To remove EdgeComet, delete the four ACLs, the use_backend edgecomet line and the edgecomet, ec_internal and ec_gateway sections, then reload.

If it does not work

Crawlers get pages without EC-Source

The fallback hides an error. Check the log for edgecomet/origin_fallback and the health state of ec_gateway/render, then the render key, the CA file and DNS. A wrong key gives 401 Authentication failed: invalid render key or domain mismatch, which falls back while the health check stays UP L7OK, because /health needs no key.

ec_gateway/render is down with "SSL handshake failure"

Certificate verification failed: the CA file is wrong or empty. Use the bundle path for your distribution.

"Couldn't open the ca-file" or "0 CA were loaded from '@system-ca'"

Replace @system-ca with the full path of your CA bundle.

ec_gateway/render is down with L4CON or L4TOUT

HAProxy cannot reach the render host on port 443. Check outbound firewall rules, and keep resolve-prefer ipv4.

HAProxy does not start: "Failed to initialize server(s) addr."

none is missing from init-addr on the server render line.

"invalid or incomplete action 'set-var-fmt(txn.ec_url)'"

Your HAProxy is 2.4. Use the 2.4 variant.

Every crawler request takes about a second and lands on your origin

EdgeComet refuses connections and the health check has not marked it down, or the check is missing: check on server render, or track on server ec_hop.

Crawlers get 503

origin_fallback cannot be reached, or the request is larger than one buffer and cannot be retried.

Related

• How EdgeComet routes bot traffic
• Nginx and Apache
• HAProxy for the self-hosted engine, which covers the open-source engine you run yourself