Route search and AI bots to EdgeComet with IIS

IIS routes bots to EdgeComet with one URL Rewrite rule that proxies through Application Request Routing (ARR). Crawler page requests go to EdgeComet's render endpoint, and every other request keeps your current IIS configuration.

This is the IIS setup for prerendering or dynamic rendering, plus EdgeComet's caching and bot analytics. It works on a single IIS server, on an ARR server in front of a web farm, and behind Microsoft load balancers such as Azure Load Balancer, Azure Application Gateway and Azure Front Door.

How it works

• A URL Rewrite rule matches when the method is GET or HEAD, the request uses HTTPS, the request has no X-Edge-Render header, the path is not a static file and the user agent is a crawler.
• Its rewrite action points the request at https://render-de.edgecomet.com/render?url={encoded URL}. The UrlEncode function puts the full page URL, query string included, into the url parameter.
• The rule sets two request headers: X-Render-Key with your render key, and X-Forwarded-Proto.
• ARR proxies the request to the render host. It sends the render host as Host and adds the bot's IP address to X-Forwarded-For.
• ARR passes EdgeComet's response to the bot, including redirects, 404 and 410.

The conditions, headers and status codes are the same for every platform. See How EdgeComet routes bot traffic.

Where to add the rule

Add the rule on the first IIS server that bot requests reach. That server sees the bot's user agent, the original scheme and the bot's IP address.

Microsoft's load balancers cannot run the rule in place of IIS. Application Gateway and Front Door can match a user agent, but per Microsoft's documentation their rewrites cannot URL-encode the page URL into a parameter.

Your setup	Where the rule goes	What to check
One IIS server	A global rule on that server	Nothing else
ARR server in front of a web farm	A global rule on the ARR server, above the farm's ARR_<farm>_loadbalance rule	The farm rule stops rule processing. A rule after it never runs
Windows NLB or Azure Load Balancer in front of IIS servers	A global rule on every IIS server	These balancers work at layer 4 and pass the bot's IP address through to IIS
Azure Application Gateway in front of IIS	A global rule on the IIS servers	Match HTTPS on X-Forwarded-Proto, set the request timeout, check bot IPs
Azure Front Door in front of IIS	A global rule on the IIS servers	Turn off caching for pages, match HTTPS on X-Forwarded-Proto, check the host and bot IPs

Behind Azure Application Gateway

• HTTPS: Application Gateway can end TLS, and IIS then sees {HTTPS} as off. Match {HTTP_X_FORWARDED_PROTO} against ^https$ instead, the header Application Gateway sends.
• Timeout: the backend setting's request timeout defaults to 20 seconds. Set it to 60 seconds, which covers a render.
• Bot IP: Application Gateway sends the bot's address in X-Forwarded-For as IP:port, which EdgeComet reads. After setup, check that Data Explorer > Bot log shows bots' addresses, not Application Gateway's.

Behind Azure Front Door

• Caching: Front Door cannot use the user agent in its cache key. With caching on for pages, Front Door can answer a bot with a cached visitor page, which never reaches IIS, or serve EdgeComet's copy to visitors. Turn caching off on routes that serve pages.
• Host: Front Door sends the origin's host name as Host by default, not your domain. Leave the origin host header empty, which passes your domain on, or build the URL in the rule from {HTTP_X_FORWARDED_HOST}.
• HTTPS and bot IP: match {HTTP_X_FORWARDED_PROTO}, and check the Bot log as for Application Gateway.

Render hosts

Cluster	Render host
Germany	render-de.edgecomet.com
United States	render-us.edgecomet.com

The examples on this page use the German cluster. For the US cluster, replace render-de.edgecomet.com with render-us.edgecomet.com.

ARR must send the render host as Host, and your IIS servers need outbound HTTPS on port 443 to it. If a firewall limits outbound traffic, allow the render host by name, not by IP address.

Before you start

• IIS 10 with URL Rewrite 2.1 and Application Request Routing 3.0, installed from Microsoft's packages on iis.net.
• Administrator access to the server. Global rules and ARR's proxy settings live in applicationHost.config.
• Your Render Key and Render Host. The dashboard has no IIS guide: take them from the Generic Integration guide under Configuration > Connections > CDN integration.
• The domains your rule covers must be domains of your EdgeComet website. EdgeComet answers 401 for any other domain.

Configure ARR

ARR's reverse proxy is off until you turn it on. In IIS Manager, Enable proxy is under Application Request Routing Cache > Server Proxy Settings. All four settings are in the system.webServer/proxy section of applicationHost.config:

Setting	Value	Why
enabled	true	Turns on ARR's reverse proxy
preserveHostHeader	false, the default	ARR sends the render host as Host. With true, it sends your domain, which the render host refuses with 403
reverseRewriteHostInResponseHeaders	false	With true, ARR rewrites the host in Location headers, which can change EdgeComet's redirects to another domain
timeout	60 seconds or more	EdgeComet's render timeout defaults to 15 seconds. The ARR default is 2 minutes

From an administrator PowerShell:

& "$env:windir\system32\inetsrv\appcmd.exe" set config -section:system.webServer/proxy /enabled:"True" /preserveHostHeader:"False" /reverseRewriteHostInResponseHeaders:"False" /commit:apphost

These are server-wide settings. Web farms have their own preserveHostHeader and timeout, which this change does not touch. If another reverse proxy rule on the server needs preserveHostHeader set to true, move it to a web farm.

URL Rewrite 2.1 also has a setting, useOriginalURLEncoding, which is true by default and double-encodes {UNENCODED_URL}: a path with %2F reaches EdgeComet as %252F. Set it to false in Configuration Editor, as Microsoft's IIS team advises for reverse proxies.

The routing rule

Add the rule to the global rules in applicationHost.config, first in the list. Replace the domain pattern and YOUR_RENDER_KEY:

<system.webServer>
  <rewrite>
    <globalRules>
      <!-- EdgeComet: send crawler page requests to EdgeComet. Keep it first -->
      <rule name="EdgeComet bot routing" stopProcessing="true">
        <!-- Not a static file -->
        <match url="\.(avif|css|eot|gif|gz|ico|jpeg|jpg|js|json|map|mp3|mp4|ogg|otf|pdf|png|svg|ttf|txt|wasm|wav|webm|webmanifest|webp|woff|woff2|xml|zip)$" negate="true" />
        <conditions logicalGrouping="MatchAll">
          <!-- Domains of your EdgeComet website -->
          <add input="{HTTP_HOST}" pattern="^(www\.)?example\.com$" />
          <add input="{REQUEST_METHOD}" pattern="^(GET|HEAD)$" />
          <!-- HTTPS only. Behind Application Gateway or Front Door: input="{HTTP_X_FORWARDED_PROTO}" pattern="^https$" -->
          <add input="{HTTPS}" pattern="^on$" />
          <!-- Loop guard: EdgeComet's own requests carry X-Edge-Render -->
          <add input="{HTTP_X_EDGE_RENDER}" pattern=".+" negate="true" />
          <!-- Crawler user agents -->
          <add input="{HTTP_USER_AGENT}" pattern="bot|crawl|spider|slurp|WhatsApp|Snapchat|facebookexternalhit|AMZN-User|Claude-User|Perplexity-User|ChatGPT-User" />
        </conditions>
        <serverVariables>
          <set name="HTTP_X_RENDER_KEY" value="YOUR_RENDER_KEY" />
          <set name="HTTP_X_FORWARDED_PROTO" value="https" />
        </serverVariables>
        <action type="Rewrite" url="https://render-de.edgecomet.com/render?url={UrlEncode:https://{HTTP_HOST}{UNENCODED_URL}}" appendQueryString="false" />
      </rule>
      <!-- Your existing global rules, such as ARR_<farm>_loadbalance, follow -->
    </globalRules>
  </rewrite>
</system.webServer>

Notes on the rule

• Global rules run before site rules and before ARR's farm rule. They keep your render key in applicationHost.config, out of site web.config files that often sit in source control.
• {HTTP_HOST}: global rules apply to every site on the server. List the domains of your EdgeComet website only.
• HTTPS only: an http:// request goes to your site, which redirects it to HTTPS, and the bot's next request is routed. The https:// in the action relies on this condition.
• {UNENCODED_URL} is the path and query string exactly as the bot sent them. UrlEncode turns the full URL into one parameter value, and appendQueryString="false" keeps IIS from adding the query string a second time, outside that value.
• Letter case: URL Rewrite ignores case in patterns by default, which the static file and crawler checks need.
• X-Forwarded-For: ARR adds the client's address with its port, such as 66.249.66.1:52344. EdgeComet reads that form and verifies the bot by its IP address.
• Output caching: a rule that reads request headers, such as {HTTP_USER_AGENT}, makes URL Rewrite turn off kernel-mode output caching for all requests. Check this against your caching setup.
• A site rule in web.config works on a single server too. It can set only server variables on the allowed list: add HTTP_X_RENDER_KEY and HTTP_X_FORWARDED_PROTO under URL Rewrite > View Server Variables at server level.

What bots get when EdgeComet cannot answer

IIS has no per-request fallback. ARR cannot retry a failed request on your site, and URL Rewrite cannot check a response for EC-Source.

Case	Bots get
EdgeComet answers with EC-Source, including redirects, 404 and 410	EdgeComet's response
401 Authentication failed: invalid render key or domain mismatch	The 401
400 Invalid URL, for example a page URL longer than 2,048 characters	The 400
EdgeComet answers 502 or another 5xx	The 5xx
EdgeComet cannot be reached, or does not answer within ARR's timeout	502 from ARR, logged as 502.3

The Cloudflare, Nginx, Apache and HAProxy integrations send bots to your origin in the last four cases. On IIS, protect against them:

• Test the key and domain before you turn the rule on, with the direct request in Check the integration. Run Test connection and Verify install after every key or domain change.
• Remove the rule before you set the website to Inactive or rotate the render key.
• If bots must never get these errors, route at a layer that falls back, such as Cloudflare, Nginx or HAProxy in front of IIS.

Let EdgeComet through IIS rules

EdgeComet's own requests carry X-Edge-Render, which sends them past the routing rule to your site, and X-Render-Key with your key. IIS Request Filtering and Dynamic IP Restrictions cannot make exceptions by header.

If you block bots with URL Rewrite rules, add a condition that skips requests with your exact render key:

<rule name="Block unwanted bots" stopProcessing="true">
  <match url=".*" />
  <conditions logicalGrouping="MatchAll">
    <add input="{HTTP_USER_AGENT}" pattern="BadBot|HeadlessChrome" />
    <!-- EdgeComet fetches carry your render key -->
    <add input="{HTTP_X_RENDER_KEY}" pattern="^YOUR_RENDER_KEY$" ignoreCase="false" negate="true" />
  </conditions>
  <action type="AbortRequest" />
</rule>

Dynamic IP Restrictions counts EdgeComet's requests like any other client's. If it blocks renders, raise its limits. A WAF in front of IIS, such as Azure Web Application Firewall on Application Gateway or Front Door, needs its own exception. See Let EdgeComet through your firewall.

Check the integration

Run these checks on one domain before you send all bots through the rule. They cover query strings, escaped characters and the loop guard:

# Crawler: expect EC-Source and the other EC- headers
curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/ | grep -i '^ec-'

# Browser: expect no EC- headers
curl -sI -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36" https://example.com/ | grep -i '^ec-'

# Loop guard and static file: expect no EC- headers
curl -sI -A "Googlebot" -H "X-Edge-Render: test" https://example.com/ | grep -i '^ec-'
curl -sI -A "Googlebot" https://example.com/favicon.ico | grep -i '^ec-'

# Query string and escaped characters: compare the URL in Data Explorer > Bot log with this one
curl -s -o /dev/null -A "Googlebot" "https://example.com/search?q=a+b%2Fc&page=2"

# Key and domain, bypassing IIS: 401 means a wrong key, domain or an inactive website
curl -i -A "Googlebot" -H "X-Render-Key: YOUR_RENDER_KEY" \
  "https://render-de.edgecomet.com/render?url=https%3A%2F%2Fexample.com%2F"

On each IIS server, check outbound HTTPS to the render host. Expect TcpTestSucceeded : True:

Test-NetConnection render-de.edgecomet.com -Port 443

To see which requests EdgeComet answered, add EC-Source to the site's IIS log. In IIS Manager, select the site, open Logging, click Select Fields... and Add Field..., and pick Response Header as Source Type with EC-Source as Source. For a failing request, Failed Request Tracing shows the rule that matched, the rewritten URL and ARR's error code.

Then run Test connection and Verify install in the Generic Integration guide, and see Check that search and AI bots reach EdgeComet.

Update or remove

After a render key rotation, update HTTP_X_RENDER_KEY and your exemptions. To stop sending bots to EdgeComet, delete the rule from the global rules.

If it does not work

Bots get 401 Authentication failed: invalid render key or domain mismatch

The render key is wrong, the domain in the URL is not on your EdgeComet website, or the website is inactive. Compare the key with the dashboard guide. Behind Front Door, check that the URL carries your domain, not the origin's host name.

Bots get 403 with server: cloudflare and no EC-Source

ARR sent your domain as Host to the render host. Set preserveHostHeader to false in system.webServer/proxy.

Bots get 502 and the IIS log shows 502.3

ARR could not reach the render host or timed out. Run Test-NetConnection, check DNS and outbound firewall rules, and check ARR's timeout. Failed Request Tracing shows the WinHTTP error code.

HTTP Error 500.50 - URL Rewrite Module Error.

A site rule sets a server variable that is not on the allowed list. Add HTTP_X_RENDER_KEY and HTTP_X_FORWARDED_PROTO at server level, or use a global rule.

Escaped characters arrive double-encoded, such as %252F

useOriginalURLEncoding is true. Set it to false in Configuration Editor.

The rule has no effect

A global rule with stopProcessing, such as ARR's farm rule, runs before it: move the EdgeComet rule to the top. Behind a load balancer that ends TLS, {HTTPS} is off: match {HTTP_X_FORWARDED_PROTO} instead.

Bot IPs in your reports are your load balancer's

The bot's address does not reach EdgeComet in X-Forwarded-For. Check what your load balancer and ARR send, and see What if every Googlebot request shows as a fake bot?

Test connection fails

A rule in IIS, or a WAF in front of it, blocks EdgeComet's request. Add an exemption.

Related

• How EdgeComet routes bot traffic
• Connect EdgeComet to any CDN or reverse proxy
• Let EdgeComet through your firewall
• Nginx, Apache and HAProxy