How EdgeComet Works

Three services. Clear responsibilities. Production-grade JavaScript rendering for bots.

Architecture

Request Flow

EdgeComet sits behind your reverse proxy.
Only bot traffic is routed through the system.

All
Traffic
Load Balancer
CDN/LB
Only Bots
Traffic
EdgeComet
Gateway 1
Gateway 2
Gateway N
Cache
Render 1
Render 2
Render 3
Render N
Your Website
Components

Three Services, Clear Responsibilities

Each component does one thing well. Deploy together or scale independently.

Edge Gateway

Entry point that intercepts bot requests, manages caching, and coordinates rendering.

  • Under 50MB RAM, minimal CPU
  • Filesystem cache + Redis metadata
  • Graceful fallback to bypass mode
  • Automatic peer discovery via Redis

Render Service

Manages headless Chrome pool to render JavaScript-heavy pages on demand.

  • Multiple Chrome instances, isolated tabs
  • Auto-restarts to prevent memory leaks
  • Configurable wait strategies
  • 3-second failover on crash
Optional

Cache Daemon

Background service that keeps cache fresh without impacting live traffic.

  • Three queue priorities (urgent, normal, auto)
  • Rate-limits to preserve render capacity
  • Recaches only visited pages
  • Exposes invalidation API
Edge Gateway

Bot Detection & Request Routing

20+ Pre-configured Bot Patterns

Google (Search, Ads, Mobile), Bing, and AI crawlers: GPTBot, ClaudeBot, PerplexityBot, plus messaging apps (WhatsApp, Telegram, Slack, Discord).

Flexible URL Pattern Matching

Exact paths, wildcards (/blog/*), regexp (~/api/v[0-9]+/), and query parameter matching. Patterns auto-sorted by specificity.

Three-Level Configuration

Global defaults, per-host overrides, and per-URL pattern rules. Deep merge semantics with predictable inheritance.

Device Dimensions

Render as desktop or mobile. Each dimension cached separately. Serve appropriate content based on crawler type.

Diagnostic X-Headers

X-Render-Source, X-Cache-Age, X-Matched-Rule for debugging. Trace every request through the system.

URL Actions

Render, bypass to origin, block (403/404/410), or return custom status codes. Control what bots see at each URL.

Render Service

Chrome Without the Pain

EdgeComet manages Chrome instances proactively so you don't get paged at 2am.

Event What Happens Your Involvement
Memory bloat Scheduled restart after N renders None
Instance hang Auto-detected and replaced None
Chrome crash New instance spawned within 3 seconds None
Render timeout Soft timeout captures partial HTML None
Security update Update Chrome binary, restart service ~30 min, few times/year

Wait Strategies

Choose when to capture: networkIdle (recommended), load, or DOMContentLoaded. Add extra delay for late-executing JavaScript.

Resource Blocking

22+ analytics and tracking scripts blocked by default. Images, fonts, media skipped. 30-50% faster renders, reduced origin load.

Isolated Instances

One browser per instance, ephemeral tab per render. No cross-request leakage. Tab context destroyed immediately after capture.

Pool Sizing

10-15 instances for 8-core servers. Scale horizontally by adding more Render Service nodes. Edge Gateway load-balances automatically.

Two-Tier Storage for Speed & Scale

Redis

Metadata, distributed locks, service registry

  • Cache key existence checks
  • TTL and expiration tracking
  • Distributed lock coordination
  • Cluster peer discovery

Filesystem

HTML content, date-organized directories

  • Fast local reads (~15ms)
  • Date-based directory structure
  • Bulk cleanup via directory removal
  • No Redis memory pressure
Expiration: delete

Remove expired cache immediately. Force fresh renders for next bot visit.

Recommended
Expiration: serve_stale

Keep expired content as fallback. If fresh render fails, bots still get cached content.

~15ms
Cache Hit Response
High Availability

Cache Sharding & Replication

For high-traffic sites, distribute cache across multiple Edge Gateway instances.

hash_modulo

Consistent hashing maps same URL to same instance set. Predictable cache placement.

random

Shuffles healthy instances for load balancing. Good for write-heavy workloads.

primary_only

No replication. Cache stored only on rendering instance. Simplest setup.

Replication Model

Push on Render
Render
Local
Peer
Peer

Immediate replication after render completes

Pull on Access
Local
MISS
Peer
HIT
Peer

Lazy fetch when cache miss occurs

Fallback Chain

When serving a request, EdgeComet tries each source in order until successful:

Fresh Local
Fresh Remote
Re-render
Stale Local
Stale Remote
Bypass
Reliability

Graceful Degradation

Bots always receive a response. EdgeComet never becomes a single point of failure.

Failure Scenario What Bot Receives SEO Impact
Redis down Bypass to origin (direct fetch) Normal origin behavior
Website down Stale cache (if available) OR bypass Cached content protects indexation
Render Service down Stale cache (if available) OR bypass Cached content protects indexation
Chrome crash Auto-failover to next instance Minimal - automatic recovery
Cache file missing Fresh render triggered Slight delay, then cached
Built For Performance

Technology Stack

Go 1.21+

High-performance, low memory footprint

FastHTTP

10x faster than net/http

chromedp

Chrome DevTools Protocol

Redis 6.0+

Coordination & metadata

Prometheus

Metrics & monitoring

Zap

Structured JSON logging

Ready to Deploy?

Get EdgeComet running in your infrastructure in 15 minutes.

Get a Quote Browse Docs