The global entry

Where a tenant lives. Nothing remote is asked on the way: a request carries Fenec-Home: 1, and the router that forwards it to the tenant's node answers with Fenec-Home: <region>, the node's — a router relaying it to another region names nothing of its own, and the answer it passes back carries the home router's. The entry takes the header off before the answer goes to the client: a router names the home on a refusal too, so to anyone holding a tenant's name. The entry keeps 50 000 tenants' homes an isolate (HOMES), forgotten first-heard first, and shares each one it learns with the colo's other isolates through the Cache API, read only by an isolate that has not heard the tenant (HOMES_CACHE; it does nothing on workers.dev, so the entry wants a custom domain). A tenant heard nowhere goes to the region REGIONS names for the client's continent (Cloudflare's request.cf.continent), else DEFAULT_REGION, then to each other region in turn where one's routers cannot be reached — any region's router relays the tenant home, and only the home's router places it on a node, so no write lands anywhere else. A region whose routers failed is asked last: by a read or a keyed write for ten seconds, then first again; by any other write until one of its routers answers, since a write that may have reached a router cannot be sent to another. A guessed region's router that answers 5xx naming no home — cut off from the home's region, say — sends a read on to the next region too; a write it answered is its answer. Asking the global router at each miss would have needed its --token, which moves and deletes every tenant, in the Worker, a round trip to the global router's region, and would have turned every unknown name a client tries into a question for that one router; a table in Workers KV is read eventually consistently and billed by the read, and still needs something to write it. A router answers the header only to a request asking for it, so no other client is told one; a router from before it answers none, and the entry keeps sending such a tenant to the nearest region, relayed each time.

Streams and Cloudflare's limits. The answer is handed back as its body arrives, so a server-sent event reaches the client as the router sends it. Cloudflare's Workers limits: waiting on a fetch is not CPU time, and an HTTP-triggered Worker has no limit on its duration while the client stays connected, so a subscription or a held write is not cut; an invocation may have six connections waiting for their headers, and the entry has one. A router behind a hostname Cloudflare proxies must start its answer within the proxy read timeout (100 seconds, fixed below Enterprise): a Fenec-Wait holds a write 30 s at most, and a silent stream writes a keep-alive every 20 s (--http-keepalive). The Worker's request.signal (the enable_request_signal flag) is handed to the fetch, so a client gone cancels the request to the router. Under wrangler dev a stream is let go of at a stand-in router 2 s after its client goes, workerd noticing as the next keep-alive is written, and at a node through a router in 4 to 8 s, each hop noticing at its next. A client gone before its answer's head is not told to the Worker there at all — workerd notices a client only as it writes to it, past wrangler's own proxy too — so that case is held in Node, the handler over Node's fetch. Past the entry it goes on to the node: a router waiting on a node's held answer watches its client and passes the client's end on, as the node would read it from a client of its own — a region's router relaying the request to the home router, which passes it to the node — so a claim whose client went hands its job to the next claim held, as it does asked of the node itself, rather than lease it to nobody; a client that only half-closed is answered as the node answers it.

Headers. Everything a client sends goes on — Authorization, Idempotency-Key, Fenec-*, traceparent, X-Request-Id (Cloudflare's ray id where the client sent none, so the routers' and the nodes' lines carry it) — but the hop-by-hop headers, the routers' own marks (Fenec-Router, Fenec-Global, Fenec-Verified), which a router believes from another router alone, a Fenec-Client of the client's own, and every way a client names an address (X-Forwarded-For, X-Real-IP, Forwarded, True-Client-IP, CF-*). Fenec-Verified and Fenec-Home are dropped from answers. The routers' CORS headers come back as they sent them; with CORS set to their --http-cors, the entry answers a preflight itself with the same headers, a round trip saved and a region that is down no reason for a page to see a network error, and puts them on its own answers alone — two Access-Control-Allow-Origins are refused whole by a browser. A path is judged as the router and the node will read it, decoded as fenec-http decodes one: /t/acme/%5Freplication and /t/acme%2F_replication are a tenant's replication stream to them, which a URL's own parsing leaves encoded.

The client's address. Behind the entry every request reaches a router from Cloudflare's addresses, so the wait after a refused token would count every client's refusals together, and any good token would start the count again. The entry names its client in Fenec-Client: <address> <seconds> <mark>, the address Cloudflare's CF-Connecting-IP, the seconds the Unix time the entry made it at, the mark 16 bytes of HMAC-SHA256 of fenec-client <address> <seconds> under ENTRY_TOKEN, base64url. Bound to the address and the time, a mark seen names that one address for a minute, where a mark of the token alone would let whoever saw one name any address for good. Not CF-Connecting-IP or X-Forwarded-For believed from Cloudflare's addresses: a Worker sets CF-Connecting-IP and X-Real-IP as it likes on a request to an origin outside Cloudflare (Cloudflare's HTTP headers page), so any account's Worker could claim any address from those ranges.

A router started with the entry's token — fenec-shard --entry-token, or FENEC_SHARD_ENTRY_TOKEN — checks the mark against its own HMAC, in constant time, and the time within 60 seconds of its clock either way, and takes the address for that request's in the connection's place. A refused token waits by it, and a good one starts its count again and no other's; the audit log names it as peer, the entry's address beside it as via, and a trace as client.address, the connection's as network.peer.address; and it is the client the router names to the tenant's node, or to the home region's router it relays the request to, which names it to the node in turn. A header seen on the wire can be replayed for that minute, and names that one address: what the minute costs, where a mark of the token alone would name any address for good. A header that does not check, or a router without the flag, counts the connection's address, and fenec_router_entry_clients_total says which (named, bad_mark, stale, or absent for a tenant's request carrying none, as every one from an entry whose ENTRY_TOKEN is unset does), the router's log a minute apart at most of each kind not believed — not the audit log, which a header costing its sender nothing would fill at the router's request rate. On a router only the entry reaches, alert on all but named (Monitoring). Every router drops the header from what it forwards, flag or not, so a node never reads a client's claim. A mark costs a request one HMAC-SHA256, 0.75 µs on this machine, which a round trip does not show: through a router with the flag and one without, in turns, a request the entry named its client in took 41.7 µs p50 against 42.1, and one with no header 42.0 against 42.05 — a look among its headers and a count — the runs' differences a few tenths of a microsecond either way (make shard-bench ARGS=latency, eight runs). Without the flag the router's added p50 stayed 20.4 µs, as before.

The entry's token is its own. The Worker holds it, so a router refuses it where it holds it for anything else: equal to its --token, which moves and deletes tenants, to its --replication-token, whose stream carries every node's admin token, or to the admin token of a node in its directory, at the start and as a node is added. The nodes' data and replication tokens the router never sees: keep the entry's apart from those too. A router takes one entry token at a time. To change it, start each router with the new one, then put it in the Worker (npx wrangler secret put ENTRY_TOKEN): in between every request is served, and its client counted by Cloudflare's address, as bad_mark shows — a few minutes of the counting there was before, against a second token held by every router.

When a router cannot be reached. A region's routers are tried in turn from the one that answered last. Every answer a router writes carries its mark, Fenec-Shard: 1, and an answer at a router's URL without it is not the router's — not X-Request-Id, which a router's answers carry too and a load balancer may write on its own 502. Cloudflare's own answers for an origin it never reached (521, 523, 525, 526, and 530, a tunnel with no connector) go on to the next router whatever the request. One that may have reached the router — a fetch that fails, which workerd says alike of a connection refused and of one reset after the request was written; Cloudflare's 520, 522 (a request whose bytes went unacknowledged for 90 seconds is among its time-outs) and 524; and any other 5xx without the router's request id, such as a tunnel's connector answering 502 for a router it cannot reach — goes on only for a GET, HEAD, OPTIONS or a request carrying an Idempotency-Key, as the router's own rule towards a node; any other is a 502 saying it may have run, and the region's next request goes to the next router. A router's own 502 — a node it could not reach — names the home and is its answer, passed on, no other region tried. A body up to a mebibyte is kept for the next router; a longer one is streamed to one, its length kept (a router takes no chunked body), and one sent with no length past a mebibyte is refused with 411: read whole, it was as large as a client cared to send, in isolates whose 128 MB are shared.

What it costs. Under wrangler dev on this machine, a read by id of a tenant in us took 81 µs p50 asked of us's router directly, and through the entry with its home heard 1.18 to 1.22 ms, 0.66 to 0.68 past wrangler's own proxy Worker — workerd taking the request, the entry and its fetch, all on loopback; with the client named besides (ENTRY_TOKEN, an HMAC a request), 1.23 to 1.24 ms (0.68 to 0.70). Never heard, it went to eu first, whose router relayed it: 1.29 to 1.31 ms (0.71 to 0.75), eu's router asked directly relaying it in 114 to 115 µs. Heard only through the colo's cache, 1.60 to 1.61 ms (1.00), miniflare's cache a service of its own (make entry-bench: two runs of six rounds of 2 000 in turns, at loads of 3.4 to 5). That is not what Cloudflare adds: there a client reaches its nearest colo and the router is a region away, the leg it would pay to reach the router itself; a home not heard costs a relay between regions, where the cache is read inside the colo. A router asked for the home answered in 82 to 83 µs p50 against 81 not asked; not asked, its mark on every answer, its added p50 was 23.75 µs against main's 23.9 (23.4 to 24.1 and 23.7 to 24.0, six runs a side in ABBA turns, make shard-bench ARGS=latency).