Replicas, standbys and failover

Every tenant file has a feed of its own, so a node's tenants are followed by another node's — the standby — tenant by tenant, through /t/<tenant>/_replication.

The router creates a tenant on both, so the replica has a file to follow into, and deletes it from both; a move takes the replica with it. Recording a pair gives the standby a following copy of every tenant already on the node. A replica node opens every tenant it holds at start and keeps them open — a follower that is not running is a replica falling behind — so --idle-close does not apply to them. No tenant is placed on a standby or moved to one: it would open there as a replica of a tenant the other node does not have.

A tenant's role is its file's, not the node's. On a node started with --replica-of, a replica's file follows and a new one follows, but a primary's file stays a primary — a tenant a failover promoted there included, whatever the node restarts with. Were every file on such a node made to follow, a restart with the same flags would undo the failover, and the old primary's copy would wipe the writes taken since.

When a node is gone, POST /_shard/nodes/<n>/failover promotes its tenants on the standby, one at a time, and points the directory at it. They are separate databases: there is nothing to make atomic between them, so the answer lists what was promoted and what was not. A copy that is not its tenant as the directory knows it — of another lineage, or an old primary's copy that never heard from the new one — is refused by its node and listed as not promoted, the tenant left where its rows are; with -d '{"force": true}' it is promoted whatever it holds, which the audit log records. Under --auto-failover the router grants each node the tenants went to its lease over them before it answers, so they take writes as soon as the answer comes: left to the next round, a promoted copy refused them for up to a third of a lease.

curl -X POST localhost:8090/_shard/nodes/n1/failover
{"node":"n1","to":"n2","promoted":["acme","beta"],"failed":[]}

Measured with make shard-bench, node, standby and router in one process on loopback: a write through the router is answered in 2.93 ms (--sync always on the primary, which is an fsync a write) and is on the standby 0.089 ms later, 0.448 ms at p99 — timed by reading the standby's own copy. A failover of 20 tenants took 63.5 to 79 ms, the medians of three rounds of four to twelve runs, 3 to 4 ms each, and the first write on the promoted node 0.18 ms.

A promoted tenant's history forks where the replica stood, so what the primary had not sent is gone — the same trade a database's failover makes, and --sync always on the primary is what makes it none. A failover that moved every tenant ends the pair; one that left a tenant behind keeps it, so a second call can take it. The old node rejoins as the standby of the new one: point its --replica-of there and record the pair the other way round. The router then has each of its tenants follow (POST /_admin/tenants/<t>/follow) — its own files are a primary's, and follow only when told to — and the writes it took that were never sent are gone.

fenec-server --dir /data/n1 --http 127.0.0.1:8081 --admin-token a1 \
         --replication-token r1 --replica-of http://127.0.0.1:8082
curl -X PUT localhost:8090/_shard/nodes/n2 \
  -d '{"addr":"127.0.0.1:8082","token":"a2","standby":"n1"}'
{"node":"n2","unreplicated":[]}

A tenant file that still follows on a node that follows nobody — a failover that left it behind, and the node restarted without --replica-of — refuses writes, says so in the log, and is promoted where it stands with POST /_admin/tenants/<t>/promote (with -d '{"force": true}' where it holds no write and has heard nothing from its primary since it opened).

A standby waits idle, and takes every tenant of its node at once when the node goes. With --replicas the router gives each tenant a replica of its own instead, on another node in no pair: the one holding the fewest replicas of that tenant's node, then the fewest tenants and replicas altogether. Each node's tenants then follow on all the others, and every node holds primaries and replicas both.

The replica's node is told which node to follow (POST /_admin/tenants/acme/follow with {"from": "http://127.0.0.1:8081"}) and keeps it in acme.follows beside the file, so the tenant follows again when the node starts over. The replication token is the nodes', one for the cluster. A move takes the replica along: the copy goes on following the tenant on its new node, since the image carried its history, and when the tenant moves onto the node holding its replica, that copy goes and another is placed. A delete removes both.

A failover of a node in no pair promotes each of its tenants on its own replica, one at a time, and routes it there. The tenants whose replicas it held are named as unreplicated. POST /_shard/replicas gives every tenant without a replica one, on a node holding a copy of it first: a node back from a failover has its old primaries follow the nodes that took them over, and nothing of theirs is left lying there.

curl -X POST localhost:8090/_shard/nodes/n1/failover
{"node":"n1","promoted":[{"tenant":"acme","to":"n2"},{"tenant":"beta","to":"n3"}],"failed":[],"unreplicated":["gamma"]}
curl -X POST localhost:8090/_shard/replicas
{"replicated":[{"tenant":"acme","replica":"n1"},{"tenant":"beta","replica":"n1"},{"tenant":"gamma","replica":"n1"}],"left":[],"failed":[]}

A replica that did not take is tried again. A create, a move or a repair that gives a tenant its replica makes the file on the other node and tells it to follow the primary; where that node refuses, or takes the follow past the router's --upstream-timeout — a rename on a node under load took 13 s, longer than the deploy kit's 10 s — the answer names the tenant unreplicated and a repair runs again 2 s later, then 4, 8 and so on up to a minute apart, eight times, about four minutes in all. It asks again whether the tenant still needs one, so it is placed once, and it adopts a copy whose follow landed after the router stopped waiting: the node already holding the copy comes first among those of the zone the rule picks, so no second copy is made. A node that did not answer the repair's look is given no replica. A tenant whose region has no other node is not tried again: add a node, then POST /_shard/replicas.

A file the router made on a node, or meant to take off one, that no placement names — a create taken back (its schema refused, say) whose DELETE the node did not take, a create whose PUT went unanswered, a move undone whose target kept the copy, a move's source that would not go, a copy a failed try made or found, a deleted tenant's copy that stayed, an old primary on a node that was silent as its tenant's replica went elsewhere — is noted in the directory with why, and reported: every repair lists it under left ({"tenant","node","why"}), fenec_router_files_left counts it, and FenecFilesLeftOnNodes alerts after half an hour. The router never deletes one. Neither it nor the node can tell the file it left from a later one at the same place: the tenant moved back onto that node, or made there again, or a file put there by hand. A create of the name on that node is answered 409 while the file is there. A repair's look forgets a file it finds gone, or one the directory now places, so an operator removes it by hand after looking at it, then runs a repair (deploy/RUNBOOK.md, Files left on nodes). Only a tenant left without its replica makes a repair answer 502. Tenants left without a copy are counted by fenec_router_tenants_unreplicated{region}, which FenecTenantsWithoutReplica in deploy/alerts/ reads.

Measured with make shard-bench, three nodes and the router in one process, 30 tenants: each is created with its replica in 33 ms, the replicas 10 a node, and a write is on its replica 0.179 ms after it was answered, 0.421 ms at p99. A failover of one node's 10 tenants took 57 ms, 5 onto each of the other two, and the repair after it gave 20 tenants a replica in 176 ms, all on the returning node's copies. Placed by the fewest altogether alone, every tenant of one node followed on the same other: a tie went by name each time, and its failover put them all there.

The directory is a fenecdb file, so it replicates the way any other does. A second router follows it and forwards from the same placements; the primary is the one that changes it.

Both forward /t/<tenant>/...; the standby's /_shard/ answers reads and refuses changes with 409, since its directory takes no write of its own. When the primary is gone, POST /_replication/promote on the standby forks its directory's history and it starts taking them — the same promotion a replica of a database gets, and the same rule after it: the old primary rejoins with --replica-of pointed at the new one, or its file is refused.

A standby is behind by whatever the last directory change has not reached it yet, which is one write and a sync: placements change when someone places, moves or deletes a tenant, not when clients read and write. Nothing of the tenants' own data goes through the router's directory, so a standby that is a second behind still forwards every request correctly. One whose directory has not arrived at all — just started, its primary not yet reached — answers 503 with Retry-After rather than 404: it knows of no tenant, which is not knowing there is none.

Each change the primary makes reaches the standby's maps through the change ring, as a subscription's rows do: the placements it put or deleted since the last one, 4 µs at 100 000 tenants, where reading every map again took 38 ms under the router's write lock. An image adopted, or more changes than the ring holds, and the maps are read again whole.