Working with tenants

1. The directory marks the tenant moving; it is still routed to the source.
2. The source freezes it: writes in flight finish, new ones get 503 with Retry-After: 1. Reads go on being served.
3. The image is copied from the source and installed on the target.
4. The directory points the tenant at the target, in one statement.
5. The source copy is deleted — once every standby router keeping up has been sent the new placement, a second at most, so one that forwards from it sends the tenant's requests to the target rather than to a node that no longer holds it. Its open subscriptions end with an error event, and the client reconnects through the router to the target. A client that had seen everything resumes from its cursor without a reseed: the change sequence travels in the image and the source was frozen, so nothing happened in between. One that was behind is reseeded, as after any restart — the target has the sequence but not the history before it.

The router refuses nothing itself during a move, so there is no window in which the tenant is nowhere — only one in which writes are asked to retry. If a step before the flip fails, the source is thawed and the directory put back. A router that stops mid-move comes back with the tenant still served from the source and marked moving; moving it again replaces whatever copy the interrupted attempt left on the target, which is stale by definition, since the directory never pointed at it.

Tenants are the isolation boundary, and this crosses it, so it is a /_shard/ endpoint behind the router's --token. A JSON Web Token is refused there as any token other than the router's is: the router holds no keys, and a token bound to one tenant must not reach the others through it. The router asks each tenant through its node's POST /_admin/tenants/<t>/query with the node's admin token, which it holds already: a token that can take a tenant's whole image (GET /_admin/tenants/<t>/file) learns nothing more by reading it, where the nodes' data token, which nothing else here needs, would reach every tenant and write. The node refuses a statement that writes before it runs, and runs the rest in read-only mode besides.

The router still parses no FenecQL. The statement and its parameters go to every node as they came, and what is done with the rows is said in the request, never read out of the query:

Three merges, each exact from the tenants' answers alone:

• One after another, the default: every row, its tenant named.
• An order: a k-way merge by the named columns, each asc or desc, of rows each tenant gives in that order. With each tenant's own order ... limit 20, the first twenty of the merge are the first twenty of all of them. The router checks each tenant's rows come in the order, and a tenant whose do not — its statement orders otherwise — fails, never merged wrong. Values compare as the engine compares them, text by its bytes; a timestamp travels as ISO text, which leaves its milliseconds out where they are 0, so a text that is one compares as its time. Ties go in tenant name order. A key whose text is not ordered by its bytes says how it is (below).
• A fold: sum, min and max of the named columns, by the group columns, groups in order. These are the aggregates that combine across tenants: a count is a sum of counts. An average of averages is not an average, so there is none: sum its sum and its count. Every column of a row must be grouped by or folded, or the tenant fails: a column the merge would drop is an answer cut. A sum of whole numbers past 64 bits is refused, as the engine refuses one. A min, max or group column may be an object naming its text order as an order's key is: a tenant's min over a collated field is the collation's least, so the least of those is too, and a nocase group is one for every text folding to it, under the first tenant's text.

A column whose text the tenants compare otherwise than by its bytes is an object in place of its name: {"column": "name", "collate": "tr", "nocase": true, "desc": true} where "name desc" would be. The router reads no schema, so what the field declares — collate tr or und, nocase — or what the statement's order name collate tr names is said again here, and the column compares by the engine's own rule: the collation over the case folds, the texts that fold to one tied, and a text read as text even where it looks like a timestamp. "nocase": true folds as a field in the column's collation does, Turkish's under tr (I the capital of ı) and Unicode's otherwise; where the statement's collate names another than the field's own, "nocase" names the field's, as "nocase": "und" for a plain nocase field ordered collate tr. A collation other than tr or und is refused (400), named, and desc is an order's alone. Not told, the router compares bytes: the tenants whose rows come in another order fail, and a min or max is the bytes' least or greatest of the tenants' own.

A tenant that fails — its node unreachable, its call past the timeout, an answer that does not fit the merge, or one the node refused — is named with its node, the status and why. Without "partial": true the query fails with every failure named; the first stops the asking, and the answer says how many tenants were not asked. Its status is the tenants' own where they all failed alike — a statement every node refused is 400 or 403 — and 502 otherwise. A tenant moved while the query runs is asked again on the node the directory names now; one deleted meanwhile is a failure. The tenants come from the directory as the query begins: one created after that is not in it.

The rows gathered from the tenants together are bounded by --fanout-max-rows (100 000) and the bytes of their answers by --fanout-max-bytes (64 MiB), counted as the answers arrive: a query past either is refused (413), its partial rows or not, and the asking stops. The calls go out over the router's connections to the nodes, --fanout-concurrency (16) at once, the tenants taken node by node in turn, and each tenant's answer is read for the merge on the thread that fetched it, beside the calls still out.

Each tenant's answer is one statement's, as /query's is. Tenants are separate databases read at different moments, so two answers are of no one moment, and a write landing in one while the query runs may or may not be in its answer. A read that opens a tenant opens it on its node, which closes it again once idle.

A query across tenants is a span of the router's trace, a span a node under it, and each tenant's read on its node under that node's span, all with the request's id; the router counts them in fenec_router_fanouts_total and two more (Monitoring), no label naming a tenant. Each is a line of the router's audit log; the nodes' reads for it are not, as a read is not.

Over 100 tenants of 200 rows on three nodes, on one machine (make shard-bench ARGS=fanout):

p50 over 300 rounds in three runs, the names' in six beside other work on the machine; the client asking 16 at once merges nothing. A merge in a collation over case folds costs what one by a timestamp does. A request forwarded to one tenant costs what it did.