Serverless and Cloudflare

The browser module runs wherever V8 runs a WebAssembly module, a Cloudflare Worker among them. A Durable Object gives it what a file gives a server: one writer, and storage that outlives the process. So one object holds one database, as a tenant is one file on a fenec-server node.

The second needs nothing new: Fenec.connect(url), the Python Client or fetch, and a function's connections are HTTP requests, so there is no pool to exhaust. Servers in several regions take one URL through a Worker in front of them, which sends each tenant's requests to its region: the global entry. The rest of this page is the first.

A Worker imports a .wasm file compiled, and cannot compile bytes at run time; Fenec.open takes the module so. The open and the restore run inside blockConcurrencyWhile, once: opened by whichever request found the object cold, two requests could each restore an engine of their own and append to the storage as if alone. persist runs in a finally, since a statement that failed may follow ones in the same text that wrote. A persist that fails leaves the database in memory holding writes its storage never took, so the object lets it go and the next request restores what storage holds, rather than answer from writes nobody kept. Its refusal, and a failed maintain's once release has let the database go, come only once nothing of that database is on its way to storage, and nothing more of it will be: the database restored next reads all the first stored and is the only one writing. An error setting the alarm fails neither a write, which is stored, nor maintain, which answers it as error for the alarm to be tried again. integrations/cloudflare/example is this Worker with a tenant from the path, POST /t/<tenant>/query, and its test runs it under wrangler dev — workerd, as Cloudflare runs it — then stops and starts it over the same storage and reads the rows, a near's answer and each tenant's own database back, and sends a cold object 32 writes at once.

persist keeps what a file would hold: an image the first time, then only the writes since the last call — the engine's journal, as persist in the browser takes it for IndexedDB — until they outgrow half the image and 64 KiB, and a new image replaces them.

A SQLite-backed object — the only kind a new account makes — has ctx.storage.sql, and the database is a table a generation, fenec:1, fenec:2 and on (the key, then the generation): the image's rows, cut under the 2 MB a row takes, then a row each time writes are stored. Nothing names where the database stands beside the rows: the table there is says it, so a write is the one row. A new image is written, and the table before it dropped, in one transactionSync: workerd counts a table made as two rows written and a table dropped as none, where a DELETE of the rows before counts each, and storage that stops part way holds the generation before, whole. The rows are an INTEGER PRIMARY KEY table with no other index, since an index is a row written more.

A write goes out at once, unless the one before it is still on its way to being durable (storage.sync()): then it waits for that, and every request whose writes land meanwhile — a Durable Object takes the next request in while one waits — goes out with it, in one row. Each persist resolves only once its own writes are durable, and a read that may have seen writes on their way waits for them, so no answer leaves before what it shows is kept, and a lone write answers as soon as before: 2.0 to 2.3 ms p50 against 2.4 to 3.1. Waiting a turn of the event loop instead, for requests to land, cost a lone write 1.7 to 2.4 ms more. The figures above are make cloudflare-bench's, under wrangler dev with every storage call counted: a write before was a value, a record naming it and an alarm set again, and a checkpoint a value each 128 KiB of the image and a delete each of the values before.

A key-value backed object, made before SQLite with new_classes, has no sql; there the image and the writes are cut into values under its 128 KiB, every image goes under a generation of its own, and the record naming the whole one is written last. A test stops either kind of storage after every number of operations, an image's, an append's and a compact's, and restores the last whole database each time; and a write that fails fails every write that went out with it.

A database kept by an earlier version on a SQLite-backed object is in values, which restore no longer reads there. To move one, restore it through the key-value methods alone — an object holding get, put, delete and list of ctx.storage and no sql — and persist through ctx.storage.

An object evicted, or a deploy, starts the next request with no database in memory: the module instantiated, the image and the writes after it read out of storage and loaded. The graphs the image holds are restored as they were; the writes after it are applied as the write path would, their vectors linked into the graph one after another — the browser module has one thread. The first near after a restart, against the ones after it, under wrangler dev on an M1 (make cloudflare-bench, median of five restarts; rows of a text, an int and a 128-dim vector, 1 000 a request, each kept before it was answered, no alarm) are in the table above; a second run was within 2 ms of each.

Linking the writes after the image is no longer what a start waits for: when this page was first written it took the first near over 10 000 rows to 792 ms and over 50 000 to 1 455, and the same measurement on today's module gives 39 and 175, so a checkpoint takes 50 000 rows from 175 ms to 157, 10 000 none quicker, and is mostly worth the rows it folds. The rows are read into the module a row at a time rather than joined first: 50 000 rows as written started in 175 ms against 186 to 196, after a checkpoint 157 against 168 to 171. A request's round trip through wrangler dev is in every figure. The rows went in at 5 200 to 15 000 a second, a request of 1 000 at a time with its vectors linked and kept before the answer.

The figures under “after a checkpoint” are a new image taken by hand (checkpoint); an object that keeps its database from its alarm takes one by the rule below.

maintain deletes the rows a minute (grace) past their time, as a server's sweep does, so a reader of expired() as frequent sees each before it goes; 10 000 a call (sweep) in statements of 1 000 ordinary deletes, stored as any write is, the alarm set again at once past the bound. The object answers nothing while it runs: 1 000 rows of 100 000 took 0.46 ms to delete in the module, so 10 000 hold it about 5 ms, far inside an alarm's 30 s of CPU and 15 minutes. Then it sets the alarm for the next row's time and the grace, no sooner than a minute (every) after this sweep, or sets none; each sweep is two rows at most, its deletes and the alarm, so rows expiring a few a second cost two rows a minute rather than two a row. No export of the module was added for it: a del naming expired() and walking the field's index in order says what the server's sweep does natively.

It writes a new image when the writes since the last reach half of it and 64 KiB (min), where persist folds them in a request, or when half of what storage holds is dead and a mebibyte — what a sweep or deletes left, which the writes' size alone does not show. Woken to sweep and nothing more, it stores the deletes as one row and writes no image. Once schedule is called, a request folds the writes only past the whole image: the alarm takes the image at half, and no request waits for it. A threshold of 64 KiB, as the alarm used, wrote a 27.6 MB image as 16 rows for writes that were one or two; at a quarter of the image, rewrites of 1 000 rows of 128 dimensions a request, a row each, brought an 11 MB image, 8 rows, every 6.

The module never gives memory back, and holds every version of a record written since it was opened. Once the dead versions reach half of what it holds — at least 4 MiB, at most 16 — maintain makes sure storage holds an image without them and answers reload: the example lets the database go and restores a new module, as it does on any failure — calls arriving wait while the ones holding it finish, a write on its way to storage among them, then it is released and closed and another restored. Let go of at once, a database an alarm dropped while a request's write waited behind another's stored it after the next request had restored a second database and written, under the same id: an answered write lost. In Node, 50 000 rows of 128 dimensions with three in five past their time held 42.7 MB of module through two sweeps of 10 000; the third left half of storage dead, its image took the module to 51.6 MB, and the module restored from it held 19.2. A compact in the module would have rebuilt every graph holding tombstones and built its image whole, and given nothing back.

A Worker's isolate has 128 MB, the WebAssembly memory in it, and wrangler dev does not hold a Worker to it: the figures above reach past it. What the module holds, measured in Node on the same module (rows of a text, an int and a vector under HNSW):

A WebAssembly memory grows and never gives back, so an object holds the most it ever held. The module keeps an index's vectors in chunks of about a mebibyte and lets a segment of documents grow no further than the record that seals it: grown by doubling, old and new side by side, 5 000 rows of 768 written 1 000 a request held 100 MB, 58 now, and 10 000 of them 150, 84 now. A restore keeps the image it reads and the documents in it, rather than copy each out: 10 000 rows of 768 restored held 97 MB, 66 now; and it copies the rows into that image as it reads them, one at a time (loadParts): read whole, joined, then copied in, a 27.6 MB image — 50 000 rows of 128 — held 60 MB of JavaScript beside the module's 42 at its peak, and holds 4, the process 105 MB more resident against 49 to 55 (make cloudflare-bench, in Node). A checkpoint takes the image a mebibyte at a time (snapshotChunks), each chunk made as it is taken out of the documents the stores hold, not a copy of them, and let go once it is in storage: made all at once, the image stood beside the rows, and 50 000 rows of 128 went from 73 to 109 MB at it, 83 now. So an object holds about 60 000 rows of 128 dimensions and 12 000 of 768, however many a request brings; past that, a fenec-server node and Fenec.connect, or a tenant's data spread over more objects.