Indexes
One index makes one kind of question fast: hash, ordered, unique, over several fields, and rows that expire.
An index makes one kind of question fast and gives the rows that
reading every row would. Declare it on a field when the collection is
made, or add it later with create index.
Run in the shell, answers and all; the timings are from one run.
fenec> create collection events (user int, kind text, at timestamp);
collection `events` created
fenec> put events [{user: 1, kind: "view", at: "2026-05-03T09:00:10Z"}, {user: 1, kind: "buy", at: "2026-05-03T09:05:30Z"}, {user: 2, kind: "view", at: "2026-05-03T09:01:00Z"}, {user: 3, kind: "view", at: "2026-05-03T10:20:00Z"}, {user: 3, kind: "buy", at: "2026-05-03T10:40:00Z"}];
5 records (196.17µs)
fenec> explain get events where kind = "buy" order at desc limit 2
plan
-----------------------------------------------------------
filter: a full scan, 5 of 5 rows tested, 2 matched
order: at desc, every key read, 2 matches, all put in order
rows: 2
(3 rows, 188.29µs)
fenec> create index on events (kind) @hash;
index built on `events.kind`
fenec> create index on events (at) @sorted;
index built on `events.at`
fenec> explain get events where kind = "buy" order at desc limit 2
plan
-----------------------------------------------------------------------
order: the ordered index on at not walked, an equality names fewer rows
filter: the hash index on kind, 2 rows, which is the answer
order: at desc, every key read, 2 matches, all put in order
rows: 2
(4 rows, 327.04µs)
fenec> explain get events where at >= "2026-05-03T10:00:00Z" order at desc limit 2
plan
--------------------------------------------------
order: walked the ordered index on at desc, 2 rows
rows: 2
(2 rows, 253.75µs)
fenec> describe events
events
user int
kind text @hash
at timestamp @sortedWith no index the filter tested all five rows. With a
@hash on kind it names the two rows holding
"buy", and with a @sorted on at a range
and an order walk the index and stop at the page. Text and vector indexes
are on Text search and
Vector search.
Indexes
An index makes one kind of question fast. Declare it on a field when you
create the collection, or add it later with create index.
create collection articles (
title text,
body text @text(k1=0.9, b=0.4, prefix=0),
year int @hash,
slug text @unique,
published timestamp @sorted,
embed vector<768> @hnsw(cosine, m=16, ef_construction=200)
)- One index per field:
@hashfor equality,@sortedfor ranges and order,@textfor ranked text search,@hnswfor nearest neighbours,@geofor points near a place. A path into ajsonfield takes@hash,@uniqueor@sortedof its own (Objects and paths), and an index over several fields is the collection's, beside them (below). @uniqueis a@hashthat refuses a second document holding a value one holds already.nullis no value: any number of documents may hold it.- Only filters joined by
anduse an index. Anything underor, and!=,~orhas, reads every row. - For Turkish, Chinese or Japanese text, see
prefixandcharsbelow.
Index options and what they cost, measured
create index on articles (embed) @hnsw(cosine)
create index on articles (body) @text
create index on articles (body) @text(prefix=6) -- for an inflected language
create index on products (title) @text(chars) -- short Chinese or Japanese text
create index on articles (title) @sorted
create index on users (email) @unique -- refused if a value is held twiceA field takes one index. @hash answers equality,
@sorted ranges and order, @text ranked text
retrieval, @hnsw nearest neighbours. @sorted is
for int, float, timestamp and
text — the types with an order.
@unique is a @hash index that refuses a
second document holding a value one holds already: a put, an
insert or a set that would write one is refused,
and nothing of the statement is written — 409 over HTTP, as a
taken id is. It answers equality and a lookup as
@hash does. A block — a /batch, a run of several —
is checked against its own earlier writes, so it may free a value and take
it again, and a document keeping its own value is no second one.
null is no value, as in SQL: any number of documents hold it.
Values are one where = finds them one, so -0.0
and 0.0 collide. create index ... @unique over
documents that hold a value twice is refused and names the value and two
of its documents. A replica takes what its primary took and does not ask
again; describe shows the index as @unique. The
refusal names the document holding the value and the value, but to a
scoped token it names the field
alone (`users.email` is unique, and the value is taken): the
row is another user's, which the token may not read.
The check asks the bucket the value would go in, so the index has to
hold every document: the first write after an open builds it, where a
plain @hash waits for the first read that wants it. An open
costs what it did — 19 ms for an image of a million rows either way — and
that first write takes 15 ms over 100 000 rows and 259 ms over a million;
after it a put takes 940 ns against 850 for the same field
under @hash, the price of the lookup. A collection with no
unique field writes as fast as before (848 and 852 ns).
An @hnsw graph holds a vector once. Documents written with
the same vector — the embedding of an empty chunk, a line of boilerplate,
a zero vector — share its node, and near hands back each of
them at the one distance. A node each, a vector written hundreds of times
filled its neighbours' lists with copies of itself: over 20 000 rows of 128
dimensions, a third of them copies of 20 vectors, 1 812 copies were found by
no search and other queries' recall fell from 0.82 to 0.67; one node each,
none is missed and it is 0.87.
prefix=N indexes each word's prefixes alongside it
(3..N by default; prefix_min moves the lower bound). It is a
stemmer that needs no dictionary and no language setting, and it is off
unless asked for because it is a trade the corpus decides. Word-boundary
matching suits English; it does not suit a language that inflects by gluing
suffixes on, where kitap, kitabı and
kitapların are three terms sharing a stem the index never
sees.
Measured on Turkish WebFAQ (144 846 documents, 10 000 queries): nDCG@10 0.4841 → 0.5547, +14.6%, closing about two fifths of the gap to a 278M-parameter multilingual transformer (0.650). The cost is the 3.4× postings it indexes — 58 MB → 182 MB, and 82 → 485 µs per query. On English SciFact the same option is worth +0.0137.
A script written without spaces is indexed in runs of
characters. Chinese, Japanese and Thai run their words together, and
Korean hangs its particles on them, so a word is not a term there: a run of
Han, kana or Hangul is indexed as its overlapping pairs of characters —
東京都に is 東京, 京都,
都に — and one of Thai, Lao, Khmer or Myanmar, whose
characters are letters, as its triples. A query is split the same way, so
match body "東京" finds the run it is written in. There is no
dictionary and no language setting; a script that spaces its words keeps
its words.
chars indexes each character of Han, kana and Hangul as
well, so a query of one character finds every run holding it, which pairs
alone do only for a run of one. It pays on short text and costs on long:
product titles gain a fifth, news and questions lose a little, and the
index holds 1.6 to 1.9× the postings for a query 3 to 8×
slower. Off unless asked for.
| nDCG@10, BM25 | documents | words whole | runs of characters | with chars |
|---|---|---|---|---|
| C-MTEB EcomRetrieval, Chinese product titles | 100 902 | 0.006 | 0.439 | 0.528 |
| C-MTEB CovidRetrieval, Chinese news | 100 001 | 0.162 | 0.868 | 0.872 |
| WebFAQ, Chinese | 132 491 | 0.146 | 0.686 | 0.666 |
| JaGovFaqs-22k, Japanese | 22 135 | 0.119 | 0.582 | 0.576 |
| WebFAQ, Korean | 102 000 | 0.537 | 0.682 | 0.651 |
| WebFAQ, Thai (triples; pairs 0.495) | 47 429 | 0.448 | 0.547 | — |
The same pipeline as BEIR (make beir over each set in its
layout); English SciFact scores what it did, to the fourth decimal. A query
over those sets takes 0.09 to 0.66 ms at the median.
A filter reaches an index only inside an and chain.
Equality — = and in [..] — reaches a
@hash field; in is a set of equalities written
short, so it is answered as the union of one bucket per element. Either
reaches id, which the store's own id index answers without a
@hash being declared or possible. Comparisons —
<, <=, >, >=
and = — reach a @sorted field, which narrows them
to a range of keys.
Everything else — !=, ~, has,
comparisons on a field without @sorted, and anything under an
or — is a full scan: each matching row is decoded and
evaluated. ~ in particular is substring matching without
ranking and reaches no index; ranked retrieval is match over a
@text field.
A full scan reads rows in id order, and starts at the lowest id an
id > x or id >= x in the and
chain lets through, so a page read after the last id of the one before —
where id > $1 limit 1000 — reads only its own rows: a
million rows read that way a page at a time took 8.6 s and take 110 ms.
The browser module leaves this out.
Ordered indexes
@sorted makes ranges, counts over a range and
order … limit fast. It always gives the same rows as reading
every row would.
get articles order published desc limit 20
get articles where published >= "2026-09-01" order published desc limit 20await db.from('articles').order('published', 'desc').limit(20).rows();
await db.from('articles')
.where('published', '>=', '2026-09-01')
.order('published', 'desc')
.limit(20)
.rows();db.collection("articles").order("published", "desc").limit(20).rows()
(db.collection("articles")
.where("published", ">=", "2026-09-01")
.order("published", "desc")
.limit(20)
.rows())db.From("articles").Order("published", "desc").Limit(20).Rows(ctx)
db.From("articles").
Where("published", ">=", "2026-09-01").
Order("published", "desc").
Limit(20).
Rows(ctx)await db.From("articles")
.Order("published", "desc")
.Limit(20)
.RowsAsync();
await db.From("articles")
.Where("published", ">=", "2026-09-01")
.Order("published", "desc")
.Limit(20)
.RowsAsync();try await db.from("articles")
.order("published", "desc")
.limit(20)
.rows()
try await db.from("articles")
.where("published", ">=", "2026-09-01")
.order("published", "desc")
.limit(20)
.rows()db.from("articles").order("published", "desc").limit(20).rows()
db.from("articles")
.where("published", ">=", "2026-09-01")
.order("published", "desc")
.limit(20)
.rows()await db.from('articles').order('published', 'desc').limit(20).rows();
await db.from('articles')
.where('published', '>=', '2026-09-01')
.order('published', 'desc')
.limit(20)
.rows();get articles order published desc limit $1
-- [20]
get articles where published >= $1 order published desc limit $2
-- ["2026-09-01", 20]- It costs memory and write time: about 20 bytes a row for a
number, plus the text for
text. - It is rebuilt after an open by the first query that needs it; it is never written to the file.
When an ordered walk wins over the scan
get articles where title >= "k" and title < "l" count
get articles where published >= "2026-01-01" near embed $1 limit 10await db.from('articles')
.where('title', '>=', 'k')
.where('title', '<', 'l')
.count();
await db.from('articles')
.where('published', '>=', '2026-01-01')
.near('embed', [0.9, 0.1, 0])
.limit(10)
.rows();(db.collection("articles")
.where("title", ">=", "k")
.where("title", "<", "l")
.count())
(db.collection("articles")
.where("published", ">=", "2026-01-01")
.near("embed", [0.9, 0.1, 0])
.limit(10)
.rows())db.From("articles").
Where("title", ">=", "k").
Where("title", "<", "l").
Count(ctx)
db.From("articles").
Where("published", ">=", "2026-01-01").
Near("embed", []float64{0.9, 0.1, 0}).
Limit(10).
Rows(ctx)await db.From("articles")
.Where("title", ">=", "k")
.Where("title", "<", "l")
.CountAsync();
await db.From("articles")
.Where("published", ">=", "2026-01-01")
.Near("embed", new[] { 0.9, 0.1, 0 })
.Limit(10)
.RowsAsync();try await db.from("articles")
.where("title", ">=", "k")
.where("title", "<", "l")
.count()
try await db.from("articles")
.where("published", ">=", "2026-01-01")
.near("embed", [0.9, 0.1, 0])
.limit(10)
.rows()db.from("articles")
.where("title", ">=", "k")
.where("title", "<", "l")
.count()
db.from("articles")
.where("published", ">=", "2026-01-01")
.near("embed", listOf(0.9, 0.1, 0.0))
.limit(10)
.rows()await db.from('articles')
.where('title', '>=', 'k')
.where('title', '<', 'l')
.count();
await db.from('articles')
.where('published', '>=', '2026-01-01')
.near('embed', [0.9, 0.1, 0])
.limit(10)
.rows();get articles where title >= $1 and title < $2 count
-- ["k", "l"]
get articles where published >= $1 near embed $2 limit $3
-- ["2026-01-01", [0.9, 0.1, 0], 10]order over a @sorted field with a
limit walks the index in order and stops at the page, testing
the rest of the filter on each row it passes; without one, every match's key
is read and only offset + limit of them are put in order. A
filter that matches too little to fill the page within an eighth of the
collection sends the walk back to the scan: over a million rows one matching
nothing cost 152 ms against the scan's 121, while one matching 1% fills the
page in 0.29 ms against 125. A range on a @sorted field is read
from the index as a set of ids — for a filter, a count, or the
allowed set of a filtered near.
| 1 000 000 rows unless noted | scan | @sorted |
|---|---|---|
order created desc limit 20 | 26.8 ms | 0.001 ms |
the newest 1%, order created desc limit 20 | 63.1 ms | 0.001 ms |
where price >= 99900 limit 20, 0.1% of the rows | 1.44 ms | 0.008 ms |
the same range, count | 65.8 ms | 0.008 ms |
where pct < 1 near embed $1 limit 10, 1% of 200 000 × 128 | 17.9 ms | 0.16 ms |
A comparison with now() — run_at <= now(),
at >= now() - 3600000 — is a range of the index as one with a
literal is: the time is worked out once, before the query is planned, and
is the statement's one time. Left a call, a queue's claim walked past its
ready jobs into the leased and delayed ones after them, testing each.
The planner prefers a smaller set when it has one: an equality over a
@hash field, or a narrow range on another @sorted
field, names fewer rows than the walk would pass, so those are sorted instead.
A range wider than half the collection is left to the scan, which is cheaper
than gathering and sorting that many ids.
A comparison reaches the index only when its literal says exactly
what the scan would compare. price < 12.5 on an
int field, or an integer past 253 against a
timestamp, is evaluated row by row — the index narrows with the
comparisons it can express and the rest are tested — because an answer that
differs from the scan's is a wrong answer, not a fast one. Ties come out in
ascending id either way, as the scan leaves them.
The cost is paid on write and in memory: about 20 bytes a row for a
number, plus the text for text, and an insert into a sorted
run. Over a million rows two ordered indexes took a bulk load from 0.52 s
to 0.88 s, and the index is built from the documents by the first query
that reads it after an open, like the hash and text indexes — it is never
written to the file.
Indexes over several fields
@sorted(owner, created desc) keeps the rows in the order of
several fields, the first first. The page an app reads most — one user's
latest twenty, a tenant's inbox, a feed — is a run of it, read to the page
and no further. @unique(org, slug) refuses a second row holding
the same values in every part.
create collection posts (
owner int,
created timestamp,
title text,
@sorted(owner, created desc)
)
create index on pages (org, slug) @unique -- or @unique(org, slug) in the collectionget posts where owner = $1 order created desc limit 20 -- a run of the index, 20 rows read
get posts where owner in [$1, $2, $3] order created desc limit 20 -- three runs, merged
get posts where owner = $1 and created >= $2 count -- counted off the indexawait db.from('posts')
.where('owner', 7)
.order('created', 'desc')
.limit(20)
.rows();
await db.from('posts')
.where('owner', 'in', [7, 8, 9])
.order('created', 'desc')
.limit(20)
.rows();
await db.from('posts')
.where('owner', 7)
.where('created', '>=', '2026-10-01')
.count();(db.collection("posts")
.where("owner", 7)
.order("created", "desc")
.limit(20)
.rows())
(db.collection("posts")
.where("owner", "in", [7, 8, 9])
.order("created", "desc")
.limit(20)
.rows())
(db.collection("posts")
.where("owner", 7)
.where("created", ">=", "2026-10-01")
.count())db.From("posts").
Where("owner", "=", 7).
Order("created", "desc").
Limit(20).
Rows(ctx)
db.From("posts").
Where("owner", "in", []any{7, 8, 9}).
Order("created", "desc").
Limit(20).
Rows(ctx)
db.From("posts").
Where("owner", "=", 7).
Where("created", ">=", "2026-10-01").
Count(ctx)await db.From("posts")
.Where("owner", 7)
.Order("created", "desc")
.Limit(20)
.RowsAsync();
await db.From("posts")
.Where("owner", "in", new object?[] { 7, 8, 9 })
.Order("created", "desc")
.Limit(20)
.RowsAsync();
await db.From("posts")
.Where("owner", 7)
.Where("created", ">=", "2026-10-01")
.CountAsync();try await db.from("posts")
.where("owner", 7)
.order("created", "desc")
.limit(20)
.rows()
try await db.from("posts")
.where("owner", "in", [7, 8, 9])
.order("created", "desc")
.limit(20)
.rows()
try await db.from("posts")
.where("owner", 7)
.where("created", ">=", "2026-10-01")
.count()db.from("posts")
.where("owner", 7)
.order("created", "desc")
.limit(20)
.rows()
db.from("posts")
.where("owner", "in", listOf(7, 8, 9))
.order("created", "desc")
.limit(20)
.rows()
db.from("posts")
.where("owner", 7)
.where("created", ">=", "2026-10-01")
.count()await db.from('posts')
.where('owner', 7)
.order('created', 'desc')
.limit(20)
.rows();
await db.from('posts')
.where('owner', 'in', [7, 8, 9])
.order('created', 'desc')
.limit(20)
.rows();
await db.from('posts')
.where('owner', 7)
.where('created', '>=', '2026-10-01')
.count();get posts where owner = $1 order created desc limit $2
-- [7, 20]
get posts where owner in [$1, $2, $3] order created desc limit $4
-- [7, 8, 9, 20]
get posts where owner = $1 and created >= $2 count
-- [7, "2026-10-01"]- Equalities on its first fields, then a range or an order on the
next.
where owner = $1 order created descreads its run;where created > $1alone does not use(owner, created), sinceownercomes first. - An order matches it in its fields' directions, or every one the
other way round:
(owner, created desc)answersorder created descandorder createdfor one owner. - It gives the scan's answer, row for row, ties in ascending
ideither way, as@sorteddoes. - A
lookupkeyed by its first fields reads each parent's children as a run of it, walked in the level's order where that is the index's next parts:lookup lines on (org, order) = (org, id) order line asc limit 5. - Up to 8 fields of
bool,int,float,timestamp,text(in its collation) orjson, a path into one among them (meta.lang). A field keeps its own index beside it.
How a key over several fields is laid out
create index on posts (owner, created desc) @sorted -- built beside the database on a server
drop index on posts (owner, created) -- by its fields
explain get posts where owner = 7 order created desc limit 20
-- order: walked the index on (owner, created desc), owner fixed, 20 of 100 rows read, 20 keptA key is its fields' values laid end to end, each written so that the
bytes order as the values do — an int's sign turned over, a float's bits
made to order, a text escaped and ended, a descending field's bytes
inverted — so an equality on its first fields is a run of the keys and a
range on the next one is a run inside it. A text in a collation is kept as
written and compared in it. null stands below every value of a
field, as order puts it; a row holding a float's
NaN, which compares equal to everything, or a json value no
order has a place for (a boolean, a list, an object, an int past
253) is held apart: while one is, the index is walked for no
order, and a filter's run takes those rows with it, each
tested.
The planner takes the and chain's terms in the index's order
— an equality, or an in whose every element it can say exactly,
on each field as far as one is fixed, then the comparisons on the next — and
tests every other term row by row; a term it cannot say exactly (a
null, 12.5 against an int) is left to
that test, so the answer never changes. An in makes a run for
each element, 256 at most together; under an order on the
fields after it the runs are walked together, merged by those fields. Of the
plans, the narrower set wins by exact counts: the run is two searches, a
@hash bucket its length, so a filter the index says whole reads
the page and nothing more, and one it does not is walked only while its run
is no longer than any set another index names — and gives up past an eighth
of the collection, as a @sorted walk does. A
count the index says whole is counted off it without a row
read. explain names the index, the fields fixed, the runs and
the range.
@unique over several fields is asked before every
put, insert and set: a second row
holding the same values in every part is refused,
409 over HTTP, naming the fields and the other row — to a
scoped token the fields alone. A null in any part is no value.
put ... if absent passes over the row holding them, and
if absent else set sets it. create index ...
@unique over rows already holding a key twice is refused naming two of
them.
It is derived data, built from the documents by the first read after an
open, kept on every write, put back with a block that does not land, and
never written to the file. A field renamed keeps it under the new name; a
field dropped takes it off. A sync replica builds what the schema declares,
a unique one as an ordered one, as its @unique fields are
@hash.
| 1 000 000 posts over 10 000 owners | p50 | p99 | built in | memory |
|---|---|---|---|---|
where owner = $1 order created desc limit 20, @sorted(owner, created desc) | 0.006 ms | 0.011 ms | 147 ms | 40 MB |
the same over owner @hash, its bucket sorted | 0.012 ms | 0.017 ms | 75 ms | 11 MB |
the same over created @sorted, walked and given up | 24.4 ms | 40.5 ms | 34 ms | 20 MB |
| the same by the scan | 21.3 ms | 37.4 ms | — | — |
a feed, owner in ten owners, @sorted(owner, created desc) | 0.022 ms | 0.029 ms | ||
the feed over owner @hash | 0.100 ms | 0.136 ms |
Writes pay for it: a lone put into those million rows took
715 ns with no index, 851 beside created @sorted and
owner @hash, and 1 561 beside @sorted(owner, created
desc), whose first field puts each write anywhere in it. A unique one
asks its run before each write: 1 745 ns a put against 1 285 for
@unique on the two joined into one field, and a read by both
fields 2.5 µs against 1.3 by the joined field — a hash index
over several fields would answer that read sooner, by a microsecond, and was
not worth a third index's code beside this one (make
composite-bench).
Rows that expire
@ttl on a timestamp field makes a row expire that
long after its value — for sessions, caches and anything else kept for a
while.
create collection sessions (user text @hash, seen timestamp @ttl(30m))- Reads are exact at once. A row past its time is never returned, even before it is deleted: every read of the collection leaves it out.
- The server deletes it. A primary sweeps once a minute, as
ordinary deletes that replicas,
/_changes, archives and subscriptions see as any. expired()reads what the others leave out, for a reaper that gives back what a row held before the sweep deletes it (below).- Durations are whole
ms,s,m,hord:@ttl(45s),@ttl(12h),@ttl(7d).
The reaper, and what @ttl costs, measured
put sessions {user: "ada", seen: now()}
set sessions {seen: now()} where user = "ada" -- seen again: lives on
get sessions where user = "ada" -- nothing once 30 minutes passawait db.from('sessions').insert({ user: 'ada', seen: expr('now()') });
await db.from('sessions')
.where('user', 'ada')
.update({ seen: expr('now()') });
await db.from('sessions').where('user', 'ada').rows();db.collection("sessions").insert({"user": "ada", "seen": expr("now()")})
(db.collection("sessions")
.where("user", "ada")
.update({"seen": expr("now()")}))
db.collection("sessions").where("user", "ada").rows()db.From("sessions").
Insert(ctx, fenecdb.D("user", "ada", "seen", fenecdb.Expr("now()")))
db.From("sessions").
Where("user", "=", "ada").
Update(ctx, fenecdb.D("seen", fenecdb.Expr("now()")))
db.From("sessions").Where("user", "=", "ada").Rows(ctx)await db.From("sessions")
.InsertAsync(new { user = "ada", seen = Computed.Expr("now()") });
await db.From("sessions")
.Where("user", "ada")
.UpdateAsync(new { seen = Computed.Expr("now()") });
await db.From("sessions").Where("user", "ada").RowsAsync();try await db.from("sessions")
.insert(["user": "ada", "seen": .expr("now()")] as Value)
try await db.from("sessions")
.where("user", "ada")
.update(["seen": .expr("now()")] as Value)
try await db.from("sessions").where("user", "ada").rows()db.from("sessions")
.insert(mapOf("user" to "ada", "seen" to Computed.expr("now()")))
db.from("sessions")
.where("user", "ada")
.update(mapOf("seen" to Computed.expr("now()")))
db.from("sessions").where("user", "ada").rows()await db.from('sessions')
.insert({'user': 'ada', 'seen': Computed.expr('now()')});
await db.from('sessions')
.where('user', 'ada')
.update({'seen': Computed.expr('now()')});
await db.from('sessions').where('user', 'ada').rows();put sessions {user: $1, seen: now()}
-- ["ada"]
set sessions {seen: now()} where user = $1
-- ["ada"]
get sessions where user = $1
-- ["ada"]create index on events (at) @ttl(7d) -- an existing timestamp field
alter collection sessions alter field seen @ttl(1h) -- another expiry
alter collection sessions alter field seen @sorted -- none: the rows liveA read of the collection — get, count, an
aggregate, near, match, a lookup level,
an inner get, a set or del by filter,
and a subscription's shape — has not (seen <= now - ttl)
added to its filter, the way a token's rule is added: a row is gone at the
moment its time plus the ttl is reached. A row whose field is
null has no time to expire from, and lives. In the browser the
module has no clock: now is the page's, Date.now()
unless db.now is set, as every time is handed to it; the page
deletes nothing, and a row past its time stays in its file, out of every
read. A live query in the page is woken by writes, not by time passing.
@ttl is an ordered index whose rows expire, so the field
also does everything @sorted does, and a collection's rows
expire by one such field. The sweeper deletes a range
of it at a time rather than scan for the rows past their time: found under
the read lock, deleted under the write lock 1 000 at a time, each batch a
block of ordinary deletes, its fsync run once the lock is let go. A replica
never sweeps — its writes are its primary's, the sweep's among them — and
neither does a tenant that follows another node. The server's
Idempotency-Keys are kept this way.
The ttl is a kind of index in the file (kind 9, the milliseconds behind
it), so a binary from before it refuses the file rather than hand out the
rows past their time. A write takes the place of a row past its time:
an insert naming its id writes over it, and a
@unique value it holds is let go of with it, the row deleted
in the same statement — so a lock taken by insert and left to
expire can be taken again at once (Instead of
Redis). BM25's statistics count the row until the sweep.
Reading what expired: the reaper
A row past its time is out of every read, so a hold that lapsed cannot be
found to give its money back, nor a reservation its stock. expired()
in a filter reads those rows — and only those, where the filter says so —
wherever a filter goes: get, count, a
lookup level, an inner get, set and
del. It is until <= now - ttl, a range of the
field's ordered index; not expired() is what a read finds without
it.
create collection holds (account text @hash, amount int, until timestamp @ttl(1ms))
get holds select id, account, amount where expired() limit 100 -- what lapsed
-- each given back once, in one block (a /batch): the delete is the guard
del holds where expired() and id = $1 require 1
set accounts {held: held - $2} where ext = $3 and held >= $2 require 1A capture needs no guard of its own against the lapse:
get holds where id = $1 limit 1 require 1 finds no hold past
its time. Two reapers, or a reaper and a capture, racing for one hold: the
del ... require 1 lets one through, and the other's block is put
back whole. The sweeper leaves a row a minute past its time before it deletes
it, so a reaper that comes by once a minute sees every one; a row it misses
for longer is deleted unseen, and what it held is the reconciliation's to
find. A scoped token reads them only by the policy's expired
grant (Scoped access). Without
@ttl on the collection, expired() is an error.
What it costs, measured (make ttl-bench, a million
rows, half past their time). A read tests each row's time as part of its
filter, as the same test written by hand does: count becomes a
scan, 68.8 ms against 1.1 with no expiry, and a filter that scanned
already reads one field more, where s = "w7" count 68.5 ms
against 51.9 — 70.1 with the test written out. A collection with no
@ttl reads as it did. The sweep of 100 000 rows past their
time out of a file of 200 000 took 0.56 s in batches of 1 000, each
holding the write lock 1.06 ms at the median and 2.3 at most.