Walkthrough · TreeDB

The life of a file in a treedb.

A photo cannot be a field of a node. So it becomes an address — and the address is the photo.

A treedb is held in memory in full, and timeranger2 rewrites the whole record on every update. Put a photo in a column and every change of that device — a state flag, a timestamp — rewrites the photo with it, in base64, in the log and in RAM. Measured on one real census: 12 134 images, 346 MB on disk, which carried inside the records would be some 460 MB of RAM, permanently, rewritten on every touch.

So the bytes live beside the treedb and the column holds an address. What makes the design worth reading is which address: the sha256 of the content itself. Everything below follows from that one decision.

The id is the content a real fixture — check it with sha256sum

The 16 by 16 PNG fixture used throughout this page

foto.png — 16×16, 114 bytes declared image/png, and the bytes agree uploaded by operator@example.com

fc1c1152d120bf253e9efbd39b3d75c854a3d7a7e409671264f828e2b7feefd0

<treedb dir>/.blobs/fc/1c/fc1c1152…feefd0.png

The first two hex digits are one directory, the next two another: two levels of fan-out, because a flat directory of ten thousand files is a directory nobody can look at. The name starts with a dot so no scan of the treedb directory ever mistakes it for a topic. The extension comes from the stored content type, and it is part of a path a web server serves.

Four consequences, and they are the whole argument: the same bytes stored twice are one asset; a URL for that asset can be cached for ever, because nothing can ever change under it; “editing” a photo is a new blob and a re-pointed link, so the history of the node says which photo it had and when; and two nodes sharing an image share the bytes without either of them knowing.

Declaring the column

One flag on one column, in the host topic's own schema. Nothing about __assets__ is declared anywhere — the topic is created by every treedb at open, and its hooks are derived from the file columns themselves.

/*  topic "devices", in the treedb schema  */
"foto": {
    "header": "Photo",
    "fillspace": 20,
    "type": "string",          // one file per column, always a string
    "flag": ["fkey", "file", "writable", "persistent"]
}

All four words earn their place. file is what puts the column on the write path. fkey is mandatory beside it — every linking behaviour keys on that word — and so is the type string: one file per column. A column that breaks either rule is refused at create-topic and is a critical at open. writable is the one people forget. It is not required — a column that only a bulk load ever fills is legitimately without it — but a form draws a column that lacks it read-only, so no person can ever put a file in it. Since 7.18.0 the treedb warns when it derives the hook.

From that column alone the treedb derives, in memory, a reciprocal hook on __assets__:

column   devices.foto          flag ["fkey","file",…]
hook     __assets__.as_devices_foto  -> {"devices": "foto"}
value    "__assets__^fc1c1152…feefd0^as_devices_foto"

The hook is named as_<topic>_<column> and it is never persisted. A hook's content already lives only in memory — linking saves the child's fkey and never the parent — so its declaration lives in the same place. It is re-derived at every open and on every live create-topic / delete-topic, which is what keeps a topic added while the yuno runs from having a column whose hook does not exist.

One write, step by step

Below is a single create-node on devices, carrying the fixture above. The left pane is the kw — the record as it arrived, rewritten in place as it goes. The right pane is what is on disk and in memory after each step. Pick a door, or make the write fail, and watch where it stops and what it leaves behind.

treedb_store_files(), then the link

Door

Outcome

The kw, rewritten in place

The store

Two things in that sequence are deliberate and easy to get backwards.

The size is checked on the base64, before the decode. A check after the decode has already spent exactly what it was defending. The ceiling is 128 MB per treedb by default; a column may lower it in its own properties, and never raise it.

The blob goes down before the index row. That order can leave bytes that no row names — and that is the good failure. The other order leaves a row pointing at nothing, which nothing repairs, ever. Bytes nobody names are what the collector is for.

The row that indexes it

__assets__ is a system topic: every treedb creates it at open, nobody declares it, and it is written by the write path, not by hand. Six columns, and exactly one of them is writable.

__assets__ the index in memory, the bytes in .blobs/

Column	Type	What it holds
`id`	string	The lowercase sha256 of the bytes. Required, and it is the file.
`content_type`	string	Read from the bytes, and fixed by the first arrival for ever — the extension is part of the served path, and the URL is cached for ever, so the name cannot move under it. One container can be two legal types (`video/mp4` and `audio/mp4` for the same bytes), which would land on two blobs for one row.
`size`	integer	Bytes, as counted after decoding.
`t`	integer	When the row was written. Set by the treedb.
`original_name`	string	The only writable column. A second arrival is an update of this node, so the history of the row is every name the file has arrived under — the list of paths that a `source_path` column could never be. A manifest with no name says nothing and leaves the stored one alone; a manifest with the name already stored says nothing new, and appends nothing.
`uploaded_by`	string	Who put the bytes there. Never rewritten: the bytes did not change.

Plus one hook per file column of the treedb, derived and never persisted. That is where the device hangs: the asset is the parent, the device is the child, and the fkey that records the link lives on the child — which is why linking saves the device and not the asset.

Reading it back, and taking it away

A reader never touches .blobs/. get-asset answers in one of two shapes and decides which itself: a signed URL when the service has a public_url and a sign_secret (nginx secure_link, 900 s, no client address in the signature — that would break the phone that changes network), and the bytes inline when it has not. One code path in the client either way.

Removal is not automatic and never has been. gc-assets runs treedb_gc_files() on demand, and it keeps:

every asset a live node links — in any treedb of the tranger, because .blobs/ belongs to the tranger and two treedbs can share it;
every asset that activating an existing snapshot would load — per key, the newest instance under that snapshot's tag. Delete the __snaps__ row and what only that snapshot held is freed;
and it sweeps the other way too: blobs that no row names, and the .tmp of a write that never reached its rename.

Run it with dry_run=1 first; it answers the list of ids it would take. The rule the collector is written to is: not being able to prove something is an orphan is a reason to keep it.

The rules that bite

file goes with fkey, on a string, and almost always with writable. The first two are enforced. The third is not, and it is the one that goes missing: three schemas lost it the same day, and each time the form drew the control read-only and the column simply could not be filled. The treedb warns about it now, once, when it derives the hook.
The write path links the column itself — autolink or not. An ordinary fkey is edited by linking; a file column is edited by handing over a file, and the link is part of the hand-over. Passing "" unlinks. A column the record does not carry is left exactly as it was.
A full reference must name its own hook. The value is what decides which hook the link goes through, so foto holding …^as_devices_qr filed the file under qr, left foto empty and answered “Node created!”. It has to be __assets__^<id>^as_<topic>_<column> of that column, or the write is refused naming the column.
A refusal writes nothing, and a create that cannot link is undone. One message, and either it happened or it failed. Before 7.18.2 a create refused with “Node already exists” had already written the blob and the row, and left both for the collector.
An asset node is a node like any other. It is created, updated, indexed, snapshotted and deleted by the same machinery as everything else. There is no special case — which is why the history works, and why delete-node on an __assets__ row runs the same guards, with force overriding none of them.

Source: kernel/c/timeranger2/src/tr_treedb.c — treedb_store_files(), store_file_bytes(), link_file_columns(), treedb_blob_path(), derive_file_hooks(), treedb_gc_files(). The full account, including every defect the implementation and its three reviews found, is in kernel/c/timeranger2/DESIGN-treedb-files.md. Companion page: the system topics a treedb keeps for itself.