timeranger2

Append-only time-series storage for Yuneta. Records are grouped into topics, keyed by a primary id, and indexed for fast access by time range or row id.

Higher-level stores built on top¶

• tr_msg2db — dict-style message store (one value per key).

• tr_queue — message queue with at-least-once semantics (used by the MQTT broker).

• TreeDB — graph memory database with hook/fkey relationships. See the tr_treedb API and the TreeDB crash course for the full rules.

Core API (selected)¶

tranger2_startup(...) / tranger2_shutdown(...)
tranger2_open_topic(...) / tranger2_close_topic(...)
tranger2_append_record(...)
tranger2_open_iterator(...) / tranger2_iterator_get_page(...) / tranger2_close_iterator(...)
tranger2_open_list(...) / tranger2_close_list(...)

Persistence semantics¶

Every call to tranger2_append_record() writes one record to disk, increments the per-key g_rowid (monotonic, never resets) and i_rowid (row position in the .md2 index file).

Filesystem watcher¶

src/fs_watcher.c implements an inotify-based watcher (fs_event_t) used by root-linux/C_FS and by the stores themselves to react to on-disk changes. See the fs_watcher page in the sidebar.

CLI companions¶

Tests¶

tests/c/timeranger2, tests/c/tr_dt_unknown, tests/c/tr_msg, tests/c/tr_msg2db, tests/c/tr_queue, tests/c/tr_treedb and the other tests/c/tr_treedb_* directories (tr_treedb_load_failed, tr_treedb_failed_save, tr_treedb_link_events, tr_treedb_files, tr_treedb_hook_hygiene, ...), and, through C_TREEDB, tests/c/c_treedb_literal_wins. The test suite lists them, and each directory has a README.md.

Source code¶

• timeranger2.h

• timeranger2.c

Function reference¶

tranger2_append_record()¶

Appends a new record to a topic in the TimeRanger database. Master-only. If __t__ is zero a timestamp is assigned (milliseconds when the topic is sf_t_ms, else seconds). The new record’s metadata is returned through the required md_record_ex out-parameter — the return value is a status code, not the metadata. jn_record is owned (consumed, even on error).

int tranger2_append_record(
    json_t *tranger,
    const char *topic_name,
    uint64_t __t__,
    uint16_t user_flag,
    md2_record_ex_t *md_record_ex,
    json_t *jn_record
);

Parameters

Returns

Returns 0 on success, or a negative value on failure.

Notes

The function makes sure that the record is appended to the specified topic in tranger2_startup(). If the topic does not exist, it must be created using tranger2_create_topic() before calling this function.

Reading the metadata back. The metadata of the new record is returned in md_record_ex, and that is where a caller reads it: md.g_rowid is the key’s global rowid (the one that never resets), md.rowid the row’s position inside its .md2 file (i_rowid), then __t__, __tm__, __offset__, __size__, system_flag, user_flag. Hand the function your only reference:

md2_record_ex_t md;
if(tranger2_append_record(tranger, "topic", 0, 0, &md, record) == 0) {
    json_int_t g_rowid = (json_int_t)md.g_rowid;
}

A string of a record can hold NUL characters. json_dumps() writes each one as the escape \u0000, and a read hands it back (the content is loaded with JSON_ALLOW_NUL). A consumer that reads the string with kw_get_str() gets the C string up to its first NUL; json_string_length() gives the whole length:

json_t *rec = json_pack("{s:s, s:s#}", "id", "A", "blob", "a\0b", 3);
tranger2_append_record(tranger, "topic", 0, 0, &md, rec);
/*  a load hands {"id": "A", "blob": "a\u0000b"}: json_string_length() is 3  */

A content that is not json when it is read logs “Bad data, the content of the record is not json” (a CRITICAL, with topic, key, __t__, __offset__, __size__, the jansson error and its position), and the record is not handed. A content that this process has not the memory to parse logs “Cannot read the record, this process has not the memory to parse its content (MEM_MAX_BLOCK)” (the same fields; the jansson error is “not enough memory”), and the record is not handed either. The largest memory block (gbmem_get_maximum_block()) is set by MEM_MAX_BLOCK for each yuno, and a content smaller than that block can need more to parse: jansson doubles the buffer of a string as it reads it, and the table of an array or an object. A string of 40 000 bytes does not parse with a block of 64 KiB, nor does an array of 10 000 numbers. A yuno with a larger block reads the record. Up to 7.25.4 such a string crashed the process (jansson wrote past its buffer, see kernel/c/linux-ext-libs/patches/jansson/).

The function does not add a __md_tranger__ to the record for the caller. It adds one (g_rowid, i_rowid, t, tm, offset, size, system_flag, user_flag, the same dict a load attaches) only to the record it hands to a realtime list that takes it: one that wants the key and is not an only_md feed (an only_md feed is handed NULL). A list open on another key does not make the append build it. Nor is a __md_tranger__ ever stored: one the record carries in, from an earlier append or from a load, is the metadata of THAT record and is dropped before the content is written.

The record is owned by the function, and these changes are made to the object itself, not to a copy: a list hands it on without copying it, on the hot path of every append. So a caller that keeps a reference of its own sees them, and reads the metadata from md_record_ex anyway:

json_incref(record);                    // the caller keeps one
tranger2_append_record(tranger, "topic", 0, 0, &md, record);
// No list takes the key:   record has no __md_tranger__ (a carried one is gone)
// A list takes the record: record HAS the list's fresh __md_tranger__
// On an sf_rowid_key topic: record also has "__rowid__"
json_decref(record);

A __t__ that belongs to an earlier file. The record goes to the file that its __t__ selects (through the topic’s filename_mask), even if that file is not the newest one. A key’s global rowid counts the records in the order of its files, so the new record takes its place in that order, and every record of the later files moves one place up:

/*  filename_mask "%Y-%m-%d"; key 1 has A (2000-01-01), B (2000-01-02)  */
tranger2_append_record(tranger, "topic", t_2000_01_01 + 10, 0, &md, jn_c);
/*  C gets g_rowid 2, and B is now 3: an iterator serves A, C, B  */

The g_rowid in md_record_ex is its place, the same number a reload gives (and stores in the loaded record’s __md_tranger__). If you keep global rowids, for example as a page position, a record written into an earlier file makes the rowids of the later records stale. Records written with __t__ = 0 (now) always go to the last file, and move nothing. Until 2026-09-15 the cache in memory disagreed with the disk until the next reload: it served the wrong record, and a follower re-published the whole file.

An append that fails is not half written. The content (.json) is written first and the md2 row after. When the md2 cannot be opened or created, sought or written, the function returns -1 and cuts the content file back to where the record began (a row written in part is cut from the .md2 too): the file keeps no record that no row names. If the cut fails as well, it logs “Cannot cut back the content of an append whose md2 row was not written: the content file keeps a record no row names”. A kill or a power cut between the two writes still leaves that shape; the next open ignores it with a warning (see After a restart too, under tranger2_open_iterator()). Up to 7.25.4 the content was left behind.

The cut comes BEFORE the critical that reports the failure. That matters with the exit bit of the tranger’s on_critical_error (LOG_OPT_EXIT_ZERO, 2, the default of C_TRANGER, and what C_TREEDB passes with its exit_on_error): gobj_log_critical() then ends the process inside the log call, and nothing after it runs. When the md2 cannot be opened or created, the cause is logged without leaving, the content is cut back, and then one more critical exits:

CRITICAL create_file: Cannot create json file  path=<store>/items/keys/A/2000-01-06.md2
CRITICAL get_topic_wr_fd: Cannot open file to write  path=<store>/items/keys/A/2000-01-06.md2
CRITICAL tranger2_append_record: Cannot append record, its md2 file cannot be opened: its content
         was cut back  topic=items key=A file_id=2000-01-06          <- exit(0) here, with 2

The same order holds for a content write that fails part way: its part is cut back before the critical.

A write that stops part way (the file size limit, a full disk) is not said as a write FAILED with an errno: write() returned a count, and errno says nothing then. It is said as a SHORT write, with the bytes written and the bytes expected. A read that returns fewer bytes than a row is said as a SHORT read the same way:

CRITICAL tranger2_append_record: Cannot append record, short write of its content: the file size
         limit or the disk is full  topic=items written=13 expected=45
CRITICAL tranger2_append_record: Cannot save record metadata, short write: the file size limit or
         the disk is full  topic=items written=13 expected=32
CRITICAL read_md2_row: Cannot read a record of md2 file, short read
         path=<store>/items/keys/A/2000-01-02.md2 row=last offset=64 read=16 expected=32

The same holds for the files that are replaced whole (topic_var.json, topic_cols.json) and for the zero wipe of tranger2_delete_instance():

ERROR    replace_json_file: Cannot replace topic_var.json, short write: the file size limit or
         the disk is full  path=<store>/items/topic_var.json
CRITICAL tranger2_delete_instance: Cannot zero the payload, short write: the file size limit or
         the disk is full  topic=items key=A remaining=53 written=4 expected=53

A row always goes on a row boundary. When the md2 row of a failed append was written in part and the cut of that part failed too (“Cannot cut back the md2 of an append whose row was not written whole: its size is not a whole number of rows”), the md2 ends in a torn row. The next append cuts the md2 back to its whole rows first, with the same warning the open gives (see A .md2 whose last row is torn under tranger2_open_iterator()), and writes its row there:

WARNING tranger2_append_record: md2 file of the key ends in a part of a row: an append that was
        never acknowledged was cut back  topic=items key=A file_id=2000-01-02
        old_size=141 new_size=128

If that cut fails, the append is refused with -1 and its content is cut back (“Cannot append record, its md2 file ends in a part of a row that cannot be cut back: the append is refused”); the whole rows stay readable, and the next append tries the cut again. The append cuts only a tail that is a torn row after good rows, the same check as the open (see tranger2_open_iterator()). For any other tail it does not cut: it is refused with -1, its content is cut back, the md2 does not change, and the file is flagged as the open flags it. Every load of the key in the same process says load_failed, and the next append into the file counts it again first, so it is refused too (see below) until the file is repaired. The logs, after the CRITICAL that names the shape:

ERROR    flag_file_unreadable_at_append: md2 file of the key flagged unreadable at
         an append: every load of the key says load_failed
         topic=items key=A file_id=2000-01-02
CRITICAL tranger2_append_record: Cannot append record, its md2 file ends in a part
         of a row that must not be cut back: the append is refused
         topic=items key=A file_id=2000-01-02 md2_size=161

The file is flagged BEFORE that CRITICAL. With the exit bit of on_critical_error (the default, see above) the yuno ends inside the log call, and the next start reads the file again and flags it at its open. So the flag in memory matters only when on_critical_error does not exit.

When the check cannot RUN (the .json cannot be opened, for example with EMFILE or ENFILE, or cannot be read, or there is no memory), it found nothing wrong with the file. The append is refused with -1 and its content is cut back, but the file is NOT flagged: the whole rows stay readable, and the next append checks again. With the default on_critical_error the refusal’s CRITICAL exits the yuno, and the next start checks the file at its open (and flags it when the check still cannot run). The CRITICAL of the check comes first (one of the three listed in tranger2_open_iterator()), then the refusal:

CRITICAL check_torn_md2_tail: Cannot check the torn tail of a md2 file, its content file
         cannot be read: not cut  topic=items key=A file_id=2000-01-02
         path=<store>/items/keys/A/2000-01-02.json errno=24 serrno="Too many open files"
CRITICAL tranger2_append_record: Cannot append record, the torn tail of its md2 file
         cannot be checked now: the append is refused, the file is not flagged
         topic=items key=A file_id=2000-01-02 md2_size=161

Up to 7.25.4 nothing cut a torn row back: an append wrote its row after the torn bytes, and the cache build left the whole file out of the key, so no read found that row.

An append into a file flagged unreadable (a .md2 the cache build could not count, see the same section) counts the file again first. If it can be read now, it gets its cell, the flag of that file goes (“md2 file of the key readable again: it is counted, and the key is not flagged for it”), and the append goes on. If not, the append is refused with -1 and writes nothing (“Cannot append record, its file is flagged unreadable: its row would follow rows no cell counts”): its row would have been counted as the file’s first, and a load read an OLD row of the file in its place.

/*  key A: the md2 of day 2 could not be opened at the open (mode 0000)     */
tranger2_append_record(tranger, "topic", t_day2, 0, &md, rec1);   // -1, nothing written
chmod(path_md2_day2, 0660);
tranger2_append_record(tranger, "topic", t_day2, 0, &md, rec2);   // 0: day 2 counted,
                                                                  // A loads whole again

The file counted again gets its cell in the MIDDLE of the key, and every global rowid after it moves up by the rows of the file. The iterators open on the key follow: an unfiltered one takes its segments again at its next page; a FILTERED one (a paging iterator with an index, see tranger2_iterator_get_page()) takes its segments and its index again at once, as its open would build them now. So its index gains the rows of the file, with their new rowids. Like every append after the open of a filtered iterator, the append that caused the count is not in its index. For example:

/*  key A: v1 (day 1), v2 (day 2, flagged), v3 (day 3), v4 (day 4)          */
it = tranger2_open_iterator(tranger, "topic", "A",
        json_pack("{s:I}", "from_t", (json_int_t)t_day1), NULL, "pager", "me", NULL, NULL);
/*  its page:  total_rows 3, v1 v3 v4                                         */
chmod(path_md2_day2, 0660);
tranger2_append_record(tranger, "topic", t_day2, 0, &md, rec_v9); // counts day 2
/*  its page:  total_rows 4, v1 v2 v3 v4                                     */

tranger2_backup_topic()¶

Creates a backup of a topic in the TimeRanger database. If backup_path is empty, the topic path is used. If backup_name is empty, the backup file is named topic_name.bak. If overwrite_backup is true and the backup exists, it is overwritten unless tranger_backup_deleting_callback returns true.

json_t *tranger2_backup_topic(
    json_t *tranger,
    const char *topic_name,
    const char *backup_path,
    const char *backup_name,
    BOOL overwrite_backup,
    tranger_backup_deleting_callback_t tranger_backup_deleting_callback
);

Parameters

Returns

The new, empty topic (not yours), created with the description, cols and var of the old one; or NULL (logged) when the backup fails.

Notes

If overwrite_backup is true and the backup exists, tranger_backup_deleting_callback is called. If it returns true, the existing backup is not removed.

Only a master can back up a topic (the backup moves its directory): on a replica it returns NULL with “Only master can back up”. The topic name follows the topic name rule.

The backup CLOSES the topic first, then moves its directory. A failure after that point (the backup exists and overwrite_backup is false, topic_desc.json does not load, or the rename() fails -- the backup name is taken by a file, another filesystem) returns NULL and opens the topic again as it was, with a warning: “Backup of topic failed: the topic is opened again as it was, not backed up”. Every pointer to the old topic is stale all the same, as after a successful backup: take the topic again by name.

So does a failure AFTER the move, when the new topic cannot be created (ENOSPC, a mkdir that fails): what the create left is removed, the backup is moved back where the topic was, and the topic is opened again, with the same warning. Only when the backup cannot be moved back is nothing opened: a CRITICAL, “Backup of topic failed, and the backup cannot be moved back: the data of the topic is in the backup”, with both paths. Up to 7.25.4 the data stayed in the backup and nothing was opened: a queue had no topic until a restart, and, its size read as 0, its backup was never tried again (tests/c/tr_queue/test_tr_queue_backup_failed).

That holds for EVERY tranger, also one opened with on_critical_error LOG_OPT_EXIT_ZERO (the MQTT broker’s queues, C_TRANGER’s default): the create of a backup runs with the exit bits of on_critical_error off (LOG_OPT_EXIT_ZERO, LOG_OPT_EXIT_NEGATIVE, LOG_OPT_ABORT), because its failure is recovered; the CRITICALs are still logged, and the tranger’s value is back when the create returns. If the create exited there, the data of the topic would stay in <topic>.bak until moved back by hand.

json_t *topic = tranger2_backup_topic(tranger, "queue", 0, 0, TRUE, 0);
if(!topic) {
    // Error already logged: not backed up; the topic is still the old one
    topic = tranger2_topic(tranger, "queue");
}

Up to 7.25.4 a failed rename() also leaked the topic_var it had loaded, and the topic stayed closed: a queue (trq_check_backup()) was left with no topic.

tranger2_close_all_lists()¶

Closes the iterators, rt_mem and rt_disk lists of a topic that belong to a given creator. An empty creator closes all of them. A non-empty creator with an empty rt_id closes all of that creator’s. With both set it narrows to a single rt_id.

int tranger2_close_all_lists(
    json_t *tranger,
    const char *topic_name,
    const char *creator,    // if empty, remove all
    const char *rt_id       // if empty, remove all lists of creator
);

Parameters

Returns

Returns 0 on success, or a negative value on failure. No-op (0) if the topic is not opened.

Notes

This closes the iterators, rt_mem and rt_disk lists associated with a specific creator, preventing resource leaks. If creator is empty, every list of the topic is closed regardless of rt_id.

tranger2_close_iterator()¶

Closes an iterator in the TimeRanger 2 database, releasing associated resources.

int tranger2_close_iterator(
    json_t *tranger,
    json_t *iterator
);

Parameters

Returns

Returns 0 on success, or a negative value on failure.

Notes

Closing an iterator makes sure that any allocated memory or resources are properly released.

tranger2_close_list()¶

The function tranger2_close_list() closes a previously opened list, which can be a real-time memory list (rt_mem), a real-time disk list (rt_disk), or a non-real-time list.

int tranger2_close_list(
    json_t *tranger,
    json_t *list
);

Parameters

Returns

Returns 0 on success, or a negative value on failure.

Notes

This function makes sure that resources associated with the list are properly released. If the list is a real-time list, it stops receiving updates.

tranger2_close_rt_disk()¶

The tranger2_close_rt_disk() function closes a previously opened real-time disk stream in the TimeRanger database, releasing associated resources.

int tranger2_close_rt_disk(
    json_t *tranger,
    json_t *disk
);

Parameters

Returns

Returns 0 on success, or a negative value on failure.

Notes

This function must be called when a real-time disk stream is no longer needed to free resources.

The directory of the feed, <topic>/disks/<id>/, is renamed away first (disks/.closing.<pid>-<start>.<seq>) and then removed. The master keeps hard-linking the new records into that directory until it hears the feed is gone; removed where it was, a key directory the master made in between made the removal fail (“rmdir() FAILED ... Directory not empty”), the directory stayed, and the master went on feeding it for ever (up to 7.25.21, seen at 3000 records/s). Renamed, the master cannot reach it, and its watch of disks/ (FS_FLAG_MOVED_AS_DELETED) hears the rename as the close of the feed. A .closing.* left by a reader that died while removing it is removed by the master the next time it reads disks/ (only when that process is gone). <start> is when the reader’s process started (field 22 of /proc/<pid>/stat): a pid alive with another start time is a pid reused by another process, and the leftover is removed too. Up to 7.25.21 the name had the pid alone, and such a leftover stayed while the unrelated process lived. A name of the old form (.closing.<pid>.<seq>) is judged by its pid alone.

tranger2_close_rt_mem()¶

The tranger2_close_rt_mem() function closes a real-time memory stream associated with a given TimeRanger instance.

int tranger2_close_rt_mem(
    json_t *tranger,
    json_t *mem
);

Parameters

Returns

Returns 0 on success, or a negative value on failure.

Notes

Closing a real-time memory stream using tranger2_close_rt_mem() makes sure that resources are properly released.

tranger2_close_topic()¶

The tranger2_close_topic() function closes an open topic in the TimeRanger database, releasing associated resources.

int tranger2_close_topic(
    json_t  *tranger,
    const char *topic_name
);

Parameters

Returns

Returns 0 on success, or a negative value on failure.

Notes

Closing a topic does not delete its data. It only releases resources associated with the open topic.

tranger2_create_topic()¶

The tranger2_create_topic() function creates a new topic in the TimeRanger database if it does not already exist. If the topic exists, it returns the existing topic metadata. The function makes sure that the topic is properly initialized with the specified primary key, time key, system flags, and additional metadata.

json_t *tranger2_create_topic(
    json_t              *tranger,
    const char          *topic_name,
    const char          *pkey,
    const char          *tkey,
    json_t              *jn_topic_ext,
    system_flag2_t      system_flag,
    json_t              *jn_cols,
    json_t              *jn_var
);

Parameters

Returns

Returns a JSON object representing the topic metadata. The returned JSON object is not owned by the caller and must not be modified or freed.

NULL (logged) for a refused name, a topic that does not exist on a replica, an empty pkey, no key type, a NEW topic that cannot be made whole (below), a topic_cols.json / topic_var.json that cannot be written at a version change, or a topic that cannot then be opened (see tranger2_open_topic()).

A new topic is made whole, or not at all. The master creates the topic directory, topic_desc.json, topic_cols.json, topic_var.json, keys/ and disks/. When any of them cannot be made (no space, a mkdir that fails), the directory it made is removed, nothing is opened or kept in memory, the call answers NULL and logs “Cannot create topic: it is not whole, what was made is removed” after the cause; the next create starts again from nothing. Up to 7.25.4 each failure was logged and the create went on: a topic whose keys/ (or disks/) could not be made was opened and returned as created, and a queue backup (see tranger2_backup_topic()) took that half topic as the queue’s new one.

It is whole or not at all under LOG_OPT_EXIT_ZERO too. The failures of the create (the mkdirs, the write of topic_desc.json) are CRITICALs logged without the exit bits of on_critical_error. “Cannot create topic: it is not whole, what was made is removed” is logged after the removal, as a CRITICAL at the tranger’s own on_critical_error. So a tranger that exits on a critical -- LOG_OPT_EXIT_ZERO, the default of C_TRANGER and what C_TREEDB and the MQTT broker’s queues pass -- exits there, with nothing of the topic left on disk. Up to 7.25.4 it exited in the log of the first failure: the topic_desc.json, topic_cols.json and topic_var.json stayed without keys/ or disks/, and the next start opened that half topic (or could not open it at all, when the failure was the write of topic_desc.json).

json_t *topic = tranger2_create_topic(tranger, "devices", "id", "tm", NULL,
    sf_string_key, json_pack("{s:s, s:I}", "id", "", "tm", (json_int_t)0), 0);
if(!topic) {
    return -1;  // Error already logged; nothing of "devices" on disk or in memory
}

Notes

This function is idempotent. This means that if the topic already exists, it will return the existing topic metadata instead of creating a new one. If the primary key (pkey) is not specified, the function defaults to sf_string_key if pkey is defined, otherwise it defaults to sf_int_key.

Only the master writes, and it asks first. On a replica, or on a master that lost its lock while it was stopped (see tranger2_stop()), an existing topic is opened read-only and nothing is written, and a new one is refused with “Cannot open TimeRanger topic. Not found and no master”.

Nothing about tm order is written. From 7.25.5 to 7.25.22 a new topic carried "marks_tm_unordered": true and the master marked a file whose tm went back (<file>.tm_unordered); both are gone, and a store that has them keeps them, ignored. A tm condition is a filter on every row: see tranger2_open_iterator().

{
    "topic_name": "readings",
    "pkey": "id",
    "tkey": "tm",
    "system_flag": 1
}

A new topic_version replaces topic_cols.json, then topic_var.json, each through a temporary file and a rename(), both fsync’ed with their directory, so the new version never survives a power cut its cols did not; the last_rowid_id counter of treedb is carried over. The cols go first because the version is what says they moved. A file that cannot be written refuses the topic: the call answers NULL, logs “Cannot re-create topic_cols.json for a new topic_version: the topic keeps its version, and is not opened” (or the error of topic_var.json), and the next create tries again -- the version on disk has not moved. “Re-Creating topic_var.json” and “Re-Creating topic_cols.json” are logged only for a file written. Up to 7.25.4 topic_cols.json was REMOVED first and a failed write went unnoticed: the topic opened with no cols file under the new version.

Topic names. A topic name is ONE directory under the database, and a segment of the backtick kw paths (topics`<name>`cols). Every call that takes a topic name — create, open, delete, backup, tranger2_topic_path(), tranger2_write_topic_var() / _cols() — refuses, with the error “Invalid topic name (path metacharacters not allowed)”:

• an empty name, . and ..;

• any name that holds / or `.

A leading . is accepted, unlike a key: MQTT queues are named <client_id>-IN / -OUT, and the broker accepts a client_id such as .foo.

The id of a disk feed follows the directory half of the rule. The id of tranger2_open_rt_disk() (the rt_id of a follower’s open-rt / open-list) becomes the directory <topic>/disks/<id>/, and the follower removes that directory before creating it. An empty id is refused with “Invalid rt id (empty)”; ., .., or an id holding / with “Invalid rt id (path metacharacters not allowed)”, and an id longer than NAME_MAX (255 bytes on Linux) with “Invalid rt id (longer than NAME_MAX)” (its directory cannot exist). A leading . is refused too, with “Invalid rt id (a leading dot is reserved)”: disks/.<name> is where a feed goes while it is removed (see tranger2_close_rt_disk()). A backtick is accepted, since an rt id is not a segment of any kw path (treedb names its own feeds <treedb>`<topic>`<id>). The id may come from a peer, so a refused id is logged as a warning, without a stack.

tranger2_open_rt_disk(tranger, "raw_tracks", "", NULL, cb, "db_history^1620", "", NULL); // ok
tranger2_open_rt_disk(tranger, "raw_tracks", "", NULL, cb, "a`b`c", "", NULL);           // ok
tranger2_open_rt_disk(tranger, "raw_tracks", "", NULL, cb, ".hidden", "", NULL);         // NULL: reserved
tranger2_open_rt_disk(tranger, "raw_tracks", "", NULL, cb, "../x", "", NULL);            // NULL: escapes

One id, one feed. The directory is keyed by the id alone, so an id already in use by a live feed of the topic is refused whatever the creator, with a warning: “rt disk id already in use by another creator, refused”, or “rt disk id already in use by the same creator, refused”. A second feed used to take the first one’s directory over, and its close removed it. The guard is per process: it reads the feeds of this tranger, so two processes that follow one store with the same id still take each other’s directory.

tranger2_open_rt_disk(tranger, "users", "", 0, cb, "gui-42", "gui", 0);       // OK
tranger2_open_rt_disk(tranger, "users", "", 0, cb, "../../etc", "", 0);       // refused
tranger2_open_rt_disk(tranger, "users", "", 0, cb, "", "", 0);                // refused: empty
tranger2_open_rt_disk(tranger, "users", "", 0, cb, "gui-42", "other", 0);     // refused: in use
tranger2_open_rt_disk(tranger, "users", "", 0, cb, "gui-42", "gui", 0);       // refused: in use

char long_id[NAME_MAX + 2];
memset(long_id, 'x', sizeof(long_id) - 1);
long_id[sizeof(long_id) - 1] = 0;                                              // 256 bytes
tranger2_open_rt_disk(tranger, "users", "", 0, cb, long_id, "gui", 0);        // refused: > NAME_MAX

tranger2_create_topic(tranger, "users", "id", "tm", 0, sf_string_key, 0, 0);   // OK
tranger2_create_topic(tranger, ".foo-IN", "", "", 0, sf_rowid_key, 0, 0);      // OK
tranger2_create_topic(tranger, "../other_db/users", "id", "tm", 0, 0, 0, 0);   // refused
tranger2_create_topic(tranger, "users/keys", "id", "tm", 0, 0, 0, 0);          // refused

tranger2_delete_key()¶

The tranger2_delete_key() function deletes a whole record (= a primary key, with every instance stored under it) from the specified topic in the TimeRanger database. It removes the keys/<key>/ directory and drops the key from the topic cache. Irrecoverable. Only the master can delete.

The deletion is propagated to subscribers:

• The master removes the live keys/<key>/ directory FIRST, and signals the delete only once it is done (up to 7.25.4 it signalled first, so a key whose directory could not be removed was heard deleted while it was still on disk). It signals it in the directory of every rt_disk feed: topic/disks/<rt_id>/<key>/, where it exists (the feed received records of the key), is removed, and then a directory .d<seq>.<key> is created and removed at once -- that is the signal. seq is the topic’s delete sequence (<topic>/delete_seq.json, recorded durable before the signal, growing across restarts of the master), and it is what lets a follower tell a delete it applied from a new one (see fs_watcher, the rt_disk feed). A key longer than NAME_MAX - 23 bytes is signalled as .h<seq>.<sha256 of the key>. A feed that received no record of the key since it opened hears the delete too. The sequence is taken only when disks/ has a feed to tell; its record costs two disk flushes per delete then. Since 7.26.0; up to 7.25.22 the signal was the key’s own directory (removed, or created and removed), so master and followers of a topic upgrade together. When disks/ cannot be listed (its opendir() fails with anything but ENOENT, or a readdir() fails half way), the feeds not reached do not hear it, and that is logged: “Cannot tell the rt_disk feeds that a key was deleted, opendir() of disks/ FAILED” or “Cannot tell every rt_disk feed that a key was deleted, readdir() of disks/ FAILED”. The delete itself is done and answers 0. Up to 7.25.4 both were silent.

• Each follower watches its own disks/<rt_id>/. It clears the key from its topic cache and fires the callback of that feed only, once.

• A key deleted and written again (once or more) before a follower reads the delete: the feed hears deleted (once or more), then the records of the key’s LAST life from rowid 1, and keeps the key, also behind a master in ANOTHER process. Each key directory is watched with a descriptor of its own (FS_FLAG_DIR_FDS, fs_watcher_dir_fd), and a link is taken through the descriptor of the directory its event came from (openat / unlinkat): a link of a directory gone reaches nothing. The link is the md2 of its life: the records are read through descriptors checked to be that life’s (the md2 against the link, the content once per life by path), so they are that life’s even if keys/<key>/ is another one by then. Up to 7.25.21 a follower that lagged behind such a master read and unlinked by path, and could hand a later life before an earlier delete ([R2 DEL], the live key out of the cache, the next append handing R1..R4; under a random two-process load 7-11 of 12 feed and key pairs got a life out of order, none now). The records of a life deleted before the follower read it are not handed -- they are gone from disk with it. The follower reads a key directory only if it is still the directory it looked at before it asked where its queue ended, and only once its stream is past that end: for any key, also one it never saw. A file linked in a directory that waits to be read is read with it, in order. When it hears a delete, the follower closes the descriptors it holds on the files of the key: looked up by the key, as the master does at its delete (up to 7.25.21 every open key was walked, once per feed that heard it: 4.3 ms per delete and feed with 50000 keys open, 17 us now). The first feed to hear a delete asks where the stream of each other feed ends (fs_queued_events_end()), to note what each one owes: a cost linear in the feeds, 17 us per delete and feed with one feed, 30 with sixteen. Where the filesystem keeps no birth time (some NFS, ext4 with 128-byte inodes) a key directory is told from another by its inode alone, and an inode freed by a delete can be given to the key written again: said once, with a warning (“No birth time of a directory ...”). So is a container whose seccomp profile refuses statx() (EPERM): the inode then comes from lstat(). Any other failure to tell which directory a key directory is (EIO, EACCES) is an ERROR, once per errno in a row: that directory’s scan is skipped, and its links wait for the key’s next record. Up to 7.25.21 every failure of statx() took the directory for gone, in silence. Up to 7.25.20 the directory was read early ([R1 DEL], [DEL R1 DEL], the live key out of the cache), and a key read before, deleted and written again in the same day file, was read through the old file’s descriptor (short reads, records lost). Details in fs_watcher.

• In-process rt_mem, open_iterator, and rt_disk subscribers without a watcher (no loop) whose key filter matches receive their tranger2_key_deleted_callback_t directly, if registered.

For per-instance delete (one row of a key’s .md2 index — irrecoverable, no resurrection), see tranger2_delete_instance().

int tranger2_delete_key(
    json_t  *tranger,
    const char  *topic_name,
    const char  *key
);

Parameters

Returns

Returns 0 on success, or a negative value on failure.

Notes

A key with no directory on disk (its stat() fails with ENOENT) is logged (“key directory not found”, an error) and the call still answers 0: the key is removed from memory all the same. Any OTHER failure of that stat() (EACCES on keys/, EIO) says nothing of the key: the call answers -1 with “Cannot delete key, stat() of its directory FAILED”, and nothing is deleted, dropped from memory or announced. Up to 7.25.4 it was taken as “not found”: the key left the cache, the delete was announced to every feed and follower, and the call answered 0 over files still on disk, which came back at the next open.

key A, 3 rows; stat(keys/A) fails with EIO      -> -1, A keeps its 3 rows, no callback
the same delete once keys/A can be stat'ed      -> 0, one callback

A directory that cannot be removed answers -1 and announces nothing, but it does NOT leave the memory as it was: some of the key’s files may be gone already, so the cache of the key is read again from what is left on disk (the key leaves the cache if nothing is left), and its iterators take their segments again from it: an unfiltered one at its next page, a FILTERED one at once, its index built again as an open builds it:

key A, 3 rows; a filtered pager of A (from_t)    -> total_rows 3
tranger2_delete_key(A), keys/A of mode 0550        -> -1, no file removed
the same pager, next page                          -> total_rows 3

The iterators of the key, in this process, lose what they took from it: see tranger2_iterator_get_page().

The legacy name tranger2_delete_record() is kept as a source-level alias in timeranger2.h (#define tranger2_delete_record tranger2_delete_key) so existing callers keep compiling unchanged. New code must use tranger2_delete_key().

tranger2_delete_instance()¶

The tranger2_delete_instance() function deletes a single instance (one row of a key’s .md2 index) without touching the surrounding rows. The row’s metadata is mutated in place: sf_deleted_instance (bit 0x0400 of system_flag2_t, on the inherited side of the mask so rt_by_disk followers see the same tombstone) is ORed into the system flag bits. The .json data log stays append-only.

Read paths (tranger2_open_iterator history loop, tranger2_iterator_get_page, and the rt_by_disk follower) skip rows whose tombstone bit is set. Treedb is downstream of those paths and inherits the skip with no tr_treedb change. tranger2_read_record_content() and tranger2_read_user_flag() deliberately do not filter — the caller has already located the row — so they remain available for audit / wipe-verification tooling.

Irrecoverable. Master-only. Second delete of the same row is a silent no-op (returns 0).

int tranger2_delete_instance(
    json_t      *tranger,
    const char  *topic_name,
    const char  *key,
    uint64_t    __t__,
    uint64_t    rowid,          // per-key i_rowid, slot in .md2, based 1
    BOOL        zero_payload
);

Parameters

Returns

Returns 0 on success, a negative value on failure.

Notes

Side effects to be aware of:

• rowids do not renumber. Slot N stays slot N (dark).

• iterator_size(), total_rows, pages count slots, not live rows. A page returning data.length=2 with total_rows=3 is the contract for “1 dead in this segment”.

• topic_cache cells (fr_t/to_t/fr_tm/to_tm) are not refreshed. If the deleted instance was the min/max t/tm of its file, the cell rollup can lie. Cheap to fix on next cold reload. Expensive to fix incrementally. Deferred until a consumer needs it.

• With zero_payload, the row is marked deleted BEFORE the wipe. A wipe that fails answers -1 with the row already dead; a wipe cut part way (the file size limit, a full disk) is logged as “Cannot zero the payload, short write: the file size limit or the disk is full”, with the bytes written and expected, not with an errno.

tranger2_set_rt_key_deleted_callback()¶

Registers a key-delete callback on a handle returned by tranger2_open_rt_mem(), tranger2_open_rt_disk() or tranger2_open_iterator(). The callback fires when tranger2_delete_key() runs against a key this subscriber tracks:

• On the master, directly from tranger2_delete_key().

• On rt_by_disk followers, from the inotify watcher on disks/<rt_id>/ when the master mirrors the deletion.

The subscriber’s filter is honoured, the same one its records go through: a specific key matches that one, an rkey (in match_cond) the keys its regular expression matches, and neither of them every deletion. Up to 7.25.20 only key was looked at, and a feed opened with an rkey was told every key deleted:

json_t *rt = tranger2_open_rt_mem(
    tranger, "devices", "", json_pack("{s:s}", "rkey", "^DVES_"),
    on_record, "dves-card", "", NULL
);
tranger2_set_rt_key_deleted_callback(rt, on_key_deleted, NULL);
tranger2_delete_key(tranger, "devices", "OTHER_1");     // on_key_deleted not called
tranger2_delete_key(tranger, "devices", "DVES_000000"); // called once

Additive: existing handles default to “no callback”, behavior unchanged. Passing cb=NULL clears any previously registered callback.

typedef int (*tranger2_key_deleted_callback_t)(
    json_t      *tranger,
    json_t      *topic,
    const char  *key,           // the deleted key
    json_t      *list,          // iterator / rt_mem / rt_disk entry, don't own
    void        *user_data
);

int tranger2_set_rt_key_deleted_callback(
    json_t                              *list,
    tranger2_key_deleted_callback_t     cb,
    void                                *user_data
);

Parameters

Returns

Returns 0 on success, -1 if list is NULL.

Example

A follower feed that closes itself when the key it follows is deleted:

PRIVATE int on_key_deleted(
    json_t *tranger, json_t *topic, const char *key, json_t *list, void *user_data)
{
    tranger2_close_rt_disk(tranger, list);  // allowed: see the notes
    return 0;
}

/*  The loop is the PARAMETER: a "yev_loop" key in the config is overwritten  */
json_t *tranger = tranger2_startup(0, jn_tranger, yev_loop);
json_t *rt = tranger2_open_rt_disk(
    tranger, "devices", "DVES_000000", NULL, on_record, "live-card", "", NULL
);
tranger2_set_rt_key_deleted_callback(rt, on_key_deleted, NULL);

Notes

The callback fires once per feed for each delete. An rt_disk feed that owns an inotify watcher hears the delete only from its own directory, and the master-side fan-out skips it. Until 2026-09-15 a feed was fired once for every inotify event of every feed of the topic (six times in a three-feed follower), and not at all for a key that had no record since the feed opened.

The callback may close its own feed (tranger2_close_rt_disk()). The watcher is stopped at the end of the batch it is handling, and the rest of that batch is dropped.

The feed’s watcher exists only when the tranger was started with a loop: pass it as the third argument of tranger2_startup(). Without a loop an rt_disk has no watcher, and it is fired directly by the master’s delete path, like an rt_mem.

tranger2_delete_topic()¶

The tranger2_delete_topic() function deletes a topic from the TimeRanger database, effectively removing all associated records and metadata.

int tranger2_delete_topic(
    json_t  *tranger,
    const char  *topic_name
);

Parameters

Returns

Returns 0 on success, or a negative value if an error occurs.

Notes

Deleting a topic is irreversible. Make sure that the topic is no longer needed before calling tranger2_delete_topic().

Only a master can delete: on a replica (master=0) it returns -1 with “Only master can delete”, and the directory stays. The topic name follows the topic name rule: tranger2_delete_topic(tranger, "../other_db/users") returns -1 and deletes nothing.

tranger2_dict_topic_desc_cols()¶

tranger2_dict_topic_desc_cols() retrieves the column descriptions of a specified topic in dictionary format.

json_t *tranger2_dict_topic_desc_cols(
    json_t      *tranger,
    const char  *topic_name
);

Parameters

Returns

A JSON object containing the column descriptions of the specified topic. The caller must decrement the reference count of the returned JSON object.

Notes

This function is similar to tranger2_list_topic_desc_cols(), but returns the data in dictionary format instead of a list.

tranger2_get_iterator_by_id()¶

Retrieve an iterator by its identifier. If the iterator exists, it is returned. Otherwise, NULL is returned.

json_t *tranger2_get_iterator_by_id(
    json_t      *tranger,
    const char  *topic_name,
    const char  *iterator_id,
    const char  *creator
);

Parameters

Returns

Returns a pointer to the iterator JSON object if found. Otherwise, returns NULL.

Notes

This function does not produce any error messages if the iterator is not found. The creator filters the match: pass the same creator used at open. An empty creator matches only entries that were themselves opened without a creator.

It is one hash lookup (since 7.25.0): the topic keeps iterators_by_id — {creator: {id: iterator}} — beside its iterators array. It used to walk the array, and tranger2_open_iterator() calls it to refuse a duplicate, so opening N iterators on a topic was O(N²).

It also opens the topic when it is closed (it goes through tranger2_topic()). To ask whether a handle you registered is still alive, check tranger2_topic_is_open() first, then look the handle up by its id: a topic closed and opened again has the same name and none of the old handles.

// Is my iterator "it1" still there? Never trust a pointer kept from the open.
json_t *it = tranger2_topic_is_open(tranger, "frames")?
    tranger2_get_iterator_by_id(tranger, "frames", "it1", "my_service") : NULL;

tranger2_get_rt_disk_by_id()¶

Retrieve a real-time disk instance by its identifier. If the specified real-time disk exists, it returns the corresponding JSON object. Otherwise, it returns NULL.

json_t *tranger2_get_rt_disk_by_id(
    json_t      *tranger,
    const char  *topic_name,
    const char  *rt_id,
    const char  *creator
);

Parameters

Returns

Returns a JSON object representing the real-time disk instance if found. Otherwise, returns NULL.

Notes

This function does not produce any error messages if the real-time disk is not found. The creator filters the match: pass the same creator used at open. An empty creator matches only entries that were themselves opened without a creator.

tranger2_get_rt_mem_by_id()¶

Retrieve a real-time memory instance by its identifier. If the specified real-time memory instance exists, it is returned. Otherwise, NULL is returned.

json_t *tranger2_get_rt_mem_by_id(
    json_t      *tranger,
    const char  *topic_name,
    const char  *rt_id,
    const char  *creator
);

Parameters

Returns

Returns a pointer to the real-time memory instance as a json_t * if found, otherwise returns NULL.

Notes

This function does not produce any internal logging or error messages if the requested real-time memory instance is not found. The creator filters the match: pass the same creator used at open. An empty creator matches only entries that were themselves opened without a creator.

tranger2_iterator_get_page()¶

Retrieves a page of records from an iterator in the TimeRanger database. The function returns a JSON object containing the total number of rows, the number of pages based on the specified limit, and the list of retrieved records.

json_t *tranger2_iterator_get_page(
    json_t *tranger,
    json_t *iterator,
    json_int_t from_rowid,    // based 1
    size_t limit,
    BOOL backward
);

Parameters

Returns

A JSON object containing the retrieved records, total row count, and pagination details. The caller owns the returned JSON object and must manage its memory.

Notes

This function is useful for paginating through records in an iterator. The from_rowid parameter determines the starting point, and the limit parameter controls the number of records retrieved per page.

A filtered iterator (one opened with any match_cond condition) pages over its row index, so total_rows, pages and the page contents count only the records that MATCH — and from_rowid is a position among those, not a global rowid. An unfiltered iterator has no index, and its positions are the global rowids. See tranger2_open_iterator().

An unfiltered iterator pages the key as it is: its segments are taken from the cache again when the key’s cache moved since they were taken (its rows, its number of files, its last file), so the rows appended since the open are readable and total_rows / pages (the live count) can be paged to the end. A filtered iterator keeps the index and the segments of its open.

A delete of the key drops them: an unfiltered iterator takes its segments again at its next page (a key written again with the same number of rows and files, spread another way, is not the same key), and a filtered one pages an empty index -- the rows it indexed are gone. A page that finds the key gone from disk behind its back (a replica whose master deleted it) logs “Key gone from disk while its iterator was open” and stops.

json_t *it = tranger2_open_iterator(tranger, "topic", "K", NULL, NULL, "pager", "", NULL, NULL);
// K: 2 rows on day 1, 1 on day 2
tranger2_delete_key(tranger, "topic", "K");
// K written again: 1 row on day 1, 2 on day 2
json_t *page = tranger2_iterator_get_page(tranger, it, 1, 100, FALSE);
// page["data"]: the 3 new rows (7.25.4: 1 row, and a CRITICAL)

json_t *it = tranger2_open_iterator(tranger, "topic", "key", NULL, NULL, "it1", "", NULL, NULL);
/*  ... 3 records appended to "key" since ...  */
json_t *page = tranger2_iterator_get_page(tranger, it, 1, 100, FALSE);
/*  page["data"] holds the 3 new rows too; page["total_rows"] counts them  */

tranger2_iterator_size()¶

tranger2_iterator_size() returns the number of records in the specified iterator.

size_t tranger2_iterator_size(
    json_t *iterator
);

Parameters

Returns

The number of records the iterator will return.

Notes

A filtered iterator answers with its index: the exact number of records matching its match_cond. An unfiltered one sums its segments — the same thing when nothing is filtered out.

tranger2_list_keys()¶

Returns a JSON array with the key names of a topic, read from its in-memory cache.

json_t *tranger2_list_keys(
    json_t *tranger,
    const char *topic_name
);

Parameters

Returns

A JSON array of key-name strings. The caller owns the returned array and must free it when no longer needed. Never NULL — a missing topic yields an empty array.

Notes

The keys are from the topic’s in-memory cache, not a disk scan. Slow for thousands of keys (allocates one string per key).

tranger2_list_topic_desc_cols()¶

Returns a JSON array containing the column descriptions of a topic in the TimeRanger database. The returned JSON object must be decremented after use.

json_t *tranger2_list_topic_desc_cols(
    json_t      *tranger,
    const char  *topic_name
);

Parameters

Returns

A JSON array containing the column descriptions of the specified topic. The caller must decrement the reference count of the returned JSON object after use.

Notes

This function was previously known as tranger_list_topic_desc().

tranger2_list_topics()¶

Returns a JSON array with the names of the topics currently opened in memory (the tranger["topics"] registry) — not a disk scan.

json_t *tranger2_list_topics(
    json_t *tranger
);

Parameters

Returns

A JSON array with the names of the currently-opened topics. The caller owns the returned array and must free it. Never NULL — an empty array when no topic is opened.

Notes

This lists only topics already opened in memory. To list every topic present on disk — including ones not yet opened — use tranger2_list_topic_names().

tranger2_open_iterator()¶

Opens an iterator for traversing records in a topic within the TimeRanger database. The iterator allows filtering records based on specified conditions and supports real-time data loading.

json_t *tranger2_open_iterator(
    json_t *tranger,
    const char *topic_name,
    const char *key,    // required
    json_t *match_cond, // owned
    tranger2_load_record_callback_t load_record_callback, // called on LOADING and APPENDING, optional
    const char *iterator_id,     // iterator id, optional, if empty will be the key
    const char *creator,     // creator
    json_t *data,       // JSON array, if not empty, fills it with the LOADING data, not owned
    json_t *extra       // owned, user data, this json will be added to the return iterator
);

Parameters

Returns

Returns a JSON object representing the iterator. The topic owns it: close it with tranger2_close_iterator().

NULL (logged; match_cond and extra are consumed all the same) when the topic cannot be opened, key is empty, an iterator with this iterator_id and creator is already open, or the index of a filtered paging iterator cannot be built.

Notes

The iterator supports real-time data loading and filtering based on various conditions. Use tranger2_close_iterator() to release resources when done.

A load that stops half way says so. When the LOADING (a callback, or data) meets a row whose metadata or whose CONTENT cannot be read, it stops there, logs it, and leaves "load_failed": true in the iterator. The rows before it, in the load’s direction, were handed (forward: the oldest, backward: the newest). What the callback saw, and what data holds, is then NOT the whole history: a caller that asks the history a question (is any record of this key frozen by a snapshot?) must read the flag before it takes a “no”. Up to 7.25.4 a record whose content could not be read was handed to the callback as NULL and the load went on; an only_md load reads no content, and no content fails it.

After a restart too. The cache of a topic is built from disk when it is opened. A .md2 file of a key that it cannot count flags the key: one that cannot be opened or read. It logs “md2 file of the key unreadable when its cache was built: every load of the key says load_failed” once, at the open, and then every iterator of the key -- paging ones too -- logs “The history of the key is not whole: a md2 file of it could not be read when its cache was built” and says load_failed. A loading stops where the first flagged file is in its direction, the place a running tranger stops when the damage happens behind its back. Key A with the files of days 1, 2 and 3, and the .md2 of day 2 made unreadable (mode 0000) while the yuno was down:

forward load   -> the rows of day 1, then load_failed
backward load  -> the rows of day 3, then load_failed

A key directory that cannot be LISTED is not an empty key. When the cache build cannot open keys/<key>/ (EMFILE, EACCES, ENOMEM), cannot READ it (readdir() fails: EIO, ESTALE; 7.25.4 took that as the end of the directory, and the key was read without the files not listed yet), or has no memory for an entry of the listing, none of the key’s files is known, so the key is flagged whole: it logs “key directory cannot be listed when its cache was built: every load of the key says load_failed” once, and every iterator of the key logs “The history of the key is not whole: its directory could not be listed when its cache was built” and says load_failed with no row handed. A keyless tranger2_open_list() names the key in load_failed_keys, so treedb refuses a create of that id. An append into the key lists it again first: while the directory still cannot be listed the append is refused (“Cannot append record, its key cannot be listed: its row would follow rows no cell counts”); once it can, the key’s cache is built from the disk, the flag goes (“key directory listed again”, an INFO) and the append goes on. Up to 7.25.4 the key loaded as an EMPTY key with load_failed false, and an append counted its row as the first of its file.

keys/A of mode 0000 at the open:
iterator of A              -> no row, load_failed
keyless list               -> the rows of B, load_failed, load_failed_keys ["A"]
append into A              -> -1 (refused)
keys/A of mode 02770 again:
append into A              -> 0, the key listed again
iterator of A              -> every row of A, the new one last, load_failed false

When keys/ itself cannot be listed, no key can be flagged by name: the topic is not opened (see tranger2_open_topic()).

A flag goes when its cause goes, on a replica too. A flag is not kept until the topic is opened again:

• A load tries the flags again first. tranger2_open_iterator() (and so tranger2_open_list()) of a flagged key lists the key again when its directory changed since it was flagged (its mode or its entries: ctime) or could not be opened then and opens now, and counts a flagged .md2 again when it changed since it was flagged (its mode or its content: ctime or size) or could not be opened then and opens now. A flag whose directory or file looks as it did is not tried: a directory that cannot be read is not listed, a damaged file not read, and their failure not logged, at every load. What still cannot be read stays flagged, and the load says load_failed with nothing more in the log. So a directory whose readdir() recovers without changing stays flagged until it changes, until an append of the master or a notification of a replica lists it (both list it whatever it looks like), or until the topic is opened again (tests/c/timeranger2/test_unlisted_relist_once):

  keys/A opens, its readdir() fails (EIO) at the open -> A flagged, logged once
  three loads of A                   -> load_failed, nothing more in the log
  readdir() works again, keys/A as it was -> a load: load_failed, not listed
  chmod keys/A (its ctime moves)     -> a load: INFO key directory listed
                                        again, every row, load_failed false

• A replica’s notification lists the key again. When the master appends to a key the replica could not list, the notification of the new row (an rt_disk feed) lists the key first. The rows of the notified file, which nobody was handed, are published to the feeds with their rowids in the WHOLE key; the rows of the key’s other files are history, for a load. While the directory still cannot be listed, nothing is read (“New records of the key not read: its directory cannot be listed”).

• A master’s append counts a flagged file, and lists an unlisted key, before it writes (see above).

replica opens, keys/A of mode 0000     -> A flagged unlisted, feed "rtALL" open
chmod 02770 keys/A; the master appends A (its 4th row)
replica, at the notification           -> INFO key directory listed again,
                                          rtALL gets the row, rowid 4
replica: iterator of A                 -> A@1 A@2 A@3 A@4, load_failed false

In 7.25.4 a key whose directory could not be listed loaded as an empty key (see above), and the notification counted the notified file alone as the whole key (the feed got rowid 1 for the key’s 4th row).

A .md2 of 0 rows gets no cell and flags nothing. With an EMPTY .json it loses nothing. With a .json that is NOT empty it is what an append that was never acknowledged leaves (the content is written first, the md2 row after, and the row was never written: the md2 failed, or the process died between the two). That is not damage: the file is ignored, as until 7.25.4, with a warning that names it, and the key loads from its other files:

WARNING: md2 file of the key with no rows and a content file that is not empty:
         an append that was never acknowledged, the file is ignored
         topic_directory=<store>/devices key=A file_id=2000-01-02 content_size=32
forward load   -> the rows of days 1 and 3, load_failed false

The next append into that day’s file names its content with a row of its own, and the warning goes. A .md2 cut to 0 bytes behind the yuno’s back has the same shape, and loses the rows of its file the same way it did until 7.25.4: a .json much larger than one record is the sign; put the pair back from a backup.

A .md2 whose last row is torn is not damage either. Its size is not a whole number of 32-byte rows: a power cut came during the write of a row. The md2 row is the commit point of an append, and an append is acknowledged only after the whole row is written, so a torn row is an append that was never acknowledged. A MASTER cuts the .md2 back to its whole rows (to floor(size / 32) * 32 bytes) when it first reads the file, logs one warning, and the key loads whole. The next append goes into the same file:

WARNING: md2 file of the key ends in a part of a row: an append that was never
         acknowledged was cut back
         topic=devices key=A file_id=2000-01-02
         path=<store>/devices/keys/A/2000-01-02.md2 old_size=109 new_size=96
forward load   -> the rows of days 1, 2 (3 rows) and 3, load_failed false

The cut removes fewer than 32 bytes, all after the last whole row. The .json is left as it is: its bytes after the last row belong to no row, and the next append writes at its end. If the cut itself fails, that is damage: the file is flagged as above (“Cannot cut back a md2 file that ends in a part of a row: the file is damaged”). A torn first row (a .md2 of fewer than 32 bytes) is cut to 0 bytes, and the file is then a .md2 of 0 rows, ignored with the warning above.

Not every tail on no row boundary is a torn row. 7.25.4 did not cut a torn row back: the next appends wrote their rows after the torn bytes, on no row boundary, and those rows were acknowledged. In such a file the last 32 bytes are a whole row, and a cut removes the end of it. So the tail is cut only when it is a torn row after good rows. A row is good when its content is WHOLE: inside the .json, one json value and one NUL byte after it, as an append writes it (or only zero bytes, for an instance deleted with its content zeroed, see tranger2_delete_instance()). A NUL inside a string is written as the escape \u0000, so it is part of the value and is not a NUL byte. A content that the process that checks it has not the memory to parse is not a content that is not whole: the largest memory block (gbmem_get_maximum_block()) is set by MEM_MAX_BLOCK for each yuno, and the yuno that wrote it can have a larger block. A content larger than that block is read in parts: a NUL before its last byte says that it is not whole, with no block of its size, and when its only NUL is its last byte the check cannot tell. A content that fits in the block can still need more to parse (a long string, a long array or object: jansson doubles its buffers and tables); when its parse fails for memory, the check cannot tell either. The tail is not cut on a content that the check cannot tell. The rules:

1. the last 32 bytes of the .md2, read as a row, are NOT a row of their own: their content does not end exactly at the end of the .json (__offset__ + __size__ equal to its size), and it is not whole; and

2. the last whole row is good, and so is the whole row before it, whose content ends at or before the start of the content of the last one.

Rule 1 stops the cut of every file that 7.25.4 left whose last row still names its content as 7.25.4 wrote it, also when the .json has content after the last row (an append killed between its content and its md2 row; 7.25.4 did not cut that content back): the last row is a real row, and its content is whole wherever the .json ends. When the process that checks has not the memory to parse that content, the check cannot tell, and that is enough to not cut. Only a content damaged since (its bytes changed) lets such a file reach rule 2. For a torn row, the last 32 bytes are the end of the last whole row and the start of the torn one: read as a row, their __offset__ and __size__ are bytes moved out of their fields. They name a whole record of the .json, to the byte, only by a coincidence of 64-bit values, and then the file is flagged, not cut. Rule 2 stops a cut when the rows before the tail are not good. Example: a topic with no tkey (its __tm__ is 0, as in a treedb topic) and records of 256 bytes, so every __offset__ is a multiple of 256. There, a row moved by one byte names a range INSIDE the .json (the real one divided by 256). A test of the range only takes it for a good row. It is not whole: it does not start where a record starts.

When a rule fails, the file is NOT cut: a CRITICAL names the shape and the cause, the file is flagged as above, and the key says load_failed until the file is repaired by hand (see the repair in treedb). The bytes of the file do not change:

CRITICAL: md2 file of the key ends in a whole row that is not on a row boundary:
          written by 7.25.4 after a torn row; not cut, repair it by hand
          cause="its content is a whole record of the content file"
          topic=devices key=A file_id=2000-01-02
          path=<store>/devices/keys/A/2000-01-02.md2 md2_size=173
          content_size=242 row_at=141 __offset__=160 __size__=32
CRITICAL: md2 file of the key ends in a part of a row after a last whole row
          that is not valid: not cut, repair it by hand
          cause="its content is not a record"
          topic=devices key=A file_id=2000-01-02 ... row_at=64

The cause of the first is “its content ends the content file”, “its content is a whole record of the content file” or “its content has no NUL but the one at its end, and this process has not the memory to parse it (MEM_MAX_BLOCK): it can be a record written by a yuno with a larger block”. The cause of the second is one of “its content is not inside the content file”, “its content is not a record”, “this process has not the memory to parse its content (MEM_MAX_BLOCK): it cannot be checked”, “the whole row before it is not a good row” or “this process has not the memory to parse the content of the whole row before it (MEM_MAX_BLOCK): it cannot be checked” (for these two, row_at is that row), and “its content starts before the end of the content of the row before it”.

Example: a master with a block of 64 KiB opens a file that 7.25.4 left, whose last row names a record of 100 KiB, written by a yuno with a larger block. The file is flagged, and no row is lost. The same when the record is a string of 40 000 bytes, or an array in 20 000 bytes: it fits in the block, and its parse does not (the log also has the ERROR “SIZE GREATER THAN MAX_BLOCK” of the allocation that failed):

CRITICAL: md2 file of the key ends in a whole row that is not on a row boundary:
          written by 7.25.4 after a torn row; not cut, repair it by hand
          cause="its content has no NUL but the one at its end, and this process
          has not the memory to parse it (MEM_MAX_BLOCK): it can be a record
          written by a yuno with a larger block"
          topic=devices key=A file_id=2000-01-01 md2_size=159 row_at=127
          __offset__=1024 __size__=102400

When the check cannot RUN, a CRITICAL says so, and the file is not cut. It is one of three:

• “Cannot check the torn tail of a md2 file, its content file cannot be read: not cut”: the .json cannot be opened (EMFILE, ENFILE, EACCES) or read. With the path of the .json.

• “Cannot check the torn tail of a md2 file, no memory to read a content: not cut”: with the path of the .json and the size asked for.

• “Cannot read a record of md2 file, read FAILED” or “Cannot read a record of md2 file, short read”: a row of the .md2 cannot be read. With the path of the .md2, the row (end, last or before last) and its offset.

CRITICAL: Cannot check the torn tail of a md2 file, its content file cannot be
          read: not cut
          topic=devices key=A file_id=2000-01-02
          path=<store>/devices/keys/A/2000-01-02.json errno=13 serrno="Permission denied"

At an open the file is flagged too: without the check, the whole rows can be the moved rows of the shape 7.25.4 left. A master’s next append into the file counts it again, and the check runs then. At an append the file is not flagged (see tranger2_append_record()).

A replica makes the same check, and flags the file the same way: it does not read rows that are not on a row boundary.

A REPLICA never writes. It reads only the whole rows of such a file, and logs nothing for the torn row: a replica also sees a torn row while a live master is writing it, and counts the row when the master’s next notification of the file comes. A file with no whole row (fewer than 32 bytes) is a .md2 of 0 rows for the replica, so when its .json is not empty the replica logs the 0-rows warning above, and does not cut. When the master opens the store, it cuts the file back. Up to 7.25.4 the cache build logged a CRITICAL (“Cannot read last record, md2 file corrupted”) and left the whole file out of the key, on a master and on a replica: its acknowledged rows were missing from every load, and nothing failed.

The flag of a file goes once a cell counts it again: an append into the file that finds it readable (see tranger2_append_record()), on a follower the file read whole from the disk after the master wrote it, or a load of the key once the file changed or opens again (see A flag goes when its cause goes below). tranger2_delete_key() clears every flag with the key. Up to 7.25.4 the cache build dropped an unreadable file and nothing failed: the key read as a shorter key.

json_t *data = json_array();
json_t *it = tranger2_open_iterator(
    tranger, "devices", "dev-1", NULL, NULL, "", "me", data, NULL
);
if(!it) {
    // not opened, the cause is in the log
} else if(json_is_true(json_object_get(it, "load_failed"))) {
    // data holds only a part of the history: do not decide on it
}
if(it) {
    tranger2_close_iterator(tranger, it);
}
JSON_DECREF(data)

A deleted key. A delete of the key (tranger2_delete_key(), or a replica’s key-deleted notice) drops what its iterators took from it: an unfiltered iterator takes its segments again at its next page, a filtered one keeps an empty index (its rows are gone, and an index is built only at the open).

match_cond — the conditions an iterator honors

backward
only_md                 (don't load jn_record on calling callbacks; honored by
                        the historical load AND the realtime rt_mem/rt_disk
                        feeds, which hand the callback NULL jn_record)

rkey                    (str) PCRE2 regex over keys ("" == ".*"), for a KEYLESS
                        list (no `key`) = the whole topic, narrowed to the keys
                        matching it. Honored by BOTH halves: the disk load only
                        visits matching keys, and the realtime feed only
                        dispatches appends of matching keys.

from_rowid / to_rowid
from_t   / to_t         t:  PERSISTENCE time (when the record was appended)
from_tm  / to_tm        tm: MESSAGE time (the record's tkey field)

user_flag / not_user_flag / user_flag_mask_set / user_flag_mask_notset

t and tm are two independent axes and their ranges combine (AND). Both are expressed in the topic’s own unit: seconds, or milliseconds when the topic sets sf_t_ms / sf_tm_ms.

A negative t / tm bound is relative to the key’s last record: its t, and its tm -- the tm of the last row of the file that holds the key’s highest t, not the highest tm of the key (which a replica and a reloaded master do not know: they read the first and last rows of each file). from_t = -N takes the records after last_t - N (that time excluded); to_t = -N, the ones up to last_t - N. The bound is resolved once, when the iterator opens, and written back into its match_cond; a list of several keys gives each key’s iterator its own copy, so each key is bounded by its own last record (before v7 the bound was relative to the TOPIC’s last record, of any key). The realtime half of a list of several keys keeps the bound unresolved -- there is no one last record for all of them -- so a negative from bounds nothing there and a negative to takes no new record. On a key with no record yet a negative from bounds nothing and a negative to takes no row. A db_history that starts with the last day of its key:

json_t *match_cond = json_pack("{s:I}",
    "from_t", (json_int_t)-86400        // the last 24 h before the last record
);

Up to 7.25.22 a negative bound reached the comparison raw, against an unsigned __t__, and no record matched -- without a word. (Before v7 it already meant “relative to the last record”; that is the meaning restored.)

The two axes are not ordered the same way, and the scan knows it:

• Rows are in t order, except in a file that holds a late record (a __t__ below the times the file already had). The master marks that file <file>.unordered, and inside a marked file the scan reads every row of the selected files instead of stopping at the first row past the t range.

• Nothing orders tm. It is written by the producer (a device that sends what it buffered writes it out of order), and the files are cut by t. A tm condition leaves no file out and ends no scan: it skips rows, one by one. Correct, and it reads every md2 row of the key, so its cost grows with the files of the key: ~15 ms for a query of one minute on one key of 30 day files x 20000 rows (performance/c/perf_timeranger2; the match condition is parsed once per scan, not per row -- it was ~0.15 s). Every scan (an iterator’s load, its index, its pages, a follower’s new rows) reads the md2 rows in blocks of 1024, one pread() each, never past the rows its segment counts; up to 7.25.22 it made an lseek() and a read() per 32-byte row (~0.44 s for that query). The fr_tm / to_tm of a file in list-keys come from its first and last rows unless the file was read whole (a late __t__): approximate when its tm goes back. From 7.25.5 to 7.25.22 the master marked a file whose tm went back and the scan trusted the others (7.4 ms there, after a migration mark-tm-order); the markers and the command are gone, and the ones a store has are ignored.

• The marker goes down before the row. The master writes the marker of a record whose t goes back BEFORE its md2 row, so a process that dies between the two leaves a marker with no row behind it (one whole read of the file at the next load), never a row with no marker. A marker that cannot be written is logged (“Cannot mark md2 file, a reload will misread its time range”); the master still reads that file whole, and writes the marker at the next append to the file (“md2 file marked, the marker missed earlier is written”). A file whose name leaves no room for a marker (<file_id>.unordered longer than NAME_MAX) is logged once (“Cannot mark md2 file, file_id too long”) and read whole at every load. Neither is fsync’ed, like the append itself. A marker lost all the same (a power cut, a rollback to a binary that appends without markers) is not written again: that file’s t range is read from its first and last rows until it gets a marker. (tranger2_mark_tm_order(), which could write it again, went with the tm markers after 7.25.22.)

• A file left out by t (a t range around a file whose rows are out of it) is a hole in the rowids the scan walks: the scan steps over it, and a from_rowid / to_rowid that falls in it begins at the next row the scan can read. A hole is not an error. The scan logs an ERROR only when the segments overlap the row it just read, a broken invariant: “next segment begins before the row just read” (forward) or “previous segment ends after the row just read” (backward), with cur_rowid and the segment’s first or last row (7.25.4 logged “next rowids not consecutive” for every hole, and lost the rows after it).

Both hold in both directions, and for a paged iterator too. With the rows t=100 E1, t=50000 E2, t=200 E3 (late) in one file:

json_t *data = json_array();
json_t *it = tranger2_open_iterator(
    tranger, "readings", "0000000000000000001",
    json_pack("{s:I, s:I, s:b}",
        "from_t", (json_int_t)150,
        "to_t", (json_int_t)250,
        "backward", 0
    ),
    NULL, "range", "", data, NULL
);
// data holds E3 (up to 7.25.3: nothing, the scan stopped at E2)
tranger2_close_iterator(tranger, it);
JSON_DECREF(data)

And with three day files of one key, tm 100, 5000 and 150, the middle one is out of to_tm = 300 and the scan steps over it:

json_t *data = json_array();
json_t *it = tranger2_open_iterator(
    tranger, "readings", "K",
    json_pack("{s:I}", "to_tm", (json_int_t)300),
    NULL, "range", "", data, NULL
);
// data holds D1 D3 (7.25.4: D1, and "next rowids not consecutive")
tranger2_close_iterator(tranger, it);
JSON_DECREF(data)

Every condition is honored per record, in both modes of the iterator:

• LOADING (a load_record_callback or both data): each record is matched as the history is walked.

• PAGING (neither): a filtered iterator builds its row index when it opens, so tranger2_iterator_size() and tranger2_iterator_get_page() count and return exactly the matching records, and get_page’s from_rowid is a position among THOSE rows. An unfiltered iterator builds no index — its open stays cheap regardless of the key size, and its positions are the global rowids.

tranger2_open_list()¶

tranger2_open_list() opens a list of records in memory, optionally enabling real-time updates via memory or disk.

json_t *tranger2_open_list(
    json_t *tranger,
    const char *topic_name,
    json_t *match_cond, // owned
    json_t *extra,      // owned, will be added to the returned rt
    const char *rt_id,
    BOOL rt_by_disk,
    const char *creator
);

Parameters

Returns

Returns a JSON object representing the real-time list (rt_mem or rt_disk), or, when real-time is disabled (to_rowid set), the extra given, tagged "list_type": "no_rt". Close either with tranger2_close_list().

NULL (logged; match_cond and extra are consumed) on error: no topic, no load_record_callback, a bad rkey, a real-time feed that cannot be opened, no extra for a no_rt list -- and when the history of THE key of a list of one key (key set) could not be loaded whole (the load_failed of tranger2_open_iterator()).

A keyless list with a key that cannot be loaded logs the key (“Cannot load the whole history of a key of the list: the records read before the failure were handed, the list goes on with the next key”), loads every other key, opens its real-time feed, and says what it lacks in the handle it returns:

{
    "list_type": "rt_mem",
    "load_failed": true,
    "load_failed_keys": ["B"]
}

The records already handed to the callback stay handed: of a failed key, the ones read before the failure (forward: its oldest, backward: its newest). A caller that answers a question from the list must read the flag, or it takes “not in the list” for “not on disk”:

json_t *list = tranger2_open_list(tranger, "devices", match_cond, extra, "", FALSE, "");
if(list && json_is_true(json_object_get(list, "load_failed"))) {
    json_t *keys = json_object_get(list, "load_failed_keys");
    // these keys are on disk and not in what the callback saw
}

What the callers in the SDK do with it:

Until 7.25.4 such a list was handed over silently, with the key missing.

Notes

Loading all records can introduce delays in application startup. Use filtering conditions in match_cond to optimize performance.

The history is loaded with one-shot iterators of their own creator (__tranger2_open_list__): an iterator the caller keeps open on a key does not make the load “already exist”.

json_t *list = tranger2_open_list(
    tranger, "devices",
    json_pack("{s:I, s:I}",
        "to_rowid", (json_int_t)1000000,    // one-shot, no realtime
        "load_record_callback", (json_int_t)(uintptr_t)load_cb
    ),
    json_object(),      // extra: IS the returned handle of a no_rt list
    "", FALSE, "me"
);
if(!list) {
    // refused (logged)
} else {
    if(json_is_true(json_object_get(list, "load_failed"))) {
        // some key's history could not be read: what load_cb got is not the
        // whole topic, and "load_failed_keys" names those keys
    }
    tranger2_close_list(tranger, list);
}

tranger2_open_rt_disk()¶

Opens a real-time disk-based iterator for monitoring changes in a topic. The function allows tracking new records appended to the topic by monitoring disk events.

json_t *tranger2_open_rt_disk(
    json_t *tranger,
    const char *topic_name,
    const char *key,        // if empty receives all keys, else only this key
    json_t *match_cond,     // owned
    tranger2_load_record_callback_t load_record_callback,   // called on append new record on disk
    const char *rt_id,      // rt disk id, REQUIRED
    const char *creator,
    json_t *extra           // owned, user data, this json will be added to the return iterator
);

Parameters

Returns

Returns a JSON object representing the real-time disk iterator. The caller does not own the returned object.

Notes

This function is used when monitoring real-time changes in a topic via disk events. The iterator remains active until explicitly closed using tranger2_close_rt_disk().

The rowid the callback receives is the key’s global g_rowid — the one that never resets — not the row’s position inside the .md2 file it lives in (that one is i_rowid, and it restarts at 1 with every new file). A consumer can therefore dedupe or page by it across a file rotation.

Several feeds can be open on the same key (a per-key one and a whole-topic one, say). Each is served exactly once per record: the feed keeps a watermark of the last row it was given, per key and per file -- one mark per FILE, so a batch that touches two files of a key (a late record and a current one, or a rotation while the follower is busy) serves each feed both (before 7.25.0, the second feed lost both) -- and the master hard-links each new .md2 into the directory of every feed that wants the key. A feed’s watermark dies with its key, so a key re-created later does not inherit it.

tranger2_open_rt_mem()¶

Opens a real-time memory stream for a given topic in tranger. This function enables real-time message processing for the specified key and applies filtering conditions from match_cond. The callback tranger2_load_record_callback_t is invoked when new records are appended.

json_t *tranger2_open_rt_mem(
    json_t *tranger,
    const char *topic_name,
    const char *key,        // if empty receives all keys, else only this key
    json_t *match_cond,     // owned
    tranger2_load_record_callback_t load_record_callback,   // called on append new record on mem
    const char *list_id,    // rt list id, optional (internally will use the pointer of rt)
    const char *creator,
    json_t *extra           // owned, user data, this json will be added to the return iterator
);

Parameters

Returns

Returns a JSON object representing the real-time memory stream. The caller does not own the returned object.

Notes

This function is valid when the Yuno instance is the master writing real-time messages from tranger2_append_record().

tranger2_open_topic()¶

The tranger2_open_topic() function opens a topic in the TimeRanger database. If the topic is already open, it returns the existing topic JSON object.

json_t *tranger2_open_topic(
    json_t  *tranger,
    const char  *topic_name,
    BOOL  verbose
);

Parameters

Returns

A JSON object representing the opened topic. The returned object is not owned by the caller and must not be modified or freed.

Notes

This function is idempotent. This means that calling it multiple times with the same topic_name will return the same JSON object without creating a new instance.

It returns NULL, never a critical, when the name is refused by the topic name rule, and when the directory exists but is not a topic (it has no topic_desc.json): “Not a topic: topic_desc.json not found”, logged only with verbose. A peer can send any name, and with on_critical_error=2 a critical is an exit(0) of the yuno.

It also returns NULL when the keys/ directory of the topic exists and cannot be listed (EMFILE, EACCES, ENOMEM, or a readdir() that fails): “Cannot list the keys of the topic” (with errno), then “Cannot open topic: its keys cannot be listed”. On a filesystem whose readdir() gives no d_type (XFS with ftype=0, NFS, FUSE, overlay) each key is asked with lstat(), as the entry type tells it: a symbolic link is not a key (up to 7.25.4 stat() followed it, and made a key of another key). An lstat() that fails with EACCES (keys/ can be read and not searched) takes the entry as a key, as the entry type does, and the listing of that key says the EACCES and flags it; one that fails with anything else but ENOENT (a key removed meanwhile) fails the listing the same way: “Cannot list the keys of the topic, lstat() FAILED”. Up to 7.25.4 that key was taken as “not a directory” and left out with no log, and the topic opened without it. The topic is not left open: opened with no keys, a key nobody read looked like a key that does not exist, and a treedb accepted a create of its id (so up to 7.25.4). The next open, or the next tranger2_topic(), tries again:

json_t *topic = tranger2_open_topic(tranger, "devices", TRUE);
if(!topic) {
    // Error already logged: not a topic, a refused name, or keys/ cannot be listed.
    // Nothing of the topic is in memory; try again later.
    return -1;
}

A keys/ that does not exist (ENOENT) is still a topic with no key, as before.

tranger2_print_md0_record()¶

Prints metadata of a record, including row ID, time, message time, and key, into a buffer.

void tranger2_print_md0_record(
    char *bf,
    int bfsize,
    const char *key,
    json_int_t rowid,
    const md2_record_ex_t *md_record_ex,
    BOOL print_local_time
);

Parameters

Returns

This function does not return a value.

Notes

The function formats the metadata into the provided buffer. This makes sure of it does not exceed bfsize. It prints row ID, time, message time, and key.

tranger2_print_md1_record()¶

Prints metadata information of a record, including row ID, user flag, system flag, timestamps, and key, into a buffer.

void tranger2_print_md1_record(
    char *bf,
    int bfsize,
    const char *key,
    json_int_t rowid,
    const md2_record_ex_t *md_record_ex,
    BOOL print_local_time
);

Parameters

Returns

This function does not return a value.

Notes

The function formats and writes metadata details into the provided buffer. This makes sure that the output does not exceed bfsize bytes.

tranger2_print_md2_record()¶

Prints detailed metadata of a record, including row ID, offset, size, timestamp, and file path, into the provided buffer.

void tranger2_print_md2_record(
    char *bf,
    int bfsize,
    json_t *tranger,
    json_t *topic,
    const char *key,
    json_int_t rowid,
    const md2_record_ex_t *md_record_ex,
    BOOL print_local_time
);

Parameters

Returns

This function does not return a value.

Notes

The function formats metadata details into the provided buffer. This makes sure that the output remains within the specified buffer size.

tranger2_print_record_filename()¶

Formats and stores the filename of a record in a buffer, using metadata from tranger2_print_record_filename().

void tranger2_print_record_filename(
    char *bf,
    int bfsize,
    json_t *tranger,
    json_t *topic,
    const md2_record_ex_t *md_record_ex,
    BOOL print_local_time
);

Parameters

Returns

This function does not return a value.

Notes

The buffer bf must be large enough to store the formatted filename. The function retrieves metadata from tranger2_print_record_filename() to construct the filename.

tranger2_read_record_content()¶

Reads the content of a record from a given topic in the TimeRanger database. The function retrieves the record’s data based on its metadata.

json_t *tranger2_read_record_content(
    json_t            *tranger,
    json_t            *topic,
    const char        *key,
    md2_record_ex_t   *md_record_ex
);

Parameters

Returns

Returns a json_t * object containing the record’s content. The caller owns the returned JSON object and must free it when no longer needed.

Notes

This function is useful when only metadata was loaded and the full record content needs to be retrieved.

tranger2_read_user_flag()¶

The tranger2_read_user_flag() function retrieves the user flag associated with a specific record in a given topic.

uint16_t tranger2_read_user_flag(
    json_t *tranger,
    const char *topic_name,
    const char *key,
    uint64_t __t__,
    uint64_t rowid
);

Parameters

Returns

Returns the uint16_t user flag associated with the specified record.

Notes

The function is used in writing mode to check the user flag of a record before modifying it.

tranger2_set_trace_level()¶

Sets the trace level of the tranger instance, controlling the verbosity of logging and debugging output.

void tranger2_set_trace_level(
    json_t *tranger,
    int      trace_level
);

Parameters

Returns

This function does not return a value.

Notes

Higher trace levels typically enable more detailed logging, which can be useful for debugging.

tranger2_set_system_flag()¶

Sets or clears specific bits in the system flag of a record — the metadata band reserved for the framework (for example the immutable-record bit), distinct from the user flag set by tranger2_set_user_flag().

int tranger2_set_system_flag(
    json_t *tranger,
    const char *topic_name,
    const char *key,
    uint64_t __t__,
    uint64_t rowid,
    uint16_t mask,
    BOOL set
);

Parameters

Returns

Returns 0 on success, or a negative error code on failure.

Notes

Only the bits in mask are touched. The system flag is framework-owned. User code normally goes through higher-level treedb helpers rather than setting system bits directly.

Master only: on a replica, or on a master that lost its lock (see tranger2_stop()), it writes nothing, logs “Only master can write” and returns -1. Up to 7.25.4 it asked nothing: the write of the read-only md2 fd failed, which is a CRITICAL, and with the default on_critical_error the process exited(0).

tranger2_set_user_flag()¶

Sets or clears specific bits in the user flag of a record in a topic within the TimeRanger database.

int tranger2_set_user_flag(
    json_t *tranger,
    const char *topic_name,
    const char *key,
    uint64_t __t__,
    uint64_t rowid,
    uint16_t mask,
    BOOL set
);

Parameters

Returns

Returns 0 on success, or a negative error code on failure.

Notes

This function modifies only the bits specified in mask. This leaves other bits in the user flag unchanged.

// Mark row 3 of key "dev-1" (its __t__ is 1731601280) as pending, keep the other bits
if(tranger2_set_user_flag(tranger, "messages", "dev-1", 1731601280, 3, 0x0001, TRUE) < 0) {
    // not the master, or the row is not there: logged
}

Master only: on a replica, or on a master that lost its lock (see tranger2_stop()), it writes nothing, logs “Only master can write” and returns -1. Up to 7.25.4 it asked nothing: the write of the read-only md2 fd failed, which is a CRITICAL, and with the default on_critical_error the process exited(0).

tranger2_shutdown()¶

The tranger2_shutdown() function stops the TimeRanger database, releasing all allocated memory.

int tranger2_shutdown(
    json_t *tranger
);

Parameters

Returns

Returns 0 on success, or a negative value if an error occurs.

Notes

This function must be called when the database is no longer needed to free resources.

tranger2_startup()¶

Initializes the TimeRanger 2 database, setting up its internal structures and preparing it for use. The function requires a hgobj instance and a JSON configuration object that defines database parameters.

json_t *tranger2_startup(
    hgobj      gobj,
    json_t    *jn_tranger,
    yev_loop_h yev_loop
);

Parameters

Returns

A JSON object representing the initialized TimeRanger 2 database instance.

Notes

The returned JSON object must be properly managed and eventually passed to tranger2_stop() or tranger2_shutdown() to release resources.

tranger2_stop()¶

The tranger2_stop() function closes the TimeRanger database. This makes sure that all topics and file descriptors are properly closed.

int tranger2_stop(
    json_t *tranger
);

Parameters

Returns

Returns 0 on success, or a negative value if an error occurs.

Notes

tranger2_shutdown() calls it when nobody did. Call it yourself when the event loop needs a turn between the two, to finish the watchers the topics close.

What the stop does:

• It closes every topic, and the fds of fd_opened_files: the __timeranger2__.json lock of a master is one of them, so the stop gives the single-master lock back. Each closed fd is marked -1: a second stop, or the shutdown, never closes that number again (by then it can belong to somebody else).

• A tranger stays usable after the stop. The first call that opens a topic again, or that WRITES (tranger2_create_topic(), _delete_topic(), _backup_topic(), _write_topic_var(), _write_topic_cols(), _append_record(), _delete_key(), _delete_instance(), _write_user_flag(), _set_user_flag(), _set_system_flag()), revives it: the shutdown then closes what the revival opened, and a master takes its lock again BEFORE anything is written. If another process took the store in the meantime (flock() answers EWOULDBLOCK), it is the startup’s single-master conflict: the revival logs a CRITICAL at on_critical_error, “Master lock NOT retaken after a stop: another process holds it, go on as not master”. With a yuno’s default ("on_critical_error": 2, exit(0)) the process exits there and stays down, as a second instance does at the startup. Any OTHER failure to take the lock says nothing of another master and is an ERROR, not a critical, and the process goes on: “Master lock NOT retaken after a stop: cannot open the lock file, go on as not master” (EMFILE, ENOENT...) or “... flock() FAILED, go on as not master” (ENOLCK, EINTR...). A tranger that did not take its lock back (and did not exit) goes on as a replica: it reads, every write is refused, and its json says "master": false, "master_lost": true. It never takes the lock again, not even once it is free -- its memory did not follow what the other master wrote; shut it down and start it again to be the master. Read master again after a restart: it is what the tranger holds now. A lookup of a topic that is already open revives nothing.

• A TRANSIENT failure demotes for good too. An EMFILE (too many open files for a moment), an EINTR or an ENOLCK at the revive is logged as an ERROR, not a critical: the process does NOT exit, it goes on as a replica, and it stays one after the cause is gone. Every write it is asked for is refused from then on (each with its own error), and nothing else says so. Nothing restarts it: the operator does. Watch for it:

  • the log line “Master lock NOT retaken after a stop” (any of its three endings), at ERROR or CRITICAL;

  • the master attribute of the C_TRANGER service, which answers what the tranger IS (false once demoted). Read it with view-attrs WITHOUT attribute=: that form goes through the gclass’ mt_reading; the one-attribute form answers the stored configuration.

    ycommand -c 'command-yuno id=<id> service=__yuno__ command=view-attrs gobj=<tranger service>'
    # "master": false on a yuno configured as the master = demoted: restart it

  • in C, master_lost true in the tranger’s json.

  Restart the yuno (kill-yuno + run-yuno) once the cause is found: the new process takes the lock at its startup.

This is the life of the tranger of a C_TRANGER that is stopped and started again: the tranger is built in mt_create, stopped in mt_stop, and shut down only in mt_destroy.

json_t *tranger = tranger2_startup(0, jn_tranger, yev_loop);   // master, lock taken
tranger2_open_topic(tranger, "devices", TRUE);

tranger2_stop(tranger);         // topics closed, lock given back
                                // (C_TRANGER mt_stop)

tranger2_topic(tranger, "devices");  // opens it again: revived, lock retaken
                                     // (C_TRANGER started again)

tranger2_shutdown(tranger);     // closes what the revival opened

And when another process took the store while it was stopped:

tranger2_stop(a);                                   // a gives the lock back
json_t *b = tranger2_startup(0, jn_tranger_b, 0);   // b takes the store

tranger2_create_topic(a, "devices", "id", "tm", 0, 0, 0, 0);
// a cannot take the lock: nothing written, "devices" opened read-only
kw_get_bool(0, a, "master", 0, 0);        // FALSE
kw_get_bool(0, a, "master_lost", 0, 0);   // TRUE

tranger2_str2system_flag()¶

Converts a formatted string containing system flag representations into a system_flag2_t integer. The input string can use delimiters such as ‘|’, ’ ', or ‘,’.

system_flag2_t tranger2_str2system_flag(
    const char *system_flag
);

Parameters

Returns

Returns the corresponding system_flag2_t integer representing the parsed system flags.

Notes

This function is useful for converting human-readable flag representations into their corresponding bitmask values.

tranger2_topic()¶

Retrieve a topic by its name from the TimeRanger database. If the topic is not already opened, tranger2_open_topic() is called to open it.

json_t *tranger2_topic(
    json_t      *tranger,
    const char  *topic_name
);

Parameters

Returns

A JSON object representing the topic. The returned object is not owned by the caller and must not be modified or freed.

Notes

If the topic exists on disk but has not been opened yet, tranger2_open_topic() is automatically invoked.

tranger2_topic_desc()¶

tranger2_topic_desc() retrieves the description of a specified topic from the TimeRanger database.

json_t *tranger2_topic_desc(
    json_t *tranger,
    const char *topic_name
);

Parameters

Returns

A JSON object containing the topic description. The caller must decrement the reference count of the returned JSON object.

The description is a projection of the topic, not the topic itself:

A key the topic does not carry is omitted rather than returned empty, so a topic with no secondary key has no pkey2s at all.

pkey2s matters to any reader that must NAME a record and not only identify it. A topic whose id column is flagged rowid, uuid or qualified keys its records by a value that is not the name: a rowid and a UUID mean nothing to a person, and a qualified id (<parent id>.<name>) names every ancestor together with the record. treedb_system_schema keys topics and cols with qualified ids. In all three the name lives in the secondary key. A viewer given only pkey can do nothing but print the address.

Notes

The returned JSON object must be properly decremented using json_decref() to avoid memory leaks.

tranger2_topic_key_size()¶

Retrieves the number of records associated with a specific key in a given topic within the TimeRanger database. If key is empty, the whole-topic size is returned instead.

uint64_t tranger2_topic_key_size(
    json_t  *tranger,
    const char *topic_name,
    const char *key
);

Parameters

Returns

Returns the number of records associated with the specified key, or the whole-topic total when key is empty.

Notes

Counts are from the in-memory cache. A missing topic returns 0. When key is empty the call delegates to tranger2_topic_size().

tranger2_topic_key_range()¶

Returns the time span of one key of a topic, on both axes, read from the in-memory cache totals (maintained on load and on every append) — so a client can bound a time picker to what the key holds without reading a single record.

json_t *tranger2_topic_key_range( // return is yours
    json_t *tranger,
    const char *topic_name,
    const char *key
);

Parameters

Returns

A JSON object, yours to decref:

{"fr_t": t, "to_t": t, "fr_tm": tm, "to_tm": tm, "rows": n}

NULL (silent) if the topic or the key is unknown.

Notes

t (persistence) and tm (message time) are independent, so the two spans differ whenever data was backfilled. Both are in the topic’s own unit: seconds, or milliseconds when the topic sets sf_t_ms / sf_tm_ms (read system_flag from the topic desc).

tranger2_topic_is_open()¶

TRUE if the topic is currently open in this tranger. Silent — a closed topic is a legitimate answer, not an error.

BOOL tranger2_topic_is_open(
    json_t *tranger,
    const char *topic_name
);

Parameters

Returns

TRUE if the topic is open, FALSE otherwise.

Notes

tranger2_topic_name()¶

Retrieves the topic name from the given json_t * topic object.

const char *tranger2_topic_name(
    json_t *topic
);

Parameters

Returns

A pointer to a string containing the topic name. The returned string must not be modified or freed by the caller.

Notes

If the topic parameter is NULL or invalid, the behavior is undefined.

tranger2_topic_size()¶

tranger2_topic_size() retrieves the total number of records present in a specified topic within the TimeRanger database.

uint64_t tranger2_topic_size(
    json_t *tranger,
    const char *topic_name
);

Parameters

Returns

Returns the total number of records in the specified topic as a uint64_t value.

Notes

If the topic does not exist, the function can return 0.

tranger2_write_topic_cols()¶

The tranger2_write_topic_cols() function updates the column definitions of a specified topic in the TimeRanger database.

int tranger2_write_topic_cols(
    json_t  *tranger,
    const char  *topic_name,
    json_t  *jn_cols  // owned
);

Parameters

Returns

Returns 0 on success, or a negative error code on failure.

Notes

Master-only. Replaces the topic’s columns wholesale — both the in-memory topic["cols"] and the on-disk topic_cols.json (not a merge). A columns change must bump the topic’s topic_version (and schema_version for structural changes), otherwise the persisted topic_cols.json masks the new schema on the next reload.

The file is replaced, never written in place: topic_cols.json.new is removed if a process left it, created O_EXCL|O_NOFOLLOW with the tranger’s rpermission, written, fsync’ed, renamed over the old file, and the directory fsync’ed: it survives a power cut too (it is written when a topic is created, re-versioned or re-ordered, never per record; up to 7.25.4 it was not fsync’ed, and a topic_version change could leave the new version on disk over cols lost with the power). The memory takes the new columns only when the file did; otherwise the call logs (“Cannot replace topic_cols.json, ...”) and returns -1 with the old file and the old columns in place. Up to 7.25.4 it wrote in place, put the columns in memory before writing, and returned 0 whatever happened. tranger2_write_topic_var() replaces topic_var.json the same way.

json_t *cols = json_pack("{s:s, s:s, s:s}", "id", "", "content", "", "extra", "");
if(tranger2_write_topic_cols(tranger, "readings", cols) < 0) {  // cols consumed either way
    // not written: topic_cols.json and topic["cols"] are the old ones
}

One change does not wait for the bump. When the incoming schema holds the same columns saying the same things and only their order differs, tranger2_create_topic() rewrites the file: the freeze is there so that a change to what a column declares cannot arrive unannounced, and an order announces nothing new.

tranger2_write_topic_var()¶

The tranger2_write_topic_var() function updates the variable metadata of a specified topic in the TimeRanger database.

int tranger2_write_topic_var(
    json_t  *tranger,
    const char  *topic_name,
    json_t  *jn_topic_var
);

Parameters

Returns

Returns 0 on success, or a negative value on failure.

Notes

Master-only. Merges jn_topic_var into the existing topic_var.json (update, not replace) and persists it. The in-memory topic is updated too, except the immutable descriptor fields, and only when the file was written.

The file is never rewritten in place: the merge is written to topic_var.json.new and renamed over topic_var.json, so a process that dies half way leaves the old file or the new one, never an empty one. treedb calls this on every create of a rowid-key node (the last_rowid_id counter lives here), so it does not fsync(): it is safe against the death of the process, not against a power cut. A topic_version change (tranger2_create_topic()) does fsync.

A write that fails leaves the old topic_var.json as it was and answers -1. A write cut part way (the file size limit, a full disk) is logged as “Cannot replace topic_var.json, short write: the file size limit or the disk is full”, without an errno: a short write does not set one.

tranger2_write_topic_var(tranger, "items", json_pack("{s:I}", "last_rowid_id", (json_int_t)42));
// topic_var.json: {"topic_version": 3, "last_rowid_id": 42}  (a new inode)

tranger2_write_user_flag()¶

The tranger2_write_user_flag() function updates the user flag of a specific record identified by rowid in the given topic_name within the TimeRanger database.

int tranger2_write_user_flag(
    json_t *tranger,
    const char *topic_name,
    const char *key,
    uint64_t __t__,
    uint64_t rowid,
    uint16_t user_flag
);

Parameters

Returns

Returns 0 on success, or a negative value on failure.

Notes

This function modifies the user flag of an existing record but does not alter other record attributes. Use tranger2_set_user_flag() if you need to update the flag using a mask.

Master only: on a replica, or on a master that lost its lock (see tranger2_stop()), it writes nothing, logs “Only master can write” and returns -1. Up to 7.25.4 it asked nothing: the write of the read-only md2 fd failed, which is a CRITICAL, and with the default on_critical_error the process exited(0).

tranger2_list_topic_names()¶

tranger2_list_topic_names() returns a JSON array of topic names by scanning the tranger database directory on disk. Unlike tranger2_list_topics(), which reads from the in-memory topic registry, this function reads subdirectory names from the filesystem.

json_t *tranger2_list_topic_names(
    json_t *tranger
);

Parameters

Returns

A new JSON array of strings, each being a topic name found as a subdirectory in the tranger database directory. The caller owns the returned array and must call json_decref() on it. Hidden entries (names starting with .) are excluded.

NULL when the directory cannot be listed (opendir() or readdir() fails): logged, “Cannot list the topics of the store” with errno, and the cause is left in gobj_log_last_message(). A store that cannot be listed is not a store with no topic: up to 7.25.4 it answered [] with no log, and list-queues and clean-queues of the MQTT broker answered an empty list with 0. They answer -1 now.

Notes

This function operates on disk, not in memory. It can return topic names that are not currently open, or miss topics that exist only in memory. Use tranger2_list_topics() to get the in-memory list instead.

Example

json_t *names = tranger2_list_topic_names(tranger);
if(!names) {
    // Error already logged: the store cannot be listed, NOT "no topic"
    return -1;
}
size_t idx; json_t *jn_name;
json_array_foreach(names, idx, jn_name) {
    printf("%s\n", json_string_value(jn_name));
}
JSON_DECREF(names)

tranger2_topic_path()¶

tranger2_topic_path() writes the filesystem path of a topic into the provided buffer. The path is constructed by appending the topic name to the tranger database directory.

int tranger2_topic_path(
    char *bf,
    size_t bfsize,
    json_t *tranger,
    const char *topic_name
);

Parameters

Returns

Returns 0 on success.

Notes

The resulting path has the format <directory>/<topic_name>, where <directory> is the tranger database directory stored in the tranger handle.