Debugging a yuno

This document covers what you actually do when something inside a running yuno is misbehaving: how to turn the right knobs to see what’s happening, where the output lands, how to follow a single message end-to-end through several yunos, and how the centralised log aggregator (logcenter) fits in.

Companion to YUNO_LIFECYCLE.md. That one covers how the agent manages yunos; this one covers how to look inside them.

1. Mental model¶

Three observation layers are independent. Confusing them is the first source of frustration:

Two destinations, configurable per yuno via daemon_log_handlers in the yuno config JSON:

                        ┌──────────────────────────┐
       severity logs    │     file handler         │  → /yuneta/logs/<yuno>/<mask>.log
       + traces  ──────►│     (rotatory, ~8 MB)    │
                        └──────────────────────────┘
                        ┌──────────────────────────┐
                        │     udp handler          │  → udp://host:port
                        │     (default :1992)      │  → typically the logcenter yuno
                        └──────────────────────────┘
                        ┌──────────────────────────┐
                        │     stdout (console mode)│  → terminal when not daemonised
                        └──────────────────────────┘
                        ┌──────────────────────────┐
                        │     remote_log over      │
                        │     ievent / websocket   │  → SPA "dev panel" (live viewer)
                        └──────────────────────────┘

Same log line can fan out to all four destinations at once. None of them is “the” log — they are just different sinks.

2. Severity levels (gobj_log_*)¶

These are the calls every gclass uses to record events. They are not traces — they fire regardless of trace settings. The six public ones are defined in kernel/c/gobj-c/src/glogger.c:

Plus two parallel non-syslog channels declared in glogger.c:51-52:

• LOG_AUDIT (8) — written without the standard header. Used for command audit. Tooling that filters on the standard timestamp format will skip these lines — read the audit file directly.

• LOG_MONITOR (9) — used by monitoring tools.

Per-yuno HARD RULE (see CLAUDE.md): every error-return path calls gobj_log_error or carries an // Error already logged comment. If you can’t find the error in the log, it’s a bug in that yuno, not “the log lost it”.

3. Trace categories¶

Tracing is the off-by-default running commentary the framework can emit. It is noisy — only turn it on when needed, and turn it off when done.

3.1 Global trace levels¶

Defined in s_global_trace_level[16] at kernel/c/gobj-c/src/gobj.c:328-345:

These are global bits — when on, every gobj in the yuno is affected.

3.2 Per-gclass trace levels¶

Each gclass declares its own up-to-16 levels in s_user_trace_level[16]. Example: c_tcp_s.c:97-108

enum {
    TRACE_LISTEN        = 0x0001,
    TRACE_NOT_ACCEPTED  = 0x0002,
    TRACE_ACCEPTED      = 0x0004,
    TRACE_TLS           = 0x0008,
};
PRIVATE const trace_level_t s_user_trace_level[16] = {
    {"listen",          "Trace listen"},
    {"not-accepted",    "Trace not accepted connections"},
    {"accepted",        "Trace accepted connections"},
    {"tls",             "Trace tls"},
    {0, 0},
};

The names are gclass-specific. Common ones across runtime gclasses:

• C_TCP, C_TCP_S: traffic, connect, tls, listen, accepted, not-accepted

• C_PROT_HTTP_SR, C_PROT_HTTP_CL: traffic

• C_IEVENT_SRV, C_IEVENT_CLI: ievents, ievents2 (the second dumps full kw)

• C_WEBSOCKET: gclass-specific debug for HTTP-upgrade handshakes

• C_TIMER, C_TIMER0: their own tick / periodic

Discover what a gclass actually offers with get-gclass-trace gclass=<X> (see §4).

3.3 Per-gobj trace levels¶

Same as per-gclass but scoped to a single gobj instance. Useful when you have ten TCP connections and only want the trace for one. API: gobj_set_gobj_trace() at kernel/c/gobj-c/src/gobj.c:11256.

3.4 The no_trace parallel system¶

For every “set trace” command there is a “set no-trace” counterpart. The no-trace mask is subtracted from the effective trace mask, so you can turn on a noisy level globally and silence it on specific gclasses / gobjs. Functions: gobj_set_global_no_trace() at gobj.c:11396, gobj_set_gclass_no_trace() at gobj.c:11617, gobj_set_gobj_no_trace() at gobj.c:11746.

3.5 Deep trace mode¶

gobj_set_deep_tracing(level) at gobj.c:11338-11346 forces all traces on regardless of masks. Not exposed as a ycommand — only via the C API, mostly used internally for emergency dumps. Avoid unless you are willing to deal with the volume.

4. Turning traces on and off¶

All commands go to the yuno itself, addressed to its __yuno__ service. Handlers in kernel/c/root-linux/src/c_yuno.c:403-412:

# discover what a gclass offers
ycommand -c 'command-yuno id=<yuno> service=__yuno__ command=get-gclass-trace gclass=C_TCP_S'

# enable / disable
ycommand -c 'command-yuno id=<yuno> service=__yuno__ command=set-global-trace level=machine set=1'
ycommand -c 'command-yuno id=<yuno> service=__yuno__ command=set-gclass-trace gclass=C_TCP_S level=traffic set=1'
ycommand -c 'command-yuno id=<yuno> service=__yuno__ command=set-gobj-trace gobj=<short_name> level=machine set=1'

# silence (no_trace)
ycommand -c 'command-yuno id=<yuno> service=__yuno__ command=set-global-no-trace level=ev_kw set=1'
ycommand -c 'command-yuno id=<yuno> service=__yuno__ command=set-gclass-no-trace gclass=C_TIMER level=periodic set=1'

# inspect current state
ycommand -c 'command-yuno id=<yuno> service=__yuno__ command=get-global-trace'
ycommand -c 'command-yuno id=<yuno> service=__yuno__ command=get-gclass-trace gclass=C_TCP_S'
ycommand -c 'command-yuno id=<yuno> service=__yuno__ command=get-gobj-trace gobj=<short_name>'

CLAUDE.md’s shorthand ycommand -c 'command-yuno id=<id> service=__yuno__ command=…' is exactly this. The default-yuno shortcut ycommand -c 'set-global-trace …' is in fact command-yuno to whatever yuno is registered as default.

Persistence¶

• Global traces are persisted: c_yuno.c:2021-2070 (save_global_trace) writes them to the yuno’s trace_levels attribute, and they are re-applied on the next start.

• Gclass and gobj traces are live-only. They die with the process.

Implication: a forgotten set-global-trace level=machine set=1 will survive a restart and quietly fill your disk. Always pair on/off in the same session.

5. Reading the logs¶

5.1 File paths¶

Per-yuno log file, built by yuneta_log_file() at kernel/c/gobj-c/src/yunetas_environment.c:141-156:

/yuneta/logs/<yuno_role_plus_name>/<filename_mask>

The actual mask is whatever you set in daemon_log_handlers.<handler>.filename_mask (see §5.4). Conventionally <role>-W.log (the W is replaced by a rotation counter).

Active log discovery:

ls -lt /yuneta/logs/<yuno>/
tail -f /yuneta/logs/<yuno>/<latest>.log | grep -a "keyword"

5.2 Log line format¶

Every log record is a JSON object built in glogger.c:1025-1048. Fields added automatically by discover() at glogger.c:1234-1304:

Searching is JSON-friendly:

grep -a '"priority":3' /yuneta/logs/<yuno>/<file>.log       # all errors
grep -a '"gclass":"C_TCP_S"' /yuneta/logs/<yuno>/<file>.log  # one gclass
grep -a '"msg":"Event NOT DEFINED in state"' …               # the canonical FSM bug

5.3 Rotation¶

The rotatory library rotates the file when it crosses a size threshold (default 8 MB, configurable via max_megas_rotatoryfile_size, entry_point.c:542). Old files are renamed; the active filename never moves. There is no time-based rotation. There is no cron — rotation happens on the next write that would cross the threshold.

5.4 Where to configure handlers¶

In the yuno’s config JSON, under environment.daemon_log_handlers (or console_log_handlers in non-daemon mode), parsed at kernel/c/root-linux/src/entry_point.c:507-580:

"environment": {
    "daemon_log_handlers": {
        "to_file": {
            "handler_type": "file",
            "filename_mask": "mqtt_broker-W.log",
            "handler_options": 255
        },
        "to_udp": {
            "handler_type": "udp",
            "url": "udp://127.0.0.1:1992",
            "handler_options": 255
        }
    }
}

handler_options is a bitmask of LOG_HND_OPT_* (glogger.h:25-39) selecting which severities the handler accepts. 255 = all of them; clearing bits drops DEBUG / INFO / AUDIT / etc.

Add or remove handlers at runtime via c_yuno.c:3350-3450 (add-log-handler / del-log-handler commands).

6. The FSM trace (machine)¶

Single most useful trace when debugging gobj behaviour. Defined in glogger.c:1161 (trace_machine). Called from the event dispatcher in kernel/c/gobj-c/src/gobj.c:

• Before dispatch (gobj.c:7507-7539): a 🔜 line per event entry.

• “Event NOT DEFINED” error (gobj.c:7539): a 📛 line — this is the canonical “parent FSM doesn’t declare the child’s event” failure (see CLAUDE.md “CHILD vs SERVICE” section).

• After dispatch (gobj.c:7576-7590): a 🔄 line per executed event.

• State change (gobj.c:7725 → 7775): a 🔀🔀 line.

Two output formats, switched by the integer variable trace_machine_format:

Format 1 — short, ANSI-coloured:

🔜 EV_RX_DATA !!c_tcp:open
🔄 EV_RX_DATA !!c_tcp :open from !!service_main
🔀🔀 mach(!!c_tcp), new st(:closed), prev st(:open)

Default — verbose:

🔜 mach(!!c_tcp), st: :open, ev: EV_RX_DATA, from(!!service_main)
🔄 mach(!!c_tcp), st: :open, ev: EV_RX_DATA, from(c_tcp_s^server)
🔀🔀 mach(!!c_tcp), new st(:closed), prev st(:open)

!! before a name means the gobj is not running at that moment. Two of those in a row is usually the bug.

Scoping the machine trace¶

The whole-yuno machine trace is overwhelming on anything bigger than a toy test. Two ways to narrow:

# only one gclass
ycommand -c 'command-yuno id=<yuno> service=__yuno__ command=set-gclass-trace gclass=C_TCP_S level=machine set=1'

# only one instance
ycommand -c 'command-yuno id=<yuno> service=__yuno__ command=set-gobj-trace gobj=<short_name> level=machine set=1'

Both are live-only — they vanish on restart.

7. Following a message end-to-end¶

Canonical request flow on a typical Yuneta service:

   external client
         │
         ▼
  ┌─────────────┐   gclass trace 'traffic'  (c_tcp_s.c:360, 566)
  │   C_TCP_S   │   gobj_trace_dump_gbuf(gobj, gbuf, …)
  └──────┬──────┘
         │
         ▼
  ┌─────────────────┐   gclass trace 'traffic'  (c_prot_http_sr.c:57, 285, 370)
  │ C_PROT_HTTP_SR  │
  └────────┬────────┘
           │
           ▼
  ┌─────────────────┐   gclass trace 'ievents' / 'ievents2'  (c_ievent_srv.c:88-90, 483)
  │   C_IEVENT_SRV  │   trace_inter_event2(gobj, prefix, event, kw)
  └────────┬────────┘
           │
           ▼
  ┌─────────────────┐   global trace 'machine' lights up the FSM dispatch
  │  service gclass │   gclass-specific traces fire its custom emit points
  └────────┬────────┘
           │
           ▼
  ┌─────────────────┐   global trace 'fs'  (timeranger2.c:754, 2361)
  │   timeranger2   │   record append + rowid emitted
  │   (treedb)      │
  └────────┬────────┘
           │
           ▼  outbound publish
  ┌─────────────────┐   gclass trace 'ievents' / 'ievents2'
  │   C_IEVENT_SRV  │
  └────────┬────────┘
           │
           ▼
  ┌─────────────────┐   gclass-specific trace
  │  C_WEBSOCKET    │   gobj_trace_dump frames
  └────────┬────────┘
           │
           ▼
        SPA browser

The correlation id¶

Inter-event messages between yunos carry a metadata block named __md_iev__ inside the kw. Inside it is the ievent_gate_stack — a LIFO of hops, each entry: {src_yuno, src_service, dst_yuno, dst_service, user, host, …}.

• Constant IEVENT_STACK_ID = "ievent_gate_stack" at kernel/c/gobj-c/src/msg_ievent.h:92.

• Pushed on outgoing request, popped + reversed on incoming response, at kernel/c/gobj-c/src/msg_ievent.c:327-340.

To grep the same transaction across multiple yunos’ logs:

grep -a 'ievent_gate_stack' /yuneta/logs/*/*.log | grep '<the user or src_yuno you care about>'

There is no automatic UUID propagated for non-ievent calls (a direct C function call has nothing to grep). The correlation is only available when the message actually traverses an ievent boundary.

Practical sequence to follow one HTTP request¶

YUNO=my_service_01

# 1. ingress + protocol
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gclass-trace gclass=C_TCP_S        level=traffic set=1"
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gclass-trace gclass=C_PROT_HTTP_SR level=traffic set=1"

# 2. internal FSM
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-global-trace level=machine set=1"

# 3. broker/topic write
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-global-trace level=fs set=1"

# 4. egress to SPA
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gclass-trace gclass=C_IEVENT_SRV level=ievents  set=1"
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gclass-trace gclass=C_IEVENT_SRV level=ievents2 set=1"

# trigger the request, capture the noise
tail -F /yuneta/logs/$YUNO/*.log > /tmp/$YUNO.trace &
# … reproduce …
kill %1

# disable everything
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gclass-trace gclass=C_TCP_S        level=traffic  set=0"
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gclass-trace gclass=C_PROT_HTTP_SR level=traffic  set=0"
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-global-trace level=machine set=0"
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-global-trace level=fs      set=0"
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gclass-trace gclass=C_IEVENT_SRV level=ievents  set=0"
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gclass-trace gclass=C_IEVENT_SRV level=ievents2 set=0"

8. The SPA-side “dev panel” viewer¶

When a SPA built on the JS gobj framework is connected to a yuno, it can display live what crosses the websocket — the same lines you’d see in the local log file, plus the ievent message bodies.

Wire-up¶

• C side: nothing special — inter-events flow as usual through C_IEVENT_SRV → C_WEBSOCKET.

• JS side: C_IEVENT_CLI (kernel/js/gobj-js/src/c_ievent_cli.js) parses incoming inter-events in ac_on_message() at c_ievent_cli.js:1092 and, if configured, invokes trace_ievent_callback(prefix, iev_msg, direction, size) at c_ievent_cli.js:1111.

• The SPA installs that callback by writing the attribute: gobj_write_attr(gobj_yuno(), "trace_ievent_callback", info_traffic) (kernel/js/lib-yui/src/yui_dev.js:146, 432, 537).

• The info_traffic() function (yui_dev.js:29) appends the message into the DOM container #developer-traffic-logger, which lives inside either:

  • the legacy C_YUI_WINDOW modal (yui_dev.js:405-424), or

  • the modern build_dev_panel() modal (yui_dev.js:452-548).

Both still ship — apps pick one based on the shell version.

Rendering¶

• Header bar: blue, with direction icon (white = sent, yellow = received, red = error), byte size, and HH:MM:SS.SSSS timestamp.

• Body: read-only vanilla-jsoneditor (pinned ^0.23.0, see project memory note about not bumping it).

• Footer: ON/OFF traffic counter.

• Auto-scroll to bottom on each message (yui_dev.js:130).

Filtering on the SPA side¶

None in the UI. The viewer shows whatever flows through the websocket. The only knobs are on/off — trace_inter_event boolean and the trace_ievent_callback itself. To “filter”, you change what the yuno emits using the ycommand knobs from §4.

Teardown order — the recursion gotcha¶

When the websocket closes, ac_on_close (c_ievent_cli.js:925) fires EV_ON_CLOSE. If set_remote_log_functions is still installed (it hijacks JS log_error/log_warning to the DOM callback), the warning emitted inside the teardown path is captured, mutates the DOM, which can fire more events, which log again — infinite recursion.

The fix at c_ievent_cli.js:903 is to call set_remote_log_functions(null) (clears the hooks, resets to console — see helpers.js:51-58) before anything publishes EV_ON_CLOSE. See memory note “Remote-log unwire order” for the historical incident.

9. The logcenter yuno¶

yunos/c/logcenter/ aggregates UDP-shipped logs from every yuno on the host (or LAN). It is not enabled by default — each yuno only ships to UDP if its config lists a udp handler.

How it listens¶

• UDP server (c_gss_udp_s) on udp://127.0.0.1:1992 by default (c_logcenter.c:117, 196-199).

• Wire format: <priority-digit><8hex-seq><json-payload><8hex-crc>, fragmented per udp_frame_size (default 1500, log_udp_handler.c:478-487).

What it does on receipt¶

c_logcenter.c:

• ac_on_message() at c_logcenter.c:1203 parses each packet (c_logcenter.c:1215-1260).

• Writes the JSON record to its own rotatory file W.log via write2logs() / _log_bf() (c_logcenter.c:849-854, 1290). Default size cap 8 MB (max_rotatoryfile_size, c_logcenter.c:122).

• Updates in-memory counters per severity / msgset / msg (do_log_stats(), c_logcenter.c:859-948).

What it exposes¶

Commands (c_logcenter.c:99-106):

Use it like any other yuno:

ycommand -c 'command-yuno id=logcenter service=__default_service__ command=display-summary'
ycommand -c 'command-yuno id=logcenter service=__default_service__ command=tail n=200'
ycommand -c 'command-yuno id=logcenter service=__default_service__ command=search match="EV_ON_CLOSE"'

Per-yuno vs centralised — when to use each¶

• Per-yuno tail when you know which yuno is misbehaving and want raw control over grep. /yuneta/logs/<yuno>/<file>.log is full fidelity.

• logcenter when you need correlation across multiple yunos, or when a yuno crashes too fast to read its own file, or for the rollup counters / email summaries.

Both can run at the same time — the file handler writes locally and the UDP handler ships to logcenter in parallel. They are not exclusive.

10. Sharp edges¶

10.1 Traces stack up; disable when done¶

A forgotten set-global-trace level=machine set=1 survives a restart (see §4 Persistence). Logs balloon. Always pair on/off in the same operational session, and double-check with get-global-trace before you leave.

10.2 Persistence asymmetry¶

If you persisted a global level by mistake, clear it explicitly: set-global-trace level=<name> set=0. Removing the file does not help — the value lives in the yuno’s treedb config.

10.3 ievent_gate_stack is only on inter-event hops¶

A direct C function call between gobjs in the same yuno does not carry the stack — there’s no metadata to attach. The correlation only exists across yuno boundaries. Plan your traces accordingly.

10.4 LOG_AUDIT lines have no standard header¶

glogger.c:51 — audit lines are written raw. Any line-filter that expects the timestamp prefix will miss them. Read the audit file directly when chasing operator actions.

10.5 UDP can drop¶

UDP logs are not reliable. Under burst (e.g. machine trace fully on) the kernel buffer can overflow and logcenter loses lines silently. The local file handler does not drop — when in doubt, trust the local file.

10.6 ev_kw is enormous¶

set-global-trace level=ev_kw set=1 dumps every event’s full kw JSON payload to the log. Useful on a single-shot test; ruinous on a busy service. Combine with gclass-scoped machine trace if you need it narrowly.

10.7 SPA dev-panel teardown order¶

set_remote_log_functions(null) MUST come before do_disconnect / destroy_shell. See §8 and memory feedback_remote_log_unwire_order.

10.8 Deep tracing has no ycommand switch¶

gobj_set_deep_tracing() is C-only (gobj.c:11338-11346). If you find a yuno generating apparently-unconfigurable traces, look for a gobj_set_deep_tracing call in its mt_create that someone left in.

11. Operational recipes¶

11.1 Just watch what a yuno is doing¶

YUNO=<id>
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-global-trace level=machine set=1"
tail -F /yuneta/logs/$YUNO/*.log | grep -a '"msg":'
# reproduce
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-global-trace level=machine set=0"

11.2 Watch traffic on a TCP/HTTP service¶

ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gclass-trace gclass=C_TCP_S        level=traffic set=1"
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gclass-trace gclass=C_PROT_HTTP_SR level=traffic set=1"
tail -F /yuneta/logs/$YUNO/*.log
# … done …
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gclass-trace gclass=C_TCP_S        level=traffic set=0"
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gclass-trace gclass=C_PROT_HTTP_SR level=traffic set=0"

11.3 Watch all logs from this host in one place¶

Enable logcenter. In every yuno’s config JSON add:

"daemon_log_handlers": {
    "to_udp": { "handler_type": "udp", "url": "udp://127.0.0.1:1992", "handler_options": 255 }
}

Then:

ycommand -c 'command-yuno id=logcenter service=__default_service__ command=tail n=500'
ycommand -c 'command-yuno id=logcenter service=__default_service__ command=search match="<keyword>"'
ycommand -c 'command-yuno id=logcenter service=__default_service__ command=display-summary'

11.4 Follow one request end-to-end¶

See §7 — the block of set-gclass-trace ... traffic + set-global-trace ... machine + set-global-trace ... fs + set-gclass-trace C_IEVENT_SRV ... ievents2 is the kit. Don’t forget to disable everything afterwards.

11.5 Capture an FSM bug in one gobj only¶

ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gobj-trace gobj=<short_name> level=machine set=1"
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gobj-trace gobj=<short_name> level=ev_kw   set=1"
tail -F /yuneta/logs/$YUNO/*.log | grep -a '<short_name>'
# … done …
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gobj-trace gobj=<short_name> level=machine set=0"
ycommand -c "command-yuno id=$YUNO service=__yuno__ command=set-gobj-trace gobj=<short_name> level=ev_kw   set=0"

These are live-only — even if you forget the off-step, a yuno restart clears them.

11.6 Spot the canonical “Event NOT DEFINED in state” error¶

That single string at gobj.c:7541 is the most common FSM failure (parent FSM didn’t declare a child’s published event — see CLAUDE.md “CHILD vs SERVICE”). It’s logged at LOG_ERR regardless of trace settings, so:

grep -a '"msg":"Event NOT DEFINED in state"' /yuneta/logs/*/*.log

works on any host without enabling anything.

12. Code pointers¶