Activity log: one stream, two channels
Research and recommendation for RobotDiagnostics' Activity view, modelled on rqt_console · 2026-10-23 · Claude Code (Opus 5.5), scribe lane · repo ~/io-diagnostics · vault note: a private note
The answer
- Yes, one log. The Activity view becomes the device's single event stream on one clock: plugs and kernel port events, status transitions, the fault lifecycle, action runs and producers' own events. It uses rqt_console's columns and its exclude and highlight filters.20,21 Status reports and measurements stay out. They are the tree and the health bar, and only a change of level is an event. That is what AUTOSAR's fault memory did when it fed the log, and the opposite of ROS
diagnostic_updater's verbose mode, which warns on every period.39,26 - Make the Event record the log record, not the status. Keep
DiagnosticStatus, because it is state. AlignEventto ROSrcl_interfaces/Loglevels and to OpenTelemetry's LogRecord, and give it the stock projectionto_ros_log()/from_ros_log()besideto_diagnostic_status().22,35,12 The additions arereceived, thedebugandfatalseverities, and acountfor coalesced repeats. - The fault memory is a read model of the same history. It stays the current state, shown on the entity and on the chip. Its lifecycle rows sit in the log, folded onto the event that caused them, so one fact is one row. Today 13:34:07 and 18:11:44 each show twice.17 A fault's detail shows "breadcrumbs": that entity's log for the 30 s before it.66,27
- Entity scope means the entity and its children, plus an optional path to the host. Every entity gets a Raw channel: its kernel lines, its driver's own output, and later
/rosoutby logger name. Raw is off by default, like Windows' Analytic and Debug channels.34,28,38,42 - It is the device's log, not a journal viewer. In 44 h this PC's journal wrote 56,573 lines, and 226 of them are kernel lines about USB (0.4 %). The Activity log carries coded events only. The raw journal is one click away as the exact
journalctlcommand for that entity.19 The host should also keep its own events on disk, because the fault memory lives only in RAM and today's 15:00 reboot wiped it.12
"Model the Activity log after the ROS 2 log view (rqt_console)."Yes for its table and its filter model:
# · Message · Severity · Node · Stamp · Location, exclude and highlight lists, "show highlighted only", and pause. Three changes: Node becomes the entity, Location becomes the source, and identical repeats collapse into one row with ×N. In the docs' own screenshot, rows #339 to #342 are the same wall hit.20,21"Possibly the status format could be like the ROS log as well?"No for the status, yes for the event. ROS itself keeps
/diagnostics (state, structured, "for a person") apart from /rosout (occurrences), and records both side by side.25,23 What crosses over is a status transition, and it becomes an Event."It's basically the computer logging stuff… a local log viewer just for the computer logs?"Wider and narrower than that. Wider, because it is the computer and everything plugged into it, in the host's words. Narrower, because it holds only coded events and never the whole journal. GNOME Logs and
journalctl stay the full-journal tools.48,38"Diagnostics faults could possibly log to the logs?… a single stream you can read to get an entire picture of the device."Yes. It is the norm. AUTOSAR R4.1 had the fault memory (DEM) send a log message on every status change, carrying the DTC and freeze frame. ros2_medkit goes the other way and turns
/rosout WARN+ into faults.39,27 The rule that keeps the log readable: transitions in, reports out."What is best practice in robotics… everything into logs… filters… its raw log stream?"Robotics practice is two wire streams (
/rosout and /diagnostics), one recording (a bag or MCAP holds both), and one timeline in the viewer (Foxglove plays Log and Diagnostics panels on one clock).25,32,29 So the "single source of truth" is one append-only store with one clock, not one message type. An entity's raw stream exists in the field already: medkit serves /components/{id}/logs (all its nodes) and /apps/{id}/logs.28Contents
- What the Activity view shows today
- The sketch: rqt_console, rebuilt for devices
- Criteria and the four options
- Best practice, from primary sources
- How the pieces connect
- The record format
- 19 references, with what to steal and what to avoid
- What needs you
- Deliberately left out
- Source Index
1 · What the Activity view shows today
Three things in this capture shape the recommendation:
- One fact, two rows. At 13:34:07,
board fault"CAN 15 fault 33" is followed byfault confirmed"MoteusFault.33 confirmed". At 18:11:44,bus error passiveis followed byfault confirmedNoAck. Each is the producer's event plus the host's fault memory saying the same thing. An ERROR confirms at once (DEFAULT_DEBOUNCE), so the two rows always arrive together.12,13 - "Happened" and "noticed at start" are mixed. Four WARN faults confirm in the same second, 18:11:50: two UpdateAvailable, CameraMoved and PortFlapping. WARN confirms on its 3rd report, so these are the host's memory catching up after it started, not four things that happened at 18:11:50. The kernel-log side already handles the same problem: enumerations in the first 15 s of a boot fold into one
Bootedrow ("9 USB devices enumerated").14 - The fault memory and link health sat beside the log. Round 3 already settled that link health moves to the Computer and fault memory becomes a filter of the log.16 This report says what that filter is (§5).
2 · The sketch: rqt_console, rebuilt for devices
This is a working mock with real rows. Today is this PC's own journal for 10-23, including the 15:00 reboot. Rack session F1 is the 10-21 recording the app replays, with the proposed folding applied.19,17 Things to try:
- click an entity name to scope the log to it;
- switch the scope between "+ children" and "+ path to host";
- turn Raw on;
- type
level:warn,code:Fault,-code:Pluggedorsource:kernel; - add a highlight;
- open a row for its args, breadcrumbs and journal command.
| # | Time | Severity | Entity | Code | Message | Source |
|---|
How each control maps to a reference:
- Columns come from rqt_console.21
- Filter grammar comes from Logcat and Console.app.67,47
- Severity floor: Foxglove applies it even to search matches.29
- Volume strip: Grafana and lnav HIST.51,57
- Scope: Rerun's
$origin/**, andjournalctl's device-plus-ancestors match.34,38 - Raw: Windows' off-by-default Analytic and Debug channels.42
- Fault badges: Wireshark's expert info shown on the packet.59
- Breadcrumbs: Sentry.66
- Colour appears only on warn and error, under round 3's rule that warning colours are reserved for diagnostics.16
Columns, filters and live tail, as a spec
| Element | Behaviour | From |
|---|---|---|
| # | The host's sequence number for the row. Filters leave gaps, and the count line says how many rows are hidden; a gap the filters do not explain means rows were lost, which MAVLink events detect the same way | rqt_console #, MAVLink seq41 |
| Time | HH:MM:SS; hover shows ISO, source stamp and receive stamp; a boot boundary is a separator row | journalctl -- Boot --, OTel Timestamp vs ObservedTimestamp35 |
| Severity | debug · info · warn · error · fatal; only warn, error and fatal are coloured | ROS Log levels22 |
| Entity | The entity name, not its path; a click scopes the log to it | rqt_console Node, Foxglove namespaces29 |
| Code | The machine code (Plugged, NoAck, MoteusFault.33); empty on a raw line | K8s reason, journald MESSAGE_ID44,37 |
| Message | One line, with ×N for coalesced repeats and badges for the fault lifecycle ("→ fault confirmed") | K8s series.count, Chrome "group similar"44,56 |
| Source | Who said it: kernel, host, fault memory, moteus_servo, realsense | rqt_console Location, OTel InstrumentationScope |
| Exclude / Highlight | The same filter types in both lists: containing (regex), severities, entity, time range from a selection, source; plus "highlighted only" | rqt_console filter factory21 |
| Typed search | level:warn means that level and above; -key: negates; the same key is ORed, different keys are ANDed; saved queries become chips | Logcat, Console.app67,47 |
| Live tail | Newest at the top; scrolling away or opening a row pauses; a "Resume · n new" pill; the count of hidden rows is always shown | Foxglove follow playback, Grafana Pause/Resume, lnav "lines hidden"29,51,57 |
| Entity scope | + children (default; the via/part_of subtree, plus rows whose hardware_id is the entity's) · this only · + path to host (adds the ports, hubs and controller it hangs off) | Rerun $origin/**, medkit component logs, journalctl device match34,28,38 |
| Raw | Uncoded lines: the entity's kernel lines, its driver's stdout and stderr from diagnose and act, later /rosout by logger name. Off by default; on inside an entity's own log | Windows Analytic/Debug channels, medkit /apps/{id}/logs42,28 |
| Row detail | args, path, hardware_id; for a fault, its snapshot plus breadcrumbs (this entity's rows 30 s before); for a kernel row, the exact journalctl command | Sentry breadcrumbs, AUTOSAR freeze frame in the DLT payload66,39 |
3 · Criteria and the four options
The weights come from what you have said: "single stream… entire picture of the device", the Activity view being "very busy and hard for me to read", "wrap the wire format; don't fork it", and the black-box rule that one fact counts once.5,12,8
| Criterion | Weight | A · two views (today) | B · everything in one stream | C · one store, Activity + Raw | D · stock viewer as the view |
|---|---|---|---|---|---|
| One read gives the whole picture of the device | 25 | 2 | 5 | 5 | 3 |
| Calm by default: readable, repeats and raw lines out | 20 | 3 | 1 | 4 | 3 |
| Stock interop: rqt_console, Foxglove and journalctl read it with no UI code | 15 | 2 | 3 | 4 | 5 |
| One fact counts once (no fault + event doubles, knock-ons folded) | 15 | 3 | 2 | 4 | 2 |
| Per-entity scope and raw drill-down in the app | 15 | 2 | 4 | 5 | 2 |
| Small build on what exists (Event, coalescer, FaultManager) | 10 | 5 | 3 | 4 | 3 |
| Score (of 5) | 2.65 | 3.10 | 4.40 | 3.00 |
- B is your proposal taken literally: every report and raw line goes in, and filters do the rest. It fails on calm. Its default view is what
diagnostic_updater's verbose mode produces, the same warning every period.26 - D means publishing to
/rosoutand using Foxglove or Lichtblick as the Activity view. It is the best second view, but it has no USB skeleton, no entity hierarchy and no actions.7 So it stays an adapter. - C stays first under every single-weight shift of −10, +10 or +20. Its lowest score in the sweep is 4.33, against a best runner-up of 3.42.
4 · Best practice, from primary sources
- state: status, conditions, the alarm status table;
- occurrences: the log or event stream;
- the memory of failures: DTCs, faults, Repairs.
ROS 2: /rosout and /diagnostics are two topics, recorded together
rcl_interfaces/msg/Loghas the fieldsstamp,level,name(the logger),msg,file,functionandline. Levels are spaced like Python's: DEBUG 10, INFO 20, WARN 30, ERROR 40, FATAL 50.22- Three outputs per node (console, file and
/rosout), each switchable.23- Logger names form a hierarchy:
abc.defsits underabcand inherits its level. - The default console format is
[{severity}] [{time}] [{name}]: {message}.
- Logger names form a hierarchy:
/rosoutis a small flight recorder. Its default QoS keeps the last 1000 messages, reliable and transient-local, with a 10 s lifespan. A viewer that joins late gets the last ten seconds.24- REP-107 says what diagnostics are for.
- A quick glance, then detail, then "a long term logging infrastructure for historical analysis", done by recording
/diagnosticswith rosbag. - Its data is weakly typed because "the intended consumer is a person".
- The default rate is 1 Hz. It is meant for hardware, because adding it to every software component makes "too much noise".
- A quick glance, then detail, then "a long term logging infrastructure for historical analysis", done by recording
- The anti-pattern ships in ROS itself.
diagnostic_updater'sverbose_flag callsRCLCPP_WARN("Non-zero diagnostic status…")on every update cycle. It compares nothing with the last cycle, so a check that stays failing warns every period.26 - ros2_medkit bridges both ways and scopes logs per entity.27,28
ros2_medkit_log_bridgepromotes/rosoutentries at or above WARN into faults. The code is a hash of the normalised message template. WARN needs 3 in 10 s; ERROR has a 5 s cooldown.- The gateway serves
GET /components/{id}/logs(all its nodes, prefix match),GET /apps/{id}/logs(exact match), and alogs/configurationresource you can PUT.
Foxglove and MCAP: one file, many channels, one timeline
- MCAP records. Schema, Channel, Message, Metadata (name plus a key/value map) and Attachment.
/rosoutand/diagnosticsare two channels in one file.32 - The Log panel.29,30
- It reads
rcl_interfaces/msg/Log,rosgraph_msgs/Logorfoxglove.Log;foxglove.Log's levels are UNKNOWN 0 to FATAL 5. - The minimum level is "always enforced", even over a node selection or a search match.
- On recorded data, clicking a line seeks the whole layout to that moment.
- It reads
- Events. Foxglove Events annotate time ranges on a device with typed properties.31 They are the cloud cousin of a fault occurrence.
- Your July 2026 decision still holds. Own small domain envelopes and project them to each backend; don't build a generic logger.11
OpenTelemetry Logs Data Model: events and logs are one stream
- Two timestamps.
Timestampis the source's time;ObservedTimestampis when the collector saw it. - Severity.
SeverityNumberruns 1 to 24 in six bands (TRACE, DEBUG, INFO, WARN, ERROR, FATAL), and 0 means unspecified.SeverityTextkeeps the source's own word. - Resource vs Attributes.
Resourceis the producing entity and is the same for every record from it.Attributesvary per occurrence. - Events are log records. A record with a non-empty
EventNameis an Event.35 - Mappings exist. The appendix maps syslog, the Windows Event Log, systemd-journald, ETW, Log4j and others. Syslog Warning (4) maps to 13, Error (3) to 17, Critical (2) to 18.36
- The match with your contract is exact: Resource is your
Entity, sent once; Attributes areargs; EventName iscode; ObservedTimestamp isreceived.12
journald: structured fields, and a device match that walks up
Every journal entry already carries the fields a device log needs:37
- What was said:
MESSAGE, a 128-bitMESSAGE_IDfor the message type,PRIORITY0 to 7, andCODE_FILE/LINE/FUNC. - Trusted fields set by journald:
_BOOT_ID,_TRANSPORT=kernel,_KERNEL_DEVICEand_KERNEL_SUBSYSTEM. - Two times: the source time and the reception time.
journalctl /dev/node adds a match for that device and "each of its ancestor devices". -x adds catalog help text keyed on MESSAGE_ID, which is the journal's own "what to check".38 Here it is on this PC, run read-only for this report:19
The first two commands ran at 14:55, before the 15:00 reboot, when that boot was still -b. A device-node match only searches the current boot, so it cannot be rerun on the old boot now. The counts were rerun at 15:05 with -b -1 over the whole finished boot; at 14:55 they were 55,005 and 225. The JSON is trimmed to the fields shown.
AUTOSAR: fault memory (DEM) and log and trace (DLT) are two modules, and the fault memory used to log its transitions
- R4.1.3 wired DEM into DLT. Each time an event's status changes, DEM calls
Dlt_DemTriggerOnEventStatus(). DLT compares the old and new status and only when they differ builds a log message.39- The payload is EventId, DTC, all extended data records, and the most recent freeze frame (SWS_Dlt_00474–00479).
- It is sent with ApplicationID
DEMand ContextIDSTD0(SWS_Dlt_00377).
- Release 4.4.0 removed the link. The change log for 2018-10-31 reads "Interaction DLT <> DEM removed" and gives no reason.40 I read it as a coupling choice at the module level, not as a verdict against logging transitions.
- DLT's identity is a three-level hierarchy: ECU ID → Application ID → Context ID. "Each ApplicationID can own several Context IDs". Its levels run FATAL (0x01) to VERBOSE (0x06).40
- Apex.Grace follows the same monitor → fault manager → DTC chain. Its post never says faults also go to a log, so there is nothing to borrow for logging.45
Three more conventions worth copying
- Windows channels by audience, on one infrastructure.42,43
- Admin events must have "a well-defined solution on which the administrator can act".
- Operational events are for diagnosing.
- Analytic and Debug are high-volume, disabled by default, and enabled only to investigate.
- Levels run Critical, Error, Warning, Informational, Verbose.
- Kubernetes Events. A
reasonis "a short, machine understandable string".typeis Normal or Warning, plus the object it regards and the reporting component. A repeat is one Event withseries.countandlastObservedTime. Events are "informative, best-effort, supplemental data", never state.44 - MAVLink events. Each event is an ID plus typed args, with two log levels (external for the ground station, internal for logging). The text templates live in metadata. Per-sender sequence numbers let the receiver re-request lost events.41 That is your
code + args + seq, with one more idea: two audiences per event.
5 · How the pieces connect
The fault memory relates to the log as follows:
- The fault memory answers "what is wrong now and how often". It is a projection of status reports with debounce, keyed (entity, code). That is the state Ignition puts in its Alarm Status Table and Home Assistant in Repairs.69,64
- The log answers "what happened, in order". Its Faults preset is the lifecycle history, which is Ignition's Alarm Journal.
- Both exist already:
FaultManageremitsFaultConfirmed,FaultHealedandFaultClearedas Events withseq.12 The change is to fold those rows onto the producer's row for the same fact, and to give the confirmed row the snapshot (freeze frame) as args, as AUTOSAR's DLT payload did.39
6 · The record format
The changes are additive under record v0's own rule 8: new optional fields with defaults, and new string enum values.12 The status record does not change.
| Event field | Change | ROS rcl_interfaces/Log | OTel LogRecord | journald |
|---|---|---|---|---|
stamp | keep | stamp | Timestamp | _SOURCE_REALTIME_TIMESTAMP |
received | add: the observer's clock, as on status | none | ObservedTimestamp | __REALTIME_TIMESTAMP |
severity | add debug, fatal to info|warn|error | level 10 / 20 / 30 / 40 / 50 | SeverityNumber 5 / 9 / 13 / 17 / 21 | PRIORITY 7 / 6 / 4 / 3 / 2 |
code | keep; empty means a raw line | carried in function (and [Code] in msg) | EventName | MESSAGE_ID (by catalog) |
message | keep | msg | Body | MESSAGE |
args | keep; a confirmed fault carries its snapshot | rendered into msg | Attributes | custom fields |
path, hardware_id | keep | name = the path, dotted (host.usb1.port-1-2), so ROS's logger hierarchy gives "+ children" | Resource (the Entity) | _KERNEL_DEVICE (kernel lines) |
source | keep | file | InstrumentationScope | SYSLOG_IDENTIFIER |
seq | keep | line | none | __SEQNUM |
count, last_stamp | add: a coalesced repeat (×N) | none (one Log per repeat) | none | none (K8s series) |
Sources: 22,35,36,37,44. The function/file/line placement is a proposal. It gives a lossless round trip for code, source and seq, and it is the same trick caused_by uses on DiagnosticStatus.12
What goes in, and what stays out
| Kind | In the log? | Default view | Why |
|---|---|---|---|
| Boot, shutdown, plug, unplug, re-enumerate | yes, coded | Activity, info (re-enumerate: warn) | Already Events; coalesced flaps14 |
| Kernel USB errors, driver bind and unbind, PCIe AER | yes, coded | Activity | The peer lane's io_facts/kernel_log.py (uncommitted at 14:34) already turns these into coded notices with a count18 |
| Fault lifecycle (confirmed, healed, cleared) | yes | Activity, folded onto the causing row; Faults preset | One fact, one row; "found at start" groups the host's catch-up |
| Status level transitions | yes, code LevelChanged | Activity only when silence (→ stale or unknown) has no plug event above it; otherwise debug | A warn or error transition is already the fault row; silence is not a fault, but a person wants to see it |
| Action runs (started, finished, result, duration) | yes | Activity; Actions preset | Today's "utility" row17 |
Producer events (BoardFault, BusErrorPassive…) | yes | Activity | The black box says what happened to it |
Raw lines: kernel lines, driver stdout and stderr, later /rosout | yes, uncoded | Raw, inside an entity's log | The answer to "its raw log stream"; Windows Debug channel42 |
| Periodic status reports, including repeated OKs | no | tree, history, health bar | Reports are state; diagnostic_updater's verbose mode is the cautionary tale26 |
| Measurements (volts, rates, temperatures) | no | history and sparklines | Home Assistant's Activity excludes entities with a unit of measurement62 |
| The full OS journal | no | "Open in journal" command | About 1,275 lines an hour; GNOME Logs and journalctl already do it19 |
Stock adapters (proposed functions in contract/record.py; none exist yet)
to_ros_log(event, entities) → dict, arcl_interfaces/msg/Log. Its partnerfrom_ros_log(message, **added) → Eventtakes any/rosoutline as a raw line attached to the entity whose owner node matchesname. The same round-trip proof asDiagnosticStatus: stock → v0 → stock is the identity.12to_foxglove_log(event)(levels 1 to 5) andto_otel_log_record(event, entity)are pure mappings from the table above.30,35- Later, a journald sink. The host's own events would sit in the journal with
SYSLOG_IDENTIFIER=io-diagnosticsand IO_* fields next to the kernel lines, and GNOME Logs would become a second viewer for free.37 After that, an MCAP export that writes/rosoutand/diagnosticschannels plus the Entity tree as Metadata, so one incident plays in Foxglove.32
7 · 19 references, with what to steal and what to avoid
Screenshots were captured headlessly on 2026-10-23 from each product's public docs page (or the docs repo the page is built from). Full-resolution files are in ~/io-diagnostics/reports/media/research/logs/.
1 · rqt_console (ROS 2)
The reference you named: one message table over /rosout, with exclude and highlight filter lists below it.20,21
- The columns:
# · Message · Severity · Node · Stamp · Location, and the running "Displaying 342 messages". - Two filter lists built from one filter factory: …containing (regex), …with severities, …from node, …from time range (taken from the selected rows), …from location, custom.
- "Show highlighted only", pause, save and load.
- Identical repeats as separate rows: #339–#342 are one wall hit.
- Filter editors taking two-thirds of the window; use chips and the View-options popover instead.
- "Location" (a source file path), which means nothing for a USB port.
images/warn.png.2 · Foxglove Log panel (and Lichtblick, its open-source fork)
The robotics log panel. It reads ROS 1 and ROS 2 logs or foxglove.Log and plays them on the layout's shared clock.29,30
- The log is one panel on the same clock as the Diagnostics and Plot panels. Clicking a line seeks everything to that moment.
- A minimum level that is never bypassed by a node selection or a search match.
- Per-node toggles; warn and error marks on the overview ruler; "follow playback" and scroll-to-bottom.
- A flat node list with no hierarchy, so it cannot show an entity and its children.
- Bracketed one-line text (
[ INFO][…][/node]: …) as the display. That is a terminal format.
No screenshot: the docs page has none, and the Lichtblick panel thumbnail in its repo is too small to read.
3 · macOS Console.app
The unified-log viewer: token search over process, subsystem, category and message type, with saved searches.46,47
- Right-click a row to show or hide this property. The choice becomes a search token at once.
- Include or exclude on each token; all tokens must match.
- Saved searches as one-click chips in the Favorites bar.
- Fault is distinct from Error (red dot vs yellow dot).
- Streaming starts on Start, so the window opens quiet.
- The unified log's default volume, with no notion of a device's own events.
- The admin password prompt to read logs.
4 · GNOME Logs and journalctl (this PC's own logs)
The stock viewers for the systemd journal, where the kernel's USB lines already live.48,37,38
- A category sidebar: Important, All, Applications, System, Security, Hardware. "Hardware" is the computer's own wires.
- journalctl's device match adds the device and every ancestor. That is "+ path to host".
-pseverity ranges,-bper boot,-o json, and-xcatalog help keyed onMESSAGE_ID.
- Rebuilding it. The journal runs about 1,275 lines an hour here; link to it instead.19
5 · Windows Event Viewer
One event infrastructure with channels by audience: Admin, Operational, Analytic and Debug.42,43,49,50
- Admin events must be actionable. Operational events are for diagnosing. Analytic and Debug are high-volume and off by default. This is the Activity vs Raw split.
- Event ID + Source as the code.
- Custom Views (saved filters).
- The overview's counts of administrative events per level over 1 h, 24 h and 7 days.
- A tree of hundreds of providers.
- Three dense panes on the first screen.
6 · Grafana Explore and the Logs visualization (Loki)
Server log exploration with a volume histogram, dedup, live tail and context.51,52
- A log-volume histogram above the lines, with bars per level. That is your "turn it into that bar" thought, with the health bar's clock.5
- Dedup modes: exact, numbers, signature.
- Live with Pause, Resume, and new lines marked.
- "Show context" (lines around this one).
- Filter for or out a field value from the expanded line.
- A query language (LogQL) as the way in.
7 · Datadog Log Explorer (Kibana Discover is the same shape)
Facets, patterns, saved views and live tail over structured logs.53,54,55
- Patterns: similar messages cluster into one row, and the varying part is highlighted. That is the grown-up form of ×N.
- A facet sidebar.
- Saved views.
- Live Tail includes logs that the index filters drop.
- Sampling. At high volume Live Tail shows a uniformly random sample, and a device log must never sample.
- A wall of facets on the first screen.
8 · Chrome DevTools Console
The most-used log viewer there is: levels, sidebar, URL filters and grouping.56
- A sidebar of User messages, Errors, Warnings, Info and Verbose, drilling down by script.
url:and-url:positive and negative filters.- "Group similar messages", on by default.
- "Preserve log" across reloads, which for this app means across reboots.
- Grouping that hides when each repeat happened. Keep first and last time on the ×N row.
9 · lnav (logfile navigator)
A terminal viewer that merges many log files into one timeline.57
- Messages from all files "in time order": the many-sources, one-clock model.
- A HIST view stacked by level.
- The filters panel shows how many lines are hidden, so filtering is honest.
- Out-of-order timestamps are flagged, with actual vs received time.
- Keyboard-only discoverability.
10 · Wireshark (live port and packet traffic)
The packet list plus Expert Information, a derived list of anomalies.58,59
- Expert Information is a summary derived from the stream. It is grouped by severity (Chat, Note, Warn, Error), it links back to each packet, and the packet's own tree is tinted by it. That is the fault memory as a view of the log.
- Coloring rules are display filters, and the first match wins.
- Raw frames in the person-facing view. Per-packet detail belongs to a bus's Raw channel.
- The page itself calls expert info "only a hint".
11 · Vector CANoe / CANalyzer Trace window (and Kvaser CanKing)
The CAN bus trace: the same frames shown chronologically or in a fixed position per ID.60,61
- Chronological vs Fixed Position display mode: one window, two read models. Chronological is a scrolling log. Fixed Position keeps one row per ID, updated in place, with its signals. That is your Activity log vs your status table.
- Diff time per ID: the measured cycle against
expected_period_s. - CanKing's Fixed Position button "locks messages by ID".
- Frame-level traffic in the device's Activity log. It belongs in the CAN bus entity's Raw channel, or in tview.
12 · Home Assistant Activity (formerly Logbook) and Repairs
The consumer smart-home answer: a human-readable activity feed per area, device and entity, plus a separate list of things that need fixing.62,63,64
- It excludes high-frequency entities by default: sensors with a unit of measurement, a state class or a numeric device class. Continuous values stay out.
- Filters by area, device, entity and integration.
logbook.logfor custom entries.- Repairs keeps "needs your intervention" items apart, with a badge. That is fault memory vs log.
- Sentences with no severity at all.
- Hiding in the view without excluding in the store.
13 · kubectl events (Kubernetes Event)
The cluster's event stream, also shown per object at the bottom of kubectl describe.44,65
- Type (Normal/Warning) + Reason (a machine code) + the object regarded + the reporting component.
- A repeat is one Event with
series.countandlastObservedTime. - The per-object "Events:" section is the entity log on the detail page.
- Using events as state. Kubernetes calls them "informative, best-effort, supplemental data".
14 · Sentry breadcrumbs
The trail of events before an error, shown on the issue.66
- Each breadcrumb has type, category, level, message and data.
- A drawer with search, filters by type and level, and time relative to the error.
- For this app: a fault's detail shows that entity's log for the 30 s before
first_occurred. That is your flight recorder rule, "save what just happened".10
- A buffer that lives only in the client. Here, breadcrumbs are just a query on the host's log.
15 · Android Studio Logcat
The best typed filter grammar in the set.67
level:WARNmeans that level and higher.tag:,-tag:andtag~:(regex);age:5m;is:crash;package:mine.- The same key is ORed and different keys are ANDed;
&,|and parentheses when needed. - Favourite queries; split panes, each with its own query.
- PROCESS STARTED and ENDED markers, which are your Booted and Restarted rows.
- A colour per tag. Colour for identity competes with colour for severity.
16 · Rerun TextLog view
Log lines that live on entity paths, in a viewer whose views are scoped by path rules.33,34
- A view's
originplus+ $origin/**and- $origin/…rules give entity scope with children and carve-outs. This is the model for "+ children". - A level row filter.
- A timeline column shared with every view.
- Building on it. The view is marked unstable.
17 · COVESA DLT Viewer (AUTOSAR log and trace)
The automotive log viewer for DLT: ECU → Application → Context identity, with remote control of log levels.68,40
- A three-level identity for filtering: ECU ID, Application ID, Context ID. Yours is the entity tree.
- Log levels can be set per application and context from the viewer: the equivalent of a "more detail for this entity" switch.
- Filter types Positive, Negative and Marker (with a colour).
- Four-character IDs.
- Non-verbose messages that need an external description file to read.
18 · Ignition Alarm Journal (industrial HMI, ISA-18.2 world)
SCADA keeps live alarms and alarm history in two components.69
- The Alarm Journal Table "displays the history of the alarm system": active, acknowledged and cleared events stored by a journal.
- It has state and priority filters, text search, and realtime or historical date ranges; the default is the last 8 hours.
- Live alarms live in a separate status component where you act on them. That is your fault memory vs log.
- A history that depends on a separately configured journal profile and database.
19 · ros2_medkit (SOVD for ROS 2)
The closest robotics peer to RobotDiagnostics. It serves faults, data, operations, configurations and logs per entity.27,28
/components/{id}/logscovers all of a component's nodes (prefix match);/apps/{id}/logsis exact. That is "an entity's raw log stream", already in the field.logs/configurationto change an entity's log level.- A rosbag of the seconds around each fault.
- Hashed fault codes (
LOG_NODE_1A2B3C4D). Prefer the producer's own code. - Promoting every WARN log line to a fault. In this app the log and the fault memory each keep their own job.
No screenshot. medkit's UI is the Foxglove extension, and the docs reached here show no image of its logs tab.
8 · What needs you (silence takes the default)
- Activity and Raw as the two channels, with coded events in Activity, uncoded lines in Raw, and Raw off outside an entity's own log. Recommend yes. Default: yes.
- Status transitions. A warn or error transition rides on the fault row. Silence (→ stale or unknown) gets its own warn row only when no plug or unplug event on its path explains it. All other transitions are debug. Default: that rule.
- Fold the fault lifecycle onto the causing row ("→ fault confirmed" badge), and group faults confirmed in the host's first reports into one "found at start" row. Default: yes.
- Keep the host's events on disk: append-only JSONL per boot under
~/.local/state/io-diagnostics/log/, size-capped, so faults, actions and transitions survive a restart. Today's reboot erased them. Default: yes, about a 20 MB cap. - Adapter order: the
to_ros_log/from_ros_logpair first (pure functions, tested liketo_diagnostic_status), then a journald sink, then MCAP export. Default: only the pure functions in the next build. - Order: newest first, as rqt_console and today's view do, not the tail at the bottom like Grafana and Logcat. Default: newest first.
9 · Deliberately left out
- No code, no git writes. The dispatcher limited writes to this report, its media and the vault note. A peer lane had uncommitted edits in
io_diagnostics/andio_facts/at 14:30–14:34.18 - docs.ros.org pages sit behind a bot check, which I did not bypass. I read the same pages from the
ros2_documentationrepo they are built from. - The SOVD
/logsschema (ISO 17978-3) is paywalled. medkit's endpoints stand in for it. - Apex.Grace logging has no public doc, so it gets one line.
- Message catalogs and translation (journald's catalog, MAVLink metadata) are later, the same call record v0 made for translation keys.12
- Noticed, not chased:
UpdateAvailableconfirms as a WARN fault (18:11:50). Home Assistant keeps updates out of Repairs. Worth a look when the fault memory gets aging. - No concept stubs. The brief limited vault writes to the note. Candidates: "OpenTelemetry Logs Data Model", "Event Log vs Fault Memory".
Source Index
- Vault: Software Design (Notion import, 2025-04-12): "All logs to rosout for ground truth introspection"
- Vault: Ideas for tomorrow (APPLE NOTE): "Make logs a first class citizens"
- Vault: bam_logger - Data Logging (2025-07-26): circular buffer, save retrospectively
- Vault: Timeline view of robotics data (2025-07-26): "Ok its all about event based!"
- Vault: Daily Note - Oct 23 2026: Activity feedback ("only like 1 activity colum"; "turn into that bar… like the daginstics log")
- Vault: MOC - Diagnostics
- Vault: IO Diagnostics — Prior Art and Converged Design (2026-10-22); repo
docs/converged-design-2026-10-22.md§7 (Lichtblick lacks presence, actions, USB skeleton) - Vault: Diagnostics Rollups — Five Ways to Reconcile Black Boxes with Top-Level Checks (2026-10-23)
- Vault: C - Diagnostics Record (What We Learned Since ROS DiagnosticStatus)
- Vault: C - Flight Recorder Debugging (Save the Last 30 Seconds, Not the Next)
- Vault: Foxglove vs Rerun — Logging Boundary and Backend Decision (Deep Research, 2026-07-21)
~/io-diagnostics/docs/diagnostics-record-v0.md(Event, Fault, rules 5 and 8, "Deliberately left out")~/io-diagnostics/contract/record.py(EVENT_SEVERITIES,Event,Fault,FaultManager,DEFAULT_DEBOUNCE)~/io-diagnostics/io_diagnostics/activity.py(coalesced plug notices,BOOT_WINDOW_S,Booted)~/io-diagnostics/docs/api-v1.md(/api/v1/activity)~/io-diagnostics/docs/round3-brief-2026-10-23.md(#6 colours, #9 Activity)~/io-diagnostics/reports/media/round3/a-activity-light.png(replay F1 of the 10-21 rack session)~/io-diagnostics/io_facts/kernel_log.py(peer lane, uncommitted, mtime 14:34 on 2026-10-23)- This PC's journal, read-only (
journalctl), 2026-10-23 14:55 and 15:05; boot12fdca7d…, 2026-10-21 18:38:30 → 2026-10-23 15:00:36 - ROS 2 docs, Using rqt_console: docs.ros.org, read from ros2_documentation
- rqt_console source, filter factory: console_widget.py
- rcl_interfaces/msg/Log.msg
- ROS 2 docs, About logging: About-Logging.rst
- rmw qos_profiles.h (
rmw_qos_profile_rosout_default) - REP-107, Diagnostic System for Robots Running ROS
- diagnostic_updater.cpp (
verbose_inUpdater::update()) - ros2_medkit_log_bridge (v0.7.0)
- ros2_medkit_gateway (logs endpoints, entity types)
- Foxglove, Log panel
- foxglove.Log and LogLevel
- Foxglove, Events
- MCAP specification
- Rerun, TextLogView
- Rerun, entity queries
- OpenTelemetry, Logs Data Model
- OpenTelemetry, Logs Data Model Appendix (A: mappings; B: SeverityNumber)
- systemd.journal-fields(7)
- journalctl(1)
- AUTOSAR SWS Diagnostic Log and Trace, R4.1.3, §7.3.3.3.1 (SWS_Dlt_00474–00479, 00377)
- AUTOSAR CP SWS Diagnostic Log and Trace, R25-11 (change history 4.4.0; ECU / Application / Context ID; levels)
- MAVLink, Events interface
- Microsoft, ChannelType (Admin, Operational, Analytic, Debug)
- Microsoft, Defining severity levels
- Kubernetes API, Event v1
- Apex.AI, Diagnostics Reimagined
- Apple, Console: log messages
- Apple, Console: find log messages and activities
- GNOME Logs
- Wikipedia, Event Viewer (screenshot)
- Microsoft Learn, Inside: Event Viewer
- Grafana, Logs in Explore
- Grafana, Logs visualization
- Datadog, Log Explorer
- Datadog, Live Tail
- Datadog, Log Patterns
- Chrome DevTools, Console features reference
- lnav, User interface
- Wireshark, Packet colorization
- Wireshark, Expert Information
- Vector, Analysis and Test of CAN Applications (ESA, 2017), slide 16
- Kvaser, CanKing Message Trace
- Home Assistant, Logbook (Activity)
- Home Assistant, Activity card
- Home Assistant, Repairs
- Kubernetes, Debug Running Pods
- Sentry, Breadcrumbs
- Android Developers, Logcat
- COVESA, DLT Viewer
- Inductive Automation, Perspective Alarm Journal Table