I like this, and the possibilities to polish such a feature are endless. But I think there are two separate components here: Logging and Tracing.

As a starter, I considered focusing on a trivial logging component that logs to syslog instead of stdout. Syslog comes with log rotation. Then, in the UI, we can dump the syslog buffer in a text field. The upside to this is we can also display system logs. The downside is that logs are cluttered with noise.

What you are describing sounds more like tracing to me. I know there is quite a popular tracing crate, but I have yet to use it.

Additionally, it would be great to expose/log changes in any environment-monitoring data that exists on the BMC, but also provide a simple per-node service which pushes the node environment data (temperature, voltages, perhaps even clock-speed) to the BMC as the aggregator/single-source-of-truth.

As a more ambitious extension, it would be great if the BMC WebUI were then able to graph all of this data!

Many other embedded systems also log to a tmpfs by default for live debugging. I’d love to see this even without a persistent storage option.

I would scope this issue later to logging only: notification and alerting are two seperate topics.

I personally modify the syslogd/klogd usage in the BMC to log to memory only ring-buffer (which needs to be enabled in buildroot during compile time.)
I also add support for remote logging in busybox-syslogd. This allows me to send the logs to a central server, e.g. loki, rsyslog or similar.
With the in memory ring-buffer, busybox brings logread for reading the logs.
If the bmcd is logging to syslog, too, everything would be in one central place.

#!/bin/sh

BUSYBOX=/bin/busybox

SYSLOG_OPTS="-C256"
#SYSLOG_OPTS="-C256 -R[2001:db8::514] -L"
KLOG_OPTS=""

[ ! -r /etc/default/syslogd ] ||
  . /etc/default/syslogd

start() {
  stop
  $BUSYBOX syslogd -n $SYSLOG_OPTS &
  echo $! > /var/run/syslogd.pid
  $BUSYBOX klogd -n $KLOG_OPTS &
  echo $! >> /var/run/klogd.pid
}

stop() (
  kill $(cat /var/run/syslogd.pid /var/run/klogd.pid 2> /dev/null) 2> /dev/null
)

restart() { stop; start; }
reload() { restart; }

case "$1" in
  start | restart | reload | stop ) "$@" ;;
esac

# ts=2 sw=2 et ft=sh

@j0ju i would like to integrate a syslog daemon like you mentioned. I think its too soon for the upcoming release that comes around the same time as the v2.5 boards. But i will start on this soon anyway

if there is more input on this from anyone, feel free to dump it here