Logs as metrics from embedded devices

Getting metrics out of embedded devices is often useful but also challenging due to resource constraints. What I’ve found to work quite well is attaching a debugger and just dumping metrics via printf and then parsing the output.

I always dump the logs into Loki with something like this

command-to-get-logs | promtail \
  --stdin \
  --client.url http://localhost:3100/loki/api/v1/push \
  --client.external-labels="app=my-embedded-device" \
  --server.disable

Than I can just print lines in logfmt:

logfmt: rx_bytes=100 tx_bytes=100
logfmt: rx_bytes=150 tx_bytes=100
logfmt: rx_bytes=400 tx_bytes=120

and parse into a graph in Grafana via LogQL:

avg_over_time({ ... } |= "logfmt: " | logfmt | keep rx_bytes | unwrap rx_bytes [$__interval])

Alternatively, as I also use InfluxDB, I can output lines in its Line Protocol Format:

influx: measurement,device=aabbccddeeff rx_bytes=100,tx_bytes=100
influx: measurement,device=aabbccddeeff rx_bytes=150,tx_bytes=100
influx: measurement,device=aabbccddeeff rx_bytes=400,tx_bytes=120

(You might notice I didn’t include the timestamp. If omitted, it defaults to the current time.)

That could be piped into something like

command-to-get-logs \
  | rg --replace '^influx: (.+)' '$1' \
  | { while read -r line; do influx write -b bucket_name "${line}"; done }

and then shown in a graph in Grafana.