Logging

Getting metrics out of embedded devices is often useful but also challenging due to resource constraints. What I’ve found to work quite well is attaching a debugger and just dumping metrics via printf and then parsing the output.

I always dump the logs into Loki with something like this

command-to-get-logs | promtail \
  --stdin \
  --client.url http://localhost:3100/loki/api/v1/push \
  --client.external-labels="app=my-embedded-device" \
  --server.disable

Than I can just print lines in logfmt:

logfmt: rx_bytes=100 tx_bytes=100
logfmt: rx_bytes=150 tx_bytes=100
logfmt: rx_bytes=400 tx_bytes=120

and parse into a graph in Grafana via LogQL: