C

Getting metrics out of embedded devices is often useful but also challenging due to resource constraints. What I’ve found to work quite well is attaching a debugger and just dumping metrics via printf and then parsing the output.

I always dump the logs into Loki with something like this

command-to-get-logs | promtail \
  --stdin \
  --client.url http://localhost:3100/loki/api/v1/push \
  --client.external-labels="app=my-embedded-device" \
  --server.disable

Than I can just print lines in logfmt:

logfmt: rx_bytes=100 tx_bytes=100
logfmt: rx_bytes=150 tx_bytes=100
logfmt: rx_bytes=400 tx_bytes=120

and parse into a graph in Grafana via LogQL:

At my job, I often work with embedded devices. Once, we had a particularly nasty problem with measurements from sensors sometimes returning unexpected values, debug logs getting garbled up, and then there was the occasional crash, too.

Most of our code was unit tested on a desktop. The tests were run with sanitizers, and they reported no issues.

Still, the problems kind of pointed to some kind of memory corruption. A threading issue, perhaps?

What do you think this C code prints?

struct foo first = { 0 };
struct foo second = { 0 };

if (memcmp(&first, &second, sizeof(first)) == 0) {
  printf("equal\n");
} else {
  printf("not equal\n");
}

The corrent answer is: “I don’t know.”

The reason is that we don’t know if the struct contains padding bytes inserted by the compiler. Because if it does, those bytes might not be initialized. The only way to set them is with memset.

I like Rust. I like how it forces you to do exhaustive matching in match statements (well, as long as you don’t use a catch-all like _ => {}), so when I add a new enum type, the compiler helplfully reminds me to handle it.

I don’t get to work in Rust nearly as much as I’d like; I usually find myself swimming in the C, instead.

Thankfully, modern compilers like GCC and Clang have a lot of warnings you can enable to help you avoid mistakes. For example, you can get the same kind of exhaustive matching for switch statements that you can get in Rust’s match statements, provided that you don’t add a default case.