If you have hesitations about attending Goatmire I do hope you respond to this email and let me know about it. If you can't attend, fully understand that. If you don't want to attend, also perfectly fine.

As the organizer it is of course incredibly useful to know where hesitation sits.

And on that note: workshops are filling up, speakers are being announced, get on in there :)

Making metrics

"A complex system that works is invariably found to have evolved from a simple system that worked. A complex system designed from scratch never works and cannot be patched up to make it work. You have to start over with a working simple system." 
John Gall, frequently as Gall's Law

There is a tool that's been around the Nerves space for some time that most people haven't encountered yet called Mobius. I will try to articulate the many ways in which it makes very particular choices in service of its objective to aggregate metrics locally on-device in an efficient and resilient manner.

Collecting observability data is incredibly powerful and useful. It is also expensive. Your average cloud deployment metrics scraping setup can get expensive. Now do it with 400.000 devices.

Most of the information is uninteresting most of the time. And you could do well with just sampling your fleet infrequently or at a worse granularity, there are many knobs and levers. But the moment you need to know historic information, retroactively, you are in a spicy spot.

Most of the telemetry and metrics along the lines of CPU usage, CPU temp, memory usage, those are not business data. They don't concern the purpose of the device, they concern the healthy operation of the device. Exactness is not as important as the rough observable behavior.

So we don't need to send every metric sample at sampling time to a cloud. We don't need every metric. We also might not need to send all the time. Many devices use LTE and pay as you go. This is costly. Keeping most of the metrics tracking on-device is actually quite nice. You can pull samples as needed, you can infrequently upload and so on.

Keeping a bunch of data on device requires some discipline and some care to do well. You want safely bounded memory and disk usage. You want resilience to missed events and performance degradation. You want to be efficient.

Mobius offers the on-device metrics storage and lets you tie in the regular Erlang telemetry support used throughout the BEAM ecosystem to acquire any exposed metrics.

The first building-block is related to a lesson most of us have learned. If you write a log file by just appending to it, eventually you'll fill the disk and a server goes down. I've done it. Nerves avoids this with the RingLogger backend which keeps logs in a ring buffer. And actually by default only in memory to avoid wearing on the flash storage.

Mobius also uses circular buffers. One per granularity-level. Second. Minute. Hour. Day. And by having samples land in these buffers and having them rotate after a certain number of samples.

120 seconds. 120 minutes. 48 hours. 60 days. We get various levels of granularity covering these periods of time. By default Mobius provides an average value across the period and the standard deviation. This should let you see where the value was on average and if that average was even or had a lot of variety. For 2 months. With very little data.

Each sample in each buffer entry stores about 4 integers. Total value, total value squared (IIRC, I haven't worked on the std deviation stuff), number of events registered and the timestamp at which it was sampled. So the telemetry just increments and in-memory state for where things stand and at the sample granularity we jot those down. It is dumb. Simple. Reliable.

And if you want a plot, you can derive a plot. You can upload a whole time-window of data as a batch to the cloude-side on demand. Lots of options. Because we have the data and we have it in an appropriate reasonable form we can do a lot of stuff depending on what we need.

Persistence is done on an interval to prevent wear on flash storage. Recently we added compression to ensure we were a bit more efficient as well.

This is stacking several intentionally very dumb mechanisms on top of each other in a thoughtful way. Straight up Gall's Law.

I have a PR that I hope I'll be able to tidy up and ship fairly soon that adds DDSketch histograms. They have a lot of trade-offs to balance in storage and memory usage but they add a lot more insight to the values where you track histograms. And DDSketch is also in many ways a few simple parts, adding up to a lot of very useful capability.

Which is quite exciting in terms of what questions we should be able to answer. If you want me to write more about it. Let me know and I'll be happy to share.

Have a great weekend, thanks for reading.