Injecting tracing the hot way

Underjord is an artisanal consultancy doing consulting in Elixir, Nerves with an accidental speciality in marketing and outreach. If you like the writing you should really try the pro version.

I mostly program in Elixir. So tracing is an overloaded term. This post relates to both of them and how we can use BEAM and Erlang fundamentals to do wild things for observability that wouldn’t be practical in most runtimes. Let’s load some hot code.

First is a set of (experimental) updates I’ve done to my “Erlang tracing” library Entrace. Erlang’s concept of tracing is a mechanism in which you can tell the system to send you information about execution of functions that match some pattern. The match specs give me a headache (though LLMs are very good at them) so I made a library that makes ti way more convenient to write traces. It also provided the ability to trace multiple things before this was easily support OTP. This version also updates to use more of the new tracing features from OTP 27. But the new updates are slightly different.

The biggest consistent problem with this library for me is that I never have it installed when I need it. So when I am investigating a running system I don’t have the convenient way that I know how to use.

But what if I could easily add it. At runtime. Well. The first approach is the new Entrace.Mini script which will give you a simplified version of the Entrace tracing tool as a single module that you can paste over your iex console and bam. You have easy tracing.

But if you want the full-featured one?

Well I have something akin to curling and piping to bash. With all the convenience and all the lack of security. A one-liner that will download my latest release of Entrace as pre-compiled .beam files for your version of OTP (27 & 28 currently supported), then it will load those in and you are ready to go.

This gets you from zero to Erlang tracing.

I’m considering if this is a decent approach to a plugin system for another system I’m working with where plugins might make some sense. And for pure Elixir code this seems perfectly reasonable.

The next part is one of those tools I would not have bothered to try building at this moment if I didn’t have an LLM on hand. But I knew exactly what I wanted it to do. entrace_opentelemetry is a library to deal with the other type of tracing. Open Telemetry traces creates spans in your code that you can attach metadata to and easily report to tools like Honeycomb or Sentry. Anything that supports the tracing parts of the standard I suppose.

So we do that. But we do it using Erlang tracing to trigger the traces. Meaning we can. At runtime, ask our app to start tracing a particular query, get timing, input values, output values, error information. The overhead is mild and the capabilities are massive. Maybe we only want to trace a function when a particular argument is nil. Sure thing. Throw it in your remote iex console and watch your production app spit new spans to Sentry.

No release of entrace_opentelemetry yet. I need to try it a bit and I also want to see if I can solve the limitation that it only provides root spans right now. Seems perfectly doable.

That’s what I had for this moment. Tracing for your tracing. So you can trace your traces.

By the way. I’m organizing an odd conference called Goatmire Elixir. If you enjoy Elixir you’ll enjoy the event. The Call for Talks is open at the time of writing. We’re open for sponsors. Tickets are on sale and the workshops are free. Kindly take a look.

Let me know if you have any questions.

I’m reachable on email lars@underjord.io or the Fediverse @lawik@hachyderm.io. Thanks for reading.

Underjord is an artisanal consultancy doing consulting in Elixir, Nerves with an accidental speciality in marketing and outreach. If you like the writing you should really try the pro version.

Note: Or try the videos on the YouTube channel.