Why am I interested in Elixir?

Underjord is a tiny, wholesome team doing Elixir consulting and contract work. If you like the writing you should really try the code. See our services for more information.

The juicy bits

• OTP & the BEAM
• Phoenix Presence
• Phoenix LiveView
• Nerves
• Scenic
• Rustler

I’ve had Elixir on the brain recently. And by recently I probably mean 2 years. In my defense I think it is fair to say it is blooming right now. I haven’t had much need of it, or opportunity for it, in my day-to-day of maintaining a Python legacy system, renewing another legacy or optimizing Elasticsearch. So I’ve tried it with a few hobby projects I’ve spent time on and that was fun. But mostly I really just watched the community and what they did with a feeling of “Shiiiit, I want in on some of that!”. I'll primarily touch on BEAM, OTP, Phoenix Presence, Phoenix LiveView, Nerves, Scenic and Rustler.

Why? What makes it so interesting to me? It is just a programming language and I try not to be religious in these matters. I’ll try to outline what attracted me to Elixir, made me keep yearning for it and what I’ve been digging into recently as I’ve made time to actually get more comfortable with it.

OTP, The BEAM and the Erlang roots

Elixir is built on the foundation laid by Erlang/OTP and it runs on the same VM. It is called BEAM or sometimes The BEAM. Why does the VM matter?

For a long time, doing web stuff primarily, I didn’t care much about the underlying runtime environment I was in. Then I was involved in a micro-service implementation and I did a lot of multi-threading. With Python. The story there is fairly well-known, there is a lock, the GIL. It is a solution to thread safety with some considerable, rather pragmatic, trade-offs. My experiences with this made me pay attention to concurrency, parallelism and core usage more. In addition, it sort of bothers me that a lot of desktop applications are bad at using multiple cores when it would make sense. Specifically considering the CPU core counts in modern machines. Anyway.

The BEAM is built to run in a distributed fashion, I believe this came before it was really used for multiple cores, it just happened to carry over particularly well. It has its own scheduler and a facility to spawn very light-weight processes that are not OS threads. Processes are isolated, they communicate through message passing. A bit more copying of data, a lot less problems. As I understand it a BEAM-based application launches the VM with (by default) a number of threads equal to the core count on your machine. Your processes will be scheduled across these threads.

The things that make the BEAM do really interesting things generally come from OTP. The Open Telecom Platform was designed to build telecom systems at Ericsson. For that purpose it has a high focus on resilience, recovery and some features I've never heard of elsewhere. Such as hot updates.

For resilience and recovery there is the idea of supervision trees. The idea is that you have a hierarchy of supervisor processes monitoring the parts of your application, the parts can also be supervisors with an underlying structure of processes and supervisors. So it branches out. Like a tree of supervision. A supervision tree, if you will.

When something goes wrong, if a process crashes, it is the supervisor of that part that is responsible for making sure we recover to a good state. If it can’t, or shouldn't, the supervisor itself can crash, passing the buck up the tree to some supervisor that may know more about how to recover.

This builds on the thought that errors should surface to the level that can handle them and it saves your simple code from having to handle every potential error it could suffer. At some point up the tree there is a correct place to handle the problem. And that's where it should be handled.

This has given birth to the saying "let it crash" which may be terrible in general marketing but might sound interesting to tech people. There is much nuance but I think supervision trees make you think about the architecture of an application in a healthy and layered way. You build the tree with a stable trunk and usually more ephemeral leaves. It also translates into some interesting thinking as we will see with Scenic.

Another rare feature that few need but that is incredibly rad is hot code updates. This is a way in which you can update an application while it is running by replacing which version of a module is running. I haven’t dug into it, seems rad. I guess it was annoying to bring down the telephone switches just to update the software. Not used very much in simpler systems I think, but fascinating.

The concurrency, robustness, resilience and distribution model that Erlang/OTP and the BEAM brings is making some things happen that I don’t believe quite exists in any other programming environment. Not from what I’ve seen at least. This makes for a very interesting runtime. Let's get into projects that leverage this.

Phoenix

When Elixir showed up on my radar it was all about Phoenix and Ecto. Let's put a pin in Ecto for now. Phoenix is a web framework that is to Elixir as Rails was to Ruby. Or that’s the idea. Phoenix is not as batteries-included or auto-magical as Rails, it has a different focus. Phoenix definitely feels more explicit and less magical. I like what it provides but it doesn't seem to over-provide. I haven’t worked with it a ton. Seems to be a solid project and most languages need a strong web framework. Phoenix fits that bill.

I find it more interesting what it has allowed people to do. Chris McCord and the merry crew he conspires with has been doing some brain-warping work to make very compelling ideas land as optional libraries for Phoenix. The ones I have in mind are called Presence and LiveView.

Phoenix Presence

This allows tracking of who is currently online, which different devices they are on and similar metadata to be provided to a distributed system in an eventually consistent way. It uses a particular CRDT to achieve this which limits the amount of data that needs to be passed around while allowing reliable, distributed, reporting of client connections for the Phoenix Channels system (a layer on top of WebSockets).

The typical use-case is a chat service. Who is online, who went offline, who can I actually contact right now and are they on mobile or active on desktop? How do you distribute this data so that eventually everyone gets the memo that “John has disconnected”. Even if server #5 goes down, comes back up and actually missed the memo? Well, you use Phoenix Presence and you basically don’t have to worry about it.

This could have been done with other languages and runtimes I imagine. But with the BEAM we already have distribution. The system is already made to be run in more than one place. Having that in place is likely a large part of why this was done, if you want to build a presence functionality you need to make it work in a distributed fashion or you're not living up to the stated goals of the ecosystem.

Phoenix LiveView

Only recently made available to the public and not quite in a release state last time I checked. Fully working though. This is a clever and curious one. Chris wanted to save backend developers from needing to write javascript for every trivial little piece of interactivity. Most of us know the drill, we need some progressive enhancements, such as form validation. Suddenly we need frontend javascript, backend support for that and suddenly: complexity.

LiveView is a way of rendering a page, it functions as a basic page or form. But if you have JS active it will hook up the LiveView javascript with the LiveView backend endpoint, it talks over Phoenix Channels and voila. You have a lot of frontend-scripting from the backend.

I saw Chris demo this live at ElixirConf 2019. What a treat! I already knew what it was supposed to be and do. And I was blown away by the details, the cleverness, the minimal overhead.

Ecto

This is a mostly a database library that can also be used for other validation as we’ve come to see. It is independent from its SQL roots these days but generally you will encounter it in Phoenix for database operations. I haven’t more than scratched the surface in actual use. But it approaches the ideas of a database library in a novel way I haven’t quite seen before. I recommend reading up on it. I should too.

Nerves

The Nerves Project

Oh this one I love. Take the resilience of the BEAM, put it on a minimal Linux image. Add your Elixir/Erlang application. Burn it as a firmware image to an SD card. Plug it into a computer, typically an embedded-level device, such as a Raspberry Pi and you are ready to have some fun.

Nerves was built to create connected devices (IoT, if you will) and try to provide more developer convenience along with a more resilient and less frustrating state of affairs. The BEAM is good at recovery and resilience. IoT is not famed for its durability. I would say Nerves definitely moves the needle in this area. Instead of running everything a Raspbian installation includes on your device you are starting from a minimal Linux and a bundle of Elixir code. The BEAM is basically your operating system, along with any parts of Linux you feel the need to add.

The project allows you to get started building more serious software projects, more durable solutions, with less work and a nicer workflow. The team is incredibly responsive on the Elixir Slack #nerves channel as well.

The support for Raspberry Pi-devices is great, all the common hobby-electronics-needs have libraries (I2C, GPIO, SPI, UART), networking in many shapes and forms. And importantly you can push firmware over SSH. I cannot stress enough how useful this is.

I’ve actually ported the Pimoroni Inky (an eInk display) library to Elixir, using Nerves heavily in the process. Some people have joined up to help get it presentable and useful. You can find our version of Inky here. I’ve also experimented with using Nerves as a sensor hub. An ongoing project, hampered mostly by my incompetent soldering.

Nerves is an incredibly useful project for making something beyond a toy or small Linux server on the Raspberry Pi. Though it does work great for toy projects too. It has been an absolutely fantastic resource for me to learn more about working with hardware, Linux and Buildroot. The people in the community channel are incredibly dedicated and helpful.

From what I’ve heard there is nothing quite like it out there for building embedded devices at single board computer size and up. There are also people that use it to push applications to the cloud as new firmware, which seems fun.

The Nerves team is also developing NervesHub which is a solution for managing a fleet of these devices with secure firmware updates, encryption hardware and that sort of thing. Typically difficult and expensive infrastructure to develop for your IoT devices. Seems very straight forward.

Scenic

An independent, functional, UI framework. I say independent because it runs as close to basic OpenGL as it can on the given platform. I believe it only depends on glfw to get things going on desktop operating systems. It has a custom driver for the Raspberry Pi official touch-screen. Not QT, Electron or whatever else we do cross-platform UI with these days.

This is something I look for every now and then. Native or close-to-native UI that does not require a web-browser. Chromium is slow on a Pi. A web-browser is not always the best experience and it means an open port, it means a lot of software beyond what you need. Because the web browser is an immense platform compared to what most applications need, especially for an embedded situation.

Scenic is focused towards embedded devices and fixed-size screens. It is quite minimalist but well-featured and has an interesting design that I believe in.

Components of the UI are arranged as a supervision tree, a graph of components and primitives. So it uses the resilience features of OTP to create a UI where a specific component can crash without bringing the whole UI with it. It should recover, it might lose some data, but the problem is isolated.

There are a bunch of cool aspects to its design that allows things such as multiple viewports/displays and the potential to do wild things in the future. You’ll get the best overview from Boyd’s own talk (Code Beam Sto 2019, good conf).

It is still early days for Scenic. But you can use it right now. It is fast and friendly, and the plans Boyd has for it are fascinating. I’m saying early days, because there is so much more to come.

If you are interested in getting in on the ground floor, the #scenic channel on the Elixir Slack is a good place to drop in. Good people, let us know if you want to contribute somehow or just try running it and come in and share your joys and confusion.

Rustler

Elixir and the BEAM have some limitations. There are moments when the FP paradigm is limiting or you want to call out to some native functionality that is not implemented in an Elixir-friendly way. There are a few escape-hatches for integrating with other languages and other systems built into the BEAM. The most powerful one is probably the NIF. Native Implemented Functions. Generally a way to write a function in C when you gotta go fast, NIFs are a bit risky in that they can actually cause problems with the BEAM. If you write yourself a memory issue in your NIF you’ve a memory issue in your BEAM and that nice supervision tree is unlikely to save you.

I’ve only barely touched Rust. But to my understanding it does away with a lot of the riskiest parts of systems languages while allowing most of the efficiencies. Sonny Scroggin held a talk that really piqued my interest on this at Code Beam Sto this year, sadly the video doesn’t seem to be release yet. Rustler is a project he and a cool Norwegian named Hans is working on that seems quite mature and makes NIFs using Rust work nicely and easily. Keeping your BEAM safe. Hopefully.

I have a thing where I’m likely to use it to bridge over into ALSA territory. We’ll see, looking forward to trying it out.

Hans is also doing some additionally mind-bending (he calls it Eir) stuff concerning the intermediate representations of BEAM code and making that work with LLVM. He held a talk about that which boils down to what was two proof-of-concepts. Niffy, which allows him to smack a small annotation on a function in Elixir and have it be compiled to a NIF instead of running on the BEAM. As far as I understand it at least. The other was Whirl which would let BEAM languages run as WebAssembly. Which seems wild and fun to me.

In summary

The BEAM and OTP as a runtime and platform provides some very cool possibilities that I have not seen elsewhere. Cheap and easy both parallelism and concurrency. Beyond that you have distribution, which is important both for resilience and scalability.

From this base, especially with the appearance of Elixir, which is often considered a more approachable than Erlang, you see these interesting uses of the tech pop up. Nerves and Scenic just set my mind on fire with possibilities. It is a fascinating ecosystem that is springing up around Elixir. I don't know anything quite like it. That is not intended as hyperbole or a slight to your favored language. I am simply enthusiastic and possibly ignorant of the much cooler toys you have access to. Tell me!

Do you know of languages/runtimes that do actually provide similar tools? Or have you seen similarly singular and fantastic potential in the open source space? Or did I pass over your revolutionary thing in the Elixir space? I’d love to hear about it. Get in touch at lars@underjord.io.

Underjord is a 4 people team doing Elixir consulting and contract work. If you like the writing you should really try the code. See our services for more information.

Note: Or try the videos on the YouTube channel.