Lumen - Elixir & Erlang in the browser

Underjord is a tiny, wholesome team doing Elixir consulting and contract work. If you like the writing you should really try the code. See our services for more information.

The Lumen Project is an alternative implementation of the Erlang VM, more known as the BEAM. It is designed to work in WebAssembly with the specific goal of bringing Elixir and Erlang to the browser.

Note: This guide is outdated. These instructions will not work. Lumen is at the time of this update (2020-12-10) a moving target with an early release out. I plan on making an updated post at some point but will leave this post as it is. These instructions will not work.

• The future, illuminated?
• Try it yourself

Note: Lumen is still a pre-release set of tools. Don't expect anything to work. Don't expect any special features. It is moving steadily beyond proofs of concept and prototypes. But it is not a complete solution or full-fledged experience yet. If you want to know more about the details I really recommend the announcement keynote from ElixirConf 2019, YouTube at the link, embed below.

This is not prime-time yet. With that disclaimer out of the way. The crew that is doing this work have been put together and funded by DockYard. That company is apparently hell-bent on making sure Elixir has enough cool projects to keep this blog running indefinitely. Much appreciated.

The Lumen team is working with the BEAM as their reference implementation, so while this is an alternate implementation that will differ in some details I don't expect this to be a split of the community efforts in any noticeable or detrimental way. As for my part in this, I'm just enthusiastic about it and I'm at most a community contributor. I'm not involved or paid beyond my own interest. If someone wants to pay me to work on Lumen in any capacity, I probably would. But that's not the case so the unbridled enthusiasm is my own.

The future, illuminated?

So what do I think it means? Well. Phoenix LiveView provides one way of minimizing the amount of JS we as developers have to write and allow us to potentially focus on the application and let the harsh separation of frontend and backend fade into the background. Ideally. But if you have an application where the UI is highly interactive LiveView might not be the ideal solution. Server roundtrips can make it untenable, you might have specific needs that concern some web APIs. This is one case where I can see Lumen step in.

Brian Cardarella (of DockYard) has touched on Javascript development being a fairly significant timesink in client projects compared to what it has been in the past. I don't think of this as a big push for isometric applications in Elixir. But it would allow some of that. I think of it as being able to build the web-equivalent of the Scenic framework for web UI. Native web UI elements with the resilience of a supervision tree. Functional principles, not just tacked on in a prototype OOP whatever-JS-is-these-days. It also means that we can have a single project where some things are compiled to the frontend with Lumen and the backend runs on the battle-tested BEAM. I think we can discover ways in which OTP principles allow us to create new and useful web frameworks. And for those of us who enjoy Elixir or Erlang, we can work in our language with paradigms we enjoy.

An aside. I'm trying not to bash on Javascript because the community gets enough flack and that's not what this is about for me. JS and its ecosystem is not my preference and I have a hard time getting excited about JS as the future. I write things in JS and I don't overly complain, I'm a professional, not a baby. I also don't hold it against anyone if JS is their jam. That's fine. Back to Lumen.

What are the trade-offs of Lumen? Well, WebAssembly is still fairly fresh. The whole bindings thing between WebAssembly and Javascript is still early last I heard. So battle-tested it ain't. I think the Lumen team is taking a thoughtful and healthy approach to avoiding unnecessary bloat on the client. However, I think it will still be a bit on the heavier side compared to straight JS or something that is only Rust -> WebAssembly. Time will tell.

Another trade-off is that we are leaving behind the years of proven reliability of the BEAM. Lumen should provide the same potential to build resilient systems. But it won't have the raw history of time in production of the mainline BEAM. So that's a thing.

So what are they doing to make this feasible for web frontend dev? Well, as it stands Lumen provides modules that allow interaction from BEAM languages to the DOM, they are basic right now but the principles seems sound. Size is kept down by instead of providing a (big honking) VM that runs BEAM bytecode they are compiling whatever parts of the runtime are actually needed and only those. Essentially it compiles everything ahead of time and strips out anything you don't need that is in the Erlang or Elixir standard libraries. This kills hot code pluggability but seems entirely worthwhile. So it all compiles through LLVM with a combination of BEAM languages and Rust into a WebAssembly result that you can run in the browser. This is based off of what I've read, heard and seen but I reserve the right to be wrong about the details. I haven't gone deep enough to know first-hand.

The browser is a powerful runtime and that's how a lot of web apps use it. I think the Rust WebAssembly efforts do a lot to empower us developers to start considering non-JS options. I'll happily use JS in the future for things big and small. I just don't love the language and its not my preference. And in the long term, having things like Lumen will make the web a more powerful platform. There are also a lot of new tech that is targeting WebAssembly as a new common language, I think Lumen may end up having surprising potential in environments where the BEAM is a bit big or full ahead-of-time compilation makes more sense.

Try it yourself

Note: Lumen is still a pre-release set of tools.

So can you try this bleeding-edge technology? Yes. I did. And then I hassled #lumen on Slack until I was content that there was something I could show that would be the correct level of interesting and exciting. So now I can share a fairly straight-forward approach to getting started with Lumen. It ends up running the interpreter in the browser rather than the process outlined above which would be compiling everything down to WebAssembly.

There is a Pull Request for the Lumen project as I just updated one of their examples to try making it more complete. You can find my branch and the relevant example here and try it yourself.

So lets walk through it. Lumen is built in Rust and if you want to help them with development you can follow the README at the root of their repo to get a proper project setup. But to simplify, here come the compressed install instructions for MacOS. Start out with cloning the project git repo from my GitHub, checkout the `interpreter-demo-updates` branch. We are using my branch because the PR is still in flight. I'll try to update this when it has been merged. Update: The PR for the example has been merged, you should just go ahead and clone from the main branch of lumen.

git clone https://github.com/lawik/lumen
cd lumen
git checkout interpreter-demo-updates
cd examples/interpreter-in-browser

Set up the basics you need for running the Rust stuff:

# If you have rust installed through homebrew,
# you may need to uninstall it
brew uninstall rust
# Install rustup
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# Activate the environment to not have to restart the terminal
source ~/.cargo/env
rustup default nightly
rustup target add wasm32-unknown-unknown --toolchain nightly
cargo +nightly install wasm-bindgen-cli
# Install wasm-pack
curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh

# Build the example
wasm-pack build

# Link the packages
pushd www
npm install
popd

pushd pkg
npm link
popd

pushd www
npm link interpreter-in-browser
popd

Run the thing:

cd www
npm run start
open http://localhost:8080 
# Or just open the browser to that address ;)

To try your hand at sending new code to the interpreter, just edit the Erlang code in www/ex_sample.erl and it should reload (or you refresh by hand).

I wrote some instructions for compiling Elixir code to Erlang code and using it with the interpreter, you can follow them here.

If you are interested in contributing to Lumen the team has been helpful in the #lumen Slack channel on the normal Elixir Slack and they've asked for help getting all the BIFs implemented previously. So I'm sure they'll help an enthusiastic contributor find their way.

I think the future looks bright. I'm very hopeful and optimistic about what Lumen might achieve.

If I screwed up some piece of the instructions, have an entirely inaccurate take on something or even better you have something positive to add I'm available at lars@underjord.io or on Twitter where I'm @lawik.

Underjord is a 4 people team doing Elixir consulting and contract work. If you like the writing you should really try the code. See our services for more information.

Note: Or try the videos on the YouTube channel.