The biggest reason for this is to make an ice-breaker for people that are attending the conf. People who would enjoy a way to engage with others up front in a practical workshop kind of way. We’ve done shared hacking on PRs and issues. We’ve done more or less a prepared workshop. We’ve done a few varieties. I think this also spices up the conference period a bit.

This one we’ll have some hardware by Gus. Prototypes for the Nerves Starter Kit that we can work on. We hope to also have a scheme where you can pay the cost of the hardware if you want to keep it. Or maybe you dazzle us with your contributions so hard we throw it at you. One never knows.

You can learn more about the Nerves Starter Kit in this presentation that Gus did at the EU online meetup (which is a lovely time and happening today actually, Marc Lainez is sharing his porting efforts). The point of the starter kit is to be a common target platform for playing with Nerves that we can offer for a long time, fairly priced, interesting variants built on open hardware designs. Rather than buying a bunch of disparate Pi parts and hoping for the best in terms of support you get something cohesive and supported right away.

So we plan to have variants of the hardware for this event and we’ll work on the firmware/software and application side. You can actually grab the project and hack on it in the simulator entirely. Whether you feel like contributing on-hardware functionality or making the docs really shine. We’ll sit down, play, hack and socialize.

The event is co-organized with The Erlang Ecosystem Foundation, massive thanks to Dan J for finding a venue. There will also be an Elixir Unconference happening at the same time. We hope you’ll throw in a 10 Euro donation to the EEF for our trouble. More details here.

You can register for the event here, same registration for the Nerves hack session and the Elixir Unconference.

Hope to see you there :)

If you have questions you can reach me on the Fediverse @lawik@hachyderm.io or via email lars@underjord.io.

Goatmire Elixir is my (Lars') brainchild. I like the idea of a quirky event, smaller scope and people who are deeply enthusiastic about related things. So I made one, for the Elixir programming language community. In the little swedish coastal “city” of Varberg. And it was a blast.

The photo gallery is up to get a sense of the amazing presentations and all talks are available online, also on YouTube. This kind of conference can only happen thanks to companies willing to take a gamble on an idea, like Tigris did. I’m very thankful.

We made a name badge. People do that for other conferences to. And if they make a special hardware name badge they are usually based on some reasonable microcontroller. Maybe LEDs or an OLED screen. This one is a bit wild. It is a small Linux-capable device built around the Allwinner T113-S4. A 1.2 GHz dual-core ARM SoC. The hardware design is called Wisteria and is based on a core board called Trellis both are open source. Designed, implemented, sourced and made real by Gus Workman. Importantly this thing has an eInk display, a battery, a few buttons and Wi-Fi. Because who doesn’t love eInk.

A big reason to make it a Linux-capable board is that one day of the conference was NervesConf EU. Nerves is a framework for IoT devices of the Linux-level class. So not microcontrollers, rather things with an MMU and a tractable amount of RAM. Essentially we run a barebones Buildroot-based Linux OS and the first process that boots in turn starts the legendary BEAM virtual machine. The thing Erlang runs on. Also, the thing Elixir runs on.

The reason a bunch of us like to work with Nerves is that it gives you these very strong error handling behaviors and lets you build devices at a higher level of abstraction. There are other cool things like the open source firmware update service NervesHub . That is also available as a commercial offering called NervesCloud which delivers firmware using .. Tigris!

Mostly Nerves is about Elixir and what a lovely language that is to build devices that deal with a lot of communication. Elixir inherits and builds on Erlang’s fantastic ability to deal with binary formats and networking protocols.

For this device we wanted to make it interactive and continuously helpful during the conference. So it would load the schedule for the conference. That’s helpful. It would allow you to show your name and title as a good name badge. And being eInk you could switch it off and it would just keep showing that.

It also had a photo gallery and the way this worked was that we let attendees upload photos from their phone to a Phoenix LiveView web app, the backend service for the whole thing. We’d moderate the inbound photos and whatever we approved would be converted to 1-bit black-and-white, with dithering, for display on these monochrome devices. Then they were downloaded and displayed. Rendered in the UI using Typst, a Rust-based rendering library.

Of course all media handling of uploads went through Tigris with pre-signed uploads and downloads. We didn’t really get a lot of mileage out of the distributed regions and all that considering we were the very opposite of geo-distributed at the time. Everything except our local Wi-Fi worked great.

A nice feature is that all devices that were switched on would actually show the same image as the server will push new pre-signed URLs out at an interval to all connected devices. Very neat and typical Phoenix shenanigans.

There was a lot of people poking at and hacking on the badges. The most critical thing, aside from some rather urgent bug fixes, was an implementation of Snake. The classic cell-phone game was implemented by Peter Ullrich during the conference.

The community will keep iterating on this software and this hardware. Another batch of these were made and sent to 39C3, the Chaos Computer Congress, for some people to hack on. Gus recently presented on the Nerves Starter Kit which expands on this device with more costly and capable peripherals which will be a great boon to people getting into Nerves. And with the open nature of the hardware and software may well get supported and adopted in other ecosystems.

On the 22nd we are planning a small community event to hack on prototypes for the starter kit leading up to ElixirConf EU in Malaga as Gus is presenting in depth on the development of this kit. We have people considering putting wheels on it to make it a balance robot. The idea is to make something extensible and hackable that people can push further.

The Tigris crew is nice and friendly. We’ve worked together in the past and I don’t think they had a big plan funding those boards. I think it was a fun way to sponsor a cool thing and show their support of fun community events. There just happens to be rings on the water from an act like that. The version of the device that was made for Goatmire Elixir thanks to Tigris will ensure more iterations of open hardware and more people getting their first crack at running on hardware.

While I know Tigris can ship large files, globally at massive scale. Sometimes people want to work in the small and it also works for shipping 1-bit images, locally at a human scale.

Thanks. From me, the Goatmire team and the Nerves community.

If you have questions about any of this I’m around on the fediverse @lawik@hachyderm.io and on email lars@underjord.io.

I say dense. Because massive success would make it sound large. And it wasn’t a large event. 150 attendees, with speakers and volunteers we were around 200 in total. And it went incredibly well. The feedback, the surveys, the outpouring of love and appreciation were all way above what I dared hope for. People have been graciously giving me space to consider a second one but it has also been the single most requested thing ever.

September 30th to October 2nd, 2026. You can sign up to be notified on the website. In our little town of Varberg in Sweden.

Unchanged

Lots of things will be the same as last time. Venue is the theatre. Lots of the practical bits that worked will be kept the same or be iterated on a little bit.

You may notice that the visuals have changed though. You can still see the Goatmire 2025 website and compare it to the new one. Part of what made Goatmire special is that people knew I intended to make it a bit different but they didn’t know what to expect. And I believe that one of my more important tasks is to be a creative influence and encourage speakers to take things a little bit further than they might typically do. And to scrub every ounce of sense that this is a corporate or commercial event. There are commercial aspects in terms of making the budget work out. That is not the focus. This is to be an experience for the community. I want to reinforce the connective tissue of the community. Have people connect and make memories. Feed the enthusiasm and appreciation that I see in so many spots around the Elixir space. Most of it is thematic, creative and abstract. Very hard to know if you hit the mark.

I feel fairly confident about the theme for this one. I’m not going to be putting it into words because it hasn’t been stated more than what you see on the website. Everyone can start interpreting what it will be and I’ll try to shape and mold that as we go forward.

Some of the “tricks” I pulled for the first one can’t be the same for the second one. Or at least that’s how I’m wired. And considering how many people have said they want to do a repeat visit I’m probably right to not make it too similar. And lots of things can only be done in collaboration with the speakers and their creative intent. I don’t know what we have to work with yet. It’s all very exciting.

From 2025 we had some talks that stood out in concept and character. And the Nerves Car is now well-known. Saša Jurić has already told us a story. Zach Daniel has already broken time and space using Ash Framework. We’ve done the distributed system murder mystery and we’ve ended on a phenomenal live-coded musical number. So all of those are out. Not necessarily the people. Just those presentations. I’m sure we’ll have a different mix of speakers as well though I certainly hope to see many of these in the CTF when we launch that.

I also still plan to co-locate NervesConf EU with Goatmire Elixir because it worked really well. The double-branding is a pain in my ass and I don’t have a clear answer to how I want that to work. Maybe NervesConf EU doesn’t get as much visual space this year. I don’t know.

Changes

We know catering will be different. Things have changed at the venue. Because the lady that had the contract on the food for the venue cancelled it and left to do other stuff. We’ll probably make similar choices. Maybe communicate more aggressively up front that the food is vegetarian so if that’s not your jam you might need to look around. But we don’t know much about the catering situation at the moment.

We plan to have more tickets, more people. The venue can theoretically absorb 350. But I think like 10 of them wouldn’t see the stage. We want more people to be able to attend. From the first one it seems like interest is very high. We have more people on the interested mailing list than we had tickets last year. People have suggested it will sell out like Woodstock and we just have no way of knowing what to expect. Compared to last year, I’m not worried about reaching my minimum number of attendees to run the event. But the space has limitations for how many people can comfortably hang out and socialize outside the seating area. I’m excited about upping the headcount because it ensures that we’ll have people that weren’t at the first one and it should boost the social aspects. It still won’t hit a size where it is an intractable mass of bodies and people become anonymous to each other I hope. I don’t think that’ll be an issue.

Also at 350 I imagine we’d choke to death. Just upping the headcount by 50 will mean that door discipline becomes more important to ensure air flow. It is an older theatre and while Taun Chapman actually put a (Nerves powered) air quality sensor in there during the event and it was never particularly bad it does get a bit toasty. It isn’t typical for theatre to be a full day where people are in the seats for most of it. Gotta not put everyone to sleep.

Another plan is to tweak the contents in various ways. We got some feedback that requested more technical contents and deeper technical contents and I think that’s appropriate. Some of our balance shifted due to speakers not being able to show up and others stepping in. Some talks end up not as technical as one had expected. It’s tricky. I think we can tweak that balance. The feedback felt applicable. Anyone who noted that they wanted more technical depth and who skipped the NervesConf EU day made a gross miscalculation though. NervesConf was crunchy.

We also had a very packed schedule. More than 30 speakers across 3 days, single track. We will probably leave more air in the schedule this time. This means the selection process for the CFT will probably be brutal for us in the committee and competition pretty intense for the spots. We said no to more than 40 talks last time. If interest is higher this year and slots are fewer.. oof. Good content requires good editing though and this is editing. Good talks will be rejected.

Weird and creative shit. Gonna wanna do things. Not the same things. Not gonna tell you what.

Hopes and aspirations

I want to run things on the dates leading up to the proper conference. Many conferences have training days. And I’ve personally organized a few satellite events for Elixir and BEAM conferences where we gather and hack on Nerves stuff. They are always a ton of fun. Hands on keyboard is a very good social environment for some devs. It also helps some people get their first PRs in or maybe try a project they wouldn’t have tried otherwise. So what I want to try and do is pull together about 2 days of workshops that are some mix of trainings, project sprints, getting-started and weirdly specific things. Free or at cost and to whatever level of ambitions the community maintainers and collaborators want to put into it. And I hope that local students will show up for these as well as conference attendees getting a few more days off work to get free training from the most knowledgeable people in the ecosystem. I think it’d be rad. Plans are still early.

My hope is that we have more satellite events. Like we had Ash Summit 2025 after Goatmire Elixir last time. It was very neat. Happy to repeat that and happy to have more variants.

I want to support speakers in making their best presentations in a few more ways. I’m happy with the mix of newer, less known speakers and big names we had. There are certainly ways to improve the mix. And I want to surface more folks that I’ve heard of that others have not that do cool stuff. But I also want to help people that might play it safe to take bigger swings. Ideally we can help people with polish, practice and such. I have some plans. It is a tricky topic but I think we can do more to ensure great presentations. And I’m very happy with how talks were in 2025. I just think we could potentially do more.

Closing

That’s it for now. Goatmire Elixir 2026 has a date. Sponsorship prospectus is currently being drafted. If you know you are interested in showing up as supporting the event, feel free to reach out. If you have questions, concerns, ideas or plans and want to discuss, I’m also very available. I look forward to making another one of these things. Big thanks to the community for the beautiful support of last year and of the announcement of the new one. Deeply touching :)

Thanks for reading. I’m available as email lars@underjord.io and socially @lawik@hachyderm.io.

Snap to today. The event has now happened. Results are in. I am very satisfied. This is my conference report, as the organizer. Because of my deep level of involvement it may end up being quite a long post.

Conception

I can’t put a date to how long we’ve prepared for the conference. But about 6 months at increasing intensity. I picked up the bug of wanting to run an event when I visited Gig City Elixir in Chattanooga in May 2024. All credit to Maggie and Bruce Tate for that, they have a special thing going. A few things really made it a brain bug back then. Todd Resudek offered that I could freely use the NervesConf name to run an EU event. Very kind and I definitely want to gather the EU Nerves community. And when I chatted with folks about running something in Sweden they seemed optimistic. Andrea Leopardi of Elixir core team fame confirmed himself as my first speaker then and there. In the middle of Rock City.

Bruce and Maggie Tate being lovely hosts during my trip to Chattanooga. They hosted me in their home!

Rock City in Chattanooga

It probably took most shape during the first quarter of this year (2025). The name, sourced from a historic name for Varberg, Getakärr, can be fairly translated to Goatmire. It was distinctive. In fact I think I got about 20 minutes of extra air time in the Thinking Elixir podcast just on the name. I bet it lived rent-free in many people’s heads. My particular sense of humor would be a guiding blight throughout the process.

There were a few things that confirmed it for us. Talking to some of the people around the Varberg tech hub Techarenan was encouraging. They (thanks Sara!) put me in touch with the public sector commerce folks and they were very enthusiastic and gave me a warm introduction to Sparbanksstiftelsen. This is a foundation that owns Varbergs Sparbank, the local bank. They reinvest in the local area and after some conversation we sent a suggestion, a rough budget and we applied for a stipend.

It took a minute to confirm but the year was very young when we got our base funding from them confirmed. Then we started spinning up the non-profit registration. Gathered recommendations for venues and talked to them. In the end landing on Varbergs Teater, the theater where me and my wife married. We chose it because it was the most interesting and the most flexible to custom ideas and is still smack in the center of town. There are many good venues in this little city since it is a swedish tourist-destination but I wanted something with a lot of character and I knew the theater could deliver.

The lovely Varbergs Teater. Picture provided by Varbergs Teater.

Beach in Varberg. Credit Natalie Greppi for Varbergs Kommun.

Town of Varberg. credit Natalie Greppi for Varbergs Kommun.

Some basic funding and a venue meant we had a starting point.

Event architecture

One day NervesConf EU for IoT with Elixir and two days Goatmire Elixir for regular Elixir. That’s the model of Gig City, so I yoinked it. Not reinventing the wheel in terms of structure. I grabbed the scheduling from Code BEAM Europe as well. And you’ll see on speaker’s day, I nabbed a concept from Öredev. I suppose this makes me a great artist. I hear they steal.

I wanted to make the event distinct. I don’t think we actually gave a ton of guidance to speakers towards that. The most relevant bit from the CFP was:

“We will not be taking on talks that are “Update on project X” but are fine with talks that involve new features and either how they were built or what they allowed you to do. Avoid presenting a changelog. Teach something, tell a story, blow some minds or show your work.”

I wanted to do wild things but I’m also new at making events and didn’t want to make my first time completely impossible. I didn’t push for people to take exceptionally bold swings in the CFP but in every conversation I ended up encouraging the more interesting choice at every turn. And there were a lot of conversations.

Importantly I’d heard the great things people had to say about EMPEX and The Big Elixir. A single track conference in a great venue will be a very focused affair. I had some faith that we could deliver something similar though we are decidedly not New Orleans or New York levels of cool. But the town punches above its weight.

3 days, the first probably smaller, single track. That’s the rough of it.

The crew was me, my wife and our then-employee Tomie. Tomie has about 10-15 years of event planning experience and helped get us started well but they would then leave Underjord to do great work over at Entryfy an awesome client of ours (with our whole-hearted, mildly heart-broken support). Entryfy would also end up sponsoring the event. We brought in a new acquaintance. Helene Mattisson, wife of swedish Elixir old-timer Johan Mattisson. She helped us stay secure in the knowledge that we weren’t missing crucial know-how and details. Like having insurance. She is a gem, and available to contract for event work.

Building it up

I had a single speaker (thanks Andrea!) so I started reaching out to my network to pin people down. The Nerves core team for NervesConf, good collaborators for more Nerves talks. Some solid names for the Elixir days of the event. Added myself for padding, since I’m probably the most visible Elixir dev in Sweden.

The procedure for getting sponsors was similar. Tap my network, both the local one in Varberg and the one in I have in Elixir. My sponsor journey went pretty well overall. I had a late plot twist of a sponsor pulling out shortly before payment was due but someone picked up the mantle quickly after (thanks Alembic!).

At some point I launched the waiting list. We estimated we could deal with 150 people. The waiting list climbed to 90 people very quickly. Over time it drifted up to 200 before we launched tickets. If there is something I’ve worked up a muscle for in the last 7-8 years it is “marketing”. That is, I am already actively posting on socials. I have a newsletter. I have a podcast. I know other people with podcasts. I get around.

Excitement was building way better and faster than I had dared hope. I was fully ready to run the conference at 90 people.

The appetite for a community-run less-commercial Elixir event was definitely there and another one popped up essentially in parallel with AlchemyConf in Braga, Portugal. But they were in March, we were targeting September. I had already decided on the theater when they announced their own (larger, fancier) theater. They were first on using Discord as a conference app though. I jacked that from them.

Over time design, ideas and concepts started to crystallize. The pre-CFP line-up looked great. The CFP made it fantastic. I had great help from the program committee I recruited (thanks Sophie, Rebecca and Cornelia!). The event gained steam. And speakers kept getting back to me with weird (awesome) ideas and requests which pleased me to no end. Gus Workman and I conspired to do something with hardware. He did all the work, I found the money. Supported by old compatriots Tigris.

The day before the day before

The emptiness was heavy. I felt like I had missed something. Probably everything.

The Discord was popping with people travelling, making board game plans and announcing their arrivals to the swedish west coast. Lots of cute observations about the strangeness of various parts of Sweden as people traversed it. I was on high revs and completely idle. Thankfully I had an errand I could run. It was both practical and an excuse for me to say hi to some people. We literally didn’t have more we could do at this point. Very close to “go time” but many hours until we could pull any triggers.

I shoved a bunch of Nerves hardware into the car and drove into the city (I live in the countryside). I would stash it in Gus' room until he arrived and could deal with it (oh, how he would have to deal with it …). Three boxes of devices with less-than-good 3D-printed cases and one full box of actually good 3D printed cases. For 200 devices in total.

Parking outside the speaker hotel, Gästis, I wrangled boxes in there and went to the reception desk. As I waited a bit for the receptionist I spotted a group of 6 or 7 familiar faces. From memory it was Frank Hunleth, Rebecca Le, Jonatan Männchen, Pascal C, Srinivas Ganti and Dan Janowski. Pascal was the only new acquaintance and he was subsequently everywhere. They helped me stash the boxes in Gus' room and I had already cleared permission with Stina (my wife) to grab dinner in town with people. So we acquired a Josh Price and a Sam Aaron at minimum before we sauntered out to find some swedish-style pizza for the weary travelers.

I went home shortly after the pizza, more people accrued as we ate. I got to say hi to Flora and Tom Petterson, just in from New Orleans. We tee-hee’d in secret about the massive part they still had to play in making the event memorable.

Setup & Speaker’s Day

The day before the conference started was planned for setup. We had access to the venue from 12-16 to rig stuff. Then we had plans with the speakers of the event from 16 into the evening. Me and my wife absolutely stuffed the car with stuff, including the plywood goat, drove into town. Parked at the venue and went for food.

The goat. A statement art piece by David Qvarnström. All event photo by Petter Boström.

Srinivas was a champ who showed up immediately for the prep though he wasn’t even originally a volunteer. He just wanted to help and that he did. After a while Flora and Tom rolled in and started the wildest testing for their talk. We’ll cover what they did later.

Zach Daniel showed up to test his talk which required some special effects. A few others also came to test, help and so on. Badges, lanyards and shirts were prepped, thoroughly optimized by Dan J, executed by Helene and Srinivas. Plans were made. Roll-ups placed. We were pretty much ready.

An extra wifi-network was added for Nerves-devices. Would it be sufficient? Of course not. It never is.

Essentially, setup went super. The tech, Tobias is an absolute mensch. What a solid calming presence, absolute pro. I also can’t recommend dealing with a theater enough. Where a conference hotel venue will go “You want to do WHAT?” a venue like this will go “Typical thursday, sounds good. Need lights or smoke with that?”. They are used to doing the difficult thing.

Right before 16 o' clock me and Stina left the venue to get drinks and snacks to Kallbadhuset (The Cold Bath House) where the first speaker social event was. Sauna and a swim in the ocean, rince and repeat. Good way to break the ice. Completely stolen from the Öredev conference who do the same thing in Malmö. People are various level of comfortable, makes for fun conversation and it was all optional. Good view. Nice old place. Not everyone attended this but it was a good time and a good start. Then we sauntered over to the restaurant, NAMI. Japanese fusion tapas. Great place.

Kallbadhuset, photo credit Natalie Greppi.

Also photo credit Natalie Greppi.

I was already very happy. Seeing some of my best community friends drift around my town. Getting to be their curator and having them come to the experiences I chose, definitely brought a smile to my lips. Watching the speakers gather, most of who I already know from the community. What a feeling. With some extra friends included we were around 36 people for dinner. Food was great. People had a great time connecting, reconnecting and so on. Then some of us drifted over to the bar for a bit.

That was a slight mistake on my part.

Day 1: NervesConf EU

I did end up staying out a bit longer and having a few more drinks than I had planned so I was definitely not at optimal energy when I dragged myself out of my hotel room bed around 6.15 in the morning. We slept in the hotel to not have the travel time and the kids were with my in-laws which was a life-saver. Sleeping through the night without bonus dialogue was a big win.

Get into outfit #1, trashy, my most worn metal shirt, the authentically worn-through jeans, studded belt, keychain, Converse All-Star, red plaid shirt. Throwback to my teens. Underscoring the informality of the event. Breakfast at hotel. Get to venue, setup from 7-8, doors at 8, seating area open from 8.30. Start at 9.

Trash outfit. Hair was accidentally messy but I suppose it fits.

Observe the holy jeans.

Great start. It starts on this video. Then Marc Lainez is off and delivering a talk on The Nerves Car project. An audacious electric mod to a WV Polo using a variety of Nissa Leaf, Tesla and other parts. Brains with Nerves, UI with Flutter and tons more interesting detail. I was probably still mildly hung over but at some point I just forgot about that. If you are curious they have shared a lot of detail on it.

All talks were recorded and the absolute majority will be released.

Then we are off. I won’t go blow by blow. I loved so many of the talks and I think everyone put their best foot forward. I’m very pleased that my starting strategy worked. Cold open on a talk. The initial keynote speaker is the welcome. The event is not about the event, it is about the contents of the event, talks and people. As the MC I spoke after. Kicking off the event when it was already in full swing. Having earned some of the space to talk about sponsors and what the logistics of the day look like.

Later in the day Gus Workman gave his talk. And every attendee in the audience had one of his Wisteria devices, aka. The Name Badge. A special eInk board based on his Trellis design. We hadn’t gotten the batteries yet but if you plugged it in to USB-C power, it totally worked. He told the story of the device, pretty awesome. The project of getting these devices was real tight to deadlines and Gus pulled an enormous workload (electronic, sourcing, software, manufacturer relations, all him) to do it (on his own volition). May it bring him great business.

Wisteria, the name badge.

By Gus Workman, Protolux Electronics.

Gus fixing and hacking on boards with a crew.

If there was one thing that didn’t go at all as planned it was the badge hack session during this day. It was fine. We covered the state of things, Gus shared more about how it worked. But it was planned to be more workshop and hacking. It didn’t turn out that way because we hadn’t had enough stuff prepared. It was the most shit-show thing of the entire day and it was .. fine. Not very shit at all.

I also tried not to close on logistics. “Don’t open or close on logistics.” is one of my big takeaways from The Art of Gathering by Priya Parker. Love that book. During this day I did say a few words to wrap up the day. This was just after the last speaker, Damir Batanovic, had impressed us thoroughly with his drone and video shenanigans, but I had handled the end logistics before he spoke.

MC:ing felt smooth enough. I even got a chant going for Frank Hunleth and got to introduce him as if he were a pro fighter. I had a really fun time facilitating, informing, introducing throughout the days. I got really comfortable and we had few issues, just dealing with the occasional HDMI issue.

Damir Batinovic flying one of his drones.

Damir showing the camera feed.

Frank and the golden rules of restarts.

Feedback from the first day was extremely positive. I went to my hotel room, I showered, I tried to rest. I failed. I revised MC notes to update them with some missing details for the next days. I had food alone, in quiet. Then I went to the bar for a little bit, got to hang with the people and catch the good mood. I had a single beer and was in bed by eleven. I slept well and was in good shape the next day.

Day 2: Goatmire Elixir starts

Outfit. Very fancy. Cravat, waistcoat, blazer styled like old furniture. This time to show people that we are taking it up a notch. Pack extra outfit and lots of gear because this is the weird start. I have a part to play. Breakfast. At the venue from 7.10 probably. Attendees get in at 8. We kick off at 9.

Fancy outfit 1.

In the green room (another perk of operating in a theater) activity is frantic and fun. Zach Daniel has tapped me, Sasa Juric, Shannon and Parker Selbert into speaking roles in his keynote. And then on top of that a whole Ash ensemble of Josh Price, Rebecca Le, James Harton and Barnabas Jovanovics. We are all outfitting ourselves. This talk is weird.

Sam Aaron was jamming introductory music as attendees filtered in and got settled. When he walked off stage the intro stinger played. This was my attempt at capturing the sense of Goatmire Elixir in my head. I had asked speakers for short clips to include. They obliged and I cut this together. Speakers had seen it before the event. Attendees came in cold.

Cue Zach. I’ll spoil the talk a little bit. He does not actually announce a new, real, Ash library. But he does break the timeline. One after another we bring him the stories of broken timelines related to the Elixir ecosystem.

It was a whole thing.

Zach, guilty of the whole thing.

mix new solar_system

Ash team assembled!

Future me.

It was also another cold open. I end up needing to MC in my future post-apocalyptic time traveler kit for a bit as I kick off the day and lead us into the next thing. Then I hurry off to get fancy again.

From the day before I have the feeling down for how talks go for me as the organizer. I start off on stage introducing the speaker, I make sure they get rolling and possibly help with HDMI or stalling for time. Once rolling, I leave and go check on things and let some of the nervous energy that I’m made out of simmer out by walking around a lot. Rinse and repeat.

Many great talks through the day. The sense of excitement and fun among the attendees was fantastic. We eventually received the Wisteria batteries and handed those out. The devices were not just for NervesConf. Everyone got one. The mood was stellar. Catering did well, coffee was available, people were generally not hungry.

Towards the end I gave a few notes, talked a bit of logistics. And then handed things off to Sasa Juric for the last iteration of the year of the talk he has been touring that has not yet been released on video. Some say we got the best version. Tell Me A Story.

Sasa Juric - Tell Me A Story

Feat. the Skull

It was sublime. I was riveted. When the Erlangelist asks you for 55 minute slot to present a three act monodrama you give it to him. When he asks for a skull you order the skull. And then he will enrapture your crowd and grab a standing ovation on his way out. And he really thoroughly lectured me on making better PRs…

The energy of this evening was wild. I’ve never had so many people thank me in my life. I’ve never had so many people tell me about my own event in bewildered excitement. I could not be smiling more.

Partially I was smiling because I was one of five people among the nearly 200 involved that knew how the next day starts. Everyone was curious what to expect from another day at Goatmire but no-one was ready. To make sure they didn’t discount the start, I pinged @everyone on Discord and teased that they should not miss it.

One beer. Maybe two. In bed at a reasonable hour.

Day 3: Goatmire Elixir ends

Other fancy outfit, darker, more somber. Breakfast. Head to venue. Piano music hits me the moment I enter. It is on.

Somber fancy

I knew Flora would bring a good talk. She’s an engaging person, fastidious about delivering and smart as a whip. But when she had seen Sasa give Tell Me A Story at Gig City Elixir 2025 she realized what you could actually do at a conference and she asked if she could completely change her talk.

Her mother did puppetry professionally. Flora has done a bunch of puppetry with her mother, internationally. Our conversation spiraled into shadow puppets, old-school overhead projectors and repeatedly asking “is it okay if it’s weird?” at which I would respond “fuck yes, please, kindly make it weirder”. My favorite was “can I bring my husband so he can play piano?”. I was in deep already. I was game for anything.

The grand piano was out. The light of the old projector was suitably flawed compared to the previous day. Another cold open. It could not have been more warm. As she spoke she found ways to put tears in my eyes. Odd that.

The puppetry was riveting. The scissors were .. abrupt. It was a beautiful, awe-inspiring performance. The standing ovation was a given. While overhead projectors have certainly seen use in computer science before I think this was truly unique. I could not be more pleased.

Flora's welcome

Flora's story

Flora speaking

Puppeteering

Piano music and birds

This talk was recorded (technology be willing) but it will not be published in the near term if ever. That is not under mine or Flora’s control for now.

The day continued with people somewhat stunned from that initial presentation. Later The Oban Murders also stuck out as a talk with fabulous showmanship and a very funny performance by the Selberts.

The Oban Murders

I also gave my talk during this day. Prior to it I stole a nap, or at least zonked out, in the green room for a bit. I suppose the days had caught up with me. When it was time for my talk. I had slides. I had notes. And maybe I should have used my slides, for clarity, but I felt so comfortable on this stage at this point and I knew the contents of my 5000 Erlangs topic so well that I just went for it with a demo in hand and a little bit of web browser. I think I stole those style notes from Sam Aaron’s earlier talk on Tau5 actually. I think my talk went fine or even well. I did not know that I had a good line-up at the time when I added my own name to it. If we do another one I probably won’t bother speaking though I do enjoy it.

As we approached the end I had the luxury of addressing the crowd for the last time. I was just launching into a few more personal emotional notes about what this all meant to me when the frantic technical rigging happening behind me really needed a UK-to-EU power adapter. We improvise. Thanks to Oferlund for hooking us up. He also provided about half the music used in the breaks. The other half was local metal band Tvivel. The ending notes were a pleasure to give. A bit of logistics, lots of thank yous. I was so pleased with the entire event and I was excited for the final presentation. Things were good and some adapter tech issue improv did not shake me in the slightest.

Then Sam Aaron played us home by teaching us the fundamentals of music, the fundamentals of programming music and then showed us what it can truly do. I spoke to at least 3 people in the staff who are not programmers who expressed interest in trying their hand at Sam’s Sonic Pi tool. I have already tried it. I love it. You can learn it with Sam’s course that supports his open source work.

People who’ve never seen Sam Aaron do his thing before were in for a treat. People who know him were thrilled. It was awesome.

Each day the venue staff were a bit close to the limit for how many hours they were allowed to work. Not over it to my knowledge but close. So we tried to clear out promptly. I’d shouted people out of the venue both evenings prior. When I tried it this time. Someone shouted about thanking me and suddenly, in the fully lit theatre, not full but well seated and with two balcony levels of people. I got a standing ovation. This was hard to absorb. It was moving. It was very gratifying. Thank you all.

Then I ran them out into the streets for real. We tore our stuff down and let the staff do their thing. Me and Stina, my wife who was on logistics and reception throughout the event went to eat with a couple of speakers and then joined the impromptu party at Bank 28. The bar had offered a discount for anyone with a badge so it became quite popular.

I was not as careful about bedtime this evening and I didn’t have to pay for a single drink throughout the night. People were very kind.

My heart was full. I was just happy. Satisfied. It was a very weird state of being that was thoroughly enjoyable.

Bonus Day: Ash Summit 2025

Because we accidentally accumulated most of the Ash framework’s core team in person for the first time with this event I helped Alembic put together a little shindig at another venue on the Saturday. Being responsible for the keys I showed up and checked people in, watched the door and goofed off on my computer.

I am actually quite interested in Ash but I was checked out. Not sleepy but bone tired. People seemed to enjoy the summit. I think we checked 56 people into this sideshow. Then I had thai food with some friends. Then I went home. I slept a lot that weekend.

Wrapping up

There were a few major things people brought up when they grabbed me quickly during the conference:

• “Thank you for making this happen”
• “You must be so tired!”
• “Will there be another one?”

On the first. I mean “you’re welcome” would seem a bit flippant or arrogant. But essentially that. I was also happy. I was glad they were glad.

On the second. Surprisingly not that tired throughout the event. I feed off of this type of energy. I certainly burned a lot of resources and needed to catch up on it after but it was never brutal during the event for me. Which is surprising. I thought it would feel worse but I do often thrive under pressure, it has a focusing effect on me.

I want to make another one. It just needs to not screw with my day-to-day, work-life balance and vacation time too much. This one was brutal during July especially when I’d normally be on vacation I had tons of print designs to finish, lots of random things to check off and meanwhile I was letting my team go due to the market being terribly poor, we were building a green house which took a lot of time and effort, I was trying to not lose too much income so I was trying to do billable work. It was honestly a shit-show. It wasn’t mainly because of the conference but the conference was part of the bad workload. It was a really crap moment in time. We did okay through it but neither me or my wife appreciated that part of the experience.

There are many reasons to believe we can do it with a bit less grind on a repeat because we know more about what we are doing now. We would keep the venue, the accomodations. There are things to make simpler, there are things to control better. We can hand more of it to Helene this time because we know what we want. It will never be an easy lift if it is going to be an impressive event. The effort shows in the result. We will make a decision but not quite yet.

You can find a list to be notified about any next events, if you go to the website.

I am incredibly pleased with how it came out. Thanks to all who contributed to that. My family, relatives, the staff, the sponsors, the volunteers, the speakers and the attendees. The town.

The response has been overwhelmingly positive. Thank you.

Disclosure: I am running a conference called Goatmire Elixir which Ampere is a sponsor of. This post is not part of the sponsorship exchange as such. It is prep for my talk for the conference which uses the hardware they lent me. So this is your transparency notice, but fundamentally I am not making comparisons on whether they are better or not. I’m learning and sharing about qemu and virtual Linux machines. Now I’d love if they paid me to shill them a bit later and I’d be transparent about that too. But this is not that :)

To recap. We have an Ampere One 192-core machine with 1 TB of RAM. The goal is to run as many virtual Linux IoT devices using the Nerves framework. We got 500 of them last time before I tried pushing any further. I also got a bit further on the same setup when I tried. Maybe 1000, I don’t recall exactly. But there have been developments, so read on!

Briefly on Nerves: the framework treats the BEAM virtual machine like the OS and essentially only uses Linux for a kernel, drivers and the like. This means we can write much if not all of the embedded device in a productive high-level language running on a provenly robust and reliable environment with memory safety and solid recovery strategies. And it means your cloud integration developer doesn’t risk seg-faulting the entire device while mangling JSON back and forth. Nerves also brings some best-practice tooling and conventions. Your init process is erlinit, your updates use fwup to provide A/B partitions and factory reset, auto-failback, validation of firmware viability, disk encryption, delta updates, streaming updates and a bunch more.

The most interesting development is the thing you can probably learn the most from. Frank Hunleth who has been my co-conspirator and a massive help saved me from fighting u-boot by .. writing another bootloader. Introducing little_loader. This adorable tractor will load up your ARM64 qemu device, consult the uboot environment that Nerves uses, find a Linux kernel from information in that and then boot. Consequently it enables the A/B upgrade features and everything else that makes Nerves great.

Writing a boot loader is a little bit ridiculous. Frank knows his way around C and apparently ChatGPT knows a fair bit about ARM and qemu. Enough to be dangerous. And where it was wrong he could rummage around until he found the way. How he does what he does is beyond me but the result is a very small boot loader that you can probably read through and understand. So if you are curious about booting ARM64 or about how qemu starts things the code should be a worthwhile read.

We got a bit tangled up in EL1 vs EL2 when we only ever needed EL1 to work. EL2 on ARM is what you’d run under if you want to be able to run VMs in your VMs so you can VM while you VM. And the version of qemu + KVM I got from Ubuntu doesn’t seem to support that. We weren’t interested in it either. At some point we might explore EL3 for secure boot and whatnot. Only time will tell.

One of the weirder challenges and something we haven’t disentangled yet is that we have some compilation issue where using the toolchains I was using the non-debug build would hang while the debug one runs fine. For now I run the debug build of the bootloader. I think it was fine from GCC 15? Anyway, hopefully we pin that down at some point. But it tripped us up a few times when the bootloader would hang due that issue rather than any actual problems with the implementation.

KVM didn’t really require anything extra aside from making sure we didn’t go to EL2. And when we tried it on MacOS it worked great with HVF as well. Host-based ARM64 VMs are ridiculously fast and practical. As in booting to the full IEx prompt in single-digit seconds instead of double-digit. And they use about 500Mb less memory. And see, that’s important. Because we want to shove as many as we can into this server I got access to.

Accelerated on host

My very hacky project for running this stuff is available here. This code is cribbed from simple.sh:

  qemu-system-aarch64 \
	-machine virt,accel=kvm \
	-cpu host \
	-smp 1 \
	-m 150M \
	-kernel ../little_loader/little_loader.elf \
	-netdev user,id=eth0 \
	-device virtio-net-device,netdev=eth0,mac=de:ad:be:ef:00:01 \
	-global virtio-mmio.force-legacy=false \
	-drive if=none,file=/space/disks/special.img,format=raw,id=vdisk \
	-device virtio-blk-device,drive=vdisk,bus=virtio-mmio-bus.0 \
	-nographic

To go through it. We use qemu-system-aarch64 to emulate an ARM64 machine. aarch64 is the common shortname for ARM64, except sometimes on MacOS where I hear it can be arm64. We specify the machine to be virt. Previously we’d leave it there but now we use an accelerator named KVM (Kernel-based Virtual Machine). It is the virtualization mechanism included with Linux and qemu can integrate with that to accelerate the execution. This also requires -cpu host, meaning, we are no longer trying to emulate a cortex-a53 processor. We are trying to run on the host processor, whatever that is. host means that we emulate the host CPU, or at least as much as qemu and the KVM accelerator can support of what the host can do. This is where we drop about 500Mb of memory overhead. We no longer have to have a pretend ARM chip in memory because we have an actual ARM chip to run on. That’s my understanding at least. Would love notes on that.

We only give it 1 core via -smp 1 and we give it 150Mb memory with -m 150. Skipping ahead we give it virtual Ethernet and a virtio block storage drive. You’ll see a lot of virt and virtio when doing this stuff. And with -nographic we tell it to not bother trying to pop up a GUI window, so we get our console in the terminal. I’ve done all the work over SSH so that’s definitely my preference.

The disk we provide runs from the raw disk image file special.img which was generated using fwup based on the Nerves project amproj I mentioned earlier. If you build that project with mix firmware you can then run:

fwup -a -i amproj.fw -d special.img -t complete

That’ll give you an image file that contains a full Nerves system. The stuff written to disk is:

• A uboot env formatted chunk of data. We don’t use uboot this time but we used that format.
• A linux kernel, not on a filesystem. Just written to the disk. RAW!
• An MBR and some partitions:
  • Root filesystem A (squashfs, read-only)
  • Root filesystem B (squashfs, read-only)
  • Application data partition (f2fs, read/write)

The uboot env is used to tell the bootloader important things about the A/B upgrade process as well as where to find the kernel to load as well as what kernel cmdline to use which is how we tell it what root filesystem to use.

The only config I put into the loader is to set the offset where it can expect the uboot-env and I set that at build-time.

Promising results

NervesCloud received 3389 simultaneous connected devices before the server hit me with the OOM killer. It was probably running a few more but around there. So each VM is:

• Bootloader
• Linux
• erlinit
• BEAM/ERTS
• Nerves base functionality
• NervesHubLink for connecting to NervesHub

I have had 3000 devices running stable and then I started to see “fun” challenges. For one thing, our NervesCloud hosts were looking a bit tight on memory because all these devices connect from the US west coast and we were only running a single node in that region. I scaled that up a smidge to make sure I didn’t bother any paying customers.

The VMs are super well-behaved, the Ampere CPU just works. The memory usage is roughly where I’d expect it. 150-250 total I think. There are probably things I can do to make it behave a little more tighter. Will explore that if time allows.

Then I ran my first demo workloads. As the purveyor of the finest Over-the-Air updates for embedded devices we here at NervesCloud.. I kid. But I wanted to shove lots of updates at them and see what that did. The updates process is a lot of compression, decompression and IO. Probably mostly IO-bound but if the devices would be struggling for CPU that’d be noticeable. If the memory usage exploded, that’d be noticed very quickly.

It worked. Not really any problems. I limited the concurrency of the update to 1000 and it couldn’t hand out the updates faster than they completed so it tended to hover around 200-300 concurrent updates happening. Or at least that’s my understanding of what happened. Did I mention the KVM setup is pretty fast?

I logged some issues about UI behavior as I was watching things live and trying to adjust things. It seems like good guy Nate Shoemaker already has a fix in flight for this. There may be more details. When you get a lot of progress reports the LiveView UI perhaps shouldn’t try to refresh all the things all the time.

Memory tuning

Frank gave me some tips about tuning Linux memory usage and tuning BEAM memory usage. When I looked into his advice I ended up doing a few things:

For BEAM VM, we change the allocators. This should use less memory and probably trades off in raw performance. Which is fine for this purpose.

Erlang release, use default mode instead of embedded. Which probably makes it boot a bit faster, makes it use less memory but it could lead to surprising delays and growing memory usage later if it loads code ad-hoc. The use of embedded mode is helpful in making the release behave much more consistently, is my understanding.

Made a bunch of adjustments to Linux memory usage. Using zram was suggested by Frank. Then I checked with (famous ML model) Claude to get hints about what knobs were available to tune on Linux because Frank hinted that it might be caching a bit much and I never know where to start when it comes to what I can do to the Linux kernel. It had some suggestions, I looked those up, found articles that matched the claims that this might reduce memory usage. Changing swappiness, dirty ratios and vfs_cache_pressure like I knew what I was doing and it sure seems to have improved things.

I know I could play with different allocators, my co-founder Josh has been doing that for NervesCloud recently. I think I could also do something with virtual balloons to reclaim memory and essentially over-provision VMs but I haven’t got there yet.

This memory tuning led to some interesting further runs where we ran a solid 5100 devices and I could have pushed it a bit further. I just didn’t have time and could be bothered to do more math at the time. The VMs are now started with 110 MB of RAM on the inside and they seem to run steady around 160 MB RES according to htop. The people I’ve talked to at Ampere indicate that I’m probably running the most VMs anyone has ever ran on their hardware. Which is fun. I’m not even running tiny VMs. I could make a Buildroot system that does nothing and run another gajillion probably. But this is much closer to a real device and workload.

The utility of it all

Honestly, getting a chance to run significant, not massive, but significant workloads against a SaaS is pretty useful. But the work we’ve put in now means we can tidy up this Nerves system and make it part of supported Nerves tooling. This would make it easy to run stuff “on device” without physical hardware. It would make running more detailed tests of Nerves functionality much more feasible as well. Essentially you’d need an ARM64 Linux box with KVM or an Apple Silicon Mac and you’d get the blazing fast ones. Or you can absolutely get by with the emulated, more demanding things from the x86 side of things. There is a lot we can do with a full-featured qemu-system for ARM devices.

While my experimentation is a bit of a stunt and mostly for the joy of experimentation and Frank’s bootloader is mostly about learning the end result is still that we have produced something we should get good mileage out of.

Heck that MacOS thing. I just tried the DELAY=1 COUNT=200 CHUNK=10 ./run.exs after modifying the script to use hvfinstead of kvm on my M2 MacBook Air. I think I had 50 VMs when I ran out of disk. Solvable problem, but not throwing out my photo library right this minute.

Further work

I need to look at how KVM and NUMA interact and if/how I can pin. I don’t think I’ll hit problems where caches and pinning matter all that much but it would feel better. When the VMs are at rest after booting the overall system CPU usage is generally less than 20% running thousands of VMs. Mostly idle, yes, but there are things happening in all of them.

Should run the workload with some graphs to see what is actually happening big picture. Right now I’m mostly going “hey, it is STILL running, eh!?”. Which is fine enough when figuring out if it fits in memory.

Tidying up

We are in the process of tidying up nerves_system_qemu_aarch64 and then it should get a proper release and some docs. It has a mix task for generating an appropriate qemu command for you. So this all becomes a part of Nerves. Over time we should be able to build some really nice tooling based off of this. And if you have ideas you should be able to pick it up and run with it already.

Really enjoying this deeper dive into things I’ve only been at the periphery of. Learning a lot of Linux, getting to really get into it with qemu, performance tuning for both the BEAM, Linux and virtualization. It is a ton of fun to see how far you can push the hardware.

Alright, that’s enough words. Let me know what you think and if there is anything in particular you’d like me try in and around this. Thanks for reading, hit me up on lars@underjord.io or @lawik@hachyderm.io or wherever you find me.

Background

I work on the Nerves project which is an IoT framework providing best-practice underpinnings and support so that you can build your IoT hubs, smart thermostats and the like with a safe and productive high-level language on a runtime known for reliability, resilience and consistent performance. The language being Elixir and the runtime being Erlang’s BEAM virtual machine.

If you want more about Erlang, we’ve had Björn on BEAM Radio talking a lot about his work on the compiler and runtime.

As part of my frequent collaborator GleSYS sponsoring Sweden’s first Elixir conference Goatmire Elixir (I would be shilling but we essentially don’t have any tickets left *shrug*) they suggested we might connect with Ampere as they have this particularly interesting hardware with the Ampere One server CPUs (you may have seen the 192 core, 3.2 GHz one discussed) and we turned it into a joint sponsorship. Since I didn’t have a talk topic lined up we discussed me doing something with their hardware which seemed fun. I love experimenting with impressive hardware.

Disclosure: This post is not part of the sponsorship exchange. They get some posts on socials and space in my newsletter along with branded presence at the event. This is me reporting on what I’m up to and providing the background for that. But I want to be transparent, they have supported the not-for-profit that runs the conference which I am organizing.

The runtime

If you know the BEAM you know it is highly concurrent and parallel. By default it starts one scheduler thread per core available and then does work stealing across those to ensure efficient use of the cores. Based on anecdata from a friend who tests these sorts of things (he has data, he shared it anecdotally) the BEAM does not scale arbitrarily with this amount of cores. I’ve speculated whether that’s due to NUMA but I don’t know the architecture of the chips well enough to say really. There would be overhead when running many schedulers of course and whatever coordination is needed. I know Meta has contributed recent updates to Erlang/OTP that should improve the many core performance. I need to see if I can find a really good benchmark for testing the limits of a single BEAM on this thing. Anyway, this was not what I primarily wanted to do.

Going embedded

This is an ARM64 part. And I mostly deal with ARM64 on my laptop or in the shape of various embedded Linux boards for Nerves usage. Now I could put Nerves on the server and call that good but that is just one massive Linux board. And this has been done with nerves_system_x86_64 and specifically for Vultr VPSes. While it would have some interesting challenges I’m not sure it would be all that interesting in terms of a demo. And I also have to shill NervesCloud which is the hosted version of NervesHub that me and Josh Kalderimis offer. Over-the-Air updates of firmware is a nice and lively type of demo. Also, while we know NervesHub scales to hundreds of thousands of devices we don’t very often get to run tests with a lot of full-fledged concurrent devices. We can simulate a lot of connections but having them backed by “real” devices is a different matter.

You will note that this post claims 500 virtual devices and you might think “that is not a lot” and you’d be right. Each device being single-core it shakes out to every device getting about 1.2GHz to play with. They should not need that much. I very much hope that a later post about this project will have a much higher number. But we are getting ahead of ourselves, we are not there yet.

The goal I have in mind. Running as many virtual Linux devices packed with Nerves as feasible on the hardware I’ve been handed access to.

Knowing I had a deadline of 10th of September and a lot of stuff to do until then both for putting together the conference, helping other presenters, I felt I needed insurance. So I reached out to Frank Hunleth (creator of Nerves) and kindly checked that if I got stuck on this, could he help unstick me. He graciously obliged and when I got the server credentials I went into it with his DMs as my backstop.

Making a custom Nerves system

My approach to Nerves, embedded and IoT is top down. I come from the outside, from higher up in the stack, web and cloud stuff. The parts I know the best are the parts that face the user and the parts I’m learning are the foundations and underpinnings. Nerves made embedded more approachable to me and over time I’ve unravelled the helpful structure and come to understand more and more of it. I’ve made initramfs-booting work with dm-verity for verified boot. I’ve compiled more kernels than I’d ever expected. I’ve made minor contributions upstream to buildroot. I’m getting there.

I still tend to start a custom Nerves project from some foundation. I knew I shouldn’t start from a Raspberry Pi system because those are weird, unique and do things no other ARM systems do. config.txt, cmdline.txt, the FAT /boot partition dealio. Compared to most boards I’ve seen since, it is weird. Approachable in many ways but fundamentally weird. So I looked at the BeagleBone Black system, it is lauded as a good workhorse board. Then I realized it is ARM 32-bit and also Frank mentioned that the software around that board is kind of hairy. So it would probably need a lot of adaptation.

So I grabbed a fairly modern ARM 64 system I knew worked which was the IOT Gateway iMX8 Plus by Redwire Labs. And I went to town. It had a lot of Compulab, NXP and iMX-specific stuff in there but at least it was ARM 64 and I am comfortable tearing up a Linux defconfig and a Buildroot defconfig.

Running virtually

My goal is running under qemu with KVM acceleration so I’m running ARM64 guests natively on ARM64 hardware. You have the command qemu-system-aarch64 which can run ARM 64 systems. And if you grab a buildroot you can use the qemu_aarch64_virt board config to get something that runs essentially.

So I could prove fairly easily that this was possible, confirming much of what my web searches had indicated about feasibility. I even managed to confirm that I could run a buildroot build under accel=kvm and -cpu host.

Now qemu can start Linux directly which is kind of cheating compared to most boards. Usually you need a bootloader for various reasons. But with qemu you can just throw a kernel at it, provide a root FS disk image and off it goes. Buuuuut. That removes some of Nerves' greatest features. A/B partitions, factory reset, safe updates. Those rely on boot loader features. So most Nerves systems use u-boot (hereafter known as uboot because that’s what buildroot calls it).

Failing at uboot

I said I know my way around a Linux defconfig and a Buildroot defconfig. Well. I don’t know my way around a uboot defconfig. Not really. I have done a few run-ins with uboot previously preparing for this experiment but essentially I don’t know it very well and there is a lot to it. It is featureful. I wonder how long it will take until I need to learn swupdate, I think that’s another defconfig. Anyway. I found the documentation for the uboot qemu ARM support and verified that it worked.

And then I spent untold hours trying to translate all these parts into the nerves system. I got pretty far with Linux and buildroot, the parts I already know decently. I got somewhere, multiple times, only to realize I was way off. Very often the only response on a build was that qemu would hang and print nothing. That is honestly still a common response.

This was an area where Frank chimed in, in the strangest way possible. I don’t want to spoil entirely what he got up to until it is public and I can make his part proper justice. But suffice to say he provided a solution that completely bypassed needing uboot. At least for now, maybe for the entire project. I still want to get better at uboot but I have to be pragmatic here and the solution Frank provided is quite interesting and will be public eventually I’m sure.

Meanwhile you can look at my work-in-progress system and the barebones project I use for it. The docs are all wrong, there is no guidance. Just sharing if you are curious or want to help fix my uboot I suppose.

Aside: approaches

Frank and I had a call where we resolved some weird errors I was seeing and discussed approaches. He is incredibly experienced in the embedded Linux realm and would have taken a very different approach to what I did. And I figure I should share it here because it is the wise way if you can do it.

He would not have used Nerves to get started. He would have started with just buildroot. I did verify things with buildroot and essentially backed my way into grabbing a working buildroot config for my target as I described above. But he would essentially make sure he got all of the desired Linux and system bits working through buildroot first. And then the A/B partition stuff with uboot or whatever mechanism the board might have. Before bringing those configs into a Nerves system to enjoy fwup and Nerves for testing all those firmware updates. He applies a bottom-up approach because he already knows the foundations. I am getting to know the foundations so I tend to approach from the parts I know best. My approach is definitely more trial and error.

I can totally see his approach working now, even for me. I couldn’t see starting there when I started this project. Now I feel like that possibility is closer. This is why it is hard to document “how to make a custom system with Nerves” because it ties together a bunch of embedded Linux know-how. How Nerves does things is a relatively small set of conventions on top of that which makes a compelling whole. Anyway, learn your buildroot if you want to make custom systems.

Running virtual Nerves

Yesterday we got the Nerves system running under qemu. Not with KVM, instead with -cpu cortex-a53. This should leave a lot of performance on the table. I hope we can get through the challenging bits of getting KVM and running directly on the host CPU. Let’s just say currently qemu just hangs there but Frank certainly has ideas about how to fix it and arguably it should be fixable through uboot as well. Whichever way gets us there first.

But it is interesting to see how far we can push with less-than-optimal virtualization. So I finished up the Nerves firmware project to make it connect to NervesCloud and so on.

1 device. Up and running. Gotta implement generating serial numbers.

2 devices. Good, it works.

10 devices. No problem.

50 devices. No problem. Takes a bit to start.

500 devices. Well the host machine hit more than 450 load avg for a while there. Took a bunch of time to calm down. Then no issue. So I don’t think we are taxing this machine beyond the startup.

Limits

Why stop at 500? I ran out of evening. I did that yesterday. My last bit of time went to starting to make the devices report back to their orchestrating script when they are up and running so that I could potentially bring them up more gracefully. Then I can try to push more interesting numbers.

I’ll have some figuring out to do. I’m guessing I can bring them up at about 190 or so at a time without making it unnecessarily slow. And once each device is up and reporting in they don’t spend a lot of CPU grunt. I hope that would make bringing up a couple of thousand somewhat smooth.

KVM support would almost certainly help make them boot much faster and improve things in general. My hope is to reach the point where memory is the limit. The BEAM is not particularly lean on memory, it ain’t bad, but I’m guessing I’ll need 100-150 Mb per virtual device. I’ll tune that if I see the need and have the time. I have 1 TB of RAM. In the best of worlds that ends up being the limiting factor and in rough math that means almost 7000 devices. I don’t know if we can get all the way there. I’m assuming there are bottlenecks I don’t know about. I hit a small one that I believe we’ve addressed. The base disk image was 2.4 Gb or so (mostly empty). I trimmed it down to a couple of 100 Mb. That was required to fit the 500 on the system disk. Now I think I have a new disk to play with so that fun little limitation should be thoroughly handled.

Useful?

So this is actually taking us somewhere useful. Having efficient qemu builds of Nerves means we can build test harnesses and things for some fairly intricate functionality, like A/B updates and a bunch of the resiliency-mechanism around Nerves.

It would allow people to do host development against a virtual board for cases where that is desireable. This can be particularly useful in workshops and training, though I personally tend to prefer having the hardware.

It is essentially a starting point for running on any ARM virtual servers offered by cloud providers.

This should also be doable to translate into virtualized ARM 64 on top of Apple Silicon. This may mean we can drop the creaking Docker implementation from Nerves (use UTM with a Linux VM instead if you are doing Nerves on MacOS) and maybe get something better. And it would mean that running a ton of virtual Nerves devices for testoing things is actually feasible on my laptop.

Some of the features we develop for NervesHub are quite hard to test without proper devices and being able to spin up a small IoT startup at a moment’s notice is really helpful. So yeah, I see a few different use-cases.

Next?

Let’s push it, shall we?

So a few different things. I need to make that reporting-in on boot behavior work as intended and see how efficiently I can start and stop these things. And then I need to see how many we can actually do under the current setup. Because 500 was entirely arbitrary.

I may have nerd-sniped Frank into doing work towards the KVM support working. We’ll see :)

Big thanks to Dave Cottlehuber for early help on getting to grips with this stuff. Massive thanks to Frank Hunleth as always helping me grow my understanding and enabling me to succeed when I flail. And thanks to GleSYS and Ampere for the hardware access.

This ain’t over.

Update: In truth, we have an update post.

If you have thoughts or questions you can reach me on email at lars@underjord.io or via the fediverse @lawik@hachyderm.io.

(I initially wrote this up for the Nerves Newsletter but wanted it shareable in some other way)

One thing that prompted finalizing this work was that Frank Hunleth shipped the support for disk encryption combined with deltas for fwup. And since they use encryption this enabled us to make deltas work for SmartRent. I am very excited to see what the results are there as we might get some numbers on the savings at significant scale. Also, I learned a neat trick where I can write things for Frank in english and he’ll write things for me in C. I wonder if I can scale that.

The PRs that enabled this:

Enable firmware delta delivery This included making a new library called confuse to parse the libconfuse style of config file that fwup uses. We needed to extract some information from the meta.conf files provided in .fw archives. This let us determine which resources have deltas enabled and which do not and only calculate deltas for the right ones.

This also includes a bag of tests that took a minute to get right to make sure that we produced the expected results with the deltas we generate.

This made deltas work.

If you created a deployment group on a product with deltas enabled it would immediately start making deltas for the firmwares that are in use within that deployment group. And with a bit of luck it all works out. But the control is very coarse-grained.

Coded up and worked through by me with some good review by Josh Kalderimis and Elin Olsson. Lots of help on fwup-related know-how from Frank.

Move delta setting to deployment group This lets you try deltas with a subset of devices. Thanks Elin for that one and Josh for suggesting it. And subsequently we fixed a bug in that update, thanks Nate Shoemaker.

Add information about available deltas to deployment summary A first step towards making deltas a first-class citizen is to make it visible that they exist. This makes them show up on the deployment group page.

There is more work to be done.

In progress: Safer delta updates This is a large-ish PR because it also generalizes a few fwup-specific things and tries to make for a maintainable future.

The important parts are that the Confuse library is now updated to pull out more and better information and to understand which fwup versions allow which features. This means we can disqualify a device from delta updates because it has a fwup version on device that is too old to allow the update to succeed. Then we deliver the full update.

fwup first got support for deltas for “raw” image writes. Then for FAT filesystems. And very recently for encrypted filesystems. When this lands it should mean deltas become an option for pretty much any user of Nerves.

It also add the choice between delta and full update when pushing an update manually. Which is nice.

In progress: Log transfer savings from deltas Knowing if deltas matter for you and how much they matter is kind of important. And this PR will introduce us tracking that information in and surfacing it in the UI. Showing the savings per transfer and in total.

This can be exciting for a couple of reasons. Maybe you run NervesHub and need smaller updates. Or maybe you don’t run NervesHub but want to shrink your updates using deltas. You can use this as a reference implementation for generating valid delta firmware images which is a bit finicky. In NervesHub it is just a checkbox telling the system whether it should generate deltas for you.

And when the PR for additional safety lands it will do a pretty solid job of figuring out cases where a delta could not work and just ship the full version instead.

There is also a quirk we need to look at in fwup. A delta update causes more reads than writes and this makes the progress bar kind of weird. We know :)

This has been awesome to work on. I expect it to make decent savings for companies that use it and I now understand the delta updates way better.

If you have questions or want to reach me you can use email lars@underjord.io or the fediverse @lawik@hachyderm.io.

I’m making an Elixir conference and it happens 10-12th of September in Varberg, Sweden. Tickets are still available at the time of writing. I figured it might be interesting to hear one data point of pulling something like this together.

I may mention sponsors, those are companies I have an economic relationship with. This post is not an ad for them and is not particularly part of any deal. This post is in a small way an ad for the conference. But really I find the process of learning what goes into making things fascinating so this is me offering a behind the scenes of my first attempt.

Seeding the idea

The idea for the conference formed in Chattanooga at Gig City Elixr and NervesConf US. The NervesConf pulled about 50 people and Gig City pulled around 150 I believe. Lovely conference, Maggie and Bruce Tate are great hosts and clearly care about making people happy, bringing in varied voices and putting on a cool thing without getting too pretentious. Love em. Also Todd Resudek offered me the opportunity, if I wanted it to use the NervesConf name for an EU event. That got wheels turning.

Still in Chattanooga, visiting Rock City in the evening, when I spoke to Andrea Leopardi of the Elixir core team and he immediately said he’d show up if I made it happen.

Poking around with some friends and acquaintances showed me I had the bones of a line-up. I bothered Saša Jurić about being a speaker and he accepted around the end of the year.

I also poked around both locally here where I live and remotely in my community for sponsorship and funding and found some enthusiasm and angles. There is a foundation that owns the local bank that invests a lot in local events. They were interested in contributing, assuming it was not-for-profit.

Essentially the core of making me confident that I could pull together speakers and sponsor money is that I’ve been operating in the Elixir community for about 7 years and I know enough people. If you want to put on an event, show up at events. Want good speakers? Know good people that give talks. It may sound simple and it kind of is. I haven’t done this with the goal of making a conference. It is more like I’ve accidentally accrued the raw materials required for a conference and I don’t have enough self-doubt to stop myself.

We pinned down the venue to be this nice classic theater in Varberg. Scheduled it after the main tourist season. A tourist-friendly town like this is nice because there is good eating, decent drinking and lots of accomodations but you can’t schedule during the busy season because everything is slammed.

The Point of No Return

Once we had enough confidence that we had a base line-up and some very likely money I bodged together a logo, website and announced the conference. All details TBD essentially. I first shared it privately for feedback and the feedback was excited. Then I put it out into my social feeds, Elixir Forum and the various appropriate channels for announcing things in Elixir Slack and Elixir Discord.

And the logo animates which was the important thing. Except on Safari on iOS, sometimes.

Keep watching, it does animate. All SVG.

Then it was a matter of starting to knock down the immediate needs.

Find sponsors, create a CFP/CFT (Call for Papers/Proposals/Talks) or whatever you call it.

I did a wait-list signup using my newsletter platform Campaign Monitor which I do not recommend. We fairly quickly reached 190-something people. Checking in with the lovely people at CodeSync who run conferences all day every day this actually sounded pretty good.

The Tools

I picked up Sessionize for managing speakers because I’d used it a bit as part of a Program Committee for Code BEAM in Stockholm in the past. It seemed competent. And it is. It has awkward parts but it is so incredibly useful. Superb software, does the job, has all the features I’ve needed and very few I don’t. Pricing was manageable.

Actually, Sessionize has guided me towards good thinking in how to handle sessions and speakers. It is very explicit about internal status of a session and what it shows the speaker. It is very explicit when you notify speakers about accepting and rejecting talks.

Similarly I picked up Tito for ticketing, I checked it on recommendation from a friend. Just like Sessionize, it is a sidebar of features and once you dig in it models the domain very clearly. I have a quirk for my event. I have two conferences and you can buy a ticket to both or either. Don’t do this. It messes with messaging, design and everything. Just make one conference. Anyway, Tito supports multiple activities with separate limits on tickets sold so that if one runs out the combo ticket would no longer be available.

It integrates with Stripe and does all the stuff you’d expect. I haven’t messed with the API yet but should be doing that soon enough. I’ve definitely messed with the Sessionize API…

The website is a messy static site generator I built based on this post by Fly.io and some violence. My dev server is cruuuude. But building in a language I’m very comfortable in was efficient and when it came time to list the speakers I could just pull them from Sessionize, store that JSON data and build the speaker lists from that data. Including pulling images and all that. I will eventually do the same with the agenda when we publish it. The site will be open sourced eventually I think. I want others to be able to yoink the code for this type of thing if they want it.

I went pretty deep. If there is one thing I think I’ve done pretty well on this conference effort it is marketing. I’m an active poster on Bluesky, Mastodon and LinkedIn. And so my conference stuff has gone out there. And the API data came in handy here. Because I work across a Linux desktop and a Mac laptop day-to-day I’ve settled on Inkscape for most of my scalable graphics work. And Inkscape can be automated. In this case I made a template SVG in Inkscape, changed it into an EEx template and then just generated SVGs for every speaker which I then generated PNGs from.

Examples:

Not all talks were ready to be publicized even if accepted. Some speakers had been invited and hadn’t sent in their material yet as well. No problem. Sesssionize has custom fields, I added a boolean called Announced which determined if they went on the website and later another called Presentable for whether they were okay for socials. Non-presentable speakers on the website had no page about their talk and title was “To be announced…” and that worked great.

Then I oscillated between the alpha of JustCrossPost and my Buffer account for posting. But importantly I wrote a specific blurb for each speaker. I try to automate the repetitive work that is not about the content of the conference. I don’t automate the communications.

All visual design work is, for good and bad, mine. Most conceptual design work is mine as well. I’m at least getting to stretch skills that are not in my day-to-day set. I enjoy it a lot.

We are looking to use Discord for the event app. This is not a massive conference and all conference apps seem quite bad. Whova used to be kind of practical but now is very noisy. Recent Elixir conferences have used Swapcard which seems okay. I am going for low friction with Discord that most in the community already have, it enables continued connections after the event and it is really smooth. Also it is built with Elixir. I’ve considered open alternatives but that has at best the same friction as the bad conference apps. Jury is still out on how I’ll like this but AlchemyConf did that and it seemed good. A short informal poll of the opinions also gave a lot of thumbs up for Discord.

The Content

We are a single track conference which I’ve heard many people note that they are excited about. I think multi-track is just a bit exhausting with all the options and a single track means it is way more likely that people saw the same talk as you which helps conversation. With scale more tracks are essentially necessary I imagine, but we don’t have to and our venue wouldn’t support it.

Theatre stage invites showmanship and theatrics and while I can’t spoil the plans some speakers have, let’s just say the combination of venue and my encouragement to swing for the fences has made for some really interesting plans. And because the venue is a theater it is all normal for them and their technicians.

This was the interesting bit of the CFP instructions:

“We will not be taking on talks that are “Update on project X” but are fine with talks that involve new features and either how they were built or what they allowed you to do. Avoid presenting a changelog. Teach something, tell a story, blow some minds or show your work.

The space for ideas is open-ended, the venue provides undivided attention. What do you want to share?”

Telling people what a thing is not is often more useful than trying to say what it is. Editing, cutting, constraints do more to actually make the gears turn than lofty possibilities in my experience.

I reached out to José Valim to see if he wanted to show up. If he wanted to speak I would certainly have let him. But I encouraged him to come and only participate. Because I think people would have enjoyed that. Unfortunately the start of the school year in Poland threw a wrench in that plan. I hope to see some more awesome people from the community and ecosystem even if they aren’t speaking.

And I suppose that gets to the program committee. On recommendation from my then-colleague Tomie I did what I had considered doing and formed a program committee to help me make decisions. Sophie Debenedetto, Rebecca Le and Cornelia Kelinske helped me a lot in sorting through submissions and making decisions. Unfortunately most can’t make it over here for the conference unfortunately. But they were awesome. Things got reviewed, emails flowed on one-off decisions. They pushed back on things I considered doing that conflicted with the bigger ideas. Exactly what I want.

And not entirely unimportant, it gave me some social cover when we had to reject probably 45 talks, many of which I knew the speaker personally. It would have been fine. But now I didn’t have to only reference my opinions in feedback but actually could give some wider notes.

As speakers have had questions, suggestions, ideas. I tend to just trust them. I give them the possibilities, I let them know about constraints. But I’m willing to take risks with the content. I want them to try stuff you wouldn’t see at an event that is more careful, more conventional. Most people that choose to present are creative and I want to see that creativity.

The talks

We’ve gotten so many good ones. We have the Nerves Car, we have Nerves hardware development, we have a new perspective on HCA Waterpark one of the wildest Elixir systems ever built, we have ORMs for Git which is delightful nonsense and we have someone giving a completely fresh-out-of-bootcamp view of Elixir.

And several I’m not talking about because I really shouldn’t.

Oh, and this is probably the last stop on Sasa’s “Tell me a story” tour. When a talk is described as a three-act monodrama that is as the kids say a vibe.

The Sponsors

I don’t think I have a ton of interesting stuff to share here. Sponsorship sales is just sales. You are selling a mix of brand exposure, developer interactions and for the right type of company potential customers. It all hinges on network, trust and your skills at prospecting and reaching out. My sales have all been in the realm of network and trust. Many of the sponsors are doing it because they want the event to exist, I really hope it is also beneficial in the ways it can be for them.

The Locals

The commerce people of the “city” of Varberg were quite excited when I said I could probably pull 150 developers from outside of Varberg in for a 2-3 day conference. Aside from endorsements for my stipend request with Sparbanksstiftelsen I haven’t really been able to get any public sector money, in spite of being a not-for-profit. Which is a bit disappointing. The people I’ve been dealing with have been great. They’d love to give me a bag of money or cut my venue bill in half but they don’t call those shots.

Some students at the local campus are potentially interested in attending. .Net devs the lot of them but I hope some do. And some of the campus alumni that are now professional devs in their early years are certainly coming. I’ve reserved some tickets to ensure we get some visitors that are fully local. I want the mixing. I need more Elixir in the groundwater here.

That’s a big reason I’m doing this. I engage with the local tech scene but most of it is incredibly different from what I do. They operate locally or operate some niche I’m not in. Lots of .Net around. But there are plenty of people interested in learning new stuff, in widening horizons and there are people who want more startup spirit in town. And I can bring a certain part of open source, a builder community and plop them down here for a couple of days. I want to connect my local community to my global community. Since I’m the loudest Elixir developer in Sweden I’d love for Varberg to weirdly end up being the Elixir capital of Sweden.

It should also give me some practice for running other types of events that we want to do with the not-for-profit, Goatmire International.

The Remotes

It is awesome to see people coming from all over. We have people from Nerves JP, the japanese Nerves community, coming in to speak. We have New Zeeland and Australia. We have the US of course. We have lots of Europe represented.

There is no rhyme or reason to this in terms of practicalities. I suppose people trust me from my long time and various contributions in the community. And the vibe of the conference must have hit a good spot. The ticket price is low but travel and accomodations are a whole thing. So people are definitely putting their money where their mind is and just giving this a shot.

We aren’t doing a virtual event, we plan to film everything and release videos for the global community. I don’t want an event with some split of attention towards a livestream. I want

The Dread

I hope I don’t fuck it up.

That’s it. I am not a big worrier thankfully. But this is a lot, the thing feels big to wrangle.

The Advisor

We started out having our friend and colleague Tomie Lee as our event advisor. They’ve done this type of thing for over a decade before becoming a programmer so they were a superb help. Then they got a chance to try a great job and went on leave to do that. That’s when we brought in Helene Mattisson. I know her husband from the community and saw her post about doing events and things. Reached out and she has been sanity-checking us ever since.

Every two weeks we have a meeting and figure out if we are missing anything. She has been immensely valuable, easy to work with and a very good investment for our sanity.

We don’t know events, we don’t have the learned experience. Getting some of that, borrowing those instincts and mental check-lists. Make sure you have someone that can advise you.

The Partner

My wife, my love, my diligent, awesome, capable partner in life, crime and business. She does the admin, the money-wrangling, the figuring out of taxes and all the nasty bits that must work. She makes phone calls she doesn’t want to make just to take them off my plate. She joins me in meetings with the theatre, she helps with all the practical stuff, she gives feedback on visual stuff.

I bring the headache of making and event happen to our doorstep and she just digs in. Alright, she’ll let me know I’m making our lives harder again. But she is incredibly game to dare things and that hasn’t come easy.

Couldn’t do it without her.

The Tangibles

Physical event. Can’t just do virtual stuff.

We already make shirts. Our Open Swag Platform is the most modest of modest productions. We sell high-quality Elixir and Nerves shirts currently. With more projects coming soon. So we work with a local print shop for those and will use them for the shirts for the event.

The range of cost for a good vs bad shirt is ridiculously wide. You can do DTG (Direct-To-Garment) prints on cheap blanks and get shirts that cost essentially nothing. The print will wash out as tiny rubbery particles at 40C in the machine. I find this unacceptable. I don’t want shirts like that. They make me angry.

We also want Fair Trade and ecologically sane stuff.

The shirts we’ve made for Underjord have held up for 3-4 years without really deteriorating. The blanks (base shirts) we use cost us around $20. Full color prints that last need to be some kind of transfer so that brings us near $30 for making the shirt if we make ~200 of them. We could compromise on the blanks and still get a “really good one” that I don’t like as much for about $9 per shirt. Nah. The thing I am willing to adapt is the print. You can make a single-color high quality screen print much cheaper than the transfers and since conference shirts are a bulk order we can do that efficiently. This is different from what we need for Oswag. So shirts land around $23 in cost per shirt. And they should last.

If you are unaware of this stuff as I have been you’ll pick up a DTG on some “recommended” blanks at Printful or Spreadshirt and get cheap shirts that die in the wash. Cheapest is to combine a cheap shirt and screen printing in bulk, that’s how you get really cheap marketing shirts where the shirt is trash but the print might hold up quite well.

Same print shop will also do badges, some banners and roll-ups for branding around the venue. I have a lot of design to pin down and I need every sponsor logo ready to do so. This is the real hard work of this type of thing. Remembering and making the effort until the things can be shipped.

We also have plans that I hope can happen that I won’t get into too much yet. But let’s just say NervesConf is an embedded conference and that’s a very tangible business.

The Execution

We have more prep to do but we also have a lot of execution and that’s the most exciting bit. When the half-crafted, half-cobbled vessel we’ve built goes into the hands of attendees and speakers and we just get to watch as the results take shape. While we put out any practical fires in the background. Lots of eyes, lots of beholding, at that point we can just hope for beauty.

I already know the speakers and many of the attendees and I know they’ll make an event that is a blast to hang out at. I have no idea how it will feel to be on the organizer side for 3 days straight. I hope all our fun plans come together. We have plans for the social bits that are still in flux, we have plans for bonus events that are not pinned down yet. It is incredibly fun how everyone is so game to collaborate, help and make things happen.

Hopefully this makes some of you consider organizing things. Be mindful of your energy. These things are a lot. We might regret it. We might repeat it. We don’t know. I’m excited to find out.

You can reach me via electronic mail lars@underjord.io or on the fediverse @lawik@hachyderm.io. Thanks for reading.

There are programming languages that came up in the increasingly olden times before market penetration required corporate sponsorship. PHP, Python and Ruby all seemed to just take shape in the tech world as some kind of grassroots thing with a wide variety of users as stakeholders. That seems incredibly hard to do now and we have languages like Rust, Go, TypeScript and Swift that are supported by corporate entities until they achieve flight and usually beyond that. Mozilla is not a FAANG level megacorp but they certainly helped Rust take flight and now I hear Rust is beloved in Amazon and Microsoft.

There are still exceptions that come up. I think Zig is mostly community-driven and seems to have some traction. Not Rust levels of traction but definitely relevance. And my ecosystem of preference, Elixir, seemed to boost out of the Ruby world when Ruby was somewhat losing steam but still had a lot of momentum built up. It built on the power and reputation of Erlang combined with the niceties of Ruby and found some traction.

Elixir is controlled by the Elixir core team. If there is a corporate entity behind the core team it is Dashbit that employs a few of them and contributes massively to the wider Elixir ecosystem with Nx, Livebook and tons of various libraries. Dashbit is a small consultancy shop. They essentially offer high-profile Elixir developers as a subscription service. PR review, big picture assistance, emergency help. And their commercial offerings fund significant developer time for the Elixir ecosystem.

Originally Elixir was developed at a company, Plataformatec. Hardly a massive corporation and it was acqui-hired by Nubank in a deal which left José Valim with the rights to Elixir. So we essentially have a BDFL model (benevolent dictator for life) which seems to serve the language well.

There is a wide variety of companies using Elixir but none of them have massive input or hold particularly much sway over the development of the language. Most of the development of the ecosystem happen thanks to the efforts of lots of volunteers. Some are supported, funded or helped by the companies they work at. Some are not. Some run their own business and make time for community work (hello!).

The biggest single corporation that has a massive impact on Elixir is Ericsson. Through Erlang/OTP. They fund the development of the foundations of the ecosystem of the BEAM virtual machine and all related parts. They get contributions from other large users, such as Meta. I do not like Meta as a company but the WhatsApp part of Meta that I’ve met, the people who work on the Erlang parts, have been thoroughly decent individuals and they clearly contribute back. If Ericsson stopped development of Erlang it would be a problem for Elixir. Thankfully the telecom giant actively uses Erlang and continues to fund and maintain the team.

There is also the Erlang Ecosystem Foundation that operates a kind of unifying entity for BEAM languages and related efforts. It provides a way to perform common work across Erlang, Elixir and Gleam for example where there might not have been an obvious way to handle funding previously. It seems to work okay, it is not particularly loud or in-your-face day-to-day as a user of the ecosystem. Right now there is a lot of focus on the increasing compliance burden for languages to stay relevant. So SBOM, the recent EU CRA, similar demands in the US.

Beyond the core language there are a bundle of companies that have contributed or are currently contributing significantly to Elixir as an ecosystem. I spend a lot of time in the Nerves/IoT parts of the ecosystem and SmartRent who employ the creator of Nerves and use Nerves heavily in products also contribute a lot back into the ecosystem. Tons of libraries as well as development on NervesHub and experiences from their production environments continuously improving Nerves itself.

For many years DockYard employed Chris McCord to work on Phoenix, that also lead to LiveView. They also develop various other projects for the community. These days Chris works at Fly who similarly fund a lot of his work on Phoenix and LiveView.

Membrane the media framework is entirely developed by Software Mansion, a consultancy in Poland who specialize in streaming media. Ash Framework is largely supported and funded by Alembic, a consultancy based out of Australia. You’ll find the name of some consultancy or other small company on a lot of important libraries in the Elixir space as they often pick up the maintenance of things they depend on.

This state of affairs is not a rocketship recipe. It is a steady-going concern and an awesome ecosystem to live in where you don’t have to worry about anything very drastic happening because Elixir Megacorp failed to hit Q3 targets. If you are looking for an ecosystem which has limited exposure to megacorps but is still production-grade, productive and satisfying I think you might like Elixir.

The model certainly has challenges, especially around decision-making and concerted efforts. I think it is preferable to a lot of other variants. And I’m not sure you can do the same thing today. It seems incredibly hard to establish a new language and get relevant traction. Gleam seems to be on the trajectory which is awesome. I hope we see more as well. I don’t think Rust is the most effective way to solve all problems and I definitely don’t want a future entirely built in TypeScript.

Are there languages I’m overlooking that are also recent and have had success? Are you comfortable with the megacorp languages? I’d be curious to hear your thoughts. You can reach me in the fediverse as @lawik@hachyderm.io or email lars@underjord.io.

The Officialness

This is an officially approved use of the Elixir logo trademark, by explicit permission from the Elixir core team. Usually the trademark can’t be used commercially. I was granted permission to make and sell products with the brand. We will make a small margin which if we’re lucky compensates my wife for the time spent packing and shipping. There is no software developer salary margins possible here. I did offer to kick back profits to the project but it did not seem worth anyone’s time in terms of book-keeping.

The Shirt

We don’t want to do quick and dirty when putting things in the world. We’ve tried a bit of print-on-demand and it is awful quality. Always. So this is transfer print which we’ve tried for our company shirts before and which has held up very well for more than 3 years now. On the same shirts that we know and trust. Selected with the same principles we use for ourselves. Organic, Fair Trade and long-lasting.

When Frank had tried his Nerves shirt for the first time during Code BEAM this year he literally told me, as if surprised, “this is a very high quality shirt!”. I could not have been more pleased.

The Path

I’ve had lots of fun with this. To a large degree because it is a big mess of people. Frank and Nerves was on board early which is why we made some shirts up front there and we can already sell the Nerves shirt, also we know it won’t move any major volume (yet?). José (Valim, creator of Elixir) was very open-minded about the whole deal and checked in with the team when I had a more complete plan and had verified I was happy with how the Nerves shirt came out.

Then there was the local print-shop that I already knew did good work and was easy to talk to. We sat down with them and talked through the possibilities, any advantages of scale and larger orders, where the costs are. They are lovely to collaborate with and I love working with someone local. A mix of emails, handshakes, phone-calls and meetings. Human-scale.

Most importantly I’ve worked on it with my practical-minded wife who checks my sanity-levels as we go but also trusts my gut a bit, leans in and helps me make a wild idea happen. And she takes on a bigger role in this working out than I do. She is the logistics department, I’m just the backup help if things take off. My main role is as the marketing department I suppose.

Putting physical things in the real world is kind of fantastic so I’m very excited to try and do more of it. And we’ve tried to do it in a way that we can stand behind fully. The process is hyper-local, small-scale and very personal. We have made choices that make for a more costly but less wasteful item.

Potential futures?

If this works out well we have many more paths, plans and options. But really. I expect this to be quite low-volume. A routine trickle if anything with occasional bumps for conference season when alchemists want to dress up perhaps?

Mostly I’m happy if people get shirts for the projects I enjoy so much and are happy with them.

Let me know how you like this little venture. I’m at @lawik@hachyderm.io and lars@underjord.io, as per usual.

This post was originally written as part of my newsletter which is a weekly thing that I’ve done for probably 5 years now. Reflections on tech, business, life and a bunch of Elixir. Easy way to keep up with my various .. things, as well.

These patterns are nearly fractal in that looking closer you see the pattern again further down. I think the queues kind of stop showing up when you go lower than the stack in memory but I don’t know enough about processors to be sure what happens down there. Maybe the electrons are really just lining up but I kind of doubt it.

I think this is why people confuse Kubernetes and Erlang for example. They operate and orchestrate systems made up of these patterns. But Kubers does it at the level of machines and containers (mostly, maybe call it fleet level) and Erlang does it at a level just slightly outside of the application level (system level? System is very overloaded as a term. I should consult the OSI model, but I won’t.). They share strategies, and some architectural shapes but the zoom dial is set to very different levels. Kubes is zoomed out a fair bit. Erlang is closer to the action.

And people correlate microservices and GenServers which is kind of accurate but not 1:1 either. A microservice can be a worker, a broker, or a server doing request/response. A GenServer can also be all those things. They can even be at the same level of abstraction, doing the same workload and then they are at the same level on the zoom dial. But of course making very different trade-offs in terms of overhead and isolation.

If you get familiar with fundamental patterns you will see them everywhere. And that becomes a way to grasp the architecture of things, whether internal to an application, parts of a larger system or just the pseudo-architecture sketched out on a whiteboard for how something needs to fit together.

This has a similar mouthfeel to me as some other unavoidable computer things do. There are no continuous streams in the digital realm, just bigger and smaller batch sizes and the associated latency/efficiency trade-offs. There is no difference between something being slow to respond or gone and in the end we just make a decision about the appropriate timeout. Just weird stuff where I want to go “but surely that can’t be it”.

Computers are kind of silly that way. Thankfully the can be very silly, very fast so it works out.

Did I miss a fundamental idea or pattern that you care about? Let me know on the fediverse via @lawik@hachyderm.io (or Bluesky) or via good ol email at lars@underjord.io. Thanks for reading.

It boiled down to this gist as a proof of concept. Should work on most Elixir installs. These days Membrane will even try to pull pre-compiled dependencies for the libraries it wants. This was pleasant news as it can otherwise be a hassle to pull the right shared libraries for media processing. I had completely missed that this was added. Update: Added the URL to the correct older version of the model. You can also fix it and use the newer version.

Using Silero VAD from Elixir is not something I discovered or was first to write about. I leaned on this post by Sean Moriarty for DockYard. Finding the right version of Silero VAD to match Ortex was a bit of a hassle because the model has changed significantly and is out of step with Ortex, the Elixir library based on the Rust library Ort which provides and ONNX runtime. Got some good help in the Erlang Ecosystem Foundation’s Slack, the #machine-learning channel is where that stuff happens. Shout-out and thanks to Travis Morton and Andrés Alejos.

I’ve covered Membrane before and I’ve used it in talks. For that talk I did voice controlled slides and a bunch of fun little tricks where Membrane ran the pipeline that fed my voice into the Whisper model. A VAD model would actually have been great there.

Voice Activity Detection or VAD, is the weird art of determining if audio signal is voice or not. A hotdog or not for the human voice. And this VAD model is a lot lighter to run than Whisper, it is smaller and faster. And it tends to do a good job rejecting keystrokes and various other non-voice noises. And by running this in Elixir it becomes very doable to get near-realtime messages to your UI about whether there is voice activity. More importantly we only need to send the chunks that seem voice-like on for processing or only transmit those to a listener on the other side. No data is the best compression.

Time to unroll that gist. The important dependencies are, annotated below with comments:

Mix.install([
  # ONNX Runtime bindings to Rust library, adapted for Elixir
  {:ortex, "== 0.1.9"},
  # Numerical Elixir  (Nx) used for dealing with ML data types
  {:nx, "== 0.7.0"},
  # Membrane itself
  {:membrane_core, "~> 1.0"},
  # Membrane element for the cross-platform audio device library
  # portaudio, used for opening the microphone
  {:membrane_portaudio_plugin, "~> 0.19.2"},
  # Membrane element for resampling audio using ffmpeg
  {:membrane_ffmpeg_swresample_plugin, "~> 0.20.2"},
  # Membrane element for the LAME MP3 encoder, to produce an
  # output .mp3 file.
  {:membrane_mp3_lame_plugin, "~> 0.18.2"}
  # Used to write a file as the final output
  {:membrane_file_plugin, "~> 0.17.0"},
])

Next we can look at the Membrane pipeline. A Membrane pipeline is mostly a GenServer talking to a pile of element GenServers. There might be a Supervisor in there somewhere, I haven’t really checked. I will annotate the pipeline with comments for what I think is worth pointing out:

defmodule Membrane.Demo.SimplePipeline do
  use Membrane.Pipeline
  @impl true
  def handle_init(_ctx, _) do
    # This is a simplistic pipeline, each element only feeds into the
    # next one and it doesn't need anything else
    spec =
      # We configure the PortAudio source to open the microphone
      # at the specs that Silero VAD actually wants
      # signed 16-bit samples at 16000 Hz
      # the buffer size is specified in frames
      # a frame is according to the docs:
      # "For a stereo stream, a frame is two samples."
      # This is a mono stream.
      # So we have 16000 / 1000 = 16 frames per ms
      # If I want 100ms chunks to process a buffer size of
      # 1600 frames will do.
      # PortAudio produces RawAudio (PCM)
      child(:source, %Membrane.PortAudio.Source{
        channels: 1,
        sample_format: :s16le,
        sample_rate: 16000,
        portaudio_buffer_size: 1600
      })
      # My custom VAD element, will cover that later
      |> child(:vad, VAD)
      # As the audio gets passed through we upsample it to suit
      # conversion to MP3
      |> child(:converter, %Membrane.FFmpeg.SWResample.Converter{
        output_stream_format: %Membrane.RawAudio{
          sample_format: :s32le,
          sample_rate: 44_100,
          channels: 2
        }
      })
      # Change the raw audio into an MP3
      |> child(:encoder, Membrane.MP3.Lame.Encoder)
      # Write it to a file on disk
      |> child(:file, %Membrane.File.Sink{location: "local.mp3"})

    {[spec: spec], %{}}
  end
end

Let’s look at the element. I over-complicated this because I hadn’t tuned the PortAudio config when I built it. So I needed to capture the samples up until my desired chunk-size and only then process them. This could be simplified but I know this runs and does the job so you get what you get:

defmodule VAD do
  # This is a Filter, meaning it takes input AND produces output
  # otherwise it would be a Source (only output) or Sink (only input)
  use Membrane.Filter

  # An input pad designs what the expected input is
  # technically Membrane can be used to sling any data
  # and I'd be curious to do weirder things with that
  # but now we stick to raw audio in buffers
  def_input_pad :input,
    availability: :always,
    flow_control: :manual,
    demand_unit: :buffers,
    accepted_format: Membrane.RawAudio

  # This element passes on RawAudio after processing
  def_output_pad :output,
    availability: :always,
    flow_control: :manual,
    accepted_format: Membrane.RawAudio

  # As we start the pipeline it will initialize the element
  @impl true
  def handle_init(_ctx, _mod) do
    # Load our model from file
    model = Ortex.load("./silero_vad_likely.onnx")

    min_ms = 100

    sample_rate_hz = 16000
    sr = Nx.tensor(sample_rate_hz, type: :s64)
    n_samples = min_ms * (sample_rate_hz / 1000)
    bytes_per_chunk = n_samples * 2

    # Set up the initial state, we will reduce over this as data comes in
    init_state = %{h: Nx.broadcast(0.0, {2, 1, 64}), c: Nx.broadcast(0.0, {2, 1, 64}), n: 0, sr: sr}
    state = %{run_state: init_state, model: model, bytes: bytes_per_chunk, buffered: []}
    {[], state}
  end

  @impl true
  def handle_playing(_ctx, state) do
    # When the pipeline is active, "playing"
    # we trigger demand which will tell the preceding
    # element that we want input
    {[demand: {:input, 1}], state}
  end

  @impl true
  def handle_demand(:output, size, :buffers, _ctx, state) do
    # If later elements in the pipeline make demands of us
    # we make demands as well. The samples must flow.
    {[demand: {:input, size}], state}
  end

  # This does most of the real work
  @impl true
  def handle_buffer(:input, %Membrane.Buffer{payload: data} = buffer, _context, state) do
    %{n: n, sr: sr, c: c, h: h} = state.run_state
    buffered = [state.buffered, data]
    # This is the now-unnecessary insurance that we've built up
    # enough buffer to do meaningful processing
    if IO.iodata_length(buffered) >= state.bytes do
      data = IO.iodata_to_binary(buffered)
      # Turn the data into a shape Silero VAD expects
      input = data
        |> Nx.from_binary(:s16)
        |> Nx.as_type(:f32)
        |> List.wrap()
        |> Nx.stack()

      # Run the model on the input data and get updated data
      {output, hn, cn} = Ortex.run(state.model, {input, sr, h, c})
      # Turn the output into a probability between 0.0 and 1.0
      prob = output |> Nx.squeeze() |> Nx.to_number()

      # Our only output right now
      IO.puts("Chunk ##{n}: #{Float.round(prob,3)}")
      run_state = %{c: cn, h: hn, n: n + 1, sr: sr}
      state = %{state | run_state: run_state, buffered: []}

      # This 0.9 is not the right threshold, I had some off calculations
      # that made the probabilities start around 0.5 but fixing that
      # made it very consistent, I could probably threshold around 0.2
      if prob > 0.9 do
        # This would be where I either trigger a Membrane notification
        # or use some other Erlang messaging mechanism to notify whoever
        # cares about voice activity that it is happening
        # We also pass the buffer forward.
        {[demand: {:input, 1}, buffer: {:output, buffer}], state}
      else
        buffer_size = byte_size(buffer.payload) * 8
        # Two fun options:
        # 1. pass nothing forward, this makes for an MP3 with no gaps between speech
        # 2. pass silence forward, this would make for clean silences between speech
        {[demand: {:input, 1}], state}
        #{[demand: {:input, 1}, buffer: {:output, %{buffer | payload: <<0::size(buffer_size)>>}}], state}
      end
    else
      %{state | buffered: buffered}
      {[demand: {:input, 1}], state}
    end
  end
end

The final part is only starting and sleeping:

Membrane.Pipeline.start_link(Membrane.Demo.SimplePipeline, [])
:timer.sleep(:infinity)

My goal with this was actually to use it in The Grand Kiosk a Nerves demo for the Seeed Studio ReTerminal DM that I am working on. It has a mic array. And Membrane works fine with Nerves even though the precompiled stuff isn’t available for Nerves targets so I do need to add it in my custom system. But I know how to do all that. Running this as a filter for hailing Whisper and having Whisper interpret speech that I can then use for text-based commands seems like fun. I want to see if I can do a nice job with vector embeddings for fuzzy-matching pre-defined voice commands as well.

Unfortunately Ortex won’t cross-compile because some underlying Rust library won’t cross-compile quite right. For now I am stuck. I toyed with trying to hack out the bad arg from the Rust deps but I don’t know Cargo well enough to do it right and might not have found the right dep to hack.

There will be a path forward eventually I’m sure. I was happy to see that once I got the right data shape into Silero VAD all my previous experience with Membrane elements made those parts just slot together quite nicely. And it is a pretty neat little example of an Elixir script.

Thanks for reading. If you have questions you can get a hold of me via the fediverse @lawik@hachyderm.io or email lars@underjord.io.

Some of the magic is a goddamn pain. Take the Membrane framework. I really like what it enables me to do but it is also a bit messy. Setting up pipelines is not trivial, writing custom elements is a bit finicky and requires some practice. But more centrally the dependencies, build requirements and underlying nastiness of video formats is really what makes everything harder than you want it to be. Membrane gives you a lot of capability to do live things, within Elixir, but you are playing with media and media as a domain is terrible.

If you spend a lot of time in the browser you are used to the platform papering over the difficulty of opening a microphone in a cross-platform manner or playing video.

Something I want to do in the near term is finish up my tech demo of the Seeed Studio ReTerminal DM and that includes making it do some unexpected stuff, all from Elixir. This actually involves “AI”, meaning Machine Learning. LLMs are mostly impractical for local use. They definitely don’t run well on a Raspberry Pi 4 CPU. But there are models that are perfectly feasible.

Take a Voice-Activity Detection model, which are lightweight models designed to determine what parts of an audio stream are speech. Sean Moriarty has covered this as a part of his experiments in voice assistance. It is a relatively small and cheap-to-run model that will save you a lot of performance where you don’t need to bother to process things not identified as speech. The device I run has microphones, so it can do this. And then we can take the voice-parts and throw them at the heavier Whisper model and get text. Then we can decide how we want to map text to action. This is all pretty useful applications of ML, quite practical and hands-on. Plumbing and pre-trained models.

ML is also goddamn painful. Whenever you paint outside the lines a bit things get really weird really quickly. The VAD model used in the guide has gone beyond the ONNX support in libraries and the input arguments have changed shape. So I have to pin down the right old model to use and wrangle the inputs just so. It is a deep domain and when the tools pave over the hard parts everything is nice and when the hard parts peak up you may want to scream. But I don’t have to shuttle data back and forth to a Python script so I prefer it.

I think there is bit more I could do in terms of Machine Learning as well. If I want something close to LLMs language semantics for recognizing commands/actions to take I could use an embedding model and make vectors of all available commands, make a vector of the command given and it should let me perform a vector search on language semantics more than just letter-level proximity. Pulling out arguments and such would still be utterly painful.

LLMs are decent at pulling out arguments but a 3B model tuned to function calling (looked at Galive Function Calling V1 for example) seems to require about 10GB of memory for your GPU, that’s a nice big enthusiast Nvidia GPU you need there. This is not feasible for an RPi. Of course I could pass the inputs off to an LLM API service but I find that uninteresting as a demo.

Suffice to say, I get to work within some creative constraints which is not all that bad. I think I can get reasonably responsive voice commands and that would be novel and an interesting demo. It will allow a lot of fun LiveView stuff since all of it is streaming. I can get events about when the VAD detects your voice and immediately start indicating it is paying attention, then as soon as the VTT model starts to spit out words I can show you what it is hearing, then I could do various interesting things in terms of searching for the closest commands, maybe show you what you are closest to at every turn. Depends on how fast the vector search is.

Membrane pipelines are just GenServers. Nx Servings that underpin Bumblebee are just GenServers. LiveViews are just GenServers. Nerves was born to start GenServers for you.

Do you ever bodge together GenServers? What is your place of joyful experimentation and where does it get painful? If you want to discuss you can reach me on the fediverse @lawik@hachyderm.io or lars@underjord.io. Also on Bluesky recently, will see how long that place lasts.

I use Elixir which builds on Erlang and has the same capabilities.

The standard way of doing Elixir releases via mix release does not support Erlang hot code updates. As in, it will not generate the necessary files for you. And if you do want to do it there are several blog posts you need to stitch together or you need to use the Erlang docs in great detail. Learn You Some Erlang has documented much of it of course. AppSignal also has a neat guide on hot code reloading in Elixir.

Bryan Hunter and Chris Keathley are both people I listen when they speak because it tends to be interesting, challenging and something I just might not have heard before. Both have described hot code updates as something that people should learn and use. I imagine Whatsapp’s initial engineering crew would agree. They did pretty well.

Of course we do use it. But mostly for trivial things.

Anyone using IEx r MyModule or recompile are doing hot code updates. But reloading a module in development on you local machine doesn’t feel all that spicy. It is cool and useful but feels like an incremental compile or watcher/builder type of thing.

I use it constantly with Nerves. When developing on an embedded Elixir device and needing to tune some numbers, or reworking a module, pasting into the IEx is faster than uploading new firmware and waiting for the reboot. I stop and start parts of the applications or just terminate the relevant GenServer if I need a state reset.

I’ve also used it for remote devices over NervesHub’s built in web console that offers an IEx prompt. When debugging a misbehaving Real Time Clock it was pretty nice to just paste chunks of utility functions and relevant I2C-calls over to the device to get clear answers about what it was doing.

I’d love to see more tooling for the full-blown delivery of hot code updates on top of Elixir’s mix release tooling. Or in the vein of the predecessor distillery. I don’t think there are any shortcuts for doing a correct hot code update. Much like database migrations they require care. And you’ll need to know how you dependencies react to hot code updates and many other interesting topics.

This was just a quick one in case people are curious how Erlang’s hot code updates are actually used, day-to-day, in Elixir.

If you have comments or questions you can find me on the fediverse via @lawik@hachyderm.io or email me at lars@underjord.io.

The Erlang of it all

The reason to do Embedded Linux with Elixir is the BEAM. And yes, I have to type Linux every time or someone goes “hurr, hurr, oh you meant Linux, not real embedded, that’s microcontrollers only”, some are friendly, some are not, all are compelled to intervene. Anyway, the BEAM virtual machine treats your application as a system, and makes it possible to operate it at runtime and while this is powerful in your web app it is really important in Nerves. Your code gets assembled into an Erlang release. A tarball of various bits along with the necessary runtime parts.

That release contains many Erlang Applications. Some are system applications such as os_mon the OS monitor, alarm_handler the .. alarm .. handler. Then we have Nerves applications that come from Nerves-related libraries. Many of these will generally work on any Linux distribution but were developed for use with Nerves. Some provide facilities that other applications usually perform in Linux. These generally allow you to integrate the system-level information into your application in a nicer way with message passing and API structure we know and love:

• nerves_time manages time, NTP, storing latest known time across power going down, other time sources.
• vintage_net Good ol' networking. Manages Linux networking but from Elixir. This does delegate work to the regular network stack and tools by calling ip or wpa_supplicant as necessary. However we can react to activity in the networking layer as message subscriptions and much more.
• nerves_runtime is probably the most Nerves-specific as it bundles a number of important bits. Among them nerves_uevent which replaces udev as a way to load drivers as requested and *gestures vaguely* other stuff.
• nerves_hub_link entirely optional but used if you use NervesHub for firmware/OTA updates.

Then there is usually your app. Let’s call it my_app. It can be one or more applications along with any applications provided by your dependencies. Generally if a dependency manages its own lifetime and state it will run an application (NervesHubLink is for example quite independent, by design). If you should manage its state and lifetime, it will run in your application supervision tree (Ecto Repo, Phoenix’s Endpoint and Phoenix PubSub are typical examples). Because applications are independent parts of your system you can start and stop them. They can depend on each other being started.

Each application has its own supervision tree meaning it can run multiple things and decide how they restart on failure and that they fail independently of other applications if the entire application fails. Your Nerves system can start the Erlang release and your main application can fail to start. The system can still connect to your update system, it can still correct the time, you may have a mechanism telling it to reboot from this broken state at some point but a programming error that breaks at runtime due to some stray piece of data lodged in an unexpected way will not prevent you from sending out an update.

The BEAM replaces systemd and any other orchestrating entity in a Nerves system. This means that if you run external binaries you probably integrate them into your supervision tree in some application using MuonTrap. You can even integrate container runtimes and run containers in a similar manner.

This is the Erlang release and the small pile of applications it ships. It includes:

• Your Elixir application(s)
• Your regular dependencies applications
• Your Nerves-related dependencies
• Built-in Erlang applications you depend on

erlinit

PID 0 in Linux, the thing that starts everything needs to do a bit of work. In Nerves we have erlinit that takes care of mounting /dev and other duties before starting the Erlang release. So this is a thin sliver under the Erlang release on top of the root filesystem.

Partitions

The official Nerves systems boot through a boot partition with the necessary files, resources, device trees and whatnot required to start Linux and then that mounts the SquashFS root filesystem and runs (erl)init from there.

This partition structure is shown in the Nerves system’s fwup.conf (rpi4 example) and if you check you will see it has Boot A, Boot B, Root A, Root B. And this is for the A/B firmware update scheme used in firmware updates. A new update is applied to the partition not currently used (let’s say B) and then we reboot with B as the primary partition. If that goes well we persist the change and all is good but we know we can switch back to the previous version if something is wrong.

Where the Linux at? Run-time vs Build-time

Between the partitions and erlinit. We’ve mostly been concerned with the run-time system now. All this stuff also has a build-time component to it of course but they are run-time abstractions in my mind. They are meant to be in the system when it runs. And Linux, the kernel, weaves in as the boot partition gets mounted by whichever bootloader you have and then you tell Linux where to get your rootfs and it will go and start erlinit helpfully for you. But explaining starting Linux without mentioning the partitioning first seemed unhelpful.

Now we mostly leave run-time concerns.

The firmware

The .fw file used by fwup is quite simple. Coming up with the ideas behind it was probably less simple. It is a zip archive with a file called meta.conf and a directory called data which contains files that will be used in writing the final image. The conf file contains some useful metadata fields and then a bunch of instructions for writing the partitions and putting the files from the data directory onto the card. fwup organized the zip archive so that it can be streamed to devices to allow for quick and memory efficient updates. It also supports firmware validation (hashes and signatures) as well as delta updates (only shipping the binary diffs between the current and desired firmware)

This is all defined in fwup.conf in the Nerves system. If you’ve read the fwup.conf the meta.conf produced will make a ton of sense.

Now we entirely leave runtime concerns.

Your project

We are now looking at build-time things, scaffolding and tooling and we need to revisit how your stuff fits in with the Nerves stuff. First time users of Nerves will just generate or clone and Elixir project with some Nerves stuff in it and write Elixir code typically. They build with a new command mix firmware when they want to run on hardware but the project is fairly normal. The config has a mildly different structure usually with a distinction between host and target but nothing wild.

Your project simply has a dependency on one or more hardware-specific Nerves systems depending on which hardware you support. When building, a pre-built version of the system will hopefully be pulled, your Erlang release gets plopped into the rootfs, the fwup.conf gets applied to generate .fw firmware file for you.

The Nerves System

Maybe the Raspberry Pi 5, the TI SK AM62B-P1, the Seeed Studio ReTerminal DM or a good ol' BeagleBone Black (or Blue, or Green). They all get a specific Nerves system for them and if you build a variant of the hardware you get to make a variant of the system to match your customizations. These highly specific systems make for a knowable setup for everything that concerns you hardware that is easy to own and maintain in the long term.

There are a bunch of bits and bobs that vary per system but typically you will find:

• nerves_defconfig Buildroot configuration for what gets included in the system, system applications, shared libraries, lots of stuff.
• linux-x.y_defconfig Linux configuration for what gets built into the kernel, drivers, modules overlays and whatchamacallits.
• linux/ A directory with any patches to Linux that the system needs.
• fwup.conf As previously discussed, defines the partition and composition of the firmware image.

The specific system builds on a more general nerves_system_br a base to hold the general buildroot config. It also contains any patches to buildroot packages of which there are typically a few. Not everything is suitable for upstreaming.

You generally don’t need to look at the Nerves system unless you need to customize drivers or ship extra code that you can’t get from your Elixir dependencies. It’s even more rare that you need to touch the base nerves_system_br.

Buildroot itself

Used to pull together a Linux system and provide a way to add and configure system-level components at build-time Buildroot is a kind of old-school project but active and going steady. From my understanding it is also very straightforward in how it is put together. I know enough to use it in Nerves and I could probably use it vanilla if I tried but I’m not good at Buildroot yet. Also not good at Linux config yet. Before I did Nerves I’d rarely even whispered about recompiling the kernel. Now I do it pretty much daily for various projects and experiments.

Tooling and framework

The concept of a System is a Nerves thing but what it does in total is not. The base system of nerves_system_br and the specific system for your board together encompass something every Embedded Linux project needs and in the end creates. Either implicitly or explicitly:

• A way to acquire a built Linux kernel with all relevant drivers for the hardware.
• A way to include other software.
• A way to specify the partition scheme, including boot concerns.

The system lays out what it does very explicitly, flattened. It has minimal indirection. It stays manageable by being as small as feasible. This is an intentional design decision. Elixir is your app layer and then when you need to pop the hood of the system it is all laid out there. There is not a sprawling system of interlocked smaller hoods to pop in the right order to get at whatever gasket is fraying.

To go beyond the fundamentals there is the tooling for integrating with Mix and further smart choices, such as using fwup, which enables further features, niceties and good practices. I should get into all the design choices made to keep your Nerves firmware from becoming a brick. There are a lot of choices that serve that specific purpose. That’s for another post.

Wrapping up

A statically defined, mostly deterministic, set of parts make up the system. Add your code on top. It compiles to a lean, compressed, firmware definition that can be signed and wrangled as need be. A single artifact that wraps up you entire device firmware and how to update it.

The easiest way to think of it is as a replacement for systemd. Though it is not only that. I will attempt to explain the different ways you can run things via the BEAM and consequently, on Nerves.

1. BEAM code (Elixir, Erlang & friends)

If you write your darned application the it was intended. In a Elixir, or Erlang, or probably Gleam, potentially even LfE. Then you’ll have the loveliest time. Not because that’s given preferential treatment, rather just because these languages are capably of utilizing the whole runtime.

You get light-weight processes that communicate through message passing. Resilient pre-emptive scheduling. Consistently low-latency. Hot code updates which we use heavily in Nerves because it is really nice to just paste code into the device during development instead of reflashing.

And you get easy access to libraries, the Phoenix web framework and the parts just fit together. Beyond that the Nerves runtime is built in this same manner and you can interact with things like your network configuration, firmware update and plenty of hardware via message passing and function calls. This makes it very easy to write systems that react live to changes.

In my world-view this is where you want to spend the bulk of your development time because you get quick iteration speed, great capabilities and a lovely language that does a decent job of keeping you from making a mess.

2. Native Implemented Functions or C + BEAM

This is number two in terms of how close to the BEAM it is, not for how recommended it is. The risk of NIFs must be stated. You segfault in a NIF and the BEAM cannot protect you and your undermine the resilience of it. The Nerves variant of the Erlang heart module might still save you. But a lot of the safeties of the higher level of abstraction go out the window. You are running code directly in the BEAM.

The various kinds of NIFs are a bit much to get into here. If you can slice the work in thin timeslices the old-school NIFs are rad because the scheduler can treat them much like it treats any BEAM process. But there are also dirty NIFs that run on a different life-cycle with “worse” guarantees but a much more manageable programming model.

C is the basic way of doing this. Then you have Zig via the tool Zigler. And of course Rust via Rustler. Both Zig and Rust to some extent reduce the risk of you segfaulting the BEAM so they are potentially preferable.

3. Ports

There are several ways of running Ports. This is an Erlang concept where a Port is an external process that you start from the BEAM and communicate with via stdin and stdout. In Nerves we typically use a Linux-specific and more featureful Muontrap library. It supports cgroup stuff to control memory and CPU usage of the processes. This can be plugged right into your BEAM supervision tree and have it’s lifecycle managed like a proper gentleprocess.

So if you have a vendor-provided binary you have to run, some Python script, some nice statically compiled Go binary. This is a good path. With tools like beam_notify you can even pass reasonable BEAM messages from shell scripts.

That’s it?

These three cover everything you need. Especially number 3, the Ports. You can run processes on Linux. What else do you need?

But there are several additional interesting things you can do.

4. C Node

You can build an application that pretends it is an Erlang node and.. I would say that this is a stretch for most Nerves usage. Let me know if you have different ideas. But it is one of the Erlang mechanisms for interop.

5. Containers

Apparently containers are cool.

Actually, they are cool. I don’t like most of the tools around them but it is very practical sometimes to just pack up and ship the developer machine to not figure out what exact combo of deps will make node happy. By default Nerves does not ship a container runtime because they are a lot of extra stuff when not needed and in most embedded projects you can just ship a firmware update when you want to ship new code. But it can make sense either because you get containers from somewhere or if you have a need for very isolated and selective updates. Or you just really like it architecturally.

Several people have run Docker and variants on Nerves for this. Steffen Deusch put together a few systems using Balena Engine which I hadn’t heard of. It is a container runtime that keeps things quite lean in the service of embedded use. I’ve tried it and had it work. It is not an officially maintained set of systems. If enough people request containers we might consider it.

6. Special language implementations

If you want a sandboxed language to run on device I would look at luerl which is an implementation of Lua in Erlang which has some really cool facets to it. Every run of Lua code returns a new state of the Lua program which means you can fork off the execution or save, load and so on. Quite neat.

The highly experimental pythonx library also offers and embedded language but this time by putting Python in a NIF. Quite wild, very early. But damn if that is not a fast path to AI/ML work.

Conclusion

You can ship your machine learning as Python if you want to. Either include Python in your system and run via a Port or run it in a container. Nerves doesn’t care. There are a lot of nice libraries for improved interop between Python and Erlang programs actually.

You can run you cool Go binaries and your uncool hardware vendor software.

For most of the work I do recommend using the Elixir that is right there at your fingertips.

Regardless the Nerves project provides you what you need.

If you have any questions about Nerves or this topic. Don’t hesitate to reach out, I am @lawik@hachyderm.io on the Fediverse and my email is lars@underjord.io. Hope you can run your code in a satisfactory way.

Getting started

Assuming you have a comfortable Elixir & Erlang installed, preferrably the most recent. Most conveniently through asdf/mise, but I don’t judge. Instructions are lifted from the official docs but I will try to keep it leaner as I assume you are comfortable with Elixir.

Install deps for your OS:

MacOS + Homebrew

brew install fwup squashfs coreutils xz pkg-config

Linux + Ubuntu/Debian

sudo apt install build-essential automake autoconf git squashfs-tools ssh-askpass pkg-config curl libmnl-dev

Have a Raspberry Pi computer (not the Picos, those are microcontrollers), with SD-card, available. It will assume a Raspberry Pi Zero, ie. using rpi0 for referring to the target system. You can replace that in any command with another supported system:

# Install the Nerves tooling called nerves_bootstrap
mix archive.install hex nerves_bootstrap

# Use the Nerves tooling to create a new Elixir + Nerves project
mix nerves.new my_thing
# it will ask to install deps, you can yes, otherwise: mix deps.get

cd my_thing
export MIX_TARGET=rpi0
# Getting deps with the target set will fetch the base linux system
mix deps.get
mix firmware
mix burn

The mix burn should prompt you for which SD-card you want to flash the firmware to.

Then start the device with that SD card in it. Give it power and it should turn on. If you have it connected with a data-cable it should show up via USB networking which is convenient. It should be announcing as nerves.local and have picked out one of your SSH public keys to embed in the firmware by default. Try ssh nerves.local to see if it is up. If not you may have to adapt a bit further and set up WiFi.

If you have networking the next update is still a mix firmware to compile but the deploy becomes:

mix upload nerves.local

That’s ridiculous.

You can also update specific modules by just pasting them over the ssh connection into IEx which is surprisingly practical.

Set up Wi-Fi

Top of the generally debunked hierarchy of needs is of course Wi-Fi. You can edit config/target.exs to edit the config for hardware devices. host.exs is for running the project locally on your computer.

The networking libraries for Nerves are known as VintageNet and we can configure the VintageNetWiFi technology like this by replacing the default "wlan0" entry:

{"wlan0",
      %{
        type: VintageNetWiFi,
        vintage_net_wifi: %{
          networks: [
            %{
              key_mgmt: :wpa_psk,
              ssid: "my_network_ssid",
              psk: "a_passphrase_or_psk",
            }
          ]
        },
        ipv4: %{method: :dhcp},
      }
    }

Re-burn the SD-card and with a little bit of time your Pi should connect to your wireless network. If you have one of the larger Pis and an ethernet cable that also works great.

Nerves Livebook

If you are a bit lost on what to do and inspiration hasn’t sparked immediately you can instead get some neat tutorials going from Nerves Livebook. There you get pre-baked firmware with special environment variables for setting up Wi-Fi. Note that it starts a bit slowly, especially on something like a Pi Zero. So be patient. Follow the instructions linked because they are friendly and practical.

Going further with Phoenix

The Nerves documentation has a part about UI where we show how to do a so-called Poncho project (less bad than an umbrella project, but it’s still raining). Redwire Labs published their approach which I know others have also done and dubbed it naked Phoenix. I like this approach so if you want to integrate Phoenix. Try it.

All just Elixir *

If you’ve dealt with your fair share of Mix projects and Elixir applications Nerves is fairly straightforward because you are just building an Elixir thing. Thing can get a little bit spicier if you need some special dependencies but usually that’s actually fairly simple. Either the library pulls it in for you or you need to install packages via buildroot. It is more involved than you might be used to for apt. Pretty much a slightly old-school Docker image situation. The nuisance is the build time and that’s the flip side of building a tiny dense BEAM-centric linux from scratch instead of shipping Ubuntu and hoping for the best. I can tell you I’ve had more luck reviving a Nerves-project 2 years down the line than I’ve had when I’ve hacked together something on top of Raspberry Pi OS.

There are some things that have fancy deps that aren’t possible to cross-compile or aren’t ready for it. But I’ve seen most of Membrane compile without issue. I’ve installed SQL Cipher as an encrypted alternative to SQLite. There’s some nuance there but it is the same that lives in your Dockerfile plus a bit of architectural nuance.

You also get some cool stuff you usually don’t poke at on a server. VintageNet for wrangling network interfaces. NervesTime for both Real-Time Clocks, ntpd and so on. The Nerves heart variant of the Erlang heart. A/B firmware partitions and firmware validation.

Things that are more fun on-device than on your average server is stuff like connecting a camera and using evision. I’ve added a Coral TPU over USB and had some fun via tflite_elixir.

Find some devices to talk to, you now have a little server that can pull down your calendar, parse it and control things based on that, or based on a public RSS feed you like. There are so many possible things. I have hacked on my Elgato Keylights and my Elgato StreamDeck.

And you can get incredible mileage out of OTP functionality like GenServers and the binary pattern matching to do things with hardware. You really get to do the stuff Erlang was made for.

More Nerves

Things to read further on:

• NervesHub Really fancy firmware updates, including binary delta updates, for Nerves devices. Me, Josh and the Nerves team are making them cooler by the day. Includes nerves_hub_link and nerves_hub_cli.
• NervesKey Opinionated abstractions on top of hardware encryption using the ATECC608-series from Microchip. This gives your device an identity and I’ve been doing stuff with it.
• Mobius On-device metrics, I want to look into this one and try to combine it with Explorer in some neat fashion.
• Scenic Not currently the liveliest project but it does allow you to make UI all in Elixir and is used in production.
• Sensors & peripherals There are many devices supported to be used directly from Elixir that you can pick up off of Adafruit or Pimoroni and just have fun with. I’ve implemented the Inky eInk display and the VCNL4040 ambient light and proximity sensor.

exit

That’s it for now. Try Nerves and let me know if you run into things that feel confusing or weird about Nerves and I’m happy to try and clarify, improve and generally help you succeed with Nerves.

I am reachable on email lars@underjord.io and on the fedi @lawik@hachyderm.io.

My latest non-newsletter issue was about The Future Stack. I think that idea is interesting and holds some water though I’m still not convinced anyone will solve the database without immense trade-offs. I’m pretty curious about geo-located tenancy via SQLite as a way to tune perf to customers but I don’t need it for what I do so not trying it. That talk was supported by Tigris and while I still work a little bit with them I am mostly wrapping that up as I have other commitments that drain my time. I really like their product and the team has been super friendly. Strong recommend. Hopefully you will see more about them from the perspectives of people I know that write well and have cool ideas.

My latest blog post was a newsletter cross-post. About Nerves adoption. Nerves being the embedded Linux framework for Elixir. These things happened in sequence:

Random company I’d never heard of reached out with a need for a Nerves consultant to show them the ropes and guide them in Elixir. Nerves Client #1.

• Work Secure Boot for CM4 devices.
• Wired 802.1x with EAP-TLS via NervesKey (ATECC608)
• And me doing a talk on securing Nerves devices and all the detours it involved.

Frank’s team at SmartRent needed som reinforcements. Nerves Client #2.

• I’ve shipped an Ambient Light sensor driver in Elixir (VCNL4040).
• I’ve tuned more ambient light vs. backlight than I could ever dream.
• Worked with ZWave PTI (package tracing interface) but thankfully they had a Ben that solved most of it.
• I’ve fought Real-Time Clocks.

At this point I made a decision to pivot Underjord towards Nerves. I like a niche and heck if that ain’t one. Not kicking out any clients. Just focusing in on that for future work. And specifically Nerves with a security tinge because I always find that interesting.

Then Josh Kalderimis shows up out of nowhere doing NervesCloud and somehow derails me into co-founding a hosted version of NervesHub with him. So I’m doing a startup. This was unexpected. This further links my destiny and success with the Nerves community and ecosystem.

If you don’t follow my other channels you may have missed my series of talking at you about Nerves. Currently releasing daily. I’ve also been active on the socials about various Nerves efforts. I’m gearing up for a talking in Berlin at Code BEAM in October where I will be covering a lot of ground. Then I have another talk in Malmö for Oredev where I will spread that Nerves flavor to the unknowing, unsuspecting general developer public.

If you follow Elixir you might see a lot of hobby-grade Nerves being done. That’s cool and fun. It is not what it was built for though. It was built, much like Erlang to be robust, production-grade, pragmatic, nuanced and practical. With that Elixir flair. And starting NervesCloud has led to seeing a new Nerves company nearly every week. Small and larger. Or companies doing Elixir on hardware that really would benefit from Nerves. Lots of interesting stuff happening out there.

And just a few days back I talked to Frank Hunleth and it was voted that I’d join the Nerves core team. It is not a shock as I’ve been running the newsletter for several years now and my work in the team will really be about communication more than anything.

I think Nerves is a fantastic way of building embedded Linux devices. It uses the simplest standard tool to get you a minimal Linux, namely Buildroot. Then it boots you into the BEAM and it takes the wheel. From there you generally write Elixir (or if you prefer, Erlang, Gleam). The supervision trees, the stateful GenServers, the binary pattern matching, the message passing. It all makes such sense in an embedded device where what you are developing is not just a one-purpose service. It is a full system.

Erlang was built for telecom switches that would now be very comparable to “edge” devices. They exist in inconvenient geographical places. They need to be reliable and they have a lot of stuff going on in them.

It seems like most embedded Linux projects grab an OS for their fundamental approach (Yocto, Buildroot, Raspberry Pi OS or a Debian derivative). Then the system orchestration is probably based on systemd. Then on top of that comes the wild west. Maybe Python? Maybe Node.js? Maybe C++ or Java.

Microcontrollers are a different conversation and that conversation is largely about Zephyr now it seems.

We’re talking about embedded Linux and equivalent. Nerves brings opinion to this space. Elixir, OTP and the BEAM give you standard ways to do things. Firmware updates? NervesHub. Bad firmware recovery? fwup with A/B partitions. Web framework? Phoenix, absolutely top of the line. Web UI? LiveView and you typically lose the latency disadvantages. Ephemeral state? GenServers. PubSub mechanism? Message passing. Message passing between jobs. Message passing.

We’re back at the graph from Sasa Juric’s book Elixir in Action about what jobs in a modern web app can be replaced with just Erlang.

And the value of pre-emptive multi-tasking for all this is hard to overstate. The value of consistently low latency. The value of fault-tolerance.

For iteration speed during development you get a whole scale of options from host-side development through pasting code over SSH, on into deploying firmware in a minute and on into deploying firmware to specific devices over NervesHub.

I’ve never been more professionally satisfied. I’m up to my ears in dev-boards, weird hardware and awesome impulse purchases. And consequently I’m highly motivated in helping this reach people so that there is more market so that I can run business in this space for the foreseeable future. We’ve got a good crew of collaborators and lots of plans.

Near team we have a Nerves Workshop in Berlin on the 13th of October before Code BEAM Berlin. No cost. And then I have my talk where the community provided me a fleet of over 100 devices. I’ve joked about my entry into embedded. It is for real though. I want to learn all this stuff, go deeper and closer the circuit board all the time.

And it is super cool to get to improve on tools the community and commercial actors are using to manage hundreds of thousands of devices. We’ll add device health information, geo-location, better deployments and so much more.

It feels weird to say but I essentially run to the office every day to get hack more on all this.

Have any thoughts or questions, you can reach me at lars@underjord.io or more casually on the fediverse as @lawik@hachyderm.io.

Building up the Nerve(s)

As we are doing NervesCloud we are thinking a lot about adoption for Nerves. Our market is tightly bound to the number of folks doing IoT/embedded projects and choosing Nerves. The business can only thrive if Nerves thrives. Nerves can only thrive if Elixir thrives.

I think there are enough people working effectively at Elixir adoption and it is a long path as a community with no massive commercial backers. It is doing well, it is healthy. Currently Nerves i mostly a smaller circle within Elixir. There is a small part of the Nerves pie that is entirely from outside of Elixir. Other embedded developers coming into Elixir by way of Nerves, rather than moving from Elixir into Nerves. Both of these are valid adoption paths.

The Elixir to Nerves one is the one I feel comfortable with. I’ve lived in that population for 6 years and know the Elixir community & ecosystem well.

The other end is more mysterious to me and I’m getting into listening to embedded podcasts, YouTube videos, blogs and trying to get familiar with that space.

Growing Nerves will require both. Partially because we do need to grow the entire pie. But also because we need the mix of experiences.

Most people moving from Elixir into Nerves are comfortable with web dev. Usually some databases, distributed systems and generally web service-oriented stuff. Most are way out of their comfort zone moving into custom Buildroot systems but can certainly learn to add their supporting libraries and compile a new system. I would imagine very few, including myself, would be confident that they could bring up a new board from one of the industry vendors.

Most people moving from embedded Linux as a general practice into Elixir are going from C or C++ and heading into a more high-level approach. They live and breathe the low-level stuff, they don’t mind dealing with Linux, device trees and so on. What Elixir offers them is a productive high-level abstraction for stuff like UI, orchestrating various processes and communication in a crash-resistant packaging. They can unleash developers that don’t know C on building features for their device without the risk of them segfaulting the hardware.

It is fairly uncommon to find people that cover this end-to-end. I think a team building with Nerves will get the advantage of these people meeting in the middle, somewhere in the Nerves firmware and cross-pollinating experiences. Shared language, shared concepts and easier shared understanding. Less need for hard silo walls and formal boundaries due to technical domains.

I think these are the fundamental paths to adoption through developers. Then there is always reaching decision makers and them pushing for adoption but I’m not convinced that that’s very useful without technical champions pushing for the tech. And arguably you could get people that arrive cleanly without Elixir experience and without embedded experience. I don’t have high expectations there but if we hit some exceptional viral vein, sure, then that could be a factor. That’s a lottery and I don’t think it is smart to bank on.

I find Nerves to have very good fundamentals. A developer from embedded who’ve used Buildroot before will find that it doesn’t actually stop you from doing anything you usually do with Buildroot. Developers from Elixir will find that most stuff is just like any Elixir project. The Nerves libraries and tools support the area linking together the embedded side and the Elixir side.

There are also tons of possiblities for improvement. The zero to prototype story is quite good. But it could be a ton better. With more capable generators, more opinionated starting points and so on. I see us getting the TTWK, Time To Web Kiosk, down to 5 minutes, counting from the moment your Seeed ReTerminal DM gets delivered.

Nerves is mildly but somewhat firmly opinionated. And the opinions it has provides a lot of capabilities for NervesHub to provide a bunch of utility. Remote IEx console is a heck of a tool. Firmware updates, firmware deltas. And we can do a lot more here. There is so much we could offer by default for a NervesHub-connected device. There is also a NervesHub CLI, that one could be improved a bunch as well.

Building the good features and functionality, improving everything for the best possible DX and productivity are all great tools for adoption. If the thing is not any good and doesn’t feel any good people will bounce off of it, if they even try.

Unfortunately the world is very noisy. You can be deep into embedded and never hear about all the fantastic tools out there. I am continuously surprised by how many folks in Elixir have not heard of Nerves, Membrane and even stuff like Nx. It is a big world and being noticed is difficult to achieve.

This is where conference talks, YouTube videos, novel stunts like the 15 minute blog in Rails come in. Just to break through and catch enough eyeballs. Inside the community and ecosystem you are in, and outside.

Both quality of the thing and outreach for the thing are important. And then we have brand.

The way the project comes across. There are many ways to do this. HTMX is an example of a very unusual brand built on memes. Elixir is known for a pragmatic and productive approach to Functional Programming. SQLite is known for the database being strong and the flesh being weak. Postgres is steady and reliable. Some projects try to be cool. Others try to be sweet and friendly. The follow-through on this is a mix of how docs and resources take care of you, how the community meets and treats you and visual aspects of branding.

Some stumble into their brand over time. I don’t know that Postgres has intentionally built a brand. They built their thing well over many years and they have a culture and process that produces a certain something. It is not particularly about the elephant in their case. It is not the Nike swoosh. Modern projects have to be a bit more forward about this stuff, at least if they want to grow and gain adoption.

I don’t know that Nerves knows what it is brand-wise yet. Curious to figure that out because it needs to speak to both the embedded Linux dev and the Elixir dev. It needs to show the clear advantages of a lot of very nuanced choices in a deeply complex field.

Challenges ahead in other words. I’m pretty excited to try and find my words on these topics and learn to speak coherent embedded. I think I speak decent Elixir at this point. Though my first talk in this realm will be in Berlin to a Code BEAM audience, so certainly starting out at home, in my own community.

Thanks for reading.

This work is sponsored and supported by Tigris. Object Storage that works like a CDN on Fly. They are also part of the subject of the post. You’ll see. It is not mainly about them though they play an interesting part.

This post is adapted from a talk I gave at GigCity Elixir in Chattanooga. In the talk I went to the trouble of trying to underline the value and difficulty of making things simple. I will abbreviate those efforts significantly in this blog post. If you don’t believe simplicity has value. Come back and read this later, when you do.

I will offer Gall’s Law.

“A complex system that works is invariably found to have evolved from a simple system that worked. The inverse proposition also appears to be true: A complex system designed from scratch never works and cannot be made to work. You have to start over, beginning with a working simple system."

This doesn’t just tell us that simple systems are the way to go when creating new things. It should also help you see the high value of any existing complex systems that work.

Opinions and trade-offs

The software way to make a thing simple is abstraction. A higher-level of abstraction should be simpler than the levels below it. Using it should produce more or better outcomes for the same effort. This means sacrificing something in translating the complex to something simpler.

Either we place constraints on the allowed solutions and restrict future flexibility, remove functionality or we make trade-offs in performance, in correctness or in other aspects. I would call all of these “opinionated” approaches.

Phoenix LiveView is a highly opinionated approach to web application development with significant and clear trade-offs. It collapses several concerns into one solution: the backend logic, the API contract, the frontend API client and the DOM rendering. If LiveView fits your needs it removes a lot of concerns.

Databases are a bit of a special case here. They are the worst. And the most important. They are where trade-offs matter the most. This is also why they are the most common bottleneck. There are things we refuse to trade off.

Apparently we want to keep our data. We want it to be the same when we read it back as when we wrote it. We want it to be easy to query and manipulate. We want it to be fast. Fast can mean retrieval time for a single record, a whole-ass query with joins and also when many queries are made at the same time. We place high demand and are willing sacrifice very little with databases.

History-time

I didn’t come up in Rails but to my understanding Rails is a web framework that was willing to trade off clarity and rigour for expressiveness and productivity. It was built in a language that was willing to trade away performance for simplicity and a certain type of elegance.

Heroku came up and offered something that fit with Rails. The trade-offs made for Heroku didn’t constrain Rails at all and offered immense conveniences. They were complementary and harmonious. You could shove a Rails-shaped piece in the Heroku-shaped slot without friction.

Rails became a darling of the startup space. The startup is a very opinionated solution as well. Quickly explore a domain with disposable solutions. Create MVPs, ship prototypes, find product-market-fit then achieve growth. Performance, rigour and maintainability are not even on the list of startup priorities usually. There was no friction between the startup way and the Rails way.

The trade-offs are well aligned. The Rails startup on Heroku achieves their solution in the simplest possible way. Without friction or contradiction.

About 40 years ago now in Sweden there was a wild software lab. They had the task of creating a platform to operate telecom systems robustly. The requirements included performance aspects such as low latency, concurrent operation, operational aspects around deploying without downtime and strong focus on reliability. It must not go down. This was a very different industry. Decidedly not a startup and it had very different trade-offs to consider. And from this we got Erlang and OTP.

For completeness we must quote Dijkstra.

“Simplicity is a prerequisite for reliability”. - Edsger W. Dijkstra

Erlang sacrifices some types of performance to get immutability. Immutability is a simpler view of the world. And it uses immutability to enable a set of simple building blocks. The Process. Send. Receive. This enabled safe concurrency. From this they built up more and more abstractions in service of the desired solution. And the result lives across Erlang, OTP and the BEAM virtual machine. It is a very complex system that is proven to work. And it is clearly derived from simpler parts. It has made very different choices from Ruby and Rails. And it certainly made different trade-offs from what it was competing against: C++

Changes

I wish the story was that the world realized Heroku, Rails and startups were too inefficient and came together to push for more efficient use of our resources.

Instead someone put JavaScript on a server and it ate Rails' lunch.

Salesforce bought Heroku.

Startups have remained a thing. Off and on. And off. And on.

Startups did focus more on growth though. And I believe the interest in massive scalability, large engineering teams and unicorn valuations did shift technical priorities. This is not where Rails or Heroku shine. And it is where cloud native got rolling.

The trade-offs made for Ruby and Rails had direct impact on performance and scalability. And Heroku was never the cheapest option. At larger scale and explosive growth your priorities change significantly. The performance you had traded off becomes a priority. The efficient scalability your MVP didn’t need is suddenly strangling your progress. High cost at great scale is very concerning to a business. Even with massive VC funding.

Constraints and friction

When we talk about a good abstraction it is always contextual to what we are doing. The Rails on Heroku startup was very happy, the hit company trying to scale on top of Heroku and Rails might be quite frustrated.

If only someone would build an ecosystem that is really productive but scales differently from Rails…

…

A good abstraction necessarily imposes constraints on you. The best constraints sacrifice things you don’t want to be doing anyway. A lot of Elixir devs find that immutability makes their code easier to understand and follow. It reduces confusion. But it IS a constraint, when we apply that constraint we have lost a potential way to build things.

We certainly want nice things. In fact i think we want all the nice things.

We want elastic scalability but we don’t want to operate Kubernetes. We want a horizontally scalable database but nobody actually likes eventual consistency. We want the speed of a cache but we don’t want to deal with getting it right. We want fantastic UI and UX capabilities but we don’t WANT a single-page app framework.

If we opt in to any of that complexity it ought to be because we consider the upside important enough.

A mismatched stack of trade-offs removes all of your space to manoeuvre. If you bring in Elixir and write it like enterprise Java you are taking the worst of both worlds. Meanwhile aligned trade-offs feel like you are getting away with something. This is how a lot of people feel about LiveView, a lot of upside for no discernable downside, to them.

What do you need?

We can only have everything if we have infinite time. And we are all very human. What do you want to spend your time on? What feels important to control? What will you pay to ignore? What power will you give up for convenience? What capability do you give up for productivity?

A project that will only run at massive scale might benefit from Bazel, Kubernetes and all that aspirational scale from day one. Assuming you can get to day two and three. Consult John Gall about that. If your project will never run at immense scale, or only ever in one country, you can trade off geo-distribution, you can trade-off future scalability and you can get a simpler system for that sacrifice.

Erlang was built for industry. It traded off particular performance and strictness aspects for an incredibly powerful runtime that excels in building robust services. It is a practical tool that doesn’t even try to be idealistically perfect.

Elixir is also an industry language. It doesn’t pick a fight with the trade-offs that Erlang made. It invests in them to enable more in the realm of services. Phoenix doubles down on that for web development. LiveView triples down on that for web app UI.

At any point one of us can feel that one of these constraints chafe. That’s the point at which our specific need might diverge from the most opinionated approach. This means it is on us to either peel back a layer and accept more complexity or find another compatible approach with a better match in trade-offs. We can usually shift a trade-off or two. The canonical case is probably Discord bringing in Rust to optimize a particularly demanding data structure in their Elixir system.

Elixir and Opinions

If you take the constraints brought by LiveView you will find that it is a sweet-spot for a solo or small team building a web-based SaaS. An Elixir-developer can thrive end-to-end in such a project and reap many of the intended benefits of the abstraction. Bumblebee from Nx sneaks in and starts to expand this picture for many machine learning needs as well.

Phoenix is not extremely heavy in opinions. At least not technically. We debate Contexts as a design decision and they are fundamentally quite open-ended. They don’t impose constraints and they don’t deliver much guidance.

I have seen a fair number of developers point to this lack of opinion as a problem. To them this is a wide open space that should be filled with opinions to give direction and achieve further gains. Other developers appreciate the space to do whatever they prefer. Some find the conceptual guidance on contexts enough to work off of.

I just recently watched Keathley’s talk from last year and I appreciate his willingness to just collapse the layers at the controller. Some people will love it and some will hate it. A clear signifier of an opinionated approach and active choice.

If you want more direction and are willing to accept a lot of new opinion from elsewhere you can add Ash Framework to the stack. It provides constraint, guidance and plenty of opinion. It will fill out everything between Bandit and Ecto if you want it to and it brings ideas on how authentication, authorization and tenancy works. Ash is interesting in this regard because you will find people that really dislike the approach and people that are all in. This indicates a framework that puts forth a strong opinion.

You want to make trade-offs that cost you as little as possible. Technically, which means humanly.

And depending on your team you may find that some technical opinions and constraints land very differently. It seems that generally developers with less experience benefit from more clarity, guidance and opinion. They are still learning and making technical decisions is quite difficult and demanding from that vantage point. More experienced developers might find constraints they disagree with quite cumbersome and frustrating. They may have other strong opinions they’d prefer to apply. They might have pointier and sharper trade-offs they want to make.

Whether you should let your experienced devs go wild in this regard is a .. team question.

The Platform

I think this idea of low-cost trade-offs is also what makes Fly interesting to the Elixir community. If we already take on the trade-offs of LiveView and the general strengths of our platform, the BEAM, we know that latency matters. And latency has always mattered. Everywhere. The more we can cut down on latency the better our chosen trade-off with LiveView works. Fly’s whole vision is to put a web app close to the user.

But of course that is not enough. Because your app is generally not only your application server. You at the very least have data and files.

Today most developers want a managed database and S3-compatible object storage. These have become table-stakes.

S3 made object storage a thing and is a prime example of a simple solution. And it blew away vast complexity that came before it. Typically NFS. Which was terrible. S3 gives a starkly limited API that enables the cloud provider behind it to scale essentially infinitely and offer great reliability. But one thing S3 does not offer is great latency, especially if you only store your files in us-east-1. Did I mention I live in Sweden?

So do you take on the complexity of adding a CDN for your files? For my sake?

The people I work with at Tigris created the object storage solution offered on Fly now. They are essentially collapsing the concerns of object storage and a CDN into one. They make the assumption that you want both. They offer geo-distribution by default, both on upload and download. That is their contribution to the vision. Aligning with the Fly proposition.

They offer a very simple service which does a lot for you. The complexity is hidden behind 18 years of progress since S3 launched as well as aligned trade-offs.

I tried to push them on what they are sacrificing to provide something more than S3. And they didn’t feel there was a sacrifice. Which seemed suspicious. So I dug in. And I think I know why now. They are selling you something. Literally. The fundamental selling point of cloud-based object storage is that you do not have to worry about disk space and you don’t have to worry about losing data. Infinite storage, great reliability. Offering this requires taking on severe operational complexity.

I do not want their job.

The work Tigris has done for low latency centers around FoundationDB which is very capable as a scalable metadata store. Most of the challenges with FoundationDB are operational. Tigris are doubling down on managing operational complexity. Aligned trade-offs can feel like no trade-offs, to them it is all the same. Keep stuff running well.

It seems object storage is handled.

The managed SQL database part on Fly is unsolved. And Fly has clearly tried to solve this a few ways indicating that, yes, this is important to them. Fly has a managed database via Supabase now and that’s cool, good. Their old one is kinda gnarly so Supabase should be much better. But Supabase is not a magical geo-replicated database. Supabase is Postgres. It doesn’t complete the vision.

Fly are trying some stuff with SQLite and LiteFS which is interesting. But I have worked a bit with the Electric SQL folks as well and I think their approach has a lot of potential for this geo-distributed concept. CRDTs are very powerful if you accept eventual consistency. And Electric SQL launched pglite, which has the eye of both Supabase and Neon. So we might see something interesting happen there. Postgres on every server perhaps? I did that as an experiment with their SQLite variant.

Some people have brought up CouchDB (the DB that did everything first, in Erlang) which is not SQL. Cassandra, which is closer to SQL from what I gather but not quite, Discord switched to ScyllaDB from that. And related to all this Phoenix will soon ship with CockroachDB support. All of the options have fairly significant trade-offs but might be great for your workload. I think elasticity and reshaping the cluster is kind of high on the list to fit the Fly model or you need to run a node in every region you want low latency, continuously. This is not typically how these work.

So why does it matter if Fly and their partners pull this off? I think it makes for a very rare alignment of priorities and trade-offs from end to end. Platform, runtime, frameworks and language all sharing values and expressing a similar overall shape. Meaning Elixir would be unconstrained by the platform and the platform doesn’t offer a bunch of complexity that is irrelevant to a typical Elixir app. This should give us trade-offs that cost as little as possible. You should be able to put a Phoenix-shaped app in the Fly-shaped hole without friction.

And hilariously good latency for users.

It matters because the goal-posts keep moving and the minimum expectations keep going up. What has worked up to here will struggle in the near future. The need to make consistent trade-offs and leverage them as much as possible is especially important to be able to work as small, highly performant teams.

It would be a true new Heroku but for Phoenix. I will leave the part about finding an aligned business model to replace the startup as a question for you all because I don’t know what that looks like but I am very curious. I think indie hacking and open coops could both fit. But give me your suggestions.

I am not trying to sell Fly. They do that well enough. I am selling Tigris a little bit but I don’t really need to. I have high hopes that Tigris will prove themselves out regardless.

I think the overall idea has interesting implications for the mainstream web SaaS side of the Elixir ecosystem. Other applications will need very different infrastructure. I really like working with Nerves which is quite different. I know Elixir performs hilariously well on bare metal and that’s a different approach. It all depends one what you are building.

Build or Rent

Personally I am always tempted to DIY anything and everything. I don’t want to rent infrastructure. I want to build it. This is my desire for control, knowledge and the satisfaction of building something. In practice I typically go with something more sensible, more pragmatic. At work one of my constraints is that I do not have eternity to build everything. Instead I have deadlines. So I pick something where someone made choices and placed constraints. Even if I don’t want to.

And the trade-offs of cloud services, AWS, Fly, Tigris, all of them, is to take operational complexity out of your hands for money. They offer a constrained API surface you can interact with to fulfil your needs. They will leverage the constraints their system places on you to achieve efficiencies in scale and manage that operational complexity for all their customers, all at once. Hopefully as a complex system that works.

Most of us are willing to pay some amount to remove operational complexity but how much and in what way varies a lot. There is a scale from the rented dedicated server to a Heroku-style platform. Sometimes it even involves Kubernetes.

It goes back to managing complexity. The demands on our software systems only increase. As we build more software in response to those demands we need to find ways to keep things as simple as possible. To raise the abstraction level of the work. Ideally we want to keep reaping rewards from trade-offs we’ve already made.

What do you think the future holds in terms of “stack”? Do you know compelling end-to-end ecosystems or things that align trade-offs exceptionally well? I can be reached and argued with on via email at lars@underjord.io or fediverse @lawik@hachyderm.io.

This blog post, my trip to Chattanooga and the work on my keynote presentation was all supported and enabled by Tigris. They offer a new approach to object storage in collaboration with Fly.io and I think you should check them out. Just run the command and see how you like it: fly storage create

Travel

From Sweden I took the train to Copenhagen a few hours south. Almost forgot my jacket with the passport in it on the way but we caught it just in time to go back and fetch it. Copenhagen to Amsterdam was no trouble. Amsterdam to Atlanta was pretty smooth as I’d ponied up for extra leg space. Atlanta to Chattanooga was delayed so instead of midnight I arrived 02.00 (2 AM) which I didn’t love. Bruce was kind enough to get me anyway. So I got to the Tate residence, was shown the guest room and slept the sleep of the dead. I had two good nights of sleep, with two days of doing a lot of very little. I hung out with good people. I ate things. This was good recovery before the conference and good social time with people I’ve known for about 5 years or so but have not had a ton of time to hang with.

Setup

The core concept of GigCity Elixir is hospitality.

Bruce and Maggie are graceful projections of that facet of southern culture. They work it as deep into their conference as they can. I felt very welcome, I know one attendee lost their place to stay, was going to not come and the Tates insisted they show up. That attendee took over the guest room I’d had as I went to the Moxy hotel. They take care of people. And they proactively support and encourage a diverse set of folks.

In the days before I was given a lovely tour of Chattanooga, got vegan junk food and had a great time chilling and resting.

GigCity Elixir as an idea is arguably Bruce’s baby. GigCity Elixir as executed is inarguably Maggie’s doing. She is a force and a delight. What a human. In the days leading up to the conf she helped me shop for cool sneakers! Apparently she’d already pinned down everything for the conference and could spend time kindly goofing off with me and showing me novel things to eat and drink.

If you want a sense of a small-scale event, put together well with a strong local connection. Consult them, visit their conference. See how they do it.

Day 1: NervesConf

With Maggie Tate logistics and a Todd Resudek schedule NervesConf became the start of my three conference days. Other people got some Ash training by Zach Daniels and Josh Price of Alembic.

Nerves is special to me. It was where I entered the community for real. I did Elixir, I did Phoenix. But I talked to people in Nerves much, much more. It was right-sized compared to the more busy Slack channels. Meeting a bunch of the people that I’ve run into over the years around Nerves was such a thrill. And getting to dig deeper into topics I’ve explored recently around security, boards and parts was thrilling. What a high-bandwidth and high-knowledge environment. I learned a ton and any question had experienced answerers right next to you.

Jon gave a very nice rundown of the recent work in the Nerves project. Some old faces spoke, some new faces spoke. All good talks. We had a talk from pojiro of the Nerves JP delegation (three people!) and I guess I brought in the European touch. We had Powell Kinney talking about the autonomous boat he’d worked on. I shared my learnings and explorations around security from recent client work on the Raspberry Pi CM4 and the challenges therein. I think it was well received and I had a good time giving it. Alex Mclain gave a talk that should serve as a foundation for anyone who wants to build products with Nerves. There should be recordings for all of these that will come out eventually.

I can’t imagine a friendlier and more chill group to present to. Darlings all of them and just a good vibe of people caring in a similar direction.

It was such a good time. I got some custom hardware (thanks Alex) and am now very keen to try and make some Nerves event happen over this side of the pond.

The only weirdness of the day was the storm preventing a bunch of people from arriving when they had planned. Many were re-routed, delayed or otherwise bothered by tornados, lightning and other fun flight-unfriendly phenomena.

A first late evening with drinks and friends.

Day 2: Gig City Elixir starts

I slept as long as I could without entirely missing breakfast.

A lot more people. I think NervesConf was 30-50 people. Gig City was up towards 150 I think. And a lot of friendly faces. Some I only knew the nickname, some I didn’t recognize in 3D. Some were very familiar and nice to meet again.

Randall Thomas started us off well of course. His talks are always good energy, possibly bad feels, but good energy.

Many lovely conversations. Suprisingly tasty catering (Maggie again) with really good vegan options.

And we did my second appearance in the conference schedule. A live episode of BEAM Radio. We tapped some guests and people we knew to prepare hot takes. Then me and Bruce just invited people up on stage. One of them was Quinn. She blew the roof off of the conversation in terms of where it could go and then it went. And we brought in people from the audience, whoever wanted to bring something to the table. I hope the recording comes out really well because I am very happy with how it all came together and the discussions that were had. People went all kinds of ways and did it well. I hope we also make a video of it.

Bruce and I didn’t have to do much, just keep shifting in new people. Thank you all who participated, I’m proud of y’all.

Another good evening. I restrained myself somewhat in preparation for my closing keynote the next day but still ended up going to bed a decent bit after midnight.

Day 3: Gig City Elixir wraps

Another morning even more tightly optimized for sleep. I wanted to be in sane shape for speaking and have a chance of lasting all day. Some nerves in me for sure but I also felt very comfortable with the vibe of the whole event.

I didn’t see a ton of talks, I picked some musts but also updated some slides that I’d left open for what the conference would bring. I went off and rehearsed. And as I felt myself slipping into a deep tiredness I scampered away the 5 minutes to the hotel, took a long shower, a short nap and recovered a lot of energy. I headed back in time to see Chris Keathley speak. I’ve followed the guy for so long and I find his thinking incredibly useful, informative and challenging, in the best way.

I went right after.

Preparing for my talk

This was the hardest talk I’ve given so far. In a keynote it didn’t seem right to hide behind technical curiosities or trixy, trixy code. I wanted to talk about idea-level stuff, I wanted to touch on community and as per usual I wanted to produce something somewhat novel. I think I pulled it off. I would not be doing this stuff if I wasn’t somewhat self-critical so there are plenty of things I could have done better. Better delivery, more practice, tighter slides, a bunch of things. But I’m satisfied with how it went and I don’t regret anything in particular.

I had an interesting constraint as I wanted the talk to touch on my sponsors field. It was not a constraint they placed but something that made sense to me. And I lined that up with things I’ve been thinking about anyway with Fly and Elixir vs Heroku and Ruby. I will try to gather those ideas in a blog post as well, people did say they felt they clarified something.

You’ll have the recording of the talk fairly soon.

Ending it

I ended the conference. My keynote was the closer. Bruce and Maggie were brave enough to let that be the final note.

That’s an experience. I recommend it. Go to some conferences and end them. It is quite fun :) Get permission first. There is a whole schedule. Ending them willy-nilly will make people upset.

Talk done I felt empty and drained for a good 15-20 minutes and just lumbered around and got encouraging words from people. This often happens after I’ve done something with a decent bit of build-up. A vacuum forms where nothing happens, intensely.

Then Flora (of Nerves puppetry fame) suggested shots so I tagged along out of the ending of the conference. Flora hustled a bunch of us off for tequila shots way too early in the day and then peaced out on an airplane. Boss move if you can pull it off I guess :)

Me and Josh (Price of Alembic) then went to Rock City which is a very cool mountain area tourist thing which the Tates and some other folks had planned a trip to. Josh accidentally acquired a Smith & Wesson trucker cap. Andrea Leopardi compelled me to promise to start using dbg instead of IO.inspect.

The rest of my evening was good conversation at the Moxy with various people and sniping Frank and Jon of the Nerves core team into trying to get ChatGPT to make flails which it does notoriously poorly at.

The next day

Most people had left. Most people were from the states. Me and Josh Price went for a sunday hike with the Tates. An “easy” “mile” according to the initial announcement. Tennessee Easy? It was decidedly at least mediumly strenuous and significantly more than a mile. And very sunny compared to what I’m used to. But we got to see just about all of Chattanooga from above and a long stretch of the Tenessee River. A lovely time.

Then we kept doing nice social things for the rest of the day. What lovely hosts and what lovely hospitality. Josh is also a really good travel-buddy.

Reflections

I’m writing this on the flight home and my head is a bit tired, a bit worn. But there is a lot of idea-matter, a lot of inspirations and a lot stuff bubbling from this conference.

I like software development, Elixir and all that. But I am also quite interested in how people make things. How do you make a community? How do you make a conference? One of the best parts about engaging as a speaker or similar is that you have one foot behind the curtain. You get to see people’s design thinking, people’s people-skills and you get to engage with people who make things happen.

Much love to all I spoke to and interacted with, anyone and everyone who clapped after talks, anyone who was too shy or respectful to say hi, all that did say hi, those who mentioned something I’ve done. Invigorating. All of it. And as much love again to everyone I hung out with, days and evenings. You all made the trip very worthwhile and I think many more good things will sprout from it.

I had the best time.

Before we dig in I should shout out some other BEAM, Elixir & Erlang-related things in the embedded space. AtomVM seems super cool and lets you do Erlang and Elixir on microcontrollers. ESP32s and such. I haven’t tried it. I really should. There is also the GRiSP boards which does interesting stuff to push Erlang’s soft real-time performance closer and closer to hard real-time. It also works with Nerves if you prefer some Linux in the picture.

Erlang has been used for embedded Linux for a long time. The language started in the telco world which blurs with networking which blurs with industrial hardware. It all blurs with “embedded linux” which blurs with IoT and connected devices overall. It should not be surprising to hear that there are a decent number of people who have done custom linux + Erlang systems in the realm of embedded linux. I don’t know of any that have established themselves as the tool of choice or built much of a following though.

I guess Frank Hunleth is one of those people. He did a bunch of embedded, mostly C and C++ before he started veering off into Erlang. Since Frank is a sweetheart we’ve had him on the pod twice. I think the first one should give most detail on his professional history. When Elixir showed up there was more interest and more engagement there and Frank pivoted his IoT tooling plans towards where the collaboration and community felt most active. So Nerves was born into Elixir.

What is Nerves then?

It is Linux as mostly a hardware abstraction layer, source of drivers and OS fundamentals. Then it starts the BEAM and you take care of the rest from Elixir or Erlang. The linux variant is a custom Buildroot system. Buildroot is the less complicated of the major embedded Linux toolsets. Nerves offers a bunch of prebuilt systems adapter to particular hardware. So the Raspberry Pi 4 will have different drivers and configuration than the Beaglebone Black. It supports all the Raspberry Pi computers (not the Pi Pico microcontroller) which means that if you have one in a drawer, time to bring it out. You can also roll your own system if you know enough or are very willing to learn hardware and Linux.

The development process is just an Elixir project. Depending on how hardware-reliant your application is this can vary a bit. Nerves makes it all easy. Trivial SSH into the device. Pushing firmware over the network so you don’t need to swap SD cards for every change. A/B partition updates to ensure it is very hard to bork your device in production. It really does have great tooling and a clean process for what is an involved type of work. And of course you can just paste code over SSH into iex. The best deployment process ;)

I got into Nerves by porting an eInk display driver from Python (which I knew) to Elixir (which I was learning). The Elixir Slack channel for #nerves was super helpful and friendly. At the time of writing I have been paid for Nerves work several times and am currently enjoying ongoing Nerves projects with clients. I really recommend trying some Nerves if you either enjoy tinkering with hardware OR have been looking for a reason to try Elixir.

Okay. So it boots the BEAM. And that’s your systemd-equivalent or init system. Most of your system should be Elixir code with all the expressiveness, effortless performance, consistently low latencies and reliability that entails. Compared to anything I’ve set up on a Raspberry Pi before it is also a wonder of determinism. Your deps get locked. It becomes a real software project, not just a bunch of random tutorials you run through.

There is also NervesHub which is an open source, self-hostable tool for managing fleets of Nerves devices. Firmware delivery, binary diffing for minimal updates, reverse consoles for troubleshooting. Authenticating devices based on hardware device keys, aka. a secure element, hardware root of trust (see NervesKey).

Nerves helped me get into the Elixir ecosystem and community and it keeps giving me wonderful new opportunities to learn and explore. It really holds a special place for me. Part of that has been the immense generosity of Frank, Connor and the rest of the Nerves core team folks with both their time and knowledge. I’ve learned so much and scratched many itches thanks to them. I hope to pay it forward both in software, time and effort.

There are already many examples of Nerves:

• SmartRent
• SparkMeter
• Rose Point (navigation)
• Bowery
• GridPoint

Most of the companies using Nerves do so quietly. This seems to be a thing, especially in industrial automation.

The Nerves ecosystem covers a pretty reasonable amount of hardware with good Elixir-based drivers or libraries. In cases where they are missing the Circuits libraries provide all the necessary primitives with I2C, GPIO, SPI, UART. Elixir offers some really exceptional binary pattern matching and binary construction patterns that make working with hardware incredibly smooth.

I think Nerves stacks an incredibly reasonable set of practical solutions on top of each other and gives shape to what might other be hard to approach. If you are curious to learn about embedded linux I think just the abstractions and structure itself is helpful and if you experiment with it you will come away wiser.

Easiest ways to get started is with Nerves Livebook: https://github.com/nerves-livebook/nerves_livebook

I made a demo video for it a while back.

So go touch some circuitboards. Enjoy yourself.

You can watch the stream here.

Transparency notice: The livestream and this blog post have been work supported by Tigris. We collaborate on the topics, they provide resources and pay for some of my time so I can do this stuff instead of writing software for clients. I am very glad and thankful that I have someone funding my publishing and creative ideas. Expect more output from me near term, and give them a try!

Background

When I cut my teeth architecting systems we didn’t have Objects outside of Object-Oriented programming languages and we had precious little Storage. And if we did have Storage it was either on the same physical server or some kind of NFS-based abomination.

Imagine you set up your n-tier architecture. Application scales horizontally in front of the database. You can cache heavy loads with some Redis or Memcache that were the rad things then. For file storage we had to set up a shared store of some sort. And they were finicky bastards.

The mounts would come loose and files got written in a local folder. The performance would randomly degrade. I remember installing a filesystem cache thing for my servers and with NFS it would suddenly just hard-lock the system at the kernel level. Because NFS lives or at least lived then, in the kernel.

When it works well it can be very practical to have a networked filesystem pretend to be a real one. But it is a pretty little lie that you are telling yourself and the system you are building. And if you lean into the file-ish nature of the NFS lifestyle … well, there are risks. And scars you can get.

I found AWS annoying when it arrived. Unreliable VPS:es that you weren’t supposed to put files on. Bah. Humbug. So it took a while before I got into Object Storage (all object storage is essentially S3-compatible these days). But much like the dumbness of EC2-instances enforced good scalability practices the simplicity of S3 made it tremendously effective.

What people want from a network file store is usually reliability and space. The data stays where you put it and does not get lost. And your data must fit. And as data grows it must still fit. So essentially infinite unknown unbounded storage growth. And ideally you don’t want to pay more than disk utilization you currently need.

Simplicity, restraint and constraints are good for starting most things. But especially to delivering on ambitious things. Like infinite* file storage.

* not actually infinite, but for most purposes close enough

Object Storage is much more clearly a service. It is not a file system. And when you scratch the surface the name clarifies itself a bit. It is not necessarily about “files” either. It is just the most successful NoSQL DB of all time probably. Keys, values and it doesn’t sweat the rest so much.

The fundamental operations are:

• Put object
• Get object
• List objects
• Delete object

The full list is much, much longer.

With Elixir

To show how we can work with Object Storage fundamentals in Elixir I set up a mix project: mix new bla. Then I added the following deps in mix.exs:

  defp deps do
    [
      {:ex_aws, "~> 2.5"},
      {:ex_aws_s3, "~> 2.5"},
      {:hackney, "~> 1.9"},
      {:sweet_xml, "~> 0.7"},
      {:jason, "~> 1.4"}
    ]
  end

I then created a config/config.exs with the following contents:

import Config

config :ex_aws, :s3,
  scheme: "https://",
  host: "fly.storage.tigris.dev",
  port: 443

To get your deps: mix deps.get

To create a bucket, if you have the fly command-line tool ready to go it is very simple to do fly tigris create. It is also free until your really use it, 5Gb and 10k+ requests/month. Of course take whatever bucket you prefer. But that will spit out credentials.

:ex_aws_s3 will automatically slurp up your credentials if you set them as environment variables but not the bucket name. If you need multiple buckets you can wrangle them explicitly. These examples assume one set of credentials.

I started the module I called Tigris like this:

defmodule Tigris do
  alias ExAws.S3

  defp bucket!, do: System.fetch_env!("BUCKET_NAME")
end

Then I tackled the listing of objects:

#..
  def list!(prefix \\ "") do
    bucket!()
    |> S3.list_objects(prefix: prefix)
    |> ExAws.request!()
    |> then(fn %{body: %{contents: contents}} ->
      contents
    end)
  end
#..

You can try things in iex -S mix to get everything we have in this project compiled and ready to go in the prompt:

iex(1)> Tigris.list!()
[]

Next some putting and getting. I am not suggesting you do error handling like this. This is for the purposes of simplicity and the stream, not a best practice and definitely not financial or legal advice.

# ..
  def put!(key, data) do
    bucket!()
    |> S3.put_object(key, data)
    |> ExAws.request!()

    :ok
  end

  def get(key) do
    result =
      bucket!()
      |> S3.get_object(key)
      |> ExAws.request()

    case result do
      {:ok, %{body: body}} -> body
      {:error, {:http_error, 404, _}} -> nil
      {:error, error} -> {:error, error}
    end
  end
# ..

I tried those out in iex as well to show the fundamentals. Then I wanted to hint at scale I guess. Task.async_stream/2 ensures things are run in plenty parallel according to your machine.

# ..
  def put_tons!(kv) do
    kv
    |> Task.async_stream(fn {key, value} ->
      IO.puts(key)
      put!(key, value)
    end)
    |> Stream.run()
  end
# ..

And then:

iex(1)> kv = for d <- 1..100, f <- 1..100, into: %{}, do: {"dir-#{d}/file-#{f}.txt", "#{d} #{f}"}
# snip
iex(2)> Tigris.put_tons!(kv)

And then to tidy up. Don’t run this on buckets where you keep data you like or need:

# ..
  def exterminate! do
    stream =
      bucket!()
      |> S3.list_objects()
      |> ExAws.stream!()
      |> Stream.map(& &1.key)

    S3.delete_all_objects(bucket!(), stream) |> ExAws.request()

    :exterminated
  end
# ..

Presigned URLs

No we are getting a bit fancy. So an enormously beneficial thing with Object Storage is that it can have features that spare your application server from a bunch of horrible work. Have you ever let your app receive a file upload only to save it somewhere else? Ludicrous! We have a service for that! There are protocols!

# ..
  def presign_get(key) do
    :s3
    |> ExAws.Config.new([])
    |> S3.presigned_url(:get, bucket!(), key, [])
  end
# ..

Use that. Get a presigned URL (you can configure expiration and such) for either downloading or uploading which you can hand off to the client and let them deal with the upload without you. Just between them and your object storage. Fun thing about doing that with Tigris btw. Tigris will place the data globally close to the uploader. It will then replicate as a cache if needed to other regions but this means your customers in Australia get local latencies.

This is equally nice for uploads and downloads. Your application server does not have to go between. At worst a download is:

• Client asks your application server for a file.
• Your application checks if the request makes sense and should be signed for.
• Presign URL.
• Client receives a response with a redirect to the presigned URL.
• Transparent download but your application doesn’t serve the bytes.

Multipart upload (streaming up!)

Up to 5Gb can be a single upload according to AWS. But typically that gets unwieldy. We can chunk uploads at 5Mb chunks.

# ..
  def put_file!(key, from_filepath) do
    from_filepath
    |> S3.Upload.stream_file()
    |> Stream.map(fn chunk ->
      IO.puts("uploading...")
      chunk
    end)
    |> S3.upload(bucket!(), key)
    |> ExAws.request!()
  end
# ..

This is a form of streaming upload. Which means if you need to do processing and then want to offload the result immediately you can reduce your memory usage to about 5Mb (it depends) by streaming the upload instead of holding on to the whole beastly thing. This specifically builds a Stream from a file for upload but there are many variants you can do using the Elixir Stream and IO tools.

Range requests (streaming down!)

Sometimes we just want a few parts of a file. Sometimes we want to build a hell-beast that does read-only SQLite VFS over S3 API. Sometimes we want to do a graceful streaming download. For this we have straight up HTTP range requests.

# ..
  def get(key, range \\ nil) do
    opts =
      if range do
        [range: "bytes=#{range}"]
      else
        []
      end

    result =
      bucket!()
      |> S3.get_object(key, opts)
      |> ExAws.request()

    case result do
      {:ok, %{body: body}} -> body
      {:error, {:http_error, 404, _}} -> nil
      {:error, error} -> {:error, error}
    end
  end
# ..

This is incredibly useful and powerful. Sure. You might mostly be dealing with files. Until you don’t. Or until you realize how many file formats expose useful information with just the right bytes.

Zip archives are streamable with Packmatic because they allow you to keep the index of files to decompress at the end of the archive. That’s a known location. We can search the last part of the file and only get the index, and so, the file listing. And that means we can pick out files. One can do similar things with ID3 tags on MP3s and get metadata without getting the whole file.

If you use the Elixir library Explorer the underlying Rust library supports the S3 API and will let you lazily read data from .parquet files and such that are stored remotely. Same thing.

I hope this clarifies why range requests are rather useful for a file storage service.

Why the S3 API became standard

All Object Storage services I’ve used so far have exposed an S3-compatible API. It makes sense. It helps you support the expected featureset and you get a trillion clients and SDKs compatible with your thing for free.

Is the S3 API just that good?

I don’t think it is. And I had a whole spiel here about how it is simply good enough and that the value proposition of reliable “infinite” storage with a common protocol makes it worth it. It clearly has been. In discussing this post with Ovais Tariq, CEO of Tigris, he actually shared a much more interesting viewpoint. Contrarian to my lukewarm take. It makes sense for Tigris to ship as S3 API compatible to make switching and getting started simple. The S3 API turns out to be a significant constraint however. Tigris has a metadata system and infrastructure that is different from S3. There are improvements, innovations and some really great features that are not easy to implement nicely in the fairly stagnant S3 API. Sure you can finagle features into headers and other clever stuff but there is a downside to it as any new ground you break will be missing in client implementations. From my conversations with Ovais and his team we will still see innovations on Object Storage from them. I guess we’ll have to stay tuned to see if the S3 API can handle it.

If you want to share some of the more interesting usages you’ve seen of Object Storage or just want to tell me about whether you found this helpful, feel free to reach out. I am on the Fediverse as @lawik@hachyderm.io and you can just email me as well at lars@underjord.io. Thanks for reading.

To start off, I am not an AI expert. I am not an ML engineer. I’m a web and systems dev that mostly works in Elixir these days. Through the Elixir community I got connected with ElectricSQL and have done some collaboration with them. This post is done as part of my work with them and is paid work. I let them read it before publishing but this is all me and my perspectives. They have kindly paid me to endure JavaScript and experiment with their tools a bit :)

ElectricSQL are firmly in the local first ecosystem. Their open source project/product gives you a way to get active-active replication with eventual consistency between a server-side Postgres and a wherever-you-want SQLite. You handle the data model from the Postgres side with your normal tooling. Whenever your clients have a chance to reach the sync service/postgres proxy you’ll get schema updates and whatever new data the server might have to your local thing. Anyway. Cool stuff. On to the local first inference!

I really am not enjoying React Native here. Part of that is almost definitely that I’m poking around the immature ragged edge of what people have launched, like llama.rn (based on llama.cpp) and the outdated example for transformers.js. There are no real paths and I’m not deep enough on mobile dev to patch up the holes well myself. I also don’t know enough C/C++ to ship custom sqlite’s with vector extensions and whatnot. Might get there but am not there currently.

I know there are people who have made models run locally. I am not blazing a trail. But it is also not particularly mature as a space.

So the current approaches I see for getting LLMs to work on phones is mostly quantization which to my understanding is the process of reducing the precision of the numbers and calculations to save heavily on space and memory usage. Taking Llama2 7B you have this 13.5Gb model at 16bit floating point. Quantizing it with llama.cpp down to “Q2 K” I believe uses 2bit integers. Pretty lossy. But it brings it down to 2.6Gb and it can actually be run on a modern iPhone (mine is a 13 Pro). With infinite time I would explore Mistral 7B as an option along with Orca variants and whatnot. There is a lot out there.

Now there are also efforts to rework models to be smaller fundamentally and then you have something like TinyLlama-1.1B (GGUF). This model is miniscule (relatively speaking) at 747Mb. I tried the 0.3 version at one point which was generating more nonsense than sense, wrong language, pure gibberish and more. The 1.x ones seems much more coherent though it certainly makes mistakes frequently. It is plenty snappy.

The crunched up Llama has produced semi-relevant responses for me but it has also been much harder to run. A 731Mb asset is perhaps on the larger side. But whether it goes over wifi, or the lightning cable at USB 2.x speeds, npx react-native@latest run-ios is not impressively fast when slinging a 2.6Gb Llama over the wire (Nullsoft would have loved this era of computing. WinAmp should return as an LLM I guess. It would really whip ..).

And I’ve had the phone go very warm using Llama2 models. And then pause all audio playback as background tasks stop getting priority. The CPU, GPU and probably some Neural Engine gets busy doing the hard work of guessing the next word and your phone becomes warm and inert. Not a great experience. But also kind of fun to actually see what this device can do.

Overall, my sense is that the quantized models are ahead of the built-to-be-small models. I think those approaches can converge and lead to some useful stuff. With enough persistence you can probably already wrangle some good usage out of them.

A challenge with these large models is of course that every app that wants to use one ends up becoming essentially a full-on mobile game size. And they aren’t very fast. I think there might be a case for an app that manages models for people and provides practical on-device APIs to interact with. To my knowledge this can be done in a few ways under iOS specifically. You can do a bunch of hacking around with App URL schemes. Your model service app can expose Shortcuts that the user can then base integrations on, a real power user thing. And it seems you could build an Action Extension.

Essentially this is the revival of the shared library. Assuming people do want LLM-sized models doing things on their phones it will quickly become unpleasant that every app ships their own Llama or Mistral. This might be resolved if Apple (and Google on Android) exposes their own on-device models that cover the desirable use-cases. We will find out.

I haven’t gotten into it but I want to try some TensorFlow Lite/TFLite models of various types as that seems well suited for phone-level devices. I just recently ran into this library for React Native. Like any React Native library I assume it will be torture to get going but has the potential to solve all my problems. Qualcomm recently released 80-or-so edge/mobile-friendly model variants on HuggingFace so that’s a good place to look if you want some TFLite to toy with.

The work on this is paused for the foreseeable future but high on my list was to try more things with vector embeddings. Those seem perfectly feasible to generate on a phone. The TFLite version of OpenAI CLIP is 304MB.

I think there are many things that can be done and if you have some pain-tolerance you can already run a local model. It is kind of fun and a good challenge to get real use out of them. I certainly like the idea of not relying on cloud to achieve these things.

Now one reason to pause this is that the ElectricSQL folks have already made some good progress with other parts of local-first ML/AI experiments. Postgres, pg_vector, Tauri and Llama2. Who needs React Native, really? They also shortly after launched a very exciting collab with the Neon Postgres folks: pglite

You might have seen pglite if you frequent Hacker News and such. This would take the pg_vector opportunities further. An embeddable Postgres, eh? Not bad :)

There is a repo from my experiments on my github if you go digging, it may be incomplete and not build. It may have junk in it. It is not a place of honor. Overall I think I explored some good stuff but most paths definitely hit a “oof, this is too rough” end. Trying to get SQLite with VSS was hairy. Trying to get Shortcuts working of all things proved incredibly frustrating. Realizing Transformers.js is missing stuff to make it work on React Native.

Overall, if you want to get into this. I think I’d recommend doing it from Swift/Kotlin. The natives will save you from one layer of fragile indirection while you explore the bleeding edge.

Thanks for reading. If you have questions or concerns, hit me up @lawik@hachyderm.io for the Fediverse/Mastodon or via lars@underjord.io.

As for Erlang this Awesome Erlang list has a ton of web frameworks. There have been many but I have never detected a consensus on what one “should” use. Actually, when I spoke to Robert Virding over beers at a conference I asked something about this and he more or less said that Elixir and Phoenix should be the preferred web framework for the BEAM. The exact question and the exact answer are muddled by time and memory. My understanding is that he much prefers Erlang for everything else but really wishes that people would just use LfE ;)

This should not be taken as a criticism of Erlang, rather a kudos to Elixir for establishing and maintaining this useful cohesion. The Elixir ecosystem has always had a fair bit of focus on the web in a way which Erlang has not.

People have built other web frameworks in Elixir. Phoenix remains the major player. It is the Rails/Django/Laravel of the Elixir ecosystem though I would argue it is more lightweight. In good ways and possibly bad.

Plug

We shouldn’t start on Phoenix. That’s the high-level framework. The fundamentals of dealing with web requests and creating responses are handed to Plug. Plug deals with headers, bodies, query params, URLs, paths and all of that by providing the concept of composable plugs. A very simple plug:

defmodule MyApp.NotFoundPlug do
	alias Plug.Conn

	def init(opts) do
		opts
	end

	def call(conn, opts) do
		conn
		# Send a 404 to the client
		|> Conn.send_response(404, "Not found")
		# Do no further processing of this plug pipeline
		|> Conn.halt()
	end
end

Plug also provides a module for routing and building up routes based on path and method. This example is from the docs.

defmodule MyRouter do
  use Plug.Router

  plug :match
  plug :dispatch

  get "/hello" do
    send_resp(conn, 200, "world")
  end

  forward "/users", to: UsersRouter

  match _ do
    send_resp(conn, 404, "oops")
  end
end

A plug which does not halt continues to the next plug specified. This means every step of authentication, authorization, parsing and so on can be layered. Similar to a middleware approach. They all work with the Plug.Conn struct.

Plug is a rather elegant way of getting web stuff done. Good building blocks. I stole some code from Frank Hunleth and Connor Rigby of the Nerves project when I wanted to do my live visitor counter in an image (also the Plug and the project) to do chunking and produce an MJPEG. It didn’t need the rest of Phoenix. A good case for straight Plug. But also, a Phoenix Controller is just a Plug and could do it just as well.

Web servers

Historically Phoenix has leaned on a web server called Cowboy written in Erlang. It has been a very reliable workhorse for a long time and has done well in that role. It connects to Plug via the plug_cowboy library. Increasingly I see projects pick up Bandit which is intended to be a replacement written in Elixir. This allows the community a lower barrier to contribution as more people in the Elixir space know Elixir than Erlang. There is more to it as well. We covered some of that in an episode of BEAM Radio if you are curious. Supposedly Bandit also benchmarks as a bit faster than Cowboy which is of course a nice perk.

Something these web servers have in common is that they are not your Ruby or Python application web servers. No reverse proxy required unless you want one. They can actually be trusted to do real frontline work. Erlang was built for it.

Phoenix

Now I think we can tackle Phoenix. Ah yes, the “Rails of Elixir”. But not nearly so similar. Now I don’t have any significant experience with Rails but I talk to people who do. I have experience in Django. Both Django and Rails work hard to help the developer be productive for the common case. The Pareto Principle, 80/20 rule, all that. And beyond.

In Django this is achieved through “magic”. Mostly inheritance of classes that introspect what you put in them and do things based off of that. They also take metaprogramming very literally and you end up putting a bunch of things in a class-within-a-class called Meta as I recall.

These are super-dynamic languages where monkey-patching and other fun stuff is incredibly available. This of course means you should limit how much you use this fun stuff as much as possible. A code-base without discipline can get very messy on top of these languages and frameworks.

Phoenix tries not to rely on “magic”. We call them macros instead.

I kid. Macros are among the more confusing parts of Phoenix as well as Plug but they are generally there to manage some inherent complexity for you and they are much of the time still in “your” code.

Typically you start a Phoenix project using mix phx.new my_app which generates a project that you then own. Sure, you have dependencies, the code of which you don’t own, but your MyAppWeb module has macros for bringing in the necessary functions for Controllers or LiveView and you can adapt that to your way.

I’ve heard multiple people go “that’s a lot of files” when generating a Phoenix project and I agree, that’s the impression you’ll get. But most of the files have fairly clear purpose once you get to know them and they are there to make things explicit and hand you the reins instead of mysteriously and magically inheriting things at you. There are also hygiene things like gettext that you might not use in your first few projects that are there because they just ought to include it. And you’d be pissed if it was not there when you needed it.

Opinionated design, or a lack thereof

We often talk about opinionated design in web frameworks. The reason is generally that an opinionated design makes significant trade-offs for some cases in order to support the common case. Pareto principle, 80/20 rule, all that, again. By providing an opinionated design you eliminate the need for many decisions and ideally provide well-proven good-enough solutions or at least helpful simplifications.

Phoenix is not deeply opinionated. I think comes with the developers driving Phoenix had more experience and wanted to focus on good primitives. Also, I Functional Programming achieves abstraction quite differently from the OOP style of Python and Ruby. FP tends to achieve complex things with simple parts.

Erlang is incredibly opinionated at a fundamental level. It makes a ton of choices in the service of building services and Elixir inherited those opinions. We’ve traded off a number of things we don’t care about to get a fantastic foundation for a web framework. We get trivial concurrency and parallelism but have traded off small binaries and number crunching. We have consistent latencies but don’t get the speed of mutable state.

I started out considering Phoenix as an opinionated framework in the vein of Django. I don’t know what gave me that idea aside from them both being web frameworks. Sure, it brings in some opinions such as “specifying routers in a central place is nice” and “this is how you should bring in your helper functions for doing controllers”. It also abstracts away connection pools and supervision trees and there are opinions enshrined there but that’s usually not what people mean when talking about opinionated framework designs.

If you come from Django or Rails you will likely miss the extremely convenient derive-everything-from-the-model typ of schema-driven development. If you want more of that the currently most interesting option is Ash Framework which is very opinionated and quite fascinating.

There are some opinionated parts outside the core of Phoenix’s web capabilities. Channels is an opinion on top of WebSockets, Presence as well. Phoenix LiveView is heavily opinionated and makes fairly aggressive trade-offs for great wins in productivity.

Phoenix also brings in Ecto by default and Ecto is a fairly opinionated approach to relational databases. Changesets and the Query DSL are both quite flexible but they push you towards a particular way of working that the library believes is best.

The most contentious part of Phoenix seems to be Contexts. Which is the connective tissue between a Controller, Channel or LiveView and the Ecto-driven data layer. There is an approach generated by default but it is much discussed and will generally require you to apply your own designs and ideas. It it very open ended. There is no special Context code. There is an opinion coming from the Phoenix generators but it seems very softly held and it is not really in the bones of the framework.

Overall I think Phoenix has been layered well and I find this restraint of opinion in the base framework means that there isn’t a big desire for a lightweight microframework alternative to Phoenix (see Flask, FastAPI in Python) because if you just ask it to not add a database or skip the HTML bits or whatever configuration you need you will get your minimal API service or you full-trim HTML-spitting machine.

The Phoenix feature set

I will try to capture the things that make up Phoenix. Major features as it were.

Project generation

Phoenix does a bit of inversion of ownership and as I’ve mentioned, produces a bunch of files that you can then own and evolve. I run mix phx.new on something like a weekly basis as I try a new hack of some sort. The generator has a bunch of options, choose database (postgres, mysql, sqlite) or skip Ecto and database details entirely. HTML or not. LiveView or not. Assets or not. This is the normal starting point.

There are other starter templates. Legendary, Petal Pro and some others. I can’t vouch for them. I’ve worked on a Petal Pro project, it was fine, it certainly brought more opinion around templating and layouts.

Ecto

The database layer. Not locked to Phoenix in any particular way but it does ship by default.

Ecto provides relational database functionality to your app and stops you from making a bunch of mistakes that could lead to SQL injection attacks and the like. This informs Ecto’s design in unusual ways. It relies a decent bit on compile-time macros for building queries. There are also escape hatches pretty much wherever you might need them.

Repos (Ecto.Repo)

A repo is an abstraction over a supervision tree that manages database connection pooling. A typical app deals with one Repo named MyApp.Repo and it provides all the query functions and such.

If you are dealing with two separate databases you can easily set up and additional repo. And if you are dealing with multi-tenancy or some other multi-database situation you have “dynamic repos” functionality which will let you work that way as well.

Schemas (Ecto.Schema)

The Schema module exposes a DSL for specifying database tables, mostly. It can also be used to specify schemas that are not backed by a database for various purposes. But mainly, this correlates with Django or Rails models while being significantly less magical in nature. They boil down to Elixir Structs.

Changesets (Ecto.Changeset)

A way to define validation rules for schemas (and other data). This will help you produce good errors, integrate actual errors returned from the database gracefully and many other things. Changesets are used in and around inserts and updates. They are a large topic and well worth reading up on because they are quite and interesting and useful design.

Web stuff

The Endpoint (MyAppWeb.Endpoint)

This module is generated for you but when added to your supervision tree it starts a Phoenix Endpoint which contains a supervision tree for starting any Phoenix-owned process. Whether you use the Plug integration for Cowboy or Bandit it will set up your web server to listen for inbound requests and route them to Plug and the Endpoint. The Endpoint defines a bunch of fundamental plugs and config. It then typically defers to the Router for further handling of requests.

Router (MyAppWeb.Router)

The router module is where you use a combination of Phoenix and Plug plugs to handle requests and delegate them to Controllers and LiveViews. It is a nice central place for structuring this and also allows you to define pipelines for other plugs that need to be applied, such as checking authentication and authorization.

MyAppWeb

This holds your macros for controllers, channels and liveviews. These macros mostly bring in other Phoenix functionality but they are in your file and you can and should use them as an extension point for bringing in your own tools.

Controller (MyAppWeb.MyPageController)

A controller handles a request. The Controller might produce JSON API responses, HTML, file chunks or whatever else. Doesn’t matter. If you are rendering HTML you get into templating and components. A controller is actually also just a Plug.

Templating (Heex)

Heex is an evolution of the regular Eex templating that ships with Elixir. Heex is HTML-aware and will tell you when you screw up your closing tags. It also has nice syntax for arguments beyond basic string interpolation and such.

def render(assigns) do
	~H"""
		<div :for={thing <- @items} :if={thing.great?} class={@myclass}>
			<.custom_component thingie={thing} />
		</div>
	"""

Innovations

I want to shine some light on the bits that I think offer capabilities beyond what we typically see in web frameworks or that do things better than most web frameworks.

PubSub (Phoenix.Pubsub)

Functionality used as an underpinning for Phoenix Channels. PubSub is a publish/subscribe mechanism for loose coupling of communication across your application. It uses Erlang’s process groups and Erlang distribution. Unless you are on Heroku in which case you need the Redis adapter. You should be clustering or Chris McCord will be upset with you.

Phoenix PubSub is incredibly practical for letting processes track a topic and receive messages when things happen on that. And as GenServers, Channels and LiveViews are processes that can handle messages you can use these for many niceties. A common one is informing a LiveView that content it is showing has in fact changed. It can then do whatever it considers appropriate to inform the user.

Presence (Phoenix.Presence)

Built to support usage in Channels but usable as a more general tool. Phoenix Presence lets you track the ephemeral presence of things (usually users). Are they online? Busy? Away with a small message? Are they on mobile only? It uses a CRDT approach to minimize how much data it needs to keep around while creating an eventually consistent model of the world without requiring a separate storage backend.

Channels (Phoenix Channels)

An abstraction, intended to go on top of WebSockets though it will do long polling if necessary, it provides an abstraction for connecting to channels and joining rooms within them. Each WebSocket is backed by a GenServer on the server side and will let you keep state about the connected user. Typically you connect to it with a JavaScript client which handles failures, reconnects and provides some API niceties.

Fundamentally the most important bits are WebSockets and server-side actors. But the rest is nice too.

LiveView (Phoenix.LiveView)

The belle of the ball for most of us. The point of LiveView is to eliminate the need to write JavaScript for most web development tasks. It allows driving highly interactive web UI based on server state. This typically happens over a Channels-style WebSocket. You can annotate your Heex templates and components with attributes such as phx-click and similar to allow sending events to the server. The event is processed to update server state and a minimal diff is calculated and passed back to the browser which can then patch the DOM using morphdom.

You would have a very hard time getting payloads this lean using a SPA with your own API implementation. It also leans hard on an Erlangism: consistently low latencies.

There is also support for driving certain simpler JS/CSS updates without a server round-trip, animation support, streamed listings, functional stateless component, stateful components and more that I’m forgetting.

It is also trivial to push server-side changes to relevant LiveViews, thanks to the BEAM and Phoenix PubSub.

Here we have a very opinionated design. It sacrifices offline support for not needing to write as much code or maintain as many interfaces. It is an immense time-saver. There are many more posts about LiveView out there. We can leave it at this point.

Asset pipeline

Once upon a time Phoenix shipped brunch I think. Then they switched to the industry standard: Webpack. Then they grew sick of the massive support burden it was to help people keep their node and npm setups working. Phoenix switched to Esbuild delivered through an Elixir library via Hex.pm.

I think this was a very good move. I have so many fewer asset problems now.

Mailer (Swoosh)

Phoenix ships with Swoosh which is an abstraction for email. It has plug-ins for many popular transactional email providers. So once you go into production you have an easy time doing password resets or magic links.

It also has a very nice little dev tool that lets you check a mailbox for the mail that has been “sent”.

Live Dashboard (Phoenix.LiveDashboard)

Under-promoted cool thing. It is an observability dashboard web UI that you can ship by default in your admin and do simple things like:

• Investigate your running processes
• See breakdowns of memory usage
• See OS metrics
• See ETS table usage
• Capture request logs

And more. For no effort you get a practical first-look tool for investigating a misbehaving system. This is very Elixir. All the primitives and possibilities come from the BEAM and have been possible in Erlang since forever. But Elixir made it nice, simple and for most of us it is there by default (because Phoenix).

Your app is not just “web”

Most systems have more duties than just serving web traffic. Often this is delegated to queues/brokers, workers, other services, Redis, databases, cloud functions or whatever else. A Phoenix app is a BEAM application first. You can run many other workloads in it. You are building a system and it doesn’t have to be several applications under this runtime.

Importantly Phoenix does not infect your system in any particular weird way. It doesn’t warp the application away from normal Elixir application conventions.

Eliminating infrastructure

I have touched on this throughout. But fundamentally a Phoenix application is only really expected to need a backing relational database. Typically Postgres.

No Redis. No mail workers. No separate task-runners. It can all occur from your application.

Wrapping up

Phoenix is very much the canonical web framework for Elixir and it has massive gravity in the community and ecosystem. I think we benefit a lot from that. There is a lot of common, directed effort centering on Phoenix and Ecto.

It is a very solid web framework, on top a legendary runtime. And you build inside it with a very approachable language.

We get the system design and development possibilities afforded by our runtime where we have very few things limiting us in what we want to do.

And then we get the innovations that have been built in a way that is essentially unique. Other ecosystems have copied LiveView, as they should. But their variants can’t do everything that LiveView can and they will have challenges getting there if they try. And that disregards all the practicalities of making an application be live.

The Actor Model is generally a model for concurrent computation. And Erlang was not built to implement Actors. I’m sure we could be debate about whether it does or not. I have not read Carl Hewitt and I’m not going to offer an opinion. It does something similar enough in terms of message passing and spawning. It might be the case that Erlang processes has shifted what people expect an Actor to be and rather than be a realization of the theory it has become an implementation that overrides the theory. I will reference The Actor Model as something Erlang and Elixir does with regards to processes, messaging and concurrency for the purposes of this post. Please argue about this further in all comment sections and social media for traction and engagement ;)

Regardless. The terminology of an “actor” does not exist* in Erlang or Elixir. There is spawning processes, sending messages and receiving them. The typical representative of a high-level, full-featured actor in Erlang is the gen_server (there are also gen_event, gen_statem and other abstractions with different features) and in Elixir we have the GenServer module, etc. A generic server. They are almost exclusively started with a link to the spawning process which should ideally be a Supervisor. I cover more of this under the resilience part of the series.

* I actually recently found a use of “actor” in the ecosystem. The Ash Framework uses the term actor for the concept of “the acting party”. Typically a user, team or organisation taking an action on a resource. It is unrelated and not in Elixir itself. Amusing though. Ash brought Actors to Elixir.

A GenServer is a bunch of logic started as the core loop of an Erlang process. It conforms to a number of useful protocols for debuggability, introspection and as mentioned they tend to be started with a link. Meaning the processes will be aware if the other end of the link exits. The fundamental loop is to wait until it receives a message, execute a relevant code path based on the message, update held state if necessary, back to receive a message and repeat.

There are three main ways of interacting with a started GenServer. Calls are a request/response approach with a timeout. These are synchronous from the calling side but can be handled in an asynchronous fashion by the GenServer (see returning a :noreply on a call in the docs). Then we have a cast which is a fire-and-forget message sent to a GenServer. And then there is regular message passing with send/2 and related fundamentals. A GenServer is an Erlang process and it can receive messages that are not calls and casts. Similarly it can send messages that are not replies to calls.

For people coming from OOP to Elixir there is often an attempt to map a class to a GenServer module and object instantiation to start_link and init. This is generally an anti-pattern. A GenServer sets the runtime behavior of your application and is typically relevant for holding a connection, processing work from a queue, polling an API or other runtime concerns. It is a system design and architectural choice. It is not really about code structure and encapsulation. If what you want is to group data and logic, what you want for this is usually a module, probably with an Elixir struct and a bunch of functions. Pure immutable operations that can be performed in any process at any time without creating a bottleneck. Your actors can benefit a lot from modules like this to manage their internal operations.

Whenever you need serial ordering or fanning out to multiple parallel pieces of execution you should reach for GenServers, Tasks and such. Sometimes you absolutely want a bottleneck, usually when you need control of how things interact. Rate limiting, serializing writes, managing a unique resources such as a GPU or tracking information about an ongoing background process.

One example of a typical actor usage is also one of the more agressive uses of the paradigm. The Phoenix LiveView. The LiveView holds a Phoenix Channels WebSocket connection in state. It handles events received from the WebSocket, messages from the system and through these handler functions it can update the state of the LiveView. Every change to the state produces diffs that can be passed to UI over the WebSocket. Every Erlang-style actor exposes an API surface to the surrounding system, the LiveView exposes a subset of it over the WebSocket.

All Erlang processes including GenServers have a process ID, succinctly known as a PID. There are numerous ways of registering, aliasing, grouping and finally addressing them. This can be used to implement messaging patterns such as queue/broker and workers or publish/subscribe (pubsub) internally for the application. Messaging can also traverse an Erlang cluster transparently which is incredibly convenient and powerful. During messaging you don’t have to be concerned about where your processes are in the cluster. If you have the PID you can reach them.

Concretely, the Erlang :pg module handles Process groups and can register named groups across the cluster in an incredibly convenient way. This is the foundation of Phoenix PubSub. Unless you use the Redis adapter in which case you should* probably move on from Heroku to get the best use out of Elixir.

* I snark.

While all* code in Erlang and Elixir runs in processes and most libraries and tools you work with are built on top of gen_server and friends. The Actor Model as implemented in Erlang is not the dominant paradigm that determines what your code looks like. Not in the way Object-Oriented Programming does. It is not like in Python where “everything is an object” kind of dominates the language. Most of Elixir is a dynamic flavor of Functional Programming. You don’t spend all your time doing Actor-stuff. Processes and messaging are underpinnings for the code you write and when you need to deal with state or work needs to be distributed across more units of concurrency. Certainly. However I find most of what I do is basic Functional Programming.

** There are several exceptions. It is true enough but not actually absolutely true. Ports and NIFs are ways of doing things outside of regular Erlang processes.

And as anyone who has done Elixir for a bit can attest. If you isolate the places where you perform messaging a bit from other, pure logic, that makes it a lot easier to test. Some have argued that Erlang is like Kubernetes and that GenServers are like microservices. This is very incorrect. But it may be helpful for understanding the nature of an Erlang/Elixir application. It has many things going on. It is very much as system in and of itself, it is not constrained to being a single web app or a single worker. It was designed to do many things.

Where this analogy holds well is probably testing. Writing tests across Kubernetes cluster is probably even worse than writing tests across your Elixir application but I think they are similarly instructive. To test multiple moving parts like this you need a lot of control over them from the testing side. The worst case is if they are heavily reliant on time and timing. That is really painful to work with in any system and it is quite difficult to control well.

Trying to write an integration test that crosses multiple GenServer and Supervisor boundaries will tell you how well you did with abstractions. You will know where you took shortcuts or simplified things too much. And this is the reason why the pure tests are much easier. Testing a single GenServer that can be started with a non-global name is a reasonable process. Testing a few of them that are all essentially singletons probably prevents your tests from running in parallel, will make cleanup and reset a pain and will make you want to refactor almost immediately. This has influenced the library guidelines for Elixir. Under Anti-Patterns you’ll find:

Avoid application configuration

Why? Because it is global. Your user may use it and your library might offer it as a convenience. If you require it your are placing massive limitations on how useful your library can be.

Avoid using processes for code organization

Which is what I was saying above. Processes and consequently GenServers are not a tool for code organization.

You will find that Phoenix generally generates modules for you that you then own. MyAppWeb.Endpoint, MyApp.Repo (from Ecto, really), MyApp.PubSub and so on. Then these are started in your application. You own them. If they pull config they pull them based on the name they have in your application so that if you want multiple Repos you absolutely can (dynamic repos also exist, different purpose). The Endpoint gives your the web server supervision tree, the Repo gives you the database connection pool supervision tree, the PubSub manages your process groups and whatnot for the PubSub mechanism.

Phoenix needs processes for what it does, concurrent requests, queries, pubsub. But it hands you that stuff. You can run many Phoenixes in your application if you need or want. The framework hands you the reigns.

Wrapping up. The Actor Model that Erlang actually has is a mechanism mostly intended to provide a reasonable API for concurrency. And distribution. And also a way to model a system design with a highly dynamic functional language. It does shared-nothing and message passing. This is where many (perhaps most?) high-level concurrency APIs end up. The alternatives are to my understanding always more painful with much more nuanced performance trade-offs.

If you have a need to argue about the nuances of Actors I am available explicitly for that practice (also business inquiries) over email as lars@underjord.io and out there on the fedi @lawik@hachyderm.io.

The previous posts on concurrency and resilience might give useful context around how processes work and how supervision trees are structured. I will try not to lean too heavily on them but if you feel the need to understand more, consider reading them.

I don’t know if my understanding is accurate to the truth of the system, I don’t read BEAM bytecode, but I’ll take a stab. Elixir and Erlang applications do not change their shape very much in compilation. I imagine the VM-level code still has an understanding of the fundamental parts of the language we see. Modules. Functions. It certainly knows about processes, messages, mailboxes and all that.

Why would someone build it this way? It has to be fundamentally inefficient compared to say .. C++ or Rust. Your compiled C or C++ program will be very different from the code you wrote. Who knows what the compiler will feel like doing. It will be semantically equivalent, the meaning of how the code executes will be retained. There is a ton of additional debugging information required if you want to retain the human-readable meaning and ideas of the original code during execution of the end product. And that might be enough to give you a nice stack trace.

Elixir aside, other high-level dynamic languages also seem to keep more of their general structure. If the program can mutate itself significantly during runtime it limits how much the compiler can be allowed to mangle things.

Erlang was built to provide hot code updates. I think this is the fundamental reason it is so introspectable at runtime. Hot code updates, while they can be done rigorously and should be treated seriously for production systems, is essentially the biggest monkey patch facility imaginable. I guess any language that can patch itself at runtime could start implementing hot code updates. But Erlang is designed to allow it and do it gracefully. This significantly limits how much the compiler can be allowed to boil things away.

Let’s start with the tool that Elixir devs tend to use daily. The REPL, the shell, iEx. It is a great part of the developer workflow as is. More importantly though, if you package an Erlang Release of your project you also end up with my_app remote. This command pops an iex shell, connects to your app over Erlang distribution and lets you poke your application. It lets you operate your application. All your modules and functions are intact and you can just make function calls, send messages and generally poke and prod around in your cluster. Whatever you need, anything that’s available in Erlang and Elixir plus any new code your a willing to type or paste into the interpreter.

What facilities do we have to actually pull information from the system? Well. The Erlang sys.get_state/1 function lets you pull the state of an “Actor”-style process (GenServer, GenStage, gen_event, gen_statem, etc) which is usually all of them. If they are part of how Erlang runs supervision trees by implementing those protocols it should also be able to give you the state it is holding. So you can inspect the running state of your application down to the studs.

Elixir offers Process.list/0 to get a plain list of all the process IDs (PIDs) on the local node. With this you can do Process.info/1 to find out what is going on and investigate things like memory usage, reduction count (number of function calls), initial function call and much more. All things that are part of the Erlang protocol for a well-behaved process.

This is all underpinnings for higher-level tools such as Erlang’s fun desktop UI app observer which can show you a graph of your supervision tree. It also has an activity monitor for your processes (order by memory used descending, oh there’s the memory leak). You can kill processes, you can get system-level stats. It has a bit of everything.

Of course with LiveView coming to Elixir this was pushed a bit further. A default Phoenix app has a few lines to uncomment to enable LiveDashboard which is a view that gives you much of the same information as observer. Plus it collects some telemetry for Phoenix and Ecto. You also get system-level information (disk, RAM, CPU), BEAM-level information (memory allocation types, resource usage, schedulers running) and a bit more. You also get a Process listing here in a nice and neat web view. If you are new in Elixir and want to make some waves I think pushing LiveDashboard further would be a good place to poke around. It is already a very cool start.

I believe LiveDashboard established this pattern for libraries with web UI for Phoenix. It provides a plug. Meaning you can shove it in any part of your router that you like. Usually behind an admin access check. I’ve since seen this done with Oban (job processing library) web UI and I believe the same thing is done with Orion.

Let’s talk about Orion because it feeds right into this story. It is a recent development by Thomas Depierre. Dubbed as a Dynamic Distributed Profiler it does something that requires a lot of instrumentation to do in any ecosystem and which is probably impossible to do fully in some. It was entirely achieved with existing Erlang facilities. You can enter a module name, a function name and an arity, hit Run. It will start to capture the performance of that function being called across your entire cluster. It then graphs that and gives you data and statistics on how that function is performing. There are many cool directions to extend this tool, let’s get into what it was built on. Tracing.

Tracing in Erlang is a mechanism for capturing information around the execution of a function. It is not limited to performance numbers. I’ve used both raw Erlang dbg, the convenience library recon and a little bit of recon_ex to do tracing on production systems when I needed to figure something out. I don’t know of any other runtime or language that makes this possible. Maybe I’m missing a world of tools in different ecosystems. Let me know. But essentially you formulate a type of pattern match for which invocations of the function you want to capture and in what way. Typically I want the inputs and the outputs along with execution time.

Think about this for a moment. If you have a function that seems to end up getting called with the wrong value, probably nil, for some unclear reason. And of course only in production. You can, in a reasonable manner, set up a trace and either wait for the thing to happen or trigger the behavior. Then you just watch the answers come in.

Oh, you need to know what happens one function deeper? Set another trace. There are limits and considerations for how much tracing you should do at once but you have a lot of room to play. Most of the tools built on top of the Erlang primitives try to protect you a bit from overloading your system with trace messages.

There are so many tools that haven’t even been built on top of this yet. I don’t think most Elixir developers are even aware that it is possible. There is no particular reason you couldn’t trigger an automatic trace after a new type of error surfaces and try to capture more info on the next run, ship that to your devs. Or build a UI for picking modules, functions and args to match so that you can capture these traces. If you want you could probably adapt the results to go into your Open Telemetry trace storage. There is a world to explore here.

And to some it won’t be available at all. If you run your stuff on Heroku and don’t add some tool for accessing a shell through the web or something you might have no way of reaching your server to pop the shell.

I should have asked for a Fly sponsorship here. This stuff is why I’m excited about their platform. Wireguard private networking by default makes clustering much easier. It also makes connecting to servers simpler which means getting at an iex shell straightforward.

You could replicate most of it with Tailscale or if you really want to work for it, custom wireguard stuff. Either way, Fly is a very good match for Elixir. It makes sense that they anchor their presence in the Elixir ecosystem by funding Chris McCord’s work on LiveView. Their infra offering just fits so well. A bunch of asterisks on the maturity of the offering still remain and they’ve owned up to that. Feature-wise I really like it.

A regular VPS or dedicated server is also perfectly convenient to shell into.

This capability, especially with Wireguard networking, also allows you to connect a Livebook (collaborative code notebook for Elixir) to a running system. This lets you build a recipe-book of things you might want to do in your system. Some would call them playbooks or runbooks. Rather than writing ad-hoc code that will get lost in your terminal history you build up a toolset. I haven’t put this into practice but it should be perfectly feasible.

To revisit. The BEAM retains the shape of your application. You modules, your functions, they still exist. Your processes are not just abstractions that are flattened out by the compiler. They are real and exist, comparatively speaking. Erlang and OTP have protocols and systems in place that make it possible to observe the running system at a high level for an overview as well as dive in and inspect any particular part.

The unique nature of these incredibly dynamic systems warranted unusual solutions and that gave us the Erlang tracing facilities. It is not a wildly monkey-patching library from some APM provider. It is not a hack to interject into the operation of the software, it is a fundamental facility of the runtime.

Playing nice with existing ecosystems and standard practices such as Open Telemetry is important. Elixir and Erlang are used for serious stuff and in mixed environments. It can’t all be special, unique and quirky. And I think the telemetry handling, metrics libraries like PromEx and the OTel implementation make great use of the BEAM and do not require external tools to operate aside from where to store the data.

Then when you look at the stuff that really is special, unique and quirky there is immense potential. We can go beyond what is feasible in other runtimes and languages. I think this is a big space for innovation on top of Elixir, Erlang and the runtime. The tools I’ve seen in this area are still fairly simple and there is so much potential.

This is what you get working with a higher level of abstraction that had a deeper purpose. Here the abstraction is not just about convenience and syntax but a deeply worked design that serves larger objectives. Python is a high level of abstraction primarily to make a sleek and convenient language. I am not convinced that the design of the language and runtime had clearer objectives more elevated than nice and convenient syntax. That’s fine but it has long-term consequences.

I think observability and introspectability, of arbitrary parts of the system, at runtime, is one of those things that people don’t know or really think about with Erlang and Elixir. But the magic of the BEAM is in the runtime and at runtime. It always was.

What would you want to see in observability? Am I missing fantastic contenders in other ecosystems? Let me know via email at lars@underjord.io or on the fedi @lawik@hachyderm.io. I really enjoy exchanges with my readers.

Previous episodes of this article series has unpacked Concurrency, Real-time & Latency and the syntax. In this one we talk about how Elixir and Erlang achieve reliability.

There are agressive, defensive and rigorous practices you can adopt in a variety of ways to achieve reliability and resilience. Testing, profiling, specifications, design, observability, culture, operations and so on. This is not about most of those. This is about software abstractions that serve you in making your system more resilient and allow you to construct resilient designs.

Erlang implements the Actor model, the typical actor in Erlang and Elixir terminology is the GenServer. The generic server. It is a process that has some nice facilities for receiving calls (“synchronous” messages, expecting a response within a timeout) and casts (asynchronous messages, with no expectation of reply). A GenServer is a bunch of API wrapped around the basic Erlang process. A process can receive and send messages. A GenServer (also other related types, gen_event, gen_statem) implements capturing errors that occur and the subsequent exit and passing information about the problems to the parental process as a message.

So what is the parental process. Well it is usually a special type of Erlang process called a Supervisor. A Supervisor can run multiple child processes that implement the necessary API for being a child. The Supervisor then handle problems with the children falling over by applying a strategy.

The strategies are normally one_for_one, one_for_all and one_for_rest. One for one indicates that if a child crashes it, it and only it should be considered when restarting. Leave the others alone. One for all means that the supervisor should restart all children if one child fails. This is often useful if the children are sharing a workload in some important way where restarting one would cause trouble with the others. One for rest is kind of nuanced in that the Supervisor will restart any child defined in the starting order after the crashing one. This can be used to kill a supporting process if the main one dies for example.

So a Supervisor supervises a set of child processes. This set can be dynamic, growing and shrinking as the system operates. In Elixir that would be a DynamicSupervisor. If a single crash happens within a child it will apply the strategy. If the new child crashes immediately it will try again. If there are enough failures within a given threshold the Supervisor will consider itself failing at the job and crash.

This crashing is a feature, not a bug. By crashing it will propagate a signal to whatever Supervisor started this Supervisor that there is an issue. This Supervisor can in turn apply strategies to try and recover the Supervisor through the tried and true practice of turning it off and on again. Or via custom logic if you are feeling spicy.

How many restarts, how often, how intense it is allowed to get. That’s all configurable. You might be fine with the defaults but if you have something particular in mind you have a pretty solid set of knobs to twiddle.

By having Supervisors supervise Supervisors all the way back to the root Supervisor which sits on top of your particular Erlang application we have defined a supervision tree. The idea being that any issue out in a nice leaf node of a tree is probably addressable there and should not affect the rest of your nice stateful actor-packed system. But if it isn’t, it is probably addressable at a branch closer to the tree.

A typical use-case for a Supervisor is a connection pool. And it might be the case that if any of the connections crash that indicates a problem with the server and you know from experience that you might as well re-establish every connection in the pool. Cool. Your Supervisor should do one_for_all. While if a phone-call crashes you probably don’t want to dump out every call in the system just because one had a weird issue. one_for_one should do it.

This is where the Erlang “Let it Crash” idea comes from. It is a bit of a meme (often repeated, poorly examined, often misunderstood, sometimes with funny pictures). Fundamentally, if there is nothing useful your code can do on a failure, no mitigation, no meaningful fallback, then you might as well have it blow up and rely on the underlying tree to make whatever is most useful out of it. You should probably handle the usual suspects “oh, data is missing, let’s 404”. That is an expected failure mode. “oh, hard drive has imploded and disk can’t be read” is a very different error and a 500 Internal Server Error is probably fine.

I don’t think I’ve seen an up-and-running Elixir application go down due to exhausting the Supervision tree with erroring out. I have seen it many times as a failure mode for a misconfigured deploy, typically missing an env variable, where starting the initial supervision tree fails.

Supervision is the fundamental building block in Erlang’s resilience story.

How is it used in Elixir? Not much. As in, we don’t typically need to think about it. Phoenix ships a practical and helpful Supervision tree. Ecto ships a good Supervision tree for your database connections and all that. These Supervision trees live a peaceful secretive life under the guise of library code and you get all the advantage with none of the work. And the moment you start coloring outside the lines and building your own GenServers the facilities are there for you.

With your regular generated Phoenix web application all you see is that your application.ex file contains a few entries for different purposes. These are the first branches of your supervision tree and all that you really see.

You get MyApp.Repo for your Ecto Repo which is the abstraction for all your database connectivity, hiding your connection pools and whatnot. You get MyApp.Endpoint which wraps up your HTTP server (Cowboy or Bandit) in Plug and Phoenix abstractions to respond. Under this you will build up a subtree with a variety of jobs. For HTTP requests you have processes that respond. I believe Cowboy uses a pool and Bandit spawns more ad-hoc. There are also the WebSockets and the specific Phoenix Channels or LiveView processes that belong to the state of your application. This all branches out from your Endpoint as it is all related to your web serving.

You can hear more about how Bandit operates on the BEAM Radio episode we did with Mat Trudel. We got into the process usage a decent bit there.

Separately you’ll have MyApp.PubSub which starts the Process Groups (pg2) or Redis adapter for doing cool PubSub stuff. This is not strictly web-related, so it has a separate sub-tree. Someone did a thinking.

I appreciate that Phoenix doesn’t try to hide this stuff from you any more than abstracting away the details. If you want to run your app without the web part you can quickly get to the idea of dropping the Endpoint from the list. You simply don’t start it. Same with the DB, just cut out the Repo. Likewise if you need multiple DBs or multiple web services. You can make more Repos and Endpoints. It is explicit. The code is right there. It uses the standard Erlang and Elixir way of starting things.

Anyway, that’s a detour from resilience and reliability. Fundamentally you have these supervisors, strategies as a structure to contain the blast radius of errors. The places this approach falls down is if you build a native implemented function (NIF) in C or similar. A crash in that brings down the whole VM. This is why doing work in properly managed BEAM code is preferred in most cases and why you’ll often see attempts to re-implement things in Elixir/Erlang rather than just binding to an existing C library. Rust and Rustler, Zig and Zigler both offer NIF implementations with some additional safety. Of course they don’t entirely remove the risks of native code though.

Rustler uses dirty schedulers and I expect Zigler does something similar or at least supports the old and the new options for building NIFs. So they should play nice-ish with Erlang scheduling. But fundamentally native code doesn’t quite offer the same safety or same behavior.

Beyond Supervision trees there are more aspects of Erlang that are resilient. Scheduling and pre-empting was intended to produce consistently low latencies but it also protects you from heavy workloads bogging things down, it prevents infinitely looping bugs from slowing the system to a crawl and in general makes things resilient to things not being ideal all the time.

Another thing is the Erlang heart. That’s a module that can be used to detect the Erlang application itself going down and being able to start it again. An almost-supervisor for your entire application. The IoT Nerves project also uses a variant of this to help ensure a resilient device.

Having built in other web frameworks and languages this is all a bit foreign. It is more reminiscent of building a microservice architecture than building a single application. But it does occur at the application level. It gives you tools to think about bringing up your system, bringing down your system, handling failure in your system. Those concerns all exist in other languages but the control-mechanisms tend to be quite coarse. Erlang and Elixir helps you think about how you design your application.

Do you have lessons learned that you’d like to share about building for resiliency? Does the Erlang way of doing it resonate with you? Let me know over email at lars@underjord.io or via @lawik@hachyderm.io.

Soft realtime. This part of the Unpacking Elixir series will really benefit from having read Unpacking Elixir - Concurrency as that post covers a lot about how concurrency works in Erlang and with that it covers Processes and Schedulers. This is mostly about Erlang and the historic choices made in developing the BEAM virtual machine. In the end we will get into how Elixir leverages it in the current landscape but Elixir didn’t create this capability.

Erlang’s claim for consistently low latency predates the multi-core concurrency approach a fair bit. From my understanding it ran a single scheduler in the olden times and the pre-empting was there to ensure this fair distribution of computational resources. It would prevent single pieces of CPU-intensive work from holding up concurrent, faster, pieces of work. It ensures progress is made on all work in short order. No piece of work should be waiting long for processing. A heavy piece of work will also eventually resolve.

Another interesting advantage of this design is that an accidentally infinite piece of work will not necessarily disrupt the system significantly. It will waste resources but will not block all progress.

Erlang was built for telecom and the work was, again from my understanding, about routing phone-calls without introducing noticeable delay preventing any particular misbehaving process from impacting other processes too heavily. In a way it was always about the quality of user experience. Performance often is.

It is a very high-level and dynamic system which is unusual when talking about performance. Why? By being a higher abstraction level, being very dynamic and working with immutable data structures as a Functional Programming style language Erlang leaves a lot of performance on the table in the service of other objectives. We won’t go into those other objectives now but Wikipedia lists distributed, fault-tolerant, highly available and hot swapping as additional traits beyond the soft real-time. I did say Erlang had audacious goals.

So soft real-time or “consistently low latency” is one of the objectives. And it is a performance objective. This is an interesting quirk of design. Designing for the absolutely lowest latency would require a very different approach. It would complicate or even sacrifice other objectives. It might be more costly, complicated and rigid or it might be significantly limited in capabilities. Erlang is not a one-trick pony.

Rather than the absolute lowest latency they went for consistently low latency, with tolerance for deviations and acceptance of imperfect results. They created a system that has been spoken of with some reverence in Computer Science ever since. It is not because it has the lowest latency. I am quite certain it does not. It was only one of the objectives after all.

So we have a cool high-level language, with nice abstractions, that offers low latency. What did people do with it? We mostly hear about chat and messaging because it does do that very well. The popular open source messaging queue RabbitMQ is built on Erlang. The Jabber/XMPP ejabberd software was a popular FOSS chat server. Seemingly that ended up in use at Facebook for their chat. Actually, you will find Erlang or Elixir driving chat in a ton of computer and video games. League of Legends by Riot Games comes to mind. Also beyond messaging but still in games is the heavy use of Erlang in Demonware which powers some pretty high-profile games. Like CoDBlops.

Even more famously Whatsapp built on Erlang. And Discord built on Elixir. These are two massive players in instant messaging that are deeply invested in the BEAM virtual machine. They are applications where latency matters. Not as life-and-death, not to hit the deadline for correctly driving an electron beam or anything quite so sensitive. Rather for providing a good experience to users. It doesn’t matter if there is small variance in latency. It matters that you typically avoid a large variance.

Onwards to what we can use it for. Elixir got the Phoenix web framework. Initially it was just a very responsive web framework. Especially as it pre-compiled templates in a nice way so they render really fast and it was already quick to respond due to Erlang. Even under pressure, latencies would be good. Great stuff but nothing shocking.

One of the early bigger things from that was Phoenix Channels which is a bit of abstraction on top of Web Sockets. This allowed a Single-Page Application or similar to talk to a server that could keep state, that could wrangle messaging, database communication and also was good at broadcasting to other connected clients in a really simple way. Importantly, it didn’t suck. A lot of them did back then. The latency was great, it scaled well. See The Road to 2 Million Websocket Connections in Phoenix for more from that era. This was in 2015.

The next big thing has been LiveView. Which came out with a v0.1.0 in 2019. This is again where that latency being a user experience thing comes in to play. Phoenix LiveView allows you to write Elixir instead of JavaScript frontend code to a very large extent. Components, interactivity and all you really want from a frontend framework. You only write Elixir and your state lives on one side of the connection. The server. This obviates the need for writing API and contract code.

It does require that connection and to be a great experience it requires that latency stays low. You can draw a straight line between this need for low latency and the fact that the creator of LiveView, Chris McCord, works at Fly.io who famously want to move your application closer to your user. That aside. LiveView lets you do a lot with a little. It has been copied by most major web framework ecosystems at this point. What they can’t copy is the BEAM runtime and as such they can’t quite get the same deal. Python and Ruby are both fighting to sidestep their GIL problems. Node.js is very sensitive to blocking the event loop and tanking latency. Livewire for PHP decided it didn’t want to bother holding state and consequently can’t do quite the same things as well.

LiveView is not a panacea. Not a silver bullet. Not the solution for every problem.

In that way LiveView is very much Erlang. It solves a solid set of problems people actually have with abstractions that make those problems trivial to work with. It seems unconcerned with a theoretical ideal and gets on with doing the work.

And it is quite quick about it.

Have any latency horror stories to share? Notes, questions or concerns about this thing I wrote? Feel free to reach out through @lawik@hachyderm.io or on email lars@underjord.io. Have a good one.

This is another piece of my series on “Unpacking Elixir”. The previously published part was on concurrency.

My background is Python so I wasn’t familiar with Ruby before-hand and ran into all these Ruby-isms with some confusion. Overall it was still a pretty smooth ride for me. I had done PHP, Javascript, Python, some C/C++, C#. It is a slightly new style but it didn’t scare me.

The thing that many people will probably find unnecessary coming from C-style languages or find very verbose coming from Python is how a block is defined:

do
  # block contents go here
end

if foo? do
	# true
else
    # false
end

def my_function(arg1) do
  # function body
end

It is fair to note that end has a lot more characters than } but I do think it comes across as more human. A less dense and symbol-filled syntax could be argued to be more approachable and potentially less noisy. I don’t particularly care either way.

Let’s write a basic module:

defmodule MyModule.SampleThing do
	@moduledoc """
	This is module documentation. Typically written as Markdown.
	"""
	
	# compile-time attribute, define-style
	@my_attribute 1000
	
	@doc """
	This is function documentation.
	"""
	def public_function(arg1, arg2) do
	  # do thing
	  new_arg = arg1 + arg2
	  # return the last thing done
	  new_arg + 2
	end

    # private function with an optional default argument
	defp private_function(arg1, arg2 \\ 5) do
	  # do something
	end

	defp short_function(x), do: x + 1
	defp no_args, do: 1
end

Modules are Pascal case. Functions, module attributes and bindings of values (variable names) are snake_case. Docstrings are multi-line text strings and also compile-time attributes. ExDoc can make very nice documentation with these. Both types of docs can also contain doctests which are a nice way of making simple tests, testing your sample code in the docs and encouraging code samples in the documentation.

Any do/end block can be replaced with , do: for one-liners. If a function takes no args you can skip the parentheses. Functions return the value of the last expression. There is no early return without branching and you do not make the return explicit.

Quickly some values and types:

integer = 1
float = 1.0
boolean = true # true and false are both atoms
null_value = nil # nil is also an atom
atom = :foo # aside from true/false/nil you reference atoms with a colon :
string = "lawik"
binary = <<108, 97, 119, 105, 107>> # equivalent to string above

tuple = {:ok, 5} # multiple values grouped, not limited to two
list = [1, 2, 3, 4, 5]
map_of_strings = %{"username" => "lawik", "site" => "underjord.io"}
map_of_atoms = %{username: "lawik", site: "underjord.io"}
map_of_atoms_without_sugar = %{:username => "lawik", :site => "underjord.io"} # Removing the syntactic sugar for colons
struct = %MyStruct{username: "lawik", site: "underjord.io"} # very special map

# erlang compatibility
keyword_list = [username: "lawik", site: "underjord.io"] # the old-school Erlang map, but syntatically sugared
same_but_in_lists_and_tuples = [{:username, "lawik"}, {:site, "underjord.io"}] # free from syntactic sugar
charlist = 'lawik' # the old-school Erlang string
charlist_in_lists_and_numbers = [108, 97, 119, 105, 107]

Interesting sugar around these types.

Strings are binaries that fit within the constraints of a UTF-8-encoded binary. String and binary concatenation is "foo" <> "bar". String interpolation is "Hello #{name} and welcome." where name is a value or expression that converts to a string through a particular Protocol.

Lists have a syntax convenience for prepending. new_list = [ new_value | old_list ] will do it. It will return when we talk pattern matching. Lists are implemented as linked lists (not ideal, it is known) so appending is much less performant. List concatenation is a ++ b . Most operations are found in either the List or Enum modules.

Maps have a syntax convenience for updating an existing field. new_map = %{ old_map | atom_key: new_value} makes for a pretty simple process. Also works for structs. Overall map operations live in the Map and Enum modules.

Keyword lists are very common in Erlang for options. Elixir added some nice sugar for that purpose. A list of tuples can be created with [key1: 5, key2: 6] and when calling a function which takes options as a Keyword list as the last argument you can drop into what feels like a python kwargs situation: my_function("regular value", force: true, timeout: 553) No double splat to speak of though.

Pipes are kind of neat:

defmodule MyModule.Pipage do
  def new do
    %{}
  end

  def add_defaults(thing) do
    thing
    |> Map.put(:timeout, 5000)
    |> Map.put(:weather, "rainy")
  end

  def set_name(thing, name) do
    Map.put(thing, :name, name)
  end

  def do_all_of_it do
    new()
    |> add_defaults()
    |> set_name("lawik")
  end
end

So pipes allow passing a value through and changing it. They don’t deal with failure or anything like that but in functional life there are many cases where this just becomes much more readable. Not that there is only one type of pipe |> and it feeds the output into the first argument of the next function. This worked nicely for my brain. It will upset many established FP folks that want all kind of pipes and arrows. I also know there is some history with piping to the last position and I know Elm does it that way. I think this is more approachable but it might be the Python self OOP bleeding through rather than what humans expect as a general rule.

Let’s make an example that is more scripting-oriented and uses the Elixir standard library.

"~"
|> Path.expand()
|> Path.join(".config/my_app")
|> File.mkdir_p!()
|> File.ls!()

This shows one of two interesting conventions for Elixir function names. The ! at the end of a function indicates a function that will raise an error on failure. Usually they have a sibling function without the bang that will return a tuple of {:ok, result} or {:error, reason}. The other one is ? at the end of a function to indicate it typically provides a boolean result. This is not enforced, it is only a convention.

Why these result tuples? We don’t have static typing and Result types, monads or what have you in Elixir. The ok/error tuple is a convention from Erlang and is essentially an informal Result type. Elixir didn’t upend the convention which means interop with Erlang code feels normal.

The way they are generally used is with pattern matching. To extract the value and throw a MatchError if it does not you can use a bare pattern match inline in a function.

{:ok, status} = File.stat(my_path)

Or you can do a case statement:

case File.stat(my_path) do
  {:ok, %{type: :directory}} -> # do thing
  {:ok, _unused} -> # do other thing
  {:error, :enoent} -> # specific error
  _ -> # any other result
end

You can also do nice things using the if macro, the with statement and a bunch of other things. I won’t attempt to cover all the syntax. The for macro comprehension thing is wildly feature-filled. One very nice place to use pattern matching is in function heads. Elixir supports overloading of a sort.

defmodule MyApp.MyModule do
	def eat_value(nil) do
		{:error, :hates_nil}
	end

	def eat_value(%{style: :tasty}), do: {:ok, :tasty}
	def eat_value(%{flavor: any_flavor}), do: {:ok, any_flavor}

	def eat_value(%{} = any_map_or_struct) do
		{:ok, :tasted_fine}
	end

	def eat_value([first_value, | _rest_of_list]) do
		{:ok, first_value}
	end

	def eat_value(num) when is_integer(num) or is_float(num) do
		{:ok, "I will consider it a #{num}."}
	end

    # Convention dictates _ to ignore a value in a match
	def eat_value(_), do: {:ok, :nothing_special}
end

This fundamentally compiles down to a case statement in a single function and if none of the cases match it will raise a specific error. You can also apply guards for numeric ranges and such. This example has matching for; Specific value nil. Value of a key in a map. Value of a key in a map to a binding and using it later. Head of a list.

The with is_integer(.. stuff is a guard clause. It also shows the convention of _ marking unused bindings, either with a name like _bla or just the plain _.

Anonymous/lambda functions are an important thing in most functional programming and we certainly have those.

# We can bind an existing function to a value with this syntax
referenced_function = &File.stat/1

# Anonymous function, full syntax
anon = fn arg1, arg2 ->
	arg1 + arg2
end

# Anonymous function, short syntax
anon = & &1 + &2

# Anonymous function, multiple clauses
anon = fn
  {:ok, result} -> result + 5
  {:error, reason} -> {:error, :bad, reason}
  other -> other
end

# Calling an anonymous function
anon.(my_first_arg)

These are commonly used in pipelines for things like the very important Enum module:

File.ls!()
|> Enum.map(fn filename ->
  Path.join(my_path, filename)
end)

Something that is not Elixir syntax so much as something you run into using libraries within Elixir are macros and DSLs (domain-specific-languages) built from macros. Elixir does not allow a ton of hiding things in normal code. The code is explicit and mostly goes in a straight line. To avoid some things becoming very unwieldy, such as the Phoenix web framework’s routing and the Ecto database library’s schema/migration definitions they have built small sets of macros that generate the relevant functions for you. This is one of few places where I find things often feel a bit woo-woo and magical. But it is almost exlusively where the alternative would be worse, either for a technical reason or for a human reason.

So when you run into things that don’t feel like normal Elixir code. Odds are you are dealing with macros. They can define custom blocks and they can define custom keywords that are mostly like functions but actually expanded at compile-time.

Example of an Ecto database schema:

defmodule MyApp.MyUser do
	use Ecto.Schema

	schema "users" do
		field :username, :string
		field :password, :string, redact: true
		timestamps()
	end
end

This generates a model with everything the database layer needs to know about the schema. Your MyApp.MyUser module will have a __schema__() function. use Ecto.Schema is what brings that magic into your module and your subsequent calls to the macro schema, field and timestamps do the rest.

This brings us to one thing that is a bit messy but learnable. alias, import, require and use. This is also covered in the official guide which is where I learned Elixir mostly.

• alias shortens the name of a module or lets you change the name you use for a module.
• require makes macros available from a particular function.
• import brings in functions AND macros.
• use lets the specific module inject any code functions through macro madness.

Erlang interop deserves a mention. Both Elixir and Erlang module and function names are atoms. MyApp.MyUser is actually syntactic sugar for :Elixir.MyApp.MyUser which is just an atom. So Erlang interop looks like this:

:erlang_module_name.function_name(arg1, arg2)
# OR actual example
im = :egd.create(200, 200)

This is already a bunch so I will end it. I have not covered all syntax in the language but I think I’ve given it a fair shake. I hope this is beneficial to you if you are curious to try the language or figure out if it is something you might enjoy.

If you have questions, comments or concerns you can reach me by my email at lars@underjord.io or on the socials @lawik@hachyderm.io.

My origin story is steeped in PHP. In PHP the concurrency model is pretty much whatever your web server (mod_php) or app server (php-fpm?) decides it should be. PHP generally assumes a shared-nothing model and each request is clear of whatever work is done in the others. In this regard work can run concurrently but they can also not share anything without reaching for external files, databases, queues or similar.

Then I went to Python. Over in Python .. it depends. It is really bad at concurrency. Ordinary threading is available but enormously constrained by the GIL (Global Interpreter Lock). The best solution seems to be just running more interpreters or using Python multiprocess which I believe forks new processes. I liked Python. I didn’t enjoy trying to do performant concurrency in Python though.

Elixir builds on Erlang. Fundamentally everything I’m writing about here is thanks to the Erlang virtual machine, aka. the BEAM, which has this stuff at its core. I will write in the context of Elixir. The things that are great here are also great about Erlang. Except possibly the Task module, which is kind of neat. That could also be done with Erlang but I don’t know if anyone has.

The BEAM is made to run a ton of lightweight processes. A type of green thread if you will. They take very little time to spawn, they take very little memory. A system can have millions of them. And if you need many millions of them you can bump the default limit. They are fundamentally a different thing from OS processes and threads. The BEAM itself is scheduled by the OS but on startup the BEAM creates threads for its schedulers. One per available CPU core by default. These in turn take care of scheduling processes. This is the source of concurrency and parallelism.

We won’t go into detail on the Actor model and how that works here but that is built on top of the fundamental primitive of a process. A process runs a function, the function only has access to its own local variables and private copies of whatever arguments were passed in. This is Functional Programming so everything is immutable. A process cannot change the values in another function. This function running in a process has two cool special bits of API it can use. It can receive messages and it can send messages. If the function runs to completion the Process goes bye-bye. If the process runs in an eternal loop checking for messages and only acts if a message comes in? Then we end up talking about the Actor model again which I said we won’t do. So lets move on.

A process runs on one of the schedulers. If many processes taking a long time creates a queue on one scheduler the other schedulers can start stealing work to balance it out. If a particular process goes through a lot of function calls (internally called reductions), say 4000 for example, without completing then the scheduler will pre-empt it. That means freeze it, mid-run, and send it back to the end of the work queue. This is called pre-emptive multitasking and it is what Linux does to the OS processes. In the BEAM nothing gets to hold up the line for very long. This is how it ensures low latency which is another topic I hope to unpack at a different time.

The immutable FP nature of Elixir and Erlang are pragmatic choices that were made to achieve a bunch of goals. It predates multi-core machines by a fair bit and they were targeting a different kind of concurrency, distributed systems made up of multiple machines, a cluster. The end result was accidentally convenient for achieving incredibly concurrency and parallelism on multi-core machines. Processes do not share state, they can only communicate asynchronously through message passing. This ensures it is safe to run them at the same time. Same time as in concurrent, executing the tasks during the same period of time. Also parallel, executing the tasks at the same time. All APIs that touch files or other constrained resources, stdin/stdout and such, have mechanisms for ensuring that contention is under control. The Erlang folks have spent a lot of time getting this right.

When I write my code I generally don’t need to care about this very much. Most web requests take place inside a single process. They might communicate on through to a database connection pool which involves more processes but overall I don’t need to care about the underlying primitives. I also don’t need to think about the Node.js way of racing to IO. Node.js can be highly concurrent. CPU-bound work is fine on the BEAM. It won’t disrupt the ability of other processes to get work done, both because multiple cores can do the work but also because if I spend many CPU cycles I will be pre-empted and other work can slip by. This is all handled by the web server I use, more than by me.

If I do want to do concurrent work there is the usual async/await mechanisms, implemented in the Task module. Not as weird as in JS. They just return a Task reference which is a helpful abstraction on top of the Process ID or PID. Then you can await on it to get a result or await on multiple to get multiple results. There is also a wicked function called Task.async_stream which will take an enumerable (list, map, stream or similar) and run a Task for each entry as a lazy stream. By default it has a concurrency matching the number of available cores (just like the schedulers). This is essentially a fancy shortcut for using all your machine has to offer in the service of getting the work done for tasks that are embarrassingly concurrent. It is very fun to use when bodging together scripts that you want to go fast.

Compared to PHP I have many more tools for interacting across the work that happens in my system. Process registries and groups, PubSub mechanisms for message passing, memory-based shared state accessed through safe mechanisms (ETS for example).

Compared to Python I can do multiple things at once in simple and useful ways. In a language that is as expressive and as dynamic (in the places I care about).

Compared to Javascript I can do multiple things at once easily without worrying about CPU-bound work holding up other requests and such. I get actual immutability instead of working really hard to pretend I do.

These comparisons are of course based on my understandings and experiences with the languages. I’ve stuck with languages I have some experience with but I don’t hang out with them on a regular basis currently. If I have misrepresented any of them, do let me know.

In summary; great options for concurrency and parallelism are not an afterthought in Elixir and Erlang. They are built into the very foundation of the runtime and the languages surface it to us. This matters for performance and practicality. Doing concurrency on the BEAM is not punishing, it is not risky. It is just what you end up doing by using it.

If you have questions, comments or concerns you can reach me by my email at lars@underjord.io or on the socials @lawik@hachyderm.io.

Except that I work in Elixir. Elixir is built on Erlang. I like to think Erlang being FP is mostly an accidental side-effect of trying to solve a bunch of complex requirements around distributed systems, fault tolerance and consistently low latency (this predates the concurrency story, another kind-of-accident). Elixir and Erlang are decidedly loose in structure.

I get the sense that they appeal to some specific flavors of FP enthusiasts only because they are FP and have undeniable practical applications. Accepted due to being (relatively, practically) successful and interesting. In spite of not being quite strict, rigorous or firm enough to give a true hard FP lover the rigidity they truly desire. Moving on from that imagery. Suffice to say Elixir is not pure.

You can write parts of your application that are generally pure and that has great advantages. However the language and plattform also offers a ton of tools to work differently with regards to state and mutability.

There are wild side effects for logging, IO, processes, message passing and many other things. It is a very practical language. Often we also need mutable state.

Surely not!?

Yes. Quite.

ETS

I would say that this is the blessed path to mutability in Erlang-land. If you want to store something in memory and you have to be able to read and write to it kind of ad-hoc. ETS can help you out. It is a bit of swiss army knife all told and has a bunch of different types of tables (bags, duplicate bags, sets and probably more, I’m going from memory).

A common misstep in Elixir and Erlang is to implement an in-memory store as a GenServer. This can be the right move if serial access is important, if what you need is a bottle-neck where you can do something important. But in many cases what you would probably rather have is an ETS table and you might enable options for improved read concurrency or improved write concurrency.

ETS is a super useful tool. It is also very much a way to get mutable local or global state.

I will gloss over DETS because that’s IO (ETS on file) and I will skip Mnesia because it is a whole-ass database.

Persistent Term

Have some global data that rarely, but occasionally, changes? Persistent term. You get really great read performance. Updating or deleting a term gets really expensive.

A very pragmatic solution for an in-memory store.

Process dictionary

I guess this is the Erlang process equivalent of something being thread-local. It is a key value mechanism that lives and dies with your process. Actually we had Mat Trudel of Bandit fame on BEAM Radio just now so there will be an episode soon where he outs his dirty, filthy use of this local mutable data in his fabulous web server.

These are also used for dynamic repos in Ecto if I’m not mistaken which can be kind of surprising. I believe they are also used for the Ecto sandbox which means it can be kind of funny crossing process boundaries. Depends on your options and what you need.

Very impure, very practical. I find myself reaching for it when I need to be able to switch some slight behavior for testability or similar and don’t necessarily have a good way to pass config around. Crimes.

Application environment

Now we are getting worse. An Erlang application is something like a library or particular app project. Each app can store configuration and assorted dodgy bogus in their own ”environment”.

It is intended for reading but of course you can write to it. I couldn’t tell you what the performance characteristics are because the act is a crime and we shouldn’t benchmark crimes. If you do benchmark the crime, let me know.

I have done this in Livebook where it is a viable way of setting up configuration during the setup block. I have also done it in relation to particular tests at times.

System environment

Environment variables are great. But we usually treat them like constants. No more! You can trivially change environment variables and suddenly you have an awesome KV store that you should absolutely not use in this way. Rad.

Update!

Karl-Johan Nilsson immediately mentioned counters and atomics after I posted this. Shows what I know.

—-

I think that covers the mutable states I know of that makes Elixir and Erlang practically capable but ever so impure. If I have horribly misrepresented something or smeared your favorite FP language, no worries. You can complain at lars@underjord.io, berate me on Mastodon via @lawik@fosstodon.org or even like, and subscribe my latest video.

Overall I would recommend scripting in whatever language you are most comfortable in that is at least reasonably comfortable for scripting. A quick script benefits from a low barrier between thought and execution. Use what makes sense for you. I find Elixir surprisingly good as a scripting language ever since the introduction of Mix.install.

Elixir actually answers a question that Python, to my knowledge does not, in this regard. While you do need the language and its runtime installed, how about dependencies? I usually globally installed requests in Python, maybe an AWS SDK library, because I’d need them eventually.

In Elixir each script can briefly define dependencies.

#!/usr/bin/env elixir

Mix.install([:req, :jason])

"https://api.github.com/users/lawik"
|> Req.get!()
|> IO.inspect()

The Mix.install part installs the top two things I need. An easy-to-use HTTP client (Erlang httpc works but is kinda clunky) and a JSON library. I could similarly pull in CSV-parsers or what-have-you. In Python you have JSON and CSV built in but you generally do want a better HTTP client.

I have occasionally found Elixir to be a bit less convenient for doing nice sloppy dictionaries as mutable state is less conveniently available and if I’m parsing a bunch of files or larger structures and want to build up convenient key/values for them I often want to exfiltrate some key and value out of multiple nested loops. This usually leads to Enum.reduce in Elixir and it is less convenient. It totally works though. And if you want to be really gnarly you can do Process.put/2 and Process.get/2. I bet you could do it reasonably comfortable with an ETS table but I haven’t really used those while scripting. Yet.

Let’s compare.

#/usr/bin/env python3
index = {}

with f in open("file.json"):
	for item in json.loads(f):
		for subitem in item["children"]:
			index[subitem["id"]] = subitem

Versus

#/usr/bin/env elixir

"file.json"
|> File.read!()
|> Jason.decode!()
|> Enum.reduce(%{}, fn item, index ->
    Enum.reduce(item["children"], index, fn subitem, index ->
		Map.put(index, subitem["id"], subitem)
    end)
end)

It can get a bit involved in nested cases. Especially if you need to operate on multiple data structures and such. Lacking mutability means paying more attention to data structures and how they flow through functions.

Overall though I’ve also found that whenever I do something slightly time-consuming I can optimize it significantly by using a single Task.async_stream/3 which will effectively utilize all my cores as it works through whatever list I’m processing. Getting this equivalent thing going in Python is significantly more painful.

#!/usr/bin/env elixir  
  
Mix.install([:req, :jason])  
  
"https://api.github.com/users/lawik/repos"  
|> Req.get!()  
|> then(& &1.body)
# And now we get very parallel
|> Task.async_stream(fn %{"stargazers_url" => url} ->  
 url  
 |> Req.get!()  
 |> then(& &1.body)  
end)  
|> Enum.flat_map(fn result ->  
 case result do  
   {:ok, stargazers} ->  
     stargazers  
   err ->  
     IO.inspect(err, label: "Error fetching stargazers")  
     []  
 end  
end)  
|> Enum.uniq()  
|> Enum.count()  
|> IO.inspect()

When writing out this script and trying it I hit the Github API limit for unauthenticated calls on like my third or forth run :)

I also like that most of the Elixir syntax ends up being basic function calls. Instead of with f in open to open a file I’ll call the File module and the read! function. The exclamation point (or bang) is a special convention. In most cases there are matching functions without that symbol and those generally return an {:ok, content} tuple or an error tuple on failure. The bang functions will raise an error on failure. These bang functions make for very convenient pipelines. If you need to operate on the failure case it is generally better to use the regular one and a case statement.

This is very Erlang “let it crash” but applied in a different context. By using them in a script we are saying that any failure to exeute the function has no reasonable mitigation or that we don’t care to spend time defending against small deviations from expectation. Handling all conceivable possible outcomes of attempting to read a file is generally a waste of time and effort in a script that requires the file to exist. Just error out if it doesn’t.

First run can be a bit slow if the Mix.install needs to pull down and compile the packages (just as the initial pip install would) but subsequent runs will be reasonably snappy.

If you want more examples of doing scripting in Elixir I suggest looking at Wojtek Mach’s Mix Install Examples repo. It has a lot of things. Wallaby for driving a browser. A single-file web server with Phoenix LiveView. Or why not write a NIF in C. Lots of fun stuff in there.

This is intentionally a shorter post. Hope you enjoyed it and it made you a bit curious. If you have thoughts, questions or anything of the sort you can reach me at lars@underjord.io or as @lawik@fosstodon.org on the Fediverse.

Where I do appreciate it is in very utilitarian things. Whisper is a good example. Live captions and audio-to-text-transcription are both quite labor-intensive. Considering how much audio is being produced in an ongoing fashion there is no chance to make that more accessible and searchable, by hand. Here this model can clearly be a good thing. It can also be used for ill but that’s mostly the nature of practical tools. Similarly I’ve worked with a startup that processes invoices, receipts and such with ML models to turn the structured but very inconsistent human de facto standard of emailing PDFs to each other into structured data. Understanding and more recently producing data in messy formats, that’s where ML seems to beat writing code.

I am genuinely excited that with the Nx project and it’s descendants this capability is available in Elixir. The Bumblebee project really drives home that you can use this now. I like Elixir and want to see it become more and more capable. For many simpler models it may now be easy enough to just load the model and do inference in your Elixir application instead of needing to offload it elsewhere. And also, without needing to rely on some other party offering an API.

Brief aside, I like it when I can run things myself. It is a good reassuring option to have even when you do rely on a managed provider. It can also be quite the requirement here in the EU with the GDPR and Schrems I-III tearing through what can and can’t be done with data. You can often get more polished results with proprietary services but they are not a cure-all and if you rely heavily on them your are taking on a lot of risk.

Is ML in Elixir really a good idea?

Not in Elixir itself. No. Probably not. But Nx really doesn’t just execute models in Elixir. It is a bit more involved, we’ll get there. For now, let’s talk about Python.

Python is the default for Machine Learning projects. It brings some good stuff to the table.

• High level language, expressive and productive, few low-level concerns
• Approachable language, friendly syntax and not heavy on syntax or tricky concepts
• Ubiquitous, it is everywhere
• Fun to write expressive libraries in

Doing ML in Python also has some drawbacks. Most of these are general to Python rather than specific to ML and also recur in similar languages like Ruby. I rather like Python, wrote it professionally for a number of years. That is to say, there is some love for the language while I’m writing about these problems.

• Efficiency. It can be dog slow and doing any kind of work in Python is not the most efficient.
• Concurrency. It is terrible for utilizing multiple cores.
• Raw performance. Crunching numbers in Python is not a good idea.
• Distributed computing. No aspect of Python makes it particularly suited to distributed computing. Few languages and runtimes are.
• Managing complexity. Python applications can get pretty messy and allow some really poor choices to be made.

Some of these are related to the language, most are related to the runtime and how it behaves. There are established solutions for many of these problems. You’ll see libraries such as numpy that do number-crunching and they drop into C/C++ to do this efficiently. There are also efforts to unlock the performance problems of the runtime in various ways.

Both concurrency and distribution are other areas where Python either drops into additional infrastructure tooling and/or C/C++ to achieve good results. Ray seems like a popular solution and it actually ends up pretty close to the Actor model offered by Erlang, along with distribution and serving concepts similar to what Nx offers.

What does Elixir offer?

Elixir covers similar ground to Python in general. It is a high-level language, it is dynamically typed, it is expressive and friendly. The runtimes are however, very different.

Due to the Erlang virtual machine (the BEAM) Elixir can offer fantastic concurrency and consistently low latency. Erlang was built for soft realtime, distributed systems, fault tolerance and scale.

Elixir inherits and expands on a fundamentally high level design of a Functional Programming style language with the Actor model built in. This limits and clarifies complexity. If you want or need complexity you generally need to more explicitly opt in to it. You can build some gnarly things. But if you write the code that gives the least resistance you’ll generally be writing pure functions operating on immutable data.

It is really bad at number crunching. The VM has recently gotten a JIT compiler which lets it be a fair bit better at certain raw number crunching tasks but overall it is not what it is for. This is where Nx, Axon and friends step in and provide a functional Elixir API for building up your computational graphs in a high-level language that will then be executed using XLA on either CPU or GPU as highly optimized code. Erlang has always offered NIFs and a few other escape hatches to do demanding work in native code. This is a purpose-built abstraction on top of that capability with ML as the focus and goal.

Your application will already be better at concurrency and perform more consistently at runtime under the BEAM VM. When you want to start distributing and orchestrating the workloads you should find that it is trivial compared to Python. You don’t necessarily need to bring in any particular tooling for it or add infrastructure like message queues. Elixir is already very good at orchestrating work.

Livebook. A tool for interactive, easily reproducible, real-time collaborative code notebooks. Jupyter Notebooks have all sorts of weird drawbacks and are still a great tool. Livebook is a much cleaner design, has features that are unlikely to happen in Jupyter and allows that nice and easy workflow for prototyping and exploring. You should try it out if you want to poke around with Elixir, with or without ML. Easiest way to try Elixir code.

The VM is built for runtime

Erlang is incredibly powerful in terms of observability. You can inspect your running system in depth and Elixir fully inherits this capability. You can work with the running system in a manner that is incredibly unusual if not entirely unique. Start and stop processes, find and identify bottlenecks and build your system for high levels of fault tolerance.

This will also translate to the ML work. Parts will be harder to introspect as they happen on GPUs or in native code but overall the way work flows through your application or cluster will be observable.

What is not there yet?

Tons of things are potentially missing depending on what you want to do. It is still early days but there is also a lot of activity.

There are many cool models already implemented and drop-in-ready for use in the Bumblebee project. There are already multiple backends for Nx providing options for edge inference (TFLite), alternate general backends (TorchX I believe), mature XLA support and more is coming. I also look forward to the OpenXLA efforts that seem to increasingly support Apple systems with Metal and all that. Those are in progress for Nx as well.

Importing models straight from Python has been a mixed bag from what I understand but I think recently announced Ortex broadens the possibilities significantly for ONNX-packaged models.

A lot vision models are usable through the OpenCV bindings project evision as well.

Again, why?

Python was not chosen for Machine Learning. It was an accident of academia that Python became the tool. Some was its merits as a prototyping tool, certainly. But as ML grew it mostly remained due to inertia. People built libraries to cover the deficiencies and so it grew. While I don’t mind Python I don’t think it is a particularly good tool for this job. It struggles with a number of performance problems in production environments. Every new need in ML necessitates new libraries because you cannot tackle it inside of regular Python.

Elixir has powerful abstractions for distributing work, performing well concurrently and scaling far and wide. Out of the box. And then the performant ML stuff is built as additional libraries that hook into these abstractions cleanly. The focus on immutable data structures and functional programming reduces complexity and avoids the code becoming a big ball of mud while the Actor model provides the tools to build architecturally sound and scalable systems without necessarily bringing in a ton of additional tools and infrastructure.

It is ridiculous how easily I can swing together a Phoenix web app and add some ML smarts to it. José showed some of this in his Bumblebee launch demo. I genuinely think Elixir provides strong advantages for companies that have a reason to do ML. It also puts basic ML within reach of the random web dev builder type who just wants a little special something for an app.

Examples:

• User profile photo? Crop it to the detected face or remove the background and have a transparent cutout to play with. Or turn it into a sketch, a comic style portrait or whatever. U2net has many applications.
• Text being written? Use text sentiment analysis to get some understanding of what is being communicated in your app and adapt UI accordingly.
• Audio in use? Transcriptions via Whisper are quite straightforward to achieve.

Whether you agree or disagree you can reach me over email at lars@underjord.io or you can poke me on Mastodon where I am @lawik@fosstodon.org.

The last ElixirConf I visited was Prague in 2019. Pre-pandemic. I had just gotten into Elixir and kind of decided that I should actually really engage and try to do the whole community thing with this language as it seemed like it had a community of a comprehendable size. I traveled there solo. I was mostly doing Nerves and playing around. I had a low-effort, high-yield client at that moment and plenty of time to experiment. I came home from that conference with a lot of excitement and I started this blog with some of the stuff I saw there.

Between now and then I’ve done a few Code BEAM conferences. They are cool events. But they are different. The events have distinct characters and overall I find ElixirConfs have so far hewn closer to what I want. That said, I plan on hitting as many Code BEAM events as I can.

I missed ElixirConf in London as it was rescheduled to be right at the time of the birth of my second child. I stayed home. He arrived on time. Worth.

This preamble was for context. I had a lot of built up connection with people in this community and this would be the first time I’d revisit this dedicated physical gathering since I really stepped into the community. This would be the first time I met a focused group of the Elixir community since I started my blog, newsletter, YouTube channel and the BEAM Radio podcast. I was very excited and had no idea what to expect.

I also submitted a talk and got it accepted. My first in-person conference talk. Hnnnngh…

Going to a conference and leaving your partner behind with children ages 1 and 3 is not a small ask. At least not with our children. So I didn’t. We invited the in-laws so my wife had some company and brought the whole family to Portugal. We are fortunate enough that it was feasible to do so as a vacation. While I was at the conference they were at some castle. I got lovely pictures of my daughter being a statue. While I was making poor decisions she had help putting them to bed. This was a good move all in all.

We also planned the trip with days before and after for shared vacation time.

Travel went well, kids were good, hotel was nice, Lisbon was lovely.

Since we were there early I also went to the Elixir Meetup two days before the conference. It was in the Bounce office. I encountered the first person who knew me from my online activities. A reader of the Nerves Newsletter specifically. Kinda cool :)

People gave two neat presentations at the meetup. The first one by David Elias about additional concepts you can apply on top of “contexts” and the second one about using Doom for visualizing supervision strategies and shooting your processes. Fun stuff I thought. Then we headed out for drinks and pizza. Some of the folks we headed out with apparently knew who I was so I handed out my first few stickers. They knew Maqbool which is apparently an incredibly common trait. Everyone knows him.

During pizza I was dubbed to be The Lars by André Albuquerque who gave the Doom talk. Apologies to danish Lars of the Netherlands who also spoke at the conference. André was apparently familiar with what I do as well. More stickers were shared. Then we sat around at a kiosk in a park, talking tech and drinking wine until it was quite late. Not a bad first run-in with both the locals and the Elixir community.

Day before the conference I had been invited to the Speaker Dinner in the evening. A very nice gathering of very enthusiastic Elixir people. This place had an incredible density of “oh, we’ve spoken, I had no idea you had a face”. Apparently I’m very tall. Or so people kept telling me (it’s fine, I don’t mind, interesting that it doesn’t translate in video). The food was good but comparatively forgettable when all around you are people you have so much shared mindspace with. Like talking to fish in a barrel.

I particularly ended up next to this figure of the name Benjamin Milde. Oh! lostkobrakai, everyone knows that guy! Then I also got proper chances to talk to Jeffrey Mathias (of Elixir testing fame), Matt Trudel (of Bandit fame) and Eric Meadows-Jönsson (who is apparently local to Gothenburg, I’d missed that). Perhaps least well known to me and the most novel conversation was with Guillaume Duboc. He is the guy doing the research project that will attempt to bring types to Elixir. Now I don’t particularly want types for Elixir, I don’t hate the idea but I don’t feel the need either. He seems incredibly enthusiastic, thoughtful and sincere. Great impression. I’m happy that whoever explores that tricky territory has his mindset.

I also found out before then that his talk. The biggest talk of the conference. The most important and probably contentious talk.. was concurrent with mine. Damnit. And now I knew my arch-nemesis for the conference was nice and friendly. Aagh!

Mornings and evenings I rehearsed my talk so that the few souls that couldn’t fit in that room with the types talk would hopefully enjoy it. Tears in my eyes. (no, not really)

My team trickled in throughout the days before the conference. I brought the whole Elixir gang.

For this conference I had a plan to try and record people and do some interview kind of stuff. I capture some and I might do something with it. Yes, I have the organizers permission and the permission of everyone who was a direct subject of my filming. But I learned a few things. It is a pain to try and do socializing, going to talks and filming at the same time. It is two things too many. Also the camera rig is a bit annoying. Also, people think you are crew at the event. Also, my gimbal broke? That’s under investigation. Second day I did not bring the big camera.

Socializing went fine. I skipped quite a few talks for the benefit of the hallway track. This was also where it became noticeable that people knew who I was. I am not a person who has had any degree of “fame” before. I generally deal okay with attention, I’m not particularly shy or very reserved usually so I wasn’t worried about it. It was all in all just fun. People were really nice.

What I wasn’t ready for and I don’t think I managed to respond fully to is the people who said what I do helps them. That makes me a little bit too happy and I can’t fully absorb it. Small things, big things. People had such nice things to say about what I have shared, made and written that it just makes me blush. They all got stickers, that’s all I had. Some asked for advice as well. The stickers are probably better quality than the advice but I tried at least. I hope I didn’t forget to give anyone who spoke to me about the stuff I do stickers. I probably did. It was occasionally hectic.

If you came up and talked to me, thank you. If I failed to respond or was busy when you tried. I’m sorry, don’t hesitate to reach out again.

Meeting people from the community that I know or know of was the big thing I had in mind for this conference. I also hoped that some people would recognize my work. I put things out for people to enjoy and it is nice to get some signal that isn’t just a view counter. Being a person people know of was interesting and new. I hope I did it gracefully.

The most memorable talk was certainly Evadne Wu’s “Not Hot Dog Revisited”. It was actually kind of close in content to my talk except very different in details and execution. But Membrane, LiveView and ML were all in there. When the organizers ask what you need for your talk it is a boss move to go: “Three different hotdogs.”

I gave my talk during this first day. Final talk of the day before lightning talks. I think I may have gotten about a hundred of the six hundred physically attending audience. Damn you Guillaume!

I spoke on LiveView with Membrane and actually mostly on creation and creativity in code. The slides and code is all available here. There should eventually be a recording published. It is a heavy revision of the talk I gave to Elixir Community Krakow earlier. The nervousness was under control, the track host was lovely, my friends (big shoutout to hypeman Marcel Fahle!) and team showed up and encouraged me. The ML model behaved kinda well, and it mostly failed in fun ways, I remembered most of the talk, I said most of the things, people laughed when it was appropriate. All in all. I feel like the talk landed. I’m happy with how it went.

Because of who I am as a person apparently it wasn’t a talk and a distinct demo section. I made the entire talk be a demo and tried to make it kind of a performance throughout.

Afterwards people have also given me feedback about the talk that confirmed that to some people it definitely landed. I’m thrilled about that. Actually, you over there, guy with the red beard who grabbed me just when I was leaving on the last day at that craft beer place? As far as you got in what you said absolutely made my evening. Whimsy indeed.

That wrapped up the day for me. I was too twitchy after my talk to watch the lightning talks. The evening ended in ramen, rooftop open bar and night club in that order. I hit the pillow at 03:00 (that’s AM for you monsters out there).

My daughter kindly did not wake me before 06:00 and agreed to snuggle and snooze until 07:00. I was not in great shape for day two of the conference. Oof. But as a parent I don’t often get to be irresponsible. I can’t regret it really. I might not repeat it though.

Second day I’ll admit I bailed on a few talks just because my body was rebelling against my past behavior. Hangover combined with chronic stomach problems made me a bit less chipper this particular day. I still had a lot of good conversations. Including leads for work, prospective collaborations and all that good stuff.

I also left my big camera at the hotel for the second day as the gimbal was broken and I wasn’t in any shape to do a bunch of recording. I will still see if there is a seed of a video in the material I’ve captured. I’d love it if there was. But lesson for next time is that I need someone focused on taking video if I’m going to do that at another one of these. I might also do more vlog-style stuff but then I should have a lighter gimbal and use the small camera.

The final item of the conference day was Chris McCord talking through the current state of LiveView and the path to 1.0. It was a good, rapid-fire presentation of new cool stuff. Checkboxes man. It started both great and awkwardly. Having ChatGPT or GPT4 generate intros for people is not particularly my jam. Having Jim Freeze recite rap lyrics hit an aggressively awkward spot intersecting cringe and awesome. Definitely warmed up the crowd.

The day and conference wrapped on that presentation. It was a good piece of punctuation. The fact that it was a bunch of web, code and practical usage reiterated to me that ElixirConf is very much a builder community. There are bouts of theory and moments of idea stuff. But mostly people build things and share. That’s my jam so it suits me great.

The Electric SQL folks pulled together a bunch of folks at a craft beer place after. It turned into a very chill place to have conversations. I would have stayed all night if we hadn’t planned dinner with the team and a small artisanal batch of Elixir friends, old and new. Some significant others joined us. It was an absolutely delightful dinner. After it wrapped up a few of us went on for a beer and then the night wound down.

I stayed a couple more days in Lisbon with my family. No Elixir in sight. And then I went home.

I’m still digesting the experience. I’m looking forward to ElixirConf US.

Ad-style pitch for a business-thing: I help companies find competent, capable or promising Elixir developers. If you build in Elixir and want to spend less time finding great candidates, reach out.

I was first introduced to Elgato for their lighting panels, the arms that hold them up and the Camlink 4K. I was stepping up my streaming/meeting camera situation by putting my then only good camera, a Sony ZV-1, on a stand and improving the lighting.

At the time the camera didn’t do USB streaming, recent firmware allows it to do 720p.. I got the Elgato Camlink 4K instead. It takes the HDMI out from the camera and lets me treat it as a webcam via USB. It has been essentially problem-free for me under Linux continuously. It can hang but I’d blame my USB controllers more than the hardware, other hardware has also complained.

The Elgato Keylights have been nice enough as well. Hook them up to wifi via the app, a Mac or Windows. Then they are accessible to be controlled and they provide nice diffused light that does the job. As I moved from Mac to Linux I started controlling them with a Python script using leglight and also some experiments with Nerves and my own keylight library.

I’ve been eyeing the Stream Deck because all the Mac productivity podcasts I have a bad habit of following has been hyping it up. Of course. No Linux support. So I held off. Then I saw the Stream Deck Plus which has physical knobs. And then the Stream Deck Pedal which is a pedal. I am not hard to impress. Unfortunately.

No Linux support from Elgato but I’ve been feeling up for a challenge. I checked and saw existing implementations in Python, Node, etcetera and therefore I considered it doable. I’ve ported from Pyton before with good results. And if I got hard stuck it seemed like streamdeck-ui is a thing. (note: no Stream Deck Plus support at time of writing.).

The Stream Decks are hidraw (not hi draw, hid raw) devices in Linux terminology which essentially means they can be driven with a very simplistic interface and all the knowledge about the specific device is enshrined in the userspace code that talks to it. Pretty much the same thing in other OS:es. If you are using the Stream Deck software to set things up, there is literally nothing saved on the device. The software is polling the device for activity, generating and pushing new images whenever the screens need to change and such. Even the Stream Deck Plus with a neat variety of controls and interfaces does very little beside retain an image and provide input signals.

I’d like my code to work on both Linux and the occasional Mac (I still run an Apple laptop) and I don’t mind accidental Windows support. Rather than work with Elixir libraries for straight hidraw I looked for something using the library hidapi which provides cross-platform abstractions with a simple API. I found hid on Hex.pm, unfortunately the source repo is gone.

This is already much more elaborate than the Elgato Keylight. That’s a very straight-forward JSON API. This was more challenging but also a lot more interesting.

The hidapi library provides:

• hid_enumerate - list devices
• hid_open - open a device
• hid_write - write an “output report” to a device
• hid_read - read an “input report” from a device
• hid_send_feature_report - this is another type of write
• hid_get_feature_report - this is another type of read
• hid_close

There are some other I don’t care about as well providing more info about the device and such.

The NIF library I had didn’t have all of those hooked up, I believe I needed to add sending feature reports. I imagine the original author just happened to not need that. I also fixed the makefile to work on recent MacOS and actually fixed a bug or two in the C code. As someone who doesn’t write C, essentially ever, that felt like an achievement. You can see my changes here.

This stuff happened during the process of porting. I worked off of python-elgato-streamdeck as a reference implementation. Particular props to this PR for Plus support. I also checked my work against node-elgato-stream-deck a few times as I wasn’t sure the Python one had it right. Some weird build issue with dynamic shared libraries and ctypes meant I never actually got the Python one running.

The device offers the following functionality:

• 8 buttons aka. “keys” with full color LCD screens under them. Down, Up. That’s all they do.
• An LCD touchscreen strip. 800x100, full color. Can register short tap, long press and a weird kind of drag.
• 4 knobs, turns in steps, left, right, can also send multiple steps if turned quickly. Can also be pressed as a button, up/down.
• Controllable LCD brightness (it is really just one panel).

All of this is done over that USB HID interface.

I had very little trouble with the Elixir port overall. My most long-standing wtfs where around sending images to the StreamDeck and making sure they worked. I have a stream of me struggling to update the LCD where in the end I had just flipped two fields in the messages I was sending. Buttons and knobs took very little time. Compared to when I was making Inky over 3 years ago I have written Elixir pretty much daily since. It is nice to feel that the language fades away when solving problems.

Building and matching binaries could have been more challenging if it wasn’t for all the work I did with The Changelog for chapter support where I implemented an ID3v2 library for Elixir. That made me very familiar with all of the binary pattern matching and constructing binaries.

Let’s actually have some code in here:

def set_brightness(d, percent) when is_integer(percent) do
	percent = min(max(percent, 0), 100)
	payload = rightpad_bytes(<<0x03, 0x08, percent>>, 32)

	write_feature(d, payload)
end

Not too complicated. How about reading keys?

def read_key_states(d) do
	{:ok, binary} = read(d, 14)
	binary
end

Also simple. Writing a new chunk of image to the LCD is a bit more involved..

def set_lcd_image(d, x, y, width, height, binary) do
    send_lcd_image_chunk(d, binary, x, y, width, height, 0)
end

defp send_lcd_image_chunk(_, <<>>, _, _, _, _, _), do: :ok

defp send_lcd_image_chunk(d, binary, x, y, width, height, page_number) do
    if width + x > 800 do
      raise "too wide"
    end

    if height + y > 100 do
      raise "too high"
    end

    bytes_remaining = byte_size(binary)
    payload_length = @config.image.report.touchlcd_payload_length
    length = min(bytes_remaining, payload_length)

    {bytes, remainder, is_last} =
      case binary do
        <<bytes::binary-size(payload_length), remainder::binary>> ->
          {bytes, remainder, 0}

        bytes ->
          {bytes, <<>>, 1}
      end

    header =
      [
        0x02,
        0x0C,
        <<x::size(16)-unsigned-integer-little>>,
        <<y::size(16)-unsigned-integer-little>>,
        <<width::size(16)-unsigned-integer-little>>,
        <<height::size(16)-unsigned-integer-little>>,
        is_last,
        <<page_number::size(16)-unsigned-integer-little>>,
        <<length::size(16)-unsigned-integer-little>>,
        0x00
      ]
      |> IO.iodata_to_binary()

    16 = byte_size(header)

    payload = header <> bytes

    payload = rightpad_bytes(payload, @config.image.report.touchlcd_length)

    1024 = byte_size(payload)

    case write(d, payload, "set lcd image chunk") do
      {:ok, _} ->
        send_lcd_image_chunk(d, remainder, x, y, width, height, page_number + 1)

      err ->
        err
    end
  end

Still not unreasonable. Just longer.

I shared some in-progress work both in this Short and on the livestream.

The end result is the streamdex library which supports the Stream Deck Plus right now as well as the Stream Deck Pedal (literally three keys, very simple). It is pretty rough, alpha-level, as I only took it to the point where I could use it. If you have a Stream Deck and interest in Elixir, feel free to contribute your device.

As a self-taught web dev it is wild to me that I’m writing something that can be fairly called device drivers right now. I’m not claiming they are advanced or impressive. Userspace drivers like this one live on top of many layers of more complex implementation. I just know that teenager me is nodding his head right now. He is pretty proud.

I’m also very glad that someone else has done the reverse engineering. I did fire up Wireshark a couple of times, actually how I found the final swapped byte for the LCD, but I don’t want any credit for figuring out how these work.

Ever since I got it fully operational I’ve been hacking at a crude but effective project for running a bunch of my computer .. stuff. That’s also public.

So far I’ve tried:

• Lights
  • keys for on/off
  • a knob for brightness that can be pressed for on/off
  • a knob for light temperature and pressing the knob resets that to a good default
• Calendar
  • Show current and upcoming calendar events on the LCD which I mostly ripped from my calendar gadget eInk project. Simplistic display still, much more fun to be had.
• Play/Pause
  • A key that toggles and uses dotool to send the media play/pause key event to the OS. Made a quick and nasty dotool_elixir library that pulls that together and compiles the Go code.
• Mic mute
  • Both a key for toggle and a pedal for press. Sends micmute key event to the OS. does not work currently on my KDE :(
• Key icons
  • Made a small library called bs_icons which pulls the Bootstrap Icons repository and provides those for me in my Elixir project. I then mangle into keys.

Building on the backs of the community and ecosystem like this I have gotten amazing mileage out of Kip Cole’s great Image library. I’ve also used resvg for turning SVG into PNG, which I apparently could have done with Image. On the plus side that also brought in Rust as well which meant achieving basic hipster compliance. Now I just need to change my HID library to use Zig and I’ll have aced it. That might happen actually.

Things yet to try:

• Touchscreen
  • Want to render much more interesting UI on it. Image can do so much fun stuff and I bet I could make it play nice with Brian Cardarella’s easing library. I’ve already tested a bit and Image can update the screen at just shy of 60hz. No idea if that looks reasonable though. We can expect to push 30hz with no issue I expect. Image provides operations like blur and ripple which I’d love to animate. I made a Short showing the silliness I’m after.
  • Using the touch at all. I’m most curious about the drag. I couldn’t actually see that Elgato uses that so it might be too crap. But I want to see what I can make it do. Build a whole little touch UI in there.
• Update keys while being pressed, trigger events on key up after down instead of down. Those kinds of niceties.

This is a really fun thing to poke around with when I have some moments for myself. Just making it do more and more. Exploring how I can combine more and more sources of data, inputs and outputs. I have a very simplistic GenServer-driven approach right now that I’d like to think about abstracting a bit. If I want to do better UI I need to keep the state of the UI differently and I’ll need to manage timing for animations.

I have a bunch of ideas around how to wrap libraries that provide new capabilities in a way where they plug into this in a more flexible way. Early thoughts. Might go nowhere.

When they market macro-pads like this for productivity I don’t think they usually mean that you’ll write tons of lines of code to make them do things. For me, this device has made me very productive. In a very specific way.

At some level. Probably.

This video showcases Sonic Pi. It is an open source project that I really appreciate. Sonic Pi enables live coding of music. If you are already a programmer it can turn your hard-earned, often dry, programming capabilities into musical potential. If you know music, you might just learn programming. If you know neither it is a fantastically fun place to start.

The video was entirely funded by Underjord out of love for cool open source software projects. Thanks to Sam Aaron of Sonic Pi for help on music choice and working through the idea. Mad props to DJ_Dave for the track which is in Sonic Pi’s examples, slightly modified for this presentation.

The project is focused on learning and enables any kid or teacher with a computer, even a Raspberry Pi, to get going.

• Learn more: https://sonic-pi.net/
• The Sonic Pi forum: https://in-thread.sonic-pi.net
• Want to get going? The tutorial: sonic-pi.net/tutorial.html

Well, I learned it all incrementally. HTML on Geocities, then some CSS, some JS. Oooh, PERL. Some XML. Local web server, exciting! Bam! PHP, MySQL, Linux. Years of Linux desktop use along with running my own server because young and no money. jQuery and AJAX. Onward to first paid work, paid hosting and CMS:es. WordPress, Joomla, Drupal, all of it. Tried to build web applications with those. That sucked. Better servers, Python, web frameworks, SPAs. Onwards to clouds. And on it goes to this day.

Note: This post is a refinement of an essay from my newsletter. Issue 15 to be specific. If you enjoy the read, consider signing up for a weekly hit of opinions and tech.

It has been unending learning. And most of it has built on but also replaced most of what came before. I don’t really like traditional CMS:es. I think they make empty promises and sneak up on you with hidden costs. But they are a pretty accessible place to start and learn. I don’t like cheap hosting providers, but, again, they are accessible. The choices I make today are choices I can make because of my past experiences. I already know many of the tradeoffs. My choices might not make sense to a newbie. “WordPress is a huge and influential project. Surely they know what they are doing?” Yeah, but they can’t change their design without breaking their golden goose of extensions. I bet they have some thoughts.

When a student asks me “What are your thoughts on MongoDB?” the answer is only simple if I lie. I know some people like it. I have heard enough to distrust it. The answer “Just use Postgres” isn’t exactly right, “It’s probably fine” feels dishonest and if I add “but I’d never choose it for anything” that’s hardly helpful. There is enormous context and choosing what to bring into a learning person’s world is genuinely difficult.

At the time I was learning, I didn’t have access to any kind of bigger picture. I had limited resources. Limited documentation. I had so many limitations. I ran into them constantly and eventually found the next thing I could use to do cool stuff. If you try to teach someone to use an HTML form today, you will need to decide if you are talking about the basic idea or reality. Do we need to cover CSRF, captchas and how to do it in React? How about the backend? Starting out, I didn’t hear about half of the things I didn’t know; I think that made things easier.

For example, I think every developer working in open tools benefits from being comfortable in a terminal and on a remote Linuxish server. So that brings some POSIX and some Bash. But then you get SSH and public key crypto. And then you have basic networking, potentially firewall configuration, at the very least Nginx, a database or maybe a bunch of Docker and cloud native tools that abstract everything away .. until you have to customize something and the complexity comes tumbling back in. The domain of knowledge just sprawls out into countless dimensions.

Yet people are supposed to get into this with a bootcamp. Or even with two years in school. I’m glad I started in my teens and don’t have to feel like I’m playing catch-up any more than I am. If it feels overwhelming to you, regardless of experience level, that’s because it is. It’s fractal; look closer at any of it and there is always more you could understand and benefit from. There are always more concerns you could absorb into your full-stackness.

Suddenly we run face first into the thing where companies are only hiring “senior” developers and everyone starts out being a “junior” developer. I’ve even seen “making senior” become a thing on Twitter because it is now an arbitrarily codified level in the orgs career progression. And in some ways that might be good. The profession probably needs to mature. But let me be clear and honest, I got my first senior and tech lead titles simply because I was the most experienced in the room, not because I was particularly experienced. But someone needed to wrangle the server, someone needed to stick their neck out. That was it.

But where the hell are people supposed to learn? Is it always going to be the WordPress shops of the world that teach juniors? Because websites are lower stakes and in the CMS world poor technical choices go unnoticed?

I keep trying to find better ways to bring people onboard and up to speed, and to hand them the tools they need. If I try to boil it down I get this:

If you want to do this stuff and build your skills, you have to know how to learn new things, and you have to be able to sit with discomfort.

The learning part is pretty commonly stated, the discomfort is not necessarily mentioned as frequently. You will never know the stack top to bottom to some specific granularity. You have uncertainty in every part of the profession. Your strongest guarantees build on top of systems on systems of unclear guarantees. All these things are learnable on their own, but the entire system of a full-stack web developer is not. You are dealing with uncertainty, and you need to be able to sit with the discomfort of that to get your work done with what you know right now.

I’d wager there’s a case to be made that the discomfort is what makes us move forward and try to find new solutions that solve our particular discomfort. And at the very least, we get to zoom deeper into that part of the fractal.

Hand someone a fish and they’ll never know that the ocean is vast. Teach them how to fish and they’ll be forever lost at sea.

Does this match your experience? Is it hopeful or distressing? You can reach me over email at lars@underjord.io or find me on Twitter as @lawik@hachyderm.io. If you enjoyed the read, it spawned from the newsletter.

There’s a thing I’m wary of with both companies and developers: when they seem to operate mostly on fear. In the case of companies, fear can lead to heavy screening during the hiring process, heavy-handed, detail-oriented project management or distrust towards employees. To my mind, these things either come from the basic mode of operation of some founding member or from cultural scars caused by a bad hire, a runaway project or very anxious management. Admittedly, there are projects where detailed specifications make sense. There are roles where screening is more important. There are entry-level garbage jobs where mutual distrust is possibly unavoidable at a certain level. However, I believe that, ultimately, all of these are detrimental practices.

Note: This post was originally published in the newsletter all about 90 weekly issues ago. I try to pick out some of the best ones for refinement and public publishing. Hope you enjoy. Much thanks to Cornelia Kelinske for helping me with the editing on this.

On the developer side I have seen developers, both employed and independent, that negotiate from a position of past wrong-doings. “Never this again.. I refuse to ..” They try to negotiate away future discomfort based on previous experiences and, to my understanding, a fear of repeating that bad experience. With people, I find this fear much more understandable. But whenever I act from this perspective I find it ends up being short-sighted. I should definitely learn what I dislike and want to avoid from bad experiences. I just no longer believe I can encode a perfect heuristic for that in contract terms or handshakes. There are an endless number of potential pains that can land on me in my work; enumerating them is pointless.

Actually, I think it’s even beyond pointless. Because you spend your leverage and career capital on promises that no organization is truly capable of keeping. “No crunch” or maybe “no calls during my vacation” are certainly reasonable demands. But will your employer let their business go to hell before they breach those terms? I doubt it.

Instead, I try to do what my first psychotherapist told me as I was working on recovering from burnout: I focus on “What do you feel?” When something comes up, I try not to just think it through, but instead, I ask myself: “what do you feel?” and I register those feelings. If I get the feeling that this is not how I want things, that’s a signal that I should push back, negotiate or at least air my concerns.

I can’t remove all the discomforts of my life, I can’t negotiate myself away from every imposition or nervous manager I might encounter. Organizations may have their scars; so do developers. I have mine. But I find fear makes us short-sighted and inflexible. Fear is pretty much completely ineffective as a driver for meaningful change. By leaning into new situations, despite the base level of discomfort it entails, I have achieved so many more of the things I want. I earn more, I work less, the work I do is more meaningful and I am much less supervised and/or micro-managed. And when things I don’t appreciate or accept come gliding in, I deal with them then and there.

This doesn’t remove the risk of unpleasant or bad things happening. But in my experience, neither does negotiating based on past hurt. It is better to negotiate for the things you want than to negotiate for what you want to protect yourself from. Because you can actually get the former. The latter is all about managing the situation as it arises.

I don’t want to operate on fear, worry or hesitation. I want to act based on consideration, intentional choices and steady commitments made through clear negotiation. This doesn’t mean I don’t have fear, worry or hesitation. I can be quite anxious. But for me, the way through that is not about building myself a fort. It is about making sure I’m an active participant in the outcome. That’s what I try for. I recommend others to at least give it a shot. Or get a therapist, maybe there are other recommendations for you.

If you find this interesting I cover this type of ground in my newsletter on a regular basis. You can find the newsletter here. If you have questions or feedback you can reach me at lars@underjord.io or as @lawik@hachyderm.io.

Editor’s note: This post is a refinement of the 7th issue of my newsletter. If you enjoy my thoughts on things, know that I send a small essay every week.

A while back I worked with a WordPress site. I used to do a lot of work with WordPress and Drupal many years back. For me, as someone in the open source end of the software development spectrum, WordPress tends to pop up every now and then. Typically a business owner I’m familiar with will go, “We got a new website, it is a fresh new WordPress instead of the mess we had.”

At that point, it is already too late to provide any beneficial input, so I usually withhold my commentary. If they had talked to me sooner, a WordPress site is not what I would have suggested.

But that’s what I feel and that’s been my experience. I don’t find WordPress to be a healthy way of building a website or running a software development project. WordPress has a lot of things going for it; dirt cheap hosting with standard LAMP-stack hosts. Solid UI that any moderately adventurous user can handle by default. Easy to set up if you have some technical skills. Well-known, much loved, huge ecosystem. It is open source and has respectable accessibility. These things tick some serious boxes for many organizations.

So what’s my problem?

There are hidden costs. No one budgets for maintenance work. No one plans for having access to a developer when the maintenance needs to happen. No one knows that the PHP-version their host provides may shift under their feet and kill their site whenever their host decides to upgrade. Most small organizations do not set up backups. All editing happens live on the site, which can have enormous consequences.

It’s also incredibly inefficient. WordPress does dynamic rendering by default, taxing CPU and IO for every single page load of what is usually entirely static content. There are tons of ways of caching WordPress; all of them have their own advantages and drawbacks. Expect road bumps. There are even ways of generating sites statically, but that’s not by design. Expect even more road bumps. This dynamic nature is part of the simplicity of WordPress and also enables its massive extensibility. It was generally how web applications were built in the 2000s. And even though hosting is cheap, it is usually unreliable, generally bad, or has had its capacity overbooked. You tend to get what you pay for. Many CDNs offer free tiers that would be enough for almost any small org website. Such a typical CDN offering would make for hilariously great performance for any simple website. Unfortunately, they tend to work very poorly with dynamically generated sites. There is a lot of benefit in matching your static content with a static approach to rendering.

There are also no good ways of managing WordPress workflows for more serious projects where you might want some DevOps deployment procedures or want developers to be able to work on the config and content while publishing happens on the production environment. Why? Because content and configuration are deeply intermingled in the database. Especially when you use extensions, which is one of the largest selling points of WordPress. All this boils down to maddening database merging, or else re-doing the development work on the production environment when it is time to deploy. Hope you kept diligent notes because that’s how we do software development in 2022. This flies in the face of most good practices and involves a lot of time-consuming, stressful and manual work.

So why hasn’t this issue been solved? To my understanding, most of the WordPress ecosystem is not run on in-depth software development shops. It is mostly web agency-level businesses. Their focus is usually not on the development workflow and the average customer’s experience level is low enough that many do not see the problem. The pain isn’t great enough in the short term, either. The initial project effort is usually fine since there is nothing in production. Deliver to the client, move on. Sustainability is either not in the business model or just a minimal support deal. The businesses that produce most WordPress sites don’t have a lot of incentive to shake this up, and some of them don’t even know that there’s reason to be concerned. I didn’t when I was doing it.

So can we use something else? I don’t know of any CMS that I would consider a valid replacement. Easy to use for end-users, manageable complexity for the devs, a great CDN story, smooth deployment. Static site generators are great, but they are firmly on the nerd shit end of the spectrum. Great for devs, iffy for less savvy users. Netlify has had some hype with Netlify CMS, but it does lean heavily on inviting you to use the CMS specifically with Netlify services.

There are a few CMS:es, but the ones with the most traction hit other reservations of mine. Craft and Cockpit seem to be more modern CMS:es in PHP. Cool, make use of the well-established tech WordPress builds on and use it to try something more modern. The options I hear most about and see recommended frequently, like Ghost or the headless Strapi, are built with Node.js, which in some ways is the spiritual successor to PHP in web dev popularity.

However, PHP-based projects are just not particularly interesting to me because I don’t really enjoy working with the language and I still find the execution model with files in folders to be a source of entire additional classes of security issues. WordPress, Drupal, and Joomla have been enormously efficient sources for automatic compromise, as security issues surface and many users just don’t know how to keep all the parts updated. If you can get a bad file upload through, you often have remote code execution. I’m sure some projects avoid this, but from what I’ve seen, it is still mostly the same model. I have other, additional reservations about PHP overall, but lots of that is just taste and preference. It’s a decent enough language, but it’s not for me.

Node.js-based projects don’t spark joy for me because I’ve had plenty of bad experiences maintaining projects that live with significant Node.js module dependencies. I’ve seen it happen with Angular 1, Cordova and Ionic at a previous job. I’ve had clients with React Native apps who encountered the same issue. When left alone for a few months or years, a project like this is an iffy thing to get running again. I can’t in good faith give Node high marks for health and sustainability. So while I might say, “yeah, try one of those Node CMS options,” I would do so because I can’t find any options I can heartily endorse.

I like the ideas of the JAM stack, but I find the J to be over-represented. Javascript for frontend interactivity is fine and really the best option right now, but I don’t particularly like it for the backend. I don’t think Node is necessary to manage the C10K problem. I don’t find the event-driven single-threaded approach compelling when I contrast it to Erlang, Elixir, and OTP, with which I typically work. I prefer making efficient use of resources, and I get easily frustrated with the challenges Node can have with racing towards doing IO because CPU-bound work can introduce enormous performance issues. Those are certainly trade-offs you can make for a simpler execution model, but I don’t like them.

For most businesses where you just need a working website and the website is not your entire job, you can pay for something like Squarespace and be done with it. It’ll probably be fine. But for some orgs, having an open source stack is important, either ideologically, as a point of pride in openness, or because they want to avoid vendor lock-in.

The Open Source CMS is not a solved problem in my book. There is no choice that provides what I want. Perhaps I’m a bit niche in that I filter out on both PHP and Node.js, but I don’t believe I’m alone in that sentiment. I think WordPress won its market leader role on the basis of developer mindshare. It is approachable and effective. It will do in 90% of use cases, you can get at the innards and make it do what you need, and it provides a UI that your users will not complain about. I can’t get that with current solutions; they are either trying to be too clever and target developers to the detriment of end users, or they are not built in a way I expect to hold up in the long term.

So what are my requirements?

• Sound technical stack with a good record in reliability and security
• Creates primarily static sites for efficiency, security, and reliability
• Can be hosted with a wide variety of generic providers
• Supports implementing dynamic features in a JAM Stack manner, with APIs, etc.
• Provides a fully online admin UI for managing the content of the site in a smooth way
• Implements a thought-out way of working with both production editing and development processes
• Can be automated and “devops:ed” effectively
• Provides a decent ecosystem of at least themes and preferably extensions as well
• Offers a sound approach to accessibility
• Open source

I’ve already done one run on building something like this where I tried a particular Git-based approach which I’ve since abandoned. I think the approach has some merit but it is not easy or straightforward.

A while back I started experiments with a couple of co-conspirators on developing the backbone for a headless CMS in Elixir. It’s a slow process as we all have jobs and other silly priorities, but I’m hopeful that this approach could actually be quite realistic. The first step is to build good data fundamentals. For featureset I’m leaning heavily on my experiences with Contentful at a client where very successfully leveraged that headless CMS.

I don’t need it to have the critical mass of WordPress. And I doubt I’ll even manage to check my entire list up there. Hopefully, though, we can make something that solves my actual problems and we’ll see where I can take it from there.

Thanks for reading. If I’ve missed some solution that you think I should consider, or if I’m wrong about something I wrote here, don’t hesitate to write at me via lars@underjord.io or on Twitter @lawik@hachyderm.io. I’m always open to discuss what I’m up to. It would be incredibly convenient to be wrong and not have this problem anymore.

If you want more of my more frequent writing on tech, Elixir and creating things I suggest the newsletter.

How we run dangerous commands

Have you ever had to run a potentially dangerous command on a production server? A destructive piece of SQL? A batch update/data migration script? A script that cleans up accumulated garbage files in a recursive manner?

How did you approach it?

Do you have a small alarm bell that goes off in your head when you are close to doing something that would be really painful (or impossible) to revert? Do you adjust your approach because of this signal? I know I do.

A cluster of synapses got together in your brain just to warn you. They strive to prevent me from shooting myself in the foot at the terminal. They probably first formed that time I blew away my entire music collection with a bad shell script. They hardened the first time I rm -rf:ed something that ended up resolving to /. I haven’t had this kind of catastrophic failure in a while. Thanks, brain.

When I feel that tingle of “are you sure this is the query you want to run?” I usually end up taking the SQL query, changing it to a SELECT and just eyeballing the output a bit. Does it hit the expected number of rows? Do the results look reasonable? Did I, in fact, just run a fresh backup?

Similarly, I ask myself before I run any recursive rm command on UNIX: “What files would this hit?”

If possible, I dry-run the command.

I’m also well aware that some would snort and say that this is not how it should be done. I wager that’s correct but not particularly useful feedback. If you can run testable migrations that can be rolled forward and back, that’s better. If you can do it the better way and the trade-offs make sense, do it the better way. But don’t feel bad if you don’t have those opportunities or resources. We do what we can with what we have. You can get far by just developing a good instinct for when you’re about to screw yourself over.

Is there a word for this alarm bell? This instinct? This moment of hesitation and the practices you put in place to make sure you’re not screwing up? I think it is a shared experience of many server-wrangling developers out there. I know an old colleague that always commented out any impactful command in case he’d accidentally fire it off before it was ready. Another colleague would type it out and Ctrl+C to cancel it but keep the command on screen while checking things.

It’s an interesting thing to me. A small moment of recognizing that you are fallible wetware and that the machine, software and hardware, does not care. Is this the central friction of human-computer interface design?

A moment of humility, perhaps.

Rituals around deployment

When the new thing is going out, what do you do? Do you run a specific command? Do you press a button in a Cloud console? Do you have a friend or colleague double-check your input values? What leads up to this point?

There are processes you can adopt where CI and CD will completely eliminate the final button press. I’ve found these to be rare in actual reality. I hear it’s a strong idea. Even with a very comprehensive CI/CD solution in place you aren’t really free of this. It’s not all about the tech. What does your CI/CD pipeline test for and why does it do that? Who’s notified, and what information is broadcast when an update goes out? What rules does it use for determining success and failure?

There seems to be some level of ritual around updates/releases of software wherever I’ve worked. In my own projects, I’ve created a ritual based on my habits and lessons learned. In an existing organization, I try to understand the things they already do. If I find their ritual lacking, I’ll usually try to leverage my understanding of it to improve it. Trying to change a ritual without understanding it is a fraught process.

Why do I call it a ritual rather than just a process? I’m not convinced it is only a technical, structural matter. Often we do things that aren’t strictly necessary but are reassuring.

You make sure Alex in Marketing knows the update is going out because they get pissed if they don’t feel included. You have a colleague check your thinking on particular things while you let other things right through. You have QA give the OK on the staging environment. Maybe you review the commit messages or the file changes that go into the update in full. Maybe you only check if there are migrations to run. Do you tag a release? Do you push it to a specific branch?

Much like some specific flavor of a burial ceremony, you have things that can seem arbitrary or strange to an outsider. These things are usually part of the culture (colors, songs, phrases), things that serve a clearly social purpose (saying a final goodbye to the deceased, closure) and things that are fair to consider a technical necessity (burying/lowering the casket/incinerating/putting the deceased somewhere).

In a release process some checks, steps and sign-offs can seem arbitrary but are ingrained in the culture and may have an unclear purpose or be entirely vestigial. In any release process there are checks, steps and maybe even sign-offs. These can seem arbitrary, but I believe that is because they are cultural in nature. They exclusively serve a social purpose while others are a technical necessity or clearly technical choices.

I think most dev shops end up with some kind of ritual. Sometimes, it’s only for communicating the changes to the surrounding organization. Other times it can be a technically intricate ritual with many mysterious incantations to get that particular system up and running with a new version. The latter type is usually a good target for significant scripting and automation.

As humans, I think we always attach some level of importance to ritual. Some people feel stronger about it than others. To some, it’s about safety, and changes to the ritual can feel frustrating. To some, finding an ideal ritual is the goal, and changes are inevitable. Some desire a minimal ritual, and others want the iron-clad, double-verify, none-shall-pass sort of approach. Sometimes the ritual is dictated by culture, sometimes by actual regulations, sometimes by resources.

I think some people rely on the stability of the ritual, while some view it more as a means to an end. A tool. But that might also be why it works: it is something in which many different types of people can see value. It is a point where the lines cross, where Venn diagrams overlap and where the mixed purposes of the ritual can satisfy the differences of the participants in a shared procedure. We come to it for different reasons but we share the feeling that it is necessary.

The longer a ritual has lived, the harder I think it is to maintain the protective abstractions. Some will want the ritual to remain as it always was, and some will find it stifling and want to change it. I think a lot of people experience this with Christmas, for example. For many people, the question of how to celebrate with your family is a source of potential conflict. While the general idea of the ritual is to come together with loved ones, the particular ritual is loaded with years of tradition that some love, some endure, and others try to avoid.

My family is very progressive. Our Christmas celebration requirements have scaled back quite far. So far, in fact, that I’ve had to start re-introducing some Christmas traditions for me, my wife and my daughter because I had ended up missing them. I’m also very glad we aren’t gripped by a tradition set in stone by our grandparents. I just really want a Christmas tree. Because I like it. The tree awakens fond memories.

So, what’s my point? Probably that our process is not exclusively based on cold logic and business realities— and that’s fine. We are fallible wetware and we need our creature comforts. Framing a process as a ritual changes my expectation of what it should achieve from the raw technical output to a broader consideration of the people involved. I find that useful.

UNIX as a tradition

I’m not sure whether I should say UNIX, or hacking, or what, to center the culture around. That debate is a rich source of “well, actually,” which I don’t really care about. It is not only free software, or open source; it is not just MIT, and it is definitely not exclusive to Silicon Valley startup culture. As a developer on Linux with mostly open source tech, you are working in a world wrought from a mix of the cultures I mentioned— even though I’m surely missing some. I’ll center it around UNIX, because I think that’s largely accurate.

When you use a term such as grok or borked, you’re leaning on that tradition. There is a lot of useful shorthand that stems from this common traditional lexicon. Some of this stuff is also quite the shibboleth, a way to identify if people are part of this tradition.

Tech and software development as a culture is widening, and I’m sure there are plenty of developers who have minimal exposure to this particular culture. That’s fine. Years in corporate dev might not expose you to this; I don’t know, I haven’t tried it. If you’re going through a software dev bootcamp at 30 you might have no previous interest, and you won’t absorb a culture from an X weeks long intensive course.

So while the traditional culture of UNIX is not the only culture of software development and tech, its influence can probably be found all over. And it is my culture. It is where I found my footing. A lot of the culture is also a bit old and outdated. Most of the terminology in the Jargon file is not in use or generally recognized. A lot of people who are in open source now have no particular connection to the wild 80’s hackers covered in The Underground Book (free download by the way). A lot of them have never heard of The Hacker’s Manifesto, which burned bright to me as a teenager.

That’s fine. The culture has its traditions and history, and I don’t think this culture will be forgotten. It is also fine to leave some of it behind. Some of it was only funny back then, some stuff seemed dated even when I got into it. Some of it is irrelevant. Some of it should be brought with us as good, useful, relevant ideas, interesting thoughts or inspiring history.

I am becoming increasingly aware of the fact that my way of knowing about computers, my way of getting into programming, is not the only “real” way. I continue to practice the UNIX tradition; I rather like it. I’m very fond of its history. I appreciate a lot of its culture. But it doesn’t live or die with whether newcomers know the heritage of grok. I’ll happily put a new-formed hacker in touch with these roots. But if they don’t want it, we still have plenty to share.

It is easier for me not to expect new people to know about these things and relate to them if I recognize that it is a tradition I was brought up in. To me, understanding tech and knowing programming are intimately tied to this cultural movement, the history and its current shape. This is true for me, very likely true for many others, yet absolutely not true for everyone. And it would be absurd if I didn’t acknowledge that people can come to the same skill set from entirely different cultural traditions.

Shared experiences, rituals and traditions in software

I’ve found the trope “it isn’t about $TOPIC, it is about people” pretty tiresome in the past. In the back of my mind I’d go, “sure, but really, $TOPIC is pretty important.” Whether $TOPIC concerns technology, a web framework, software quality or open source software, I still find it a bit disingenuous to claim that it is all about people when the actual thing is what brought them all here.

The trope can still feel tired but I have changed my perspective a fair bit. The people situation has to be in decent working order for anything to happen, for things to get built, delivered and used. And people come in a large variety, with many different opinions, fears, hopes and ambitions.

I think we can most quickly find common ground in shared tradition and we can build an understanding of differences more quickly by recognizing if we come from different places. This applies whether you are from enterprise .Net life in an MS-focused stack, straight UNIX with lots of dynamic scripting on top or career-switch-focused bootcamp graduation. All these places will put you into the world with different tools, priorities and often value systems.

I think the shared experiences often help bridge these differences. We all know how dumb computers are. We all know how dumb we programmers can be. This is the common ground.

Tradition is about where you come from and what feels familiar to you, where you find comfort and meet discomfort. To me, rituals are the recurring patterns in how you perform your practice. It could be the practice of your tradition, of your craft, or of your organization. Lots of ritual practice stems from your culture.

The Github Pull Request is an evolution of the patch sent via email and has firm roots in open source hacker culture and tradition. Those roots have deep implications for what a PR is and how it works. They are asynchronous, require no trust and imply low trust. It is a ritual of checking someone’s work before letting it into your project.

You likely have rituals that you built based on your experience and that are your preferred ways of working. Do you subscribe to a TDD approach in everything you do? Do you always start a piece of work by thinking it through? Or maybe you build the dumbest thing you think could work? With practice, you build up more of these patterns and tear down old ones.

Your organization forms rituals from culture and experience in a similar way. Whatever the driving values are, whatever decision-makers hold dear and press for, whatever the organization enshrines as important by its actions: these will shape the pattern of practice and turn into the rituals of that workplace. A lot of them are accidental, incidental and unintentional— the “long-tail consequences” of choices.

Shared ritual becomes a mix of habit, useful shorthand and practical routine. And the more similar your personal rituals are to your organization’s, team’s, or co-workers’, the less effort it is to settle into working with them. If the rituals are too far apart, you are very likely to experience friction. Rituals that you unknowingly share can turn into moments of strong common understanding with a colleague: “Oh, you do that too!” or “Yeah, makes total sense to me; I do a similar thing.”

There is a grounding in recognizing that you are not a technical optimization machine. You are not a robot making all your choices according to some absolute best. Likely, you are navigating according to your values and experiences. Your technical tradition is your foundation and a place of origin. Your rituals are the tools you’ve built and the experiences you’ve earned. The practice of making software is a complex human endeavor.

If you made through to here you might have reflections or feedback. I’m happy to take those in via email at lars@underjord.io or on Twitter @lawik@hachyderm.io. Thanks for reading.

Actually, I’m just outlining the tools I’d typically look at and consider right now. I hope you enjoy. I rather like the intro personally.

Also on YouTube.

If you have questions or feedback reach out at lars@underjord.io or on Twitter where I’m @lawik@hachyderm.io.

This was also more thoroughly covered in text in this post.

This video has been on the YouTube channel for some time but I’m catching up on the transcoding and publishing here.

This video lays the explanatory groundwork for a series of guides for setting up some self-hosted services.

It went up on YouTube a while back, I’m catching up on the website publishing side :)

If you have questions or thoughts to share you can reach me at lars@underjord.io or @lawik@hachyderm.io.

ID3 is my favorite kind of standard. A de facto one. MPEG, specifically our beloved MP3 didn’t have a way to embed metadata. So one was invented. Tacked on at the end of the file. With ID3v2 it goes at the start of the file. It’s a binary format. If you are used to JSON or XML for your data storage this is not so human-readable. It is pretty straight-forward as binary formats go I think. I haven’t really parsed one before. The spec documents are mostly clear and understandable (and used to be linkable at id3.org/id3v2.3.0) but not at all mobile-friendly (I suggest browsing them via mutagen instead, they are better).

This post is not about the technical intricacies of the format. There are things I could write about how it avoids conflict with MPEG synchronisation thingies. I could cover how easy it is to implement parsing in Elixir due to the powerful binary pattern matching syntax and how smooth encoding new binaries can be with IO data. That’s not what we’re doing. This is about a simpler time, where people saw the wild possibilities of music on computers and when people cared about files, damnit. This is about some of the most interesting and entertaining things I’ve run across while reading the spec.

Note: I’m not a scholar of retro computing or someone who does a lot of research typically. This will not be heavy on the footnotes or the digging, this is about vibes man (to totally butcher together two pieces slang, showing my range here).

Parts of an ID3 tag are called frames. Beyond Album titles, Song titles, Artist names and Genre the ID3 tag offers many more interesting ones. I have no clear idea which of these have seen production use but I think they paint a mighty picture of a very different future that could have been .. question mark?

There are generally useful things such as links, embedded images and embedded anythings really. In more recent times there have been additional specifications for Chapters and Table of Contents which are used to provide podcasts episodes with their chapter information and chapter art. Fabulous stuff. Genuinely useful in my day-to-day and if you keep an eye on what I do the purpose of why I dug into ID3 might lead us back to this usage. The newsletter is there after the post if you are curious for any additional notes on this.

The thing that first got to me was what really made me see the MP3 + ID3 file in a different light. Play counter (PCNT). This mighty little frame contains a number and it is intended to be the number of times the file has been played. According to spec it should be incremented when it begins playing. This means that the file changes as people “consume” the media in it. In that way it gives a single song a memory, a very low-resolution story to tell. The only files I expect to change regularly these days are typically considered documents, text files or production projects (image/video/audio production). Most files I deal with are otherwise copied and converted around now.

There is something that feels absurd about your MP3 player changing your song file. On the other hand I find it much worse that fewer and fewer people stick an MP3 file in a player of any kind these days. I bet most players didn’t implement this but imagine if they did. The weird cultural touches that could flourish. Personally I’ll never consider listening to any song with less than a 100 play count. I only want files that have clearly achieved popularity. Or finding a legit single-digit playcount copy of a popular rip. That means you got it close to the source!

I’m aware that I’m being silly but it tickles my mind to imagine that world.

The Play counter is the smaller and simpler sibling of a much wilder frame. Popularimeter (POPM). It allows storing an arbitrary number of (up to the max frame size of 16 Mb I believe) email and rating pairs. An email string, a rating from 0-255 and a personal play counter. The use of this rating was discussed on the ID3 Wikipedia page as apparently some OS:es and players use it to display a star rating for a song and someone has opinions. I think this frame seems fantastic!

Note: If you’ve heard the episode of the Regular Programming podcast where I get into this (not released at time of writing) I had forgotten about the personal playcount on the POPM frame.

Neat enough, I can rate my song. Just note. Due to the email address the file can have a lot of these ratings. If you keep passing along the same file you can build up a massive set of ratings. I can see who played it and how much. That could even be delightfully embarassing. Especially for me. I get stuck on songs.

I wonder if this was imagined to be used so that when you got a sample of a song in MP3 format it could come pre-loaded with ratings from a site or something. Or was it truly just the evolving file as a vehicle for crowd-sourced wisdom. Anyway, shove ratings and Personally Identifiable Information in more files and unleash them to the world I say. There’s something glorious about it.

Now the thought that ratings might come pre-loaded from some site, baked into a sample file is not wrought from thin air and mind stuff on my part. We have another odd one that gets to business. The Commercial Frame (COMR). “This frame enables several competing offers in the same tag by bundling all needed information.”

Every offer has the following. Prices specificed in any number of currencies, a date for how long the price is valid, a contact URL for reaching the seller, a “Received as” field indicating what the product is delivered as which has a number of options such as “Standard CD with other songs”, “File over the Internet” or “Stream over the Internet”. There are a bunch more musically inclined ones but also merchandise and a useful Other. It can also provide the seller’s name and an embedded logo.

Store front as a standard. In a file. I find it rather inspired compared to everything being a SaaS.

Then we get the Ownership frame (OWNE) which I assume makes the file officially an NFT. And the Terms of Use (USER) frame which presumably makes it into a smart contract. Or maybe not. The Ownership frame provides information about purchase price, date of purchase and the seller. Nothing about the actual owner. The Terms of Use is what you’d expect, the terms under which the file may be used. There is also a way of grouping frames and signing them cryptographically though parts of the implementation is left to the imagination.

Some honorable mentions as genuinely useful frames are Synchronized and Unsynchronized lyrics. I believe there may be a separate one for captions in an Accessibility Extension but I’m not sure.

The Event codes frame seems like fun. The event codes can be used for a number of different things, controlling lights, setting of explosives, whatever the player wants to interpret them as. There are some specific ones like start of song, bridge, end of song and a bunch of other musically related ones but you can specify a number of custom ones as well.

Since SQLite is trendy right now there is also no reason you couldn’t shove a SQLite database file into the file embedding part. 16Mb for a frame, 256Mb is the limit for the entire tag from what I gather. You an shove a lot of stuff in this thing.

I really wonder how much of this was ever used. It mostly seems sanely put together. Just hopelessly irrelevant to how things work now. It makes me really want to do some silly stuff with some MP3 files. Of course you don’t actually need an MP3 file. ID3 can work in a multitude of places. It doesn’t require media at all.

The possibilities feel endless to me. Completely unapplicable in the present day but I also desperately want to find out about current day usages. I’ve just dipped my toe in ID3, if you know the history and the present day of it, reach out. I’d love to know more and if you’ve written on it, share you stuff.

Note that this format is also quite dense. No curly brackets here unless you put them in your text field. It is impractical at times to deal with a binary format but it does make me happy that the tag header provides more info in 10 bytes than what it would take me to idiomatically express the version of the tag in JSON.

{"version": 3}  (15 bytes)

vs.

ID3 3 0 0 0001  (10 bytes)
|   | | | |____ 4 bytes of tag size (here set to 1 byte)
|   | | |
|   | | |______ 8 bit flags, here zeroed out
|   | |
|   | |________ spec revision: 0
|   |
|   |__________ spec version: 3
|
|______________ starting indicator "ID3"

This stuff has been fun to work with. I expect to share more about it as my implementation evolves and becomes useful. For now I needed to share the human aspect of the specification as I’ve enjoyed getting to know it and also share the speculations I concocted as I ran into these frames in my work.

If you have questions, comments or insight to share, do reach out at lars@underjord.io or find me on Twitter as @lawik@hachyderm.io.

“Isn’t Underjord Just You?”

Having gone recently from being a single-person consulting and contracting shop to having a small team I took a rather different approach to this conference. My team does not have deep connections in this community. They are new and much of their experience stem from different worlds than tech. I planned for two things before the conference and I scheduled a day for each.

On the first day we did “team-building”. That is, we spent the day together with a few activities, good food, had some drinks and generally got a chance to spend time together. The whole point was to make sure our remote-only team got a chance to find our footing as a group in person.

On the second day I had pulled together some interested people and co-conspirators, some which we knew and some which we didn’t. With this gang we set out to run an informal workshop where we played with Nerves and LiveView and just generally doing tech for most of the day. After that we went for a comprehensive dinner sitting in the evening. Tech for breaking the ice and having some fun and then time to get to know one another. The point was to know some other people before delving into the mass of people that is a conference.

I’m very happy with both of those days and the entire plan. They themselves would have made the trip worthwhile.

At Dawn We Confer

The conference started. The starting keynote was great! Sanne Kalkman spoke about the importance and value of bringing in more inexperienced developers. She offered some models I found useful for understanding the challenges in transmitting knowledge to them. If you run a team I definitely recommend keeping an eye out for the recording. I have one overall criticism around the keynotes that I’ll get back to further down, it had nothing to do with the speakers but rather execution around their talks.

It was followed by something I was actually quite hyped for. I crave a good ritual. Kenneth Lundin presented the new OTP 25 release. It is a tradition at these events and from my understanding it has been a bit of a meme that it is a dry read of the changelog. I don’t know if it was ever literally that but this year definitely seemed very well put together and quite succinct compared to what I remember of my first experience a few years back. Regardless, it is a presentation on the work of building Erlang that I find embodies the character of Erlang itself. It doesn’t try to be very fancy and it gets the job thoroughly done. I think the ARM 64 JIT is probably the most appealing thing that affects me in the particular changelog. In reality I was most excited because with an inspirational keynote and a technical rundown the conference was officially on.

Talks? But I’d Have to Listen to People..

Overall I was bad about going to and paying attention at talks. I think the preceding days busted my focus a bit as well as a code problem pulling me into try to fix it. I skipped a fair number of talks and didn’t pay a ton of attention so unfortunately I won’t be able to give a good rundown. I hope someone else shares their notes.

There might not have been a ton of entry-level technical talks. And understandably there was a split between Erlang and Elixir. Had timing been different we would have likely prioritized ElixirConf. There was one talk about LiveView and JS interop which was much appreciated by my team and was useful to me. Hands-on and practical with tools we use day-to-day. Super nice speakers as well, ended up talking a fair bit throughout the conference.

I had a lot of very decent hallway track interactions. Overall there was a lot of big Erlang shop employees there, your Kivra and Klarna folks that are apparently all over the place. My sympathies to the employees at Klarna by the way, they’re having no fun right now (see news about Klarna lay-offs). I spoke to some Elixir folk I know that made the trip. I would say there were significantly fewer there than last time and I think the pandemic/travel balance isn’t there yet for people. Very understandable.

The hallway track really got going at the bar after though. Those evening conversations were probably my favorite times. Great people, more time to talk.

Fame! Recognition! Finally!

One thing was novel and that was the one person that went. “Oh, you’re the Underjord guy right. I had a question.” This is the first in-person conference since I started doing anything in public with Elixir. I rather enjoyed that though I was rather urgently on my way to the restroom which meant I couldn’t nearly savor the glory of the conversation.

We Did It Twice!

Second day beginning keynote was fantastic. As expected with Quinn Wilton and she was not held back but seemed rather unleashed by having Robert Virding on her side, a solid speaker in his own right. Their talk was probably my favorite overall. I’m not sure I can apply it to anything but it scratched my brain in a good way. It gave historical context for a number of things about Erlang, it was funny and it was deeply technical and nerdy. Very enjoyable even in moments when a fair bit of the language-nerdery went over my head.

The ending keynote was interesting. Brooklyn Zelenka who I’ve seen give a very good keynote at ElixirConf in Prague a good while back gave this one about burning our laurels. Apparently it connects with her talk at EMPEX so I should check that one out too. Essentially it focused on what would be required to operate in a client-first world. It bundled client-first, edge computing and decentralization as problems in an interesting way. Currently talks about decentralization puts my guard up due to crypto-currencies, zero trust and blockchain hype which I have no interest in and do not see as a particularly promising way forward to .. anything.

Brooklyn interestingly steered clear of the coin side of crypto and blockchains almost entirely but did dig into what we might need to abandon to operate in a way that doesn’t require talking to a central source of truth. Essentially giving up data consistency was probably the central thing. I thought the talk was very interesting and probably one of the best put together and most clearly presented ones I attended. It also had some of the more unfortunate after..thing. Nothing that reflects on the talk and I’ll get to that whole keynotes-thing shortly.

I spoke to her after the talk, mostly because my team demanded Witchcraft stickers but also because I was curious how into Web 3 this Fission company actually is. From the conversation, my charitable and optimistic read is that they want to see what can be achieved in decentralization, in client-first applications and in general in tackling some of those hard problems.

I’m not convinced they aren’t a bit too tangled in Web 3 but I get the impression that they are more interested in providing options for solving fundamental problems in low-trust computing than facilitating the sale of bored apes. The type of problems they want to solve seem like they may very well give us useful tools for specific problems, whether or not they achieve their goals. That’s the optimistic, charitable read from someone who is generally very skeptical about anything that rubs up against Web 3.

On Keynotes

Okay. The keynote thing. I don’t think there should be a general prompt for questions after talks unless the speaker explicitly asks for it. Particular I think this is the case for a keynote. It would have to be a very interactive keynote indeed for it to benefit from the crowd’s input. The keynotes are supposed to set the key note for the conference. For example. Sanne did a very good job setting up the topic of how and why to open the door for juniors. She did get two questions. I don’t remember the first one clearly, it didn’t particularly detract from the talk but definitely didn’t add to it. The second one was .. more of a comment really.

If this was you, sorry to throw you under the bus, I am sure you meant to do some good. Interpreting it kindly this was someone wanting to make sure that no-one understood this to be an absolute. “You must hire juniors!” A less charitable interpretation is that they thought the speaker wasn’t aware that it is a nuanced thing. There was some unclear, unprepared, unrehearsed (because it’s a crowd comment trying to become a question) thing and essentially the question was “something, something but it isn’t always good to hire juniors right?”

Sure. Right. True. Not the point. Trying to raise a key note and then having someone decide the rest of the piano really needs to be taken into consideration is not particularly compelling. That nuance fits in the discussion after and about the keynote. Not as an improvised encore by a member of the crowd half way up on stage. This is to a very small degree the responsibility of the person asking the question, they were actively invited to.

The final keynote which I thought was very well put together and ended very well showed where the issue really lies in my eyes. The invitation to questions. Q&A are often poor ad-hoc addendums on any conference talk. They do have their place though and can be quite good for some types of talks. Even better if the speaker is comfortable to do questions during sometimes. It’s a different circumstance, a different goal. For a keynote I don’t see the point. This ended up with an MC trying to source questions from an auditorium that was satisfied and ready to wrap up if I was to hazard a guess. It was a good end note. Let it end. End it. Don’t let it linger.

Quinn & Robert’s keynote entirely avoided the issue by going over time, leaving no space-time for questions. So it went through spotlessly.

I’m not sure I would have reacted quite so much to this aspect if I hadn’t read the fantastic The Art of Gathering by Priya Parker. That book very much informed how I tried to do the small events I did before this conference and it has some important things to say about opening and closing. Overall I think if you have a keynote to actually set a key note, which I think the talks were well chosen to do, I don’t think you should invite questions unless the speakers specifically wants it as part of the performance they are putting out there.

In Conclusion

All in all a good conference and this above small criticism aside I have zero complaints and tons of appreciation for the event put on.

It felt fantastic to be back in the BEAM community. For me it felt like the start of many more good and even livelier ones. Unfortunately I won’t be able to continue that arc for a bit. I’m having a second child soon and we’ll be quite busy while the Elixir community congregates and cavorts without me. A hard life but a good one.

Am I wrong about keynotes? Did we meet at the conference? Do you go to conferences? Let me know your thoughts at lars@underjord.io or on Twitter where I’m @lawik@hachyderm.io.

It is not exceedingly hard to do. There are may compelling first demos we could arguably make to achieve it. This is just one way to do something fun with Nerves and LiveView in a fairly minimal way. I have also made a slightly more involved demo project for an event that you can find on my GitHub. That one includes Tailwind CSS, basic Ecto migrations with SQLite and some other niceties but it is another demo. The docs for that one are sparse, steps are likely missing. You have been warned.

Let’s stick to the slightly simpler demo and unpack the process.

Let’s get going

You will need a Raspberry Pi device, an SD card reader and some patience with me. I ran this on a Pi Zero W and a Pi 400.

The fundamental steps are taken from the Nerves documentation for User Interfaces which has a whole thing about setting up a “poncho” project with Phoenix.

First, make sure you have installed Nerves and have Elixir working.

mkdir keybored
cd keybored
mix phx.new keybored_ui --module KeyboredUI --no-ecto --no-mailer

This gives us a project directory for the whole thing, we create a Phoenix project inside of it. We make sure the name is properly capitalized and then we exclude Ecto (no database, plz) and the default mailer (no email, plz). Ecto would require a few extra steps to handle. The mailer just isn’t used.

I want a way of providing interesting and useful inputs to play with so we will bring in a special linux-only dependency. It will work on your host machine if it runs Linux and assuming you put yourself in the input group. It will also work very nicely on the Raspberry Pi with Nerves as Nerves is built on Linux.

In your mix.exs, find the deps function and add this one:

..
{:input_event, "~>1.0"}
..

Then run:

mix deps.get

InputEvent uses the Linux Input subsystem userspace API (apparently) and there is some neat reference here. Essentially it lets you get events from things like buttons, keyboards, mice, touchscreens and such in your system.

This calls for a GenServer

Time to make a nice and rough GenServer to capture those events. If you want to run this part of the Elixir application on a non-Linux host you’ll need to fake some events on your own.

Create the file lib/keybored_ui/inputter.ex and write up the following:

defmodule KeyboredUI.Inputter do
  use GenServer

  def start_link(_) do
    GenServer.start_link(KeyboredUI.Inputter, nil, name: Keybored.Inputter)
  end

  @impl true
  def init(_) do
    devices =
      InputEvent.enumerate()
      |> Enum.map(fn {device, info} ->
        {:ok, _pid} = InputEvent.start_link(device)
        {device, info}
      end)
      |> Map.new()

    {:ok, devices}
  end

  @impl true
  def handle_call(:fetch, _, devices) do
    {:reply, devices, devices}
  end

  @impl true
  def handle_info({:input_event, _device, _values} = event, devices) do
    Phoenix.PubSub.broadcast!(KeyboredUI.PubSub, "inputs", event)
    {:noreply, devices}
  end
end

This GenServer starts, registers with a name. In the init function it does the wildest possible thing and gets the list of devices available for InputEvent. All of them. For each device it then starts a link to it which means it will starts sending messages to use.

We store the list of devices since there is some good stuff in there. The name of the device for example. Take a look at the data though, it also provides a bunch of information about the events you can expect from the device and all.

Then we implement one call-handler which will let us fetch the device list. We’ll use it later.

The other handler is the input event handler. The only thing that it does is take the event and broadcast it to the “inputs” topic via Phoenix PubSub.

Then go to your lib/keybored_ui/application.ex and add it to the Supervision tree as:

#..
KeyboredUI.Inputter,
#..

Next we build a LiveView

Create the folder lib/keybored_ui_web/live and the file input_live.ex in it. It should look as follows:

defmodule KeyboredUIWeb.InputLive do
  use KeyboredUIWeb, :live_view

  @impl true
  def mount(_params, _session, socket) do
    devices = GenServer.call(Keybored.Inputter, :fetch)

    Phoenix.PubSub.subscribe(KeyboredUI.PubSub, "inputs")

    {:ok, assign(socket, devices: devices, events: [], dot: {50, 50})}
  end

  @impl true
  def handle_info({:input_event, device, values}, socket) do
    events = [{device, values} | socket.assigns.events]
    dot = process_movements(socket.assigns.dot, values)

    {:noreply, assign(socket, events: events, dot: dot)}
  end

  defp process_movements(dot, []) do
    dot
  end

  defp process_movements({x, y} = dot, [value | values]) do
    dot =
      case value do
        {:ev_rel, :rel_x, points} -> {x + points, y}
        {:ev_rel, :rel_y, points} -> {x, y + points}
        {:ev_key, :key_up, 0} -> {x, y - 5}
        {:ev_key, :key_down, 0} -> {x, y + 5}
        {:ev_key, :key_left, 0} -> {x - 5, y}
        {:ev_key, :key_right, 0} -> {x + 5, y}
        _ -> dot
      end

    process_movements(dot, values)
  end

  def render(assigns) do
    ~H"""
    <svg viewBox="0 0 100 100" style="position: absolute; top: 0; left: 0; height: 100vh; width: 100vw;">
      <circle cx={elem(@dot, 0)} cy={elem(@dot, 1)} r="6" />
    </svg>

    <div style="position: relative; max-height: 800px; overflow: hidden;">
    <%= for {d, e} <- Enum.take(@events,100) do %>
      <div><%= @devices[d].name %>: <%= inspect(e) %></div>
    <% end %>
    </div>
    """
  end
end

As we mount the LiveView we pull the list of devices and subscribe to the “inputs” topic. We set up some initial state with the list of devices, an empty list of events and .. a dot?

We add the handle_info/3 callback matching for the input events we expect to be working with. We add the event information to the events list and for particular events we let a function called process_movements/2 update the dot. We update the assigns from this.

Our render-function renders the assigns by showing an SVG dot on the screen and printing out some recent events.

Let’s add it to the router.ex file by replacing the existing entry for "/" with:

live "/", InputLive

That’s it for the LiveView part. If you are on Linux you can test this on your host by making sure your user is in the input group sudo usermod -a -G input <username>, restuffing your shell with newgrp input and then running mix phx.server.

On to the Nerves!

More Nerves!

We need to generate the Nerves firmware project next to the keybored_ui project. This gives us the foundation for a poncho-style project. It is a way of structuring related Elixir projects where the defining feature is that it isn’t an Umbrella project. It also let’s you run the UI part of your application without fiddling with the Nerves firmware config. In this case, it mostly speeds up project creation, we can use the Nerves generator and the Phoenix generator without needing to merge their efforts.

In our main keybored folder:

mix nerves.new keybored_firmware

Go into the new project’s mix.exs file and add your UI project as a dependency, turning these two projects into a poncho:

#..
{:keybored_ui, path: "../keybored_ui"},
#..

Now we want to slightly change our usage of esbuild in the UI project.

In our keybored_ui/mix.exs we change the esbuild line to read:

{:esbuild, "~> 0.3", runtime: Mix.env() == :dev && Mix.target == :host},

Specifically adding the bit about the Mix.target.

In the keybored_firmware project run:

mix deps.get

We do some configuration. We set up the wifi unless you know your USB tethering and want to do that. WiFi is reasonable enough to set up, tethering by USB is still immensely useful if something goes wrong so on a Pi Zero, Pi 3A+ or Pi 4 you should be able to take advantage of that.

In the firmware project we hit the keybored_firmware/config/config.exs to get our config configured out:

#..

config :vintage_net,
  regulatory_domain: "SE",
  config: [
    {"usb0", %{type: VintageNetDirect}},
    {"eth0",
     %{
       type: VintageNetEthernet,
       ipv4: %{method: :dhcp}
     }},
    {"wlan0", %{
      type: VintageNetWiFi,
      vintage_net_wifi: %{
        networks: [
          %{
            key_mgmt: :wpa_psk,
            ssid: "Kontoret",
            psk: "underjord"
          }
        ]
      },
      ipv4: %{method: :dhcp}
    }}
  ]

#..

# config from the nerves UI guide
config :keybored_ui, KeyboredUIWeb.Endpoint,
  url: [host: "nerves.local"],
  http: [port: 80],
  cache_static_manifest: "priv/static/cache_manifest.json",
  secret_key_base: "HEY05EB1dFVSu6KykKHuS4rQPQzSHv4F7mGVB/gnDLrIu75wE/ytBXy2TaL3A6RA",
  live_view: [signing_salt: "AAAABjEyERMkxgDh"],
  check_origin: false,
  render_errors: [view: KeyboredUIWeb.ErrorView, accepts: ~w(html json), layout: false],
  pubsub_server: KeyboredUI.PubSub,
  # Start the server since we're running in a release instead of through `mix`
  server: true,
  # Nerves root filesystem is read-only, so disable the code reloader
  code_reloader: false

# Use Jason for JSON parsing in Phoenix
config :phoenix, :json_library, Jason

#..

For the WiFi I suggest you change your SSID and PSK in the configuration to match your actual WiFI. OR, and this is certainly an option, you change your WiFi access point to have the same credentials as mine. I’m saying, you have options.

The later part is essentially copy-pasted from the Nerves UI guide. Keep your secret key base very secret, don’t betray your signing salt, use this for fun not function. The reason we set up the full Phoenix config in here is that the config in a dependency, such as keybored_ui which we treat as a dependency, is not inherited by your application. We need this stuff in here or Phoenix has no clue what it should do.

Back to keybored_ui and run the following:

export MIX_ENV=dev
export MIX_TARGET=host
mix deps.get
mix assets.deploy

This should build the static assets we need for the firmware project. In keybored_firmware we run the following snippet. You can prepare by putting the SD card in the card reader. Please note the MIX_TARGET and match it to your intended hardware device:

export MIX_ENV=dev
export MIX_TARGET=rpi0 # or rpi3a, rpi4 or what matches your hardware
mix deps.get # Downloads the appropriate system
mix firmware # Compiles the firmware
mix firmware.burn # Burn to the card, if you have multiple devices it asks

After burning. Shove the card in the Pi. Give it power and within a little while it should go on to the configured WiFi or show up via USB tethering if you do that. You should find it at http://nerves.local which works on everything except Android devices because … boooh. If something went wrong tethering is your best bet. You can reach it via ssh nerves.local if you have a network connection to it. Otherwise, reburn? I don’t know, troubleshooting is out of scope. Poke me in the #nerves channel on the Elixir Slack if you want.

If you plug a keyboard, mouse or other input device into this Pi it should let you steer the dot via arrows or pointer events, it should show the events captured and give you a sense of what LiveView could do for you. If you want to make code changes that’s surprisingly simple. Edit the code.

mix firmware # Recompile
mix firmware.gen.script # Generate an upload script (only necessary once)
./upload.sh nerves.local # Upload new firmware over SSH

I think that’s it!

Try LiveView, try Nerves, try ‘em together. I enjoy them both immensely. Knock yourself out. If you want more updates about Nerves, get the Nerves Newsletter. If you want more of my shenanigans, get my weekly no-tracking newsletter.

If you have questions, thoughts or more of a comment really. Let me know at lars@underjord.io or on Twitter where I’m @lawik@hachyderm.io.

So he essentially said, I’m interpreting here, that when it comes to deploying software to servers the documented manual steps for deploying something need to be the canonical reference. Then whether you build bash scripts, Ansible playbooks, Makefiles, Dockerfiles, Terraform, Kubernetes or something else to encode that procedure into something repeatable and scalable that’s a separate step. Having documented the process required to set it up means that there is an answer to the question: How do I get this running? An answer that doesn’t require you to parse the .yml files or grok Ansible roles and groups.

I feel like I can spring onward from that. I will also very likely not land on Kubernetes as the answer for anything that I’m trying to do with deployments. This still seems applicable to me. All of these deployment tools are abstractions for systematizing and automating procedures. How much abstraction you need should reflect who you are and what you are trying to do. Given that we create this documentation for the manual steps to set something up. Then when we are trying to work with a script of some sort that automates the process we can at any time return to that source of truth and see why any particular part of that process is there.

Some might say that a Dockerfile can be that step-by-step description. Typically I rarely see a Dockerfile capture the “why” in the way documentation typically does. There are also concerns that are mostly about Docker that get enshrined in a Dockerfile. There are things that are done a certain way in a Makefile because that’s the most convenient way of describing it in that DSL. An Ansible playbook has a ton of potential concerns that are not about describing how to deploy your application but rather about how to deploy it on particular infrastructure.

So if I write up what steps it takes to get an application running for production in the README and make sure those steps are working, by testing them myself, I can then take whatever automation tool I prefer for simplifying things and translate that documentation into that tool.

One of the big points he made in that conversation is that if you don’t figure out the manual process you won’t have a clear explanation of the steps involved and if you don’t understand the thing you are doing you are going to end up in a mess. You’ll may even end up in the situation of “I don’t want to touch it, I don’t know why it works”. Kubernetes is that to me and I have no compelling reason to demystify it. I don’t operate at immense scale with a massive team. Regardless of the deployment system and underpinnings you work with, someone needs to know how to deploy your application, it needs to be well understood somewhere. At the human scale. Documentation is a good way of making that information transmissible. It is not a solution for immense scale, it is a solution for the human that needs to implement it at scale.

I’d be happy to learn some k8s if I have a good reason to. One day I’m sure I’ll be paid for to work on something involving it. Hearing this view of starting with a solid foundation and only really building as high as necessary off of it was good. It was especially good to hear from someone that’s both highly regarded and deeply connected to these massively complex deployment solutions. Seems to me that he deals with massively complex deployment solutions because he deals with massively complex deployments.

Now I don’t want to perform all the manual steps myself. I certainly want to automate my deployments. I’d rather not introduce a ton of complexity to do it though. I think my ideal level of complexity is just above a bash-script. Somewhere in the range of a bash-script and an Ansible playbook. Maybe it’s time to get into Makefiles?

CI/CD isn’t exactly a thing that sparks joy for me so far. I’ve had a decent run at my current client with GitLab CI. It all feels like a Dockerfile which is sort of not offensively bad but not delightful either. I find them both easier to understand than GitHub Actions which are either extremely straight-forward or mind-bendingly weird depending on what I try to make them do. Once I have it set up it is very satisfying to see all the things I don’t have to do happen. The tools often leave something to be desired.

I’m currently considering setting up some boxes and running self-hosted things. That’s not in line with what Kelsey recommended. He spoke a fair bit about managed services. For me it is more about idealism, enthusiasm & Schrems II. I think I’ll go for GitLab because I find Git + CI tooling living together convenient. I think it could be more elegant with smaller tools, Gitea and some other CI tool perhaps, but the integration might just save me a ton of steps. If all my Git repos want CI anyway, might be a win to keep them coupled a bit more tightly.

What this means to me is, I should document the production deployment steps of my various Elixir apps properly, maybe even standardize a bit. Then I can consider automating a way to deploy them well. Right now I’m mostly doing a bunch of manual BS because I haven’t invested time in automating it and don’t love any of the options.

If you have suggestions on lightweight no-nonsense ways to ship things to a server and get them set up I’d be curious to hear and synthesize that input into whatever I end up doing. You can share your thoughts via email lars@underjord.io or on Twitter @lawik@hachyderm.io.

My writing is supported by a couple of nifty sponsors and they are looking for Elixir developers. If you are an Elixir developer or aspire to be one, please do take a look at the jobs for good positions with companies I’ve vetted.

Thanks for reading.

In a recent tweet a while back a member of the Elixir community expressed some frustration with how everything is all about LiveView. It led to some interesting and useful discussion and as someone who writes a fair bit about Elixir, Phoenix and such I have thoughts. Following on that discussions other people expressed further concern about the unreserved enthusiasm with which some of us have pushed LiveView and maybe even more, the PETAL stack. I doubt they were just thinking of me there but I’d be surprised if I wasn’t considered a bit of a cheerleader for it. So taking some of that very reasonable criticism and skepticism, let’s try to address that and also dig into some of the concerns about the LiveView hype.

The discussion can be somewhat uncomfortable if you are enthusiastic about LiveView. It is easy to get defensive and that’s not all that useful. If we look past that the discussion itself can be useful and should be very welcome. There was no drama in that thread I saw and it was all very civil. So lets continue along those lines.

My understanding

How qualified am I to write about Phoenix LiveView? Somewhat. I’ve seen all the keynotes, followed it since the inception and I’ve used it a fair bit. Many hobby projects, some production work. I’ve used it in anger and joy, run into problems and delights. So I think I can speak with some nuance on it. As I go into writing this my stance is that I find it useful, powerful and a very interesting approach for building web UI. I am enthusiastic about it. It makes some trade-offs that even I find frustrating. It is not best for everything.

Let’s line up some of the criticisms brought up around LiveView:

• It cannot work offline.
• Latency is a concern.
• Bandwidth and poor connections, are they screwed?
• It deviates from standard server-side rendering and will not by default fall back gracefully.
• It has been overhyped.

Offline

This is the major trade-off. If you need to make an offline-first Progressive Web App or similar you should either not use LiveView or only use LiveView for the parts that don’t make sense offline. There are potential ways of building on top of LiveView and the JS integration to deliver something that mixes this together but you’d be off in the wilds with a machete at that point. That’s not what is meant to do. There’s nothing good about this and there is no big point in dancing around it. The gains you get from shifting state and much of the interactivity to the server in terms of deduplication, security aspects and simplicity comes with the trade-off that it isn’t on the client.

I don’t expect this to change but I could see efforts like Lumen (Elixir to WASM compiler) or Gleam (targets the BEAM and JS) make it potentially possible to build some shared state management code for both client and server but that’s a far ways off if it ever happens. I think for now we should accept that this is the major trade-off we make when using LiveView.

Latency

LiveView communicates near-realtime events and state changes over WebSockets and that means I can do things like validate as the user types. Compared to a client-side frontend this means we have more latency concerns for similar validations. Round-trip latency affects every update and if we’re in Australia talking to the US we’ll see ping times of around 250ms which is noticeable. Bad internet weather can cause the same symptom. This is a good case why LiveView really isn’t meant for games or other latency-sensitive applications. It may be annoying that your validation takes a quarter of a second but if you are playing an action-oriented game, that’s often considered unplayable.

Now LiveView doesn’t prevent doing client-side validations and interactivity so for smaller operations you can cut that latency by doing things on the client. However, part of the idea of LiveView is that it reduces the amount of JS you need to write. So saying you need to do client-side JS is a bit of a cop out. Now for actual validation against server state and more serious interactivity LiveView is the same or better as talking to an API using a client-side JS framework. Going over WebSockets should reduce latency compared to most HTTP API calls. The BEAM is good at consistently keeping reasonably latencies so you shouldn’t run into many random latency spikes from the runtime.

The latency concerns are warranted if you have a lot of work you could do client-side. Whether you should be doing a separate JS frontend or use LiveView JS interop is really up to you at that point. The latency of LiveView should typically be lower than doing things over a full HTTP requests and responses though since LiveView runs over an open WebSocket. This also lets the server trivially push events and state. So I’d expect typically lower latencies than your REST-driven SPA for anything that requires a server roundtrip.

Inclusivity and Poor Connectivity

There were also concerns expressed about how it impacts poor connections, bandwidth usage and such. José Valim adressed some of this in the Twitter thread. LiveView sends as little data as it can figure out how to by initially calculating what parts of a template can change and then only sending diffs when state does change. So it would typically send less data than a normal JSON payload to achieve a change. It does of course send more data than performing the operation on the client. Assuming that the javascript code required for the client-side operations is not huge and a cause of bandwidth issues itself.

I will say, I don’t know the experience of using WebSocket webapps heavily on poor connections. Being stateful connections I imagine you might drop them due to your network being dodgy. LiveView should recover that pretty well but in that sense it cuts down on overhead by setting up a persistent connection but you might be paying a bit in fragility. WebSockets are more fragile in comparison to HTTP requests that are typically just request, response, done. This could hit any system that leans on WebSockets though, not just LiveView and I’ve not seen any real issues using LiveView apps while out and about on cellular data.

I think it is a sensible trade-offs for many modern web apps but it is always worthwhile to consider your audience and whether they might be negatively impacted by your technical choices. My feeling is that LiveView does more by default to keep things lean and efficient than most web apps do when actively trying, just by virtue of having the diffing in place to minimize payloads.

Graceful degradation

I don’t know if this is more a criticism someone has brought up or something I have on my mind. How do we handle falling back to the lowest common denominator? How good a citizen of HTML and HTTP can LiveView be? How good is it by default?

LiveView will by default render a normal web page if it cannot establish the WebSocket. If it is a form it will not work unless you put extra effort in. So most likely readable, not interactable.

I’d be interested to see a library that does some nice work to essentially create a normal form endpoint based on the submit event handler. It seems like it could be pretty straight-forward. I’m not sure if that’s just optimism on my part. Making more nuanced forms work well would have all the normal challenges of making something work great both ways.

I think LiveView does a fair job and a reasonable trade-offs here. What I’d wish is that it had a perfect fallback strategy if JS isn’t available. My wish to see perfect solutions isn’t really LiveView’s problem though and I’ll own that one myself.

It seems entirely possible to do great graceful degradation but as always it isn’t necessarily easy or typically a very compelling return on investment. Unfortunately.

OVERHYPE!

The Elixir community often self-identifies as a pragmatic bunch. I think that’s mostly true. I saw some concern raised that LiveView was being swung around with rose-tinted glasses. I think that certainly happens. We have plenty of people who like the shiny thing. I’m a sucker for it personally. However, in fairness to the pragmatic streak, I think the discussion of trade-offs has been with LiveView since the very start. I think José and Chris have addressed it continuously as well. I see the community being enthusiastic but not unreservedly so. I do have my own sense of this hype being a bit much sometimes and I imagine I know where the feeling comes from.

The many conference talks, the numerous articles and the general air of active excitement you might see when following the community. So much of it is about LiveView. As someone actively engaged you’ve heard more about LiveView than you can stand by now, probably. Yet every day someone’s exposure to that new talk is their first glimpse of LiveView. I think we should keep talking about it, writing about it and sharing how we use it. I’m painfully aware that I’m mostly not the audience though. It’s not new to me, building an app in LiveView is not a thing of excitement to me, it’s just a thing I do, a tool I enjoy. Much like Phoenix itself, Phoenix Channels, Ecto and all these tools that I was so excited about are now just things I apply to do what I do.

I’m still very interested in Elixir, Phoenix, Ecto, LiveView and everything else. I’m excited about what they let me do. Depending on where you are along your path you’ll have exhausted your hype about a thing. There’s also the possibility that a particular tool just never speaks to you the way it does to others. That’s fine as well. In that case the enthusiasm is probably overbearing.

In summary

I don’t find LiveView to be overhyped as in being claimed to be better than it technically is. It is cool tech that achieves some interesting capabilities by making some well-chosen trade-offs. I think it’s great, a good design, well iterated, a compelling tool. Most importantly for me, it lets me do more building, faster and with fewer frustrations.

It has also been promoted a lot and it continues to be. Some will be breathless in their praise, some will be nuanced. I’ve definitely spent my time getting out of breath shouting about it. Hopefully this weighs in on the nuanced side.

Seems good to me but isn’t a panacea. Very hot take.

Thanks for reading. If I’m very wrong, very right or somewhere in the middle and you want to have words about it, good or bad, I’m available at lars@underjord.io and @lawik@hachyderm.io.

There are some companies that support my publishing and they want to hire Elixir developers, even aspiring ones. I personally talk to these companies to figure out what they want and what they can offer, so please do check the jobs page. It really does help.

You can try Nerves Livebook yourself with any Raspberry Pi you have.

Some of my Elixir content

• Overview of Elixir
• Why am I interested in Elixir? (an early one)
• Why am I still excited about Elixir?
• Building a startup to a $50 million dollar round (interview)

Projects mentioned

• Elixir
• Phoenix
• Nerves
• Membrane
• Nx
• Livebook

It covers setting up a server with Linode (taken as an example) and then installing a webserver on it. I also cover a few bits about securing a server and potential next steps.

I hope it helps.

Links

• Iptables
• Ufw
• Creating a sudo user
• Disable SSH password login
• Julia Evans on DNS

Next step should be to show you good folks some stuff about how I set one up.

To put my thoughts into context. I know an okay amount of Functional Programming (FP). I know it mainly from the Elixir perspective, so high level, dynamic. Not a lot of focus on types and much more focus on run-time than compile-time. Overall my experience as a programmer trends heavily to high-level and dynamic (I went Perl, PHP, Python with some JS mixed in). I do have some experience poking around more type-heavy languages like ActionScript (for Flash, I think they got really OOP and typed around 4), some C# with .Net, some Java for Android and probably some I’m forgetting. But mainly dynamic stuff.

For this project I was mainly brought in to bring the Phoenix code up to snuff and get it production-ready, secure and at all deployable. One of the big things I discovered was just how prototype-level the server application was at this point. I’m not privy to what the plan was originally, the original team is not involved any more, but the charitable read is that the server was a proof of concept when I was brought in. You can envision the frontend as a domain-specific multi-user collaborative document editor of some kind. It does a bunch of fancy UI and updates a data structure. The way it was being persisted was essentially by packing the document state into a big chunk of JSON and sending it from Elm over Phoenix Channels as a “document_change” event. This event would then be distributed to every other user connected and also saved to file. File writes were serialized using a GenServer which took the new data and overwrote the file. A user checks a checkbox, they update their local document state, the ship that document to the server, the server writes it to a file and sends it to everyone else.

You do not have to have read Designing Data-Intensive Applications by Martin Kleppmann to know that this can cause issues. As it happens I was in the midst of reading the book though which made it all the more amusing. So if Bob and Ellen both check different checkboxes at essentially the same time only one change will actually be kept. The last one. And state will flicker back and forth as a result. To make it much more reliable I would need to break apart the individual operations and make sure that they could happen separately. We could deal with some contention if multiple people changed the same thing, but we didn’t want someone renaming the document clobbering another persons carefully written note and vice versa. So we didn’t need CRDTs or OT for full on collaboration. But we did need more granularity.

I figured out a way to structure this on the Phoenix and Elixir side rather quickly. I normalized the data model with Ecto and backed it with Postgres. Nothing fancy, very straightforward, typical relational database fare. Then I spelunked the Elm codebase to find which change operations actually occurred on this document. That was well structured and I found a type that described all of them. So I stubbed all of them as command events in the Phoenix Channel and made them all throw exceptions until implemented. The concept was that a command could be sent in, we’d update the data and then Phoenix would send out that it had committed a command for all connected users. Since the logic for applying the changes already existed in the Elm code on the client side I’d essentially be able to re-use that for updates from other users. I’d have to mirror it correctly on the server but that seemed tractable (and was, eventually). If a command failed for some reason it would get a rejection event in response and wouldn’t be broadcast to other users.

So I had to create JSON commands from these Elm types, produce some new Phoenix Channel events and receive some new ones as well. This seems like it should be very achievable. Maybe I’d even be able to be clever about it.

I spent two weeks more than I expected to get this done. Fixed bid though, don’t worry about my client, they did fine ;)

To get it done I had to accept a few things as undeniable truths (within Elm):

• The Compiler is Right
• The VS Code plugin is sometimes right, sometimes wrong and often breaks, when in doubt run the compiler
• Being Clever is Forbidden

I was trying to be clever a fair bit and Elm fought me all the way. Elm does not allow you to do side effects, it doesn’t take kindly to unspecified behavior and it doesn’t really do generic code very much. Especially not with your JSON.

So I had an easy time finding the changes I wanted to turn into commands. Figuring out how I could hook into it gracefully and defer application of the change until committed was significantly more difficult. A big thing was my complete inexperience reading the language and just how resistant it was to me hacking my way to success.

I was recommended the Elm Guide which is a good starting point for getting going with Elm. It is not a good starting point for getting deep into an existing, large, Elm codebase. Or it wasn’t enough for my needs at least. There were lots of |> which felt familiar, <| which I could mostly grasp, \foo -> which I’d never seen and so much andThen for both Result and Maybe. I had to get into “the continuation passing style” this particular code used for certain JSON operations, some specialty Monad things they’d implemented because apparently Result was not sufficient and some syntax I’m still not sure about.

There were choices made in this code way back that I’m not really sold on. I can’t get into the details but I think that helped make things a bit extra dense and inscrutable at times. However I think the nature of Elm is that it is simple, much like I’ve heard Go is simple, and the combos of strict, simple and explicit tends to lead to a lot of code. Not verbose code, Elm is quite lean in many ways, sometimes even terse, but you end up producing a lot of code. Since there are no side-effects to speak of the code is rather easy to grok and even change parts of in isolation. At least once you can read the syntax and common flow well enough. Getting into it I felt like I was the victim of too clever programming. Some of it was/is but much of it was just me being entirely not used to living with Result and Maybe. There is no null except the very explicit Maybe and there is no way to disregard a failure without actually properly dealing with (or defaulting on) a Result. There are very limited options for saying “I’ll do this part later”, “this will never happen” or “I don’t care about it”. You application has to compile.

This is all very unusual to me.

It is also very powerful. I’ve discussed types and typesystems on assorted BEAM Radio episodes and I don’t want Elixir to be like this. I’ve also discussed types specifically in this Regular Programming episode. I’ve been very resistant. And I mostly still am. I will say that Elm makes an interesting case though. A pragmatic use of strict Functional Programming with all the types, all the time. I bet it does a cleaner job than TypeScript + React/Vue/Angular for the same domain. With some asterisks where those frameworks have a faster path to doing dirty things and Elm requires that you formalize your dirty boundaries. But that’s what people want by switching to TypeScript right? Anyway, Elm has definitely let me see types more for the tool they are and what they can offer.

I still don’t agree with people who go all in on a dynamic language and go “this is great!” and shortly after “we need types!” because I think trade-offs are a thing. It feels close to people who discuss writing tests or test coverage as undeniable truths of things that must be done otherwise software is objectively bad. Or people that want a rewrite in Rust. Or that consider Erlang/OTP irrelevant now because they could “just be reimplemented in Go/Zig/Rust/Java/their-preferred-language”.

The most satisfying thing with Elm is when the frustrating strictness lets up because you’ve satisfied the compiler and logic itself, suddenly things work. Once I’d bitten the bullet and implemented a ton of decoders and encoders, handled Maybe and Result more than I ever wanted, then suddenly the parts I’d implemented worked. And then it was a matter of repeating through all the different commands. Fixing some nuanced behaviors. And it worked really well.

Since this frustrating, learning-heavy, start I’ve become fairly fluent at reading and writing Elm. I’ve built a number of frontend-centric features for this product, revamped UI and just written a ton of Elm. It is still the case that the amount of code can become quite unwieldy, it grows vertically rather quickly. That is unless you refactor it heavily to flow in pipelines and if you do that you get a fast proliferation of functions which explodes your understanding of things across many functions. And still a lot of vertical text, just elsewhere. So designing software and making the right trade-offs for source code that is easy to work with remains a challenge. It has a particular flavor in Elm but it is the same problem as always.

I’ve also introduced four developers that are fairly new to Elixir entirely freshly into Elm. They definitely found the length of a given module frustrating as well. I will say that their process of “this is inscrutable” over into “I guess I just do this and that, and then this” was very similar to mine. That put my own frustration into perspective and has probably eased my view on both learning the language by immersion and the code base itself.

I bet we’d all have had an easier time learning Elm on a green-field project. Now we had to learn all of it at once to be able to read what we were doing. The code already contained everything from the simple to the intermediate to the advanced but nothing looks like the basic examples from the guide because everything is more nuanced in the messy reality of an actual product. We all had to absorb all of it or we’d have large gaps in our understanding. There are parts I still haven’t dug into to be fair but I know how this code works and can work on it comfortably.

The compiler is very helpful almost all the time. It can really indicate very well what part of the code is currently incorrect. Now that can still be confusing because types and type aliases can get wild.

The VS Code dev tools have given me so many parsing errors, crashes, delays and mis-indicated errors that I essentially only use it as a soft indicator, my dev server tells me if there is a real issue. Unfortunately the parsing errors often break go-to definition and find references which are incredibly powerful in navigating this codebase. I get that it is an open source community effort and it is super helpful but I can’t say it works very well on this codebase. Large files and deep nesting seem to be issues on their tracker already so hopefully it will improve.

The best editor tip I can give when your files get long and you need to work in multiple locations is to split your editor so you are actually editing the same file in multiple panes/buffers. That’s been critical for me and I think the pattern of long files is pretty common in Elm, I’d be surprised otherwise.

A thing I missed is a quick reference of all operators and other assorted syntax where I could just look up “what does this mean?”. I usually found it on some nice helpful post someone had put together, probably from the Elm team. But I haven’t found something I’d call a reference manual to the language. There’s the guide which is useful but not very comprehensive. And then the documentation pages for the standard library and community packages are quite fine. And over time I’ve bridged most of the missing pieces.

A thing I like is that much like Elixir the Elm community is smaller than mainstream langs. This means it churns slowly, it also seems to have the quality of Elixir I like where many libraries don’t change much. If it worked three months ago odds are it still works. Maybe there’s something new you’d get by updating, but probably you won’t need to.

I don’t think Elm as a community is something I’m into and I’m not feeling myself getting deeply invested in this language and ecosystem. But I definitely would consider using it again and now I have it in my back pocket as an option. It also makes me a lot more curious to try Gleam for certain types of work. Gleam is a lang with types on the BEAM. It can also compile to Javascript. From reading the Gleam introduction it felt quite similar to Elm but also a bit more familiar as someone coming from Elixir. I really should try it sometime. Curious if it works well for writing matching backend and frontend code and if you could do some interesting optimistic UI or offline-friendly LiveView with that.

All in all. Elm has been an incredibly useful learning experience to me. It also shows just how tight and neat something can be when it is heavily designed for a tight purpose. While Elm concepts could move outside the frontend space it feels like a DSL for building web frontend apps. Capture interactions, update state, render a representation of the state. Tightly and reliably. Making web UI entirely deterministic.

However, it also highlights that this isn’t entirely true. There are a lot of odd kludges you run into to handle some of the nuances of web development. There are things Elm doesn’t know about and handling some things via Ports is the least convenient part of Elm. But it is also the trade-off that makes the magic happen. Web application development is an inherently complex thing and there is cost to simplifying parts of it. I think Elm does a good job with that tradeoff. I haven’t seen it entirely fall down yet.

• Easy: Making an interactive UI with many forms of interactivity that reacts in near-realtime and behaves consistently.
• Intermediate: Rendering complex SVG UIs that interact heavily with mouse events from a complex state.
• Unreasonably complicated: Focusing a text input on first render.

You win some, you lose some. Web development.

If I’m very wrong about Elm, or very right, or you have something to add or discuss you can reach me at lars@underjord.io or via Twitter as @lawik@hachyderm.io. If you want more of my writing my newsletter is weekly and gets into a lot of nuanced tech career stuff. Signup below, it don’t track.

This is a guest post to Underjord where Alex McLain, who I know from the Nerves community, lays out his ideas about how Elixir can fill a special role in software teams. Thanks to DockYard for supporting our work on this piece. They are a digital product consultancy with deep roots in the Elixir community and ecosystem. You’ve seen them support Phoenix, ElixirConf as well as this blog and several others. Let’s hear from Alex.

I started my professional software engineering career working with low-level control system hardware for commercial audio/video system automation. These systems contained a central controller unit that orchestrated the other devices in the system, therefore uptime and resiliency were important properties. That led me to search for programming languages that were designed with an emphasis on these properties, in which Elixir was one that rose to the top.

During this time I had also been using Ruby in areas not sensitive to system performance, due to its developer friendliness. This reduced the time it took to deliver a control system project. This led to an opportunity to do web development in Ruby, so I made the transition. While working on an enterprise IoT platform built with Ruby, we started running into performance and cost challenges at scale. Recalling that Elixir could be a good solution to these problems while maintaining the developer friendliness of Ruby, we transitioned part of the system to Elixir and were able to reduce the compute resources required for the platform. The culmination of my experience at this point drew me to IoT consulting, with a focus on utilizing Elixir across the stack for developing both software and hardware.

It’s probably no surprise that web-based platforms running at scale are complex. Today more and more devices that were once standalone are now “connected”, utilizing cloud and/or edge computing to supplement their functionality. This means that products are no longer confined to the bounds of a website or a widget. A modern product can consist of hardware, a cloud platform, data science, and web/mobile/desktop applications to deliver the full user experience. Product complexity is increasing substantially. The obvious approach is to staff engineers from this wide range of disciplines onto the product team. However, the budgets for these connected products have not necessarily grown proportionally to their complexity, which can result in fewer resources than ideal to create the product. In practice this can manifest into situations like in the opening story: The critical path for feature delivery can depend on specific team members more frequently, which can not only affect their work/life balance, but also increases overall project risk. So what are some other approaches to this situation?

If you practice Agile you’re most likely familiar with building cross-functional teams, in which each member of the team is able to perform more than one role. If Jerry’s team had been set up this way, the release wouldn’t hinge on a single team member. Mark would have been able to complete the changes to the application without being dependent on Kelly. For projects that span a small number of disciplines, like frontend and backend, this solution may be enough. However, for more complex projects that include more disciplines like DevOps, firmware, native mobile, data science, etc., then it’s likely not feasible for every team member to be able to work proficiently across every discipline. There are just too many skills for any one person to learn. To overcome this challenge we need to look at cross-functional teams from another angle: the barrier to entry of each discipline, the amounts in which these disciplines contribute to the project’s complexity, and team size.

Barriers to entry can be created when drastically different tooling is used across disciplines. One notable example is the choice of programming languages used, each which can have a substantial learning curve to reach proficiency. The greater the number of languages in a project, the more complex the project becomes, and the more difficult it can be for someone to work in different areas of the project. For example, a typical IoT project could utilize at least four different programming languages: Elixir, JavaScript, C, and Bash.

A strategy to reduce this barrier and reduce complexity is to find tooling that can be utilized by multiple disciplines. In this case, let’s say the team decided to use Elixir as their primary programming language across disciplines. Elixir’s Phoenix web framework can be used for the frontend and backend, Mix tasks for DevOps, and the Nerves Project for firmware. Given that the community is actively expanding Elixir’s capabilities, Nx and Livebook are relatively new tools that could start bringing data science into the stack as well. Now all of the disciplines share a certain amount of baseline knowledge with each other, and can understand each other’s code. There may still be a need for traces of other technologies, but stepping into another discipline is more fluid.

Fluidity comes with other benefits as well. If team members can move across parts of the project easier, this breaks down silos within the team that could have a tendency to form otherwise. Each section of code can now have more eyes on it, increasing the opportunity for engineers to help each other, and improving code quality. This also spreads project knowledge across more of the team, reducing the amount of information that could get lost in translation, and enabling more team members to be able to help solve issues that arise. If your product management methodology is to deliver features in vertical slices, this can also enable engineers to work across more or all of the slice. From a resourcing perspective, if more team members are cross-discipline, the project may require fewer resources to achieve the same output compared to a project resourced traditionally. This can also be beneficial for startups or those trying to get to MVP quickly with a small team.

For companies that maintain more than one product or are involved in consulting, not only can it be beneficial to find tooling that works across multiple areas of a given project, but that also works across multiple projects in the company. From a technical standpoint this can be an opportunity to leverage code reuse, allowing new projects to start on second base. From a staffing perspective it can reduce the friction of rotations between projects.

Although we’ve covered many advantages of this philosophy, there are situations it may not apply well to. For example, if a mobile app is required, there is currently no well-established way to do this in Elixir. Inevitably other technologies like React Native and JavaScript would need to be brought in, resulting in less reuse across disciplines. Another challenge is that engineers who are fluid across disciplines need to have a broader skill set and be able to keep more context in their head. One way to help less experienced engineers in this environment is to start them off in an area that suits their strengths, and support them in expanding throughout the project from there. However, some engineers prefer to narrow their focus rather than to expand more broadly. Finally, diminishing returns can be encountered with larger teams, especially when each discipline has its own squad. At this scale it can be more effective to run focused squads rather than having them be jacks of all trades.

In conclusion, modern products are increasing in complexity and requiring more engineering disciplines to be involved in creating and maintaining them. Cross-functional teams that find technological reuse across disciplines can maintain an advantage in this environment. I’ve personally found Elixir to be a great programming language for this due to its solid foundation, versatility across several disciplines, and the community’s drive to keep expanding it into new areas.

Big thanks to Alex for writing this up and letting me share it. If you want to reach him you can find the user amclain on the Elixir Slack or reach out to alex (at) alexmclain.com.

If you have thoughts about guest posts as far as Underjord is concerned you’ll find me as usual at lars@underjord.io and on Twitter as @lawik@hachyderm.io. Please note, I don’t take random solicitations about guest posts. That’s weird.

Being able to prepare and start livestreams, setting them up with LiveView, giving meaningful progress to users. Things are fun when you have near real-time events :)

The project is called evision and I had a great time on stream screwing around with it :)

It was a quick job so I want to improve it in a few ways.

• Make it not need to pull the new state from lights before sending commands.
• Pull state in the background occasionally to keep relatively up-to-date.
• Always fetch status after sending commands.
• If there is time, improve the lighting on the keybow to represent the state of the lights.

Spoiler, most of it got done during the stream :)

They made sure there was a chain of building blocks from RTMP to HLS (HTTP Live Streaming) and now I can actually set up a completely independent version of the Underjord livestreams. So that’s what we are going to be looking at this stream.

What parts are there? What considerations and concerns go into it? How does it work? How does it scale?

I think we’ll be able to show it MVP style on this stream as well as discuss where I’d like to take it and maybe even hack some towards that.

Note: This livestream happened back in November, I’m a bit behind on processing those livestream recordings to the site. There has since been a follow-up to this one as well. They’ll all arrive here eventually but they are on the YouTube channel already.

The winner’s have all been notified and codes for the ebooks sent out. And they are (names as permitted):

• From the old guard of newsletter subscribers, since many, many issues back, Marcelo!
• New newsletter subscriber, ryoung786!
• Podcast subscriber for Regular Programming, Manuel!

Congratulations to our three winners and better luck next time to everyone else. Thanks for giving it a shot. I’ll note that the pod-pool and the old subscriber pools were the biggest chances of winning. So maybe that’s a hint to get in on that newsletter now. I fully intend to keep offering opportunities to my existing readership as I very much appreciate their attention.

If you have question, you can as per usual reach me at lars@underjord.io by email or on Twitter @lawik@hachyderm.io.

I care about making good software happen. Even if I’m not writing it. And good software is formed from good teams.

To help this happen I want to connect companies and developers in the Elixir space. I know and connect with plenty of both.

This is the first time I mention this particular offer to the public. If the interest is at the level it seems this will also be the only chance to get in at the initial pricing. If your company is ready to take finding great Elixir developers seriously, reach out at lars@underjord.io.

The Default - Contexts in Phoenix

The Phoenix generators are very helpful and if you are generating controllers etc. they will gladly help you generate a Context. If you check the Phoenix documentation you can see what’s generated by default. I’ll add that code below for reference. You can generate your own (this was 1.6.1) with mix phx.gen.context Catalog Product products title:string or get the same as part of a mix phx.gen.html call. The thing that is called a Context is in lib/hello/catalog.ex as the Hello.Catalog module. You can see the file here, I’ll strip out docs, aliases and imports to give a more skimmable thing and then we’ll try to say something about what it is.

defmodule Hello.Catalog do
  def list_products do
    Repo.all(Product)
  end

  def get_product!(id), do: Repo.get!(Product, id)

  def create_product(attrs \\ %{}) do
    %Product{}
    |> Product.changeset(attrs)
    |> Repo.insert()
  end

  def update_product(%Product{} = product, attrs) do
    product
    |> Product.changeset(attrs)
    |> Repo.update()
  end

  def delete_product(%Product{} = product) do
    Repo.delete(product)
  end

  def change_product(%Product{} = product, attrs \\ %{}) do
    Product.changeset(product, attrs)
  end
end

So a Context Catalog with only a single type of entity called Product to manage can look like this. If we’d add another schema of concern we’d get list, get, create, update, delete and change for that as well. So this is CRUD. I’m not insulting, that’s shorthand for Create, Read, Update & Delete. Plus list which is a common need and something called change which is a bit unusual and relates to making forms from changesets in Phoenix.

The Context is just a module. That is, it is not a required portion of making the Phoenix web framework do web application stuff. Many pieces you deal with in a typical Phoenix app are load bearing functional components. Functional as in “it performs a function” or “has a job to do”, not as in Functional Programming. Your Endpoint, your routes, your controller, your views and your templates are all fulfilling a particular piece of the puzzle. And for database operations your have Ecto providing a number of pieces that do work: Schemas, Changesets, Migrations and the Repo are all there to fulfill a certain functionality requirement. If you generate using mix phx.gen.html the Context will take a load-bearing position in your code, between the Phoenix Controller and Ecto. But it is not itself a required part of making the puzzle work. You can make those same calls from a Phoenix controller and that works fine.

I think this not-absolutely-necessary aspect of the Context is a common place for people to get tripped up. You are in there learning how to make the web app work at all. You need every part, except this piece that gets a bit fuzzy around the edges. What is it? What should I put there? I’m not convinced Contexts are bad. But they introduce design choices for the developer to make into a system that is otherwise very mechanical. Phoenix is full of design choices that are already made for you. Contexts recommend a certain approach but they don’t require any particular approach, there is not forcing function to keep you in line in any manner. While a Phoenix Controller works a certain way, routes work a certain way, LiveView works a certain way. Contexts are not nearly so tangible. You don’t use Phoenix.Context, there’s no there there in a sense.

And not being a mechnically required component doesn’t invalidate anything from being useful code. We would have to strip away a lot of abstraction if we only wanted to deal with the strictly mechanically necessary. For example, Ecto.Changeset is mostly a pure abstraction but it is mechanically pretty much required for operating Ecto effectively. The reason Phoenix is a nice web framework to work with it that it manages a lot of complexity for you, allowing you to opt in to it if you need it while keeping it mostly out of sight for most of your work. So that is a way in which the Phoenix Context sticks out compared to other parts of Phoenix. It is a less clear-cut piece, it is providing a software design for you.

So a Context is not required. Why did it get introduced? What is it for?

The Context is an attempt to provide order that your apps can grow with. It is an attempt to provide a generic way of encapsulating concerns which could make managing your code-base as it grows easier. You can also see it as the bridge between HelloWeb which is your Phoenix code and Hello which is your Elixir code. Essentially Hello.Catalog being your interface into Elixir-code, devoid of web concerns.

I get why there has been a bunch of discussion and contention about whether Phoenix Contexts are good. It is a design and it is a design of “your code” and in that way it is quite different from the design of other parts of Phoenix. It goes a step beyond a minimal generated controller and tries to provide you with some scaffolding for what I imagine is a suggested best practice. It is not a suprise that people have opinions on good software design.

One common criticism of Contexts are that they are a somewhat clumsy abstraction on top of just calling Ecto. Another is that they are a leaky abstraction. So what is this about?

I think we can lean on Saša Jurić to touch on some of that.

Refining Contexts according to Saša Jurić

“The problem with the “official” approach is that the interface concerns leak into the core, blurring the border between the two layers. This defeats the main goal of the design: clear separation between the layers."

This quote is from this worthwile article on how Saša and Very Big Things have worked on making their Elixir code maintainable.

You really should read it to get the full perspective but the key point about leaking here is concerned with this:

  ..
  def create_product(attrs \\ %{}) do
  ..
  def update_product(%Product{} = product, attrs) do
  ..

These function heads indicate that they take any map of attributes to turn into a product. Usually these come straight from an API, HTML or LiveView controller and contain a map of string keys pointing at varyingly typed data. It is then, inside the Context, turned into proper data via an Ecto Changeset cast, validation happens and can fail. So it is a thin layer on top of the Ecto Changeset API and as an abstraction it doesn’t provide a lot of clarity about what you can expect to do and still have it work. If we send in %{"name" => "Foo"} that’d be correct, %{name: "Foo"} would also work fine, %{"title" => "Foo"} would only fail if the changeset or database requires the name field to exist.

It lets us send in anything and sorts it out. Which is convenient when you need to change things but it doesn’t impose any real separation between Ecto and your Controller or LiveView. And maybe you don’t want those to be separated?

So Saša’s article suggests strictening the interface of your Context by using something like def create_product(name, price, description) do instead where you break apart the attributes and you can actually even guard the values explicitly. So do you abandon Ecto Changesets for casting and validation? How will you render forms conveniently? No-no. The suggestion is to add a separate schemaless changeset in the Controller which knows how to validate and maybe even render the form. And then you wall off using the database parts of Ecto using the Boundary library.

So this takes what is a bit of a sticky layering where Ecto-friendly data structures flow in and out and separates it into something where you get well-structured input. You can have more meaningful typespecs this way as well. The article doesn’t say specifically because it focuses on slightly different things, I imagine the Context’s functions still return some Ecto data structures such as a Changeset on error and an Ecto Schema Struct on success. You could of course strip out the Ecto Schema Struct and reshape that data to be more neutral in this layer as well.

Getting rid of Contexts by listening to Chris Keathley

“I’m pretty sure Contexts are not a thing”

I don’t believe Chris Keathley has put his ideas on Phoenix Contexts in writing. I’ve picked them up from his conversations with Amos and occasionally Anna on the Elixir Outlaws podcast along with when we had him on BEAM Radio to speak of heresy. It was a good episode, he really put us to work discussing design.

Note: Chris has since ducked out of public activity in the Elixir community, if you disagree with my interpretation of his ideas, reach out to me, not him. I’ve found his thinking and ideas immensely useful through the years and that’s part of why I share my interpretation of them.

Brutally paraphrasing I understand his points to boil down to Contexts being a leaky and limiting abstraction. They leak Ecto concerns, both changesets and schema structs, but they don’t provide the power of Ecto’s full API. They also hide implementation detail behind a layer of abstraction where in many cases you’d benefit from having it clearly laid out at the call site.

Chris has mentioned the “call site” a number of times through the episodes of the Outlaws. The call-site is the place in the code where you decide to initiate a particular operation. Often in Phoenix this is a Controller or a LiveView callback. In wider discussions than the Contexts one he has mentioned that the call site is usually the only place that really knows what to do in the event of a failure. Contexts, as generated, don’t really change that, they just return what Ecto would. But if you abstract deeper than those generated contexts you might start running into that. So overall, I take his recommendation to be: give the call-site the capability to explicitly handle errors.

Beyond that he has spoken about using Ecto directly in controllers, building “fat controllers”. The idea being that the Controller action shows you what the entire operation does. Some Ecto operations, some error handling, sending an email to notify users. You’d see the entire flow in the controller. You’d know all the steps it takes by reading that page of code. It won’t require you to keep in your brain the context of additional Contexts. Does MyAccounts.create_user/1 send an invite email? Who knows. Does this?

# ..
case MyUser.create_changeset(params) do
    {:ok, user} ->
        case Repo.insert(user) do
            {:ok, user} ->
                case Mailer.deliver(MyUser.Emails.welcome(user)) do
                    :ok -> # .. render success
                    {:error, errors} -> # .. render errors
                end
            {:error, changeset} ->
                # .. render errors
        end
    {:error, changeset} ->
        # .. render errors
end

# ..

I’ve intentionally not made this super clean. It uses case statements and stair-steps heavily as a result. But is it unclear what it is doing? I don’t think so. Is it elegant? Not really. It is tempting to pack these things into abstractions and build neat functions. But I know there is benefit to being able to just read the flow directly. Many small functions can lead to a lot of jumping around and at a certain point threatens sanity in the same way that OOP inheritance does.

So this is very explicit and it is leaning on the necessary abstractions such as Ecto and a Mailer that both manage other systems. It doesn’t introduce additional abstractions that concern what we are trying to do, creating a user. I think this is important to be mindful of when creating a lot of nice abstraction layers. What is being made easier? What is being made harder?

Some kind of conclusion

I don’t actually care which one you use. I sometimes barely care which one I use. But when building a code-base which is intended to grow and be shared it is important to be mindful of what approach you are taking. The fact that multiple people with rather opposing positions see problems with the Contexts design as it is generated tells me it is definitely worth considering whether it really serves your purposes. It could also just be that it is a bit middle-of-the-road and as such not satisfying either end bell curve. When it comes to opinionated approaches I think it is helpful to know which opinion you lean towards. Mixing them is probably the worst approach. I think being consistent beats using the exact right thing in many cases. I’m partial to (my interpretation of) Keathley’s approach because it involves less code, fewer layers and actually defers the option of adding abstraction layers to a later point.

I think there is good stuff in Saša’s approach. It seems like it improves on the separation, makes the layering more meaningful and in that way fulfills more of the purpose of the original design. I especially like the honesty in suggesting you probably want a tool like Boundary to enforce the layering. It is very easy to shortcut through a layer like that and break through that design and having an automated tool tell you when you are breaking your own design is useful.

What other consequences are there? With the explicit in-controller approach you would likely only write one set of tests, for the controller. With the Context approach you’d likely be writing tests for both the controller and the Context. And with a thin controller they’d be very similar but likely not the same. Oppositely, if you provide multiple interfaces into your system, several types of API, webhooks, Web UI and they all need the exact same functionality, Contexts offer the potential for re-use which isn’t part of the explicit in-controller approach.

So what do I recommend? Try one and see how you like it. It won’t matter for a throwaway project, as the challenges here are mostly about growing and maintaining a code-base, but you might get a sense of how it is to work in that way. To me this is mostly lines up to be a division of values. Do you prefer more structure and more abstraction or less? This is a rather nuanced piece of software development and involves all the good topics around hiding complexity versus making the code clearly readable in a single function. Essentially eternal topics.

If you don’t know what to believe. You’re just trying to get your web app to work. I’d just as well roll with Contexts as generated. This type of nuance in design can be meaningful for teams and growing systems, I wouldn’t put it high on the list for a new learner. As far as making the web go brrrrr, it doesn’t matter.

Transparency note: This giveaway is supported by Manning, the publisher that published Elixir in Action. I haven’t been paid to do the giveaway but I also didn’t have to pay for the e-books I’m giving away. I think that’s cool of them. Ideally this also means that they sell some extra books, in which case they’ll likely let me give away more books. I’ll attempt to be very clear about how this all holds together and any privacy considerations you should be mindful of.

Privacy note: Links to Elixir in Action and Manning books in this post have some of those UTM parameters that let them know if I drove any traffic with this “campaign”. If you find that upsetting I’d suggest avoiding this post. I try to be clear about this stuff and I don’t sneak things like that in. If you have software installed that strips that stuff out, I don’t mind, it’s your browser.

Let’s give away some books shall we?

I’m giving away three e-book copies of Elixir in Action in this particular giveaway and I’ll divy them up a bit to be fair to old readers and new:

• One to any of my existing newsletter subscribers that decide to participate in the giveaway. You’ll get a notice about this in this week’s email.
• One to anyone who decides to sign up for my newsletter between now and until the end of the giveaway period. You are free to unsubscribe after the giveaway period.
• One to anyone who sends me an email with a screenshot of them subscribed to the Regular Programming in their podcast client before the end of the giveaway period. Include the word “giveaway” in the subject line of the email.

The giveaway period ends on the 17th of December when I sit down and pick winners at an arbitrary time. Get your submissions in before then.

Now if you are keen on some books beyond winning this I suggest you take a look at Manning’s other books and use the discount code nlunderjord21 for 35% off.

If you have some strong recommendations for books from Manning. Let me know and I’ll see if I like them and if I think I should arrange a giveaway of those. You can reach me via lars@underjord.io or on Twitter where I’m @lawik@hachyderm.io.

Special thanks to DockYard for supporting my work on this piece. They are a digital product consultancy with deep roots in the Elixir community and ecosystem. You’ve seen their efforts supporting Phoenix and ElixirConfs and with their support I can bring you more stories from companies and people using Elixir. Let’s get to it.

Vic.ai has a product that makes a lot of sense to me. My wife is an accountant (when she isn’t editing podcasts, growing vegetables, etc) and I’ve run my own small business at least three times. So I’m familiar with the domain. I know how disjointed invoicing can be. It is a field which lives and breathes PDF files sent over email. The grand interchange format of a massive industry.

PDFs and photos of documents are not a standardized format as far as what data they contain while invoices are themselves fairly structured and formal documents. Vic trains machine learning models to automate the interpretation of these document files into a structured form and offers ways to verify, make corrections and pass the results on to your ledger. I can see how this can save a lot of time. For a startup this is an unusually lucid value proposition.

I think it is a remarkably good fit for ML. The documents are somewhat formalized and fit in a certain mold. An invoice for a particular region typically contains a limited set of features to look for. Your due date, invoice number, totals, line items, vendor information and such. But it is formalized for humans not machines, there isn’t a single template being used, so it is hard to automate as is. And here ML becomes a missing piece translating human-readable to machine-readable and feeding it into whatever your process is. A mix of OCR and ML.

Vic.ai hasn’t been the most visible company in the Elixir space so far. You are perhaps more likely to know of Marcin from his work with Asciinema which is a very cool tool for recording a terminal and replaying it. You should check it out.

I sat down to talk to Marcin about the use of Elixir in the company and before I had transcribed and edited our conversation the company closed a $50 million round of investment. So that little team is already growing significantly and will likely continue to do so. If you are interested in applying for a job with Vic you can reach out to me, we can talk and if it seems like a good fit I can give a warm intro.

On to the interview, which lands us smack in the middle of going from small-talk into relevant topics.

—

Marcin I know how it is. I used to run my own business as well.

Lars Oh, really?

Marcin Yeah. Before this. So this will transition us into the history of Vic because I knew Alex before, for a couple of years. I met him in 2015 I think. I left a Ruby on Rails shop in 2013 and I started my own company doing consulting gigs.

_(editor’s note: Alex refers to Alexander Hagerup, CEO at Vic.ai) _

From a colleague at that previous company I got in touch with some people in Norway and then did some projects for them. Then I met the guys behind the Gitorious project. A project similar to Github but self-hosted and open source. That project was later acquired by Gitlab.

It was through that chain of people I got to know Alex.

Alex had this idea for a mobile app he was building and he needed an API. So that’s how we got connected. We met and I did two projects for him. During this time I still had my own company in Krakow. So like you I was managing my own time, paying my own taxes and all of that. Until 2017 I think, when I had already done a couple of months of work on Vic.

Lars Was this app Alex was building the foundation for Vic?

Marcin No, so in 2015 he had an idea for an app that was supposed to be for sharing images with your closer circles. Something like Instagram but with friends and family rather than public.

I was building the backend. We worked on this for two years but nothing came of it.

Then around the time I was getting into Elixir and Phoenix, Alex asked if I wanted to build a site for a US startup he was building. It was for booking places at restaurants and venues to use as workspaces during off hours. Essentially providing co-working space when the business is mostly empty.

So this was how Elixir was introduced to Alex. He didn’t care much about the specific technology. That has been a nice thing working with him, he trusted my tech choices. He was never going “Let’s use PHP or Python”. I built the photo app backend in Clojure.

(Editor’s note: Alex would let the record show that he did care about the choice but trusted the suggestion :)

Lars So backing up a bit, and looking at your background, how did you get started with programming?

Marcin I was in seventh grade of primary school and there was a computer shop in my hometown. I already had an Amiga 600 that I was mostly playing games on. I went to that shop one day and they had a book, with a disc, on their shelf which said AMOS and it was basically the handbook for the AMOS programming language. So that was the Amiga version of BASIC.

So I went home and asked my parents if I could have it. They saw that it was programming, figured it was too difficult for me and didn’t want to waste money on it. They assumed it would end up unused on a shelf. So I went to bed crying because my career was in ruins already.

I don’t remember how I finally got it but I think it was a year later, so in eighth grade. So I started programming on the Amiga. After eighth grade they bought me a PC because I was going to high school with a focus on mathematics. So they got me this nice PC, an Intel Pentium at 100 Mhz or so.

So after initially just playing PC games. This was the summer of 95 or 96 and Quake was out, Duke Nukem was already there so I was playing all of these shooters. But after a year I got bored and wanted to do some stuff myself.

I was looking into Turbo Pascal. The IDE was super nice, F5 to compile, F6 to run or something like that. Everything was integrated nicely and it was simple. So I learned Pascal. I was also observing the demo scene, all these crazy people who do magic things with programming. I was hooked. I collected the demos and started reading the zines of the scene. There was a lot of good insight there, in the tech corner they would cover things like doing plasma effects in assembly or casting 3D points into screen pixels and such.

So I was reading that and then I started writing my own silly demos in Turbo Pascal. There were some pieces where that was too slow because I was trying to calculate a lot of pixels, 320 by 240 so over 60.000 of them and Pascal was not ideal for some of this stuff. So I needed to go lower.

I went into C and then hit similar problems and had to do some things in Assembly. And C has this nice ability to just write some assembly in-line with some macro. So I did that and then I was just jumping between languages as I needed them. C, C++ and my first professional job was PHP.

Lars So do you have formal training in computer science?

Marcin Yeah, I do. I have a master’s degree in computer science. So that happened in parallel to my explorations. I wouldn’t say I didn’t learn anything at the university but it was mostly complimentary.

Lars It sounds like you were learning quite actively outside of school.

Marcin Yeah, exactly. I didn’t have much trouble going through university, to put it lightly.

So then around 2003 I was doing PHP and making some websites. Then I learned Python and some other things. In 2006 I got into Ruby a lot and worked at a Ruby shop in Krakow where I spent 6 years. At the end of that period I was kind of burnt out by the complexity that’s hidden under the surface. The magic of that brings you misery while you are not paying attention.

So I was looking into various languages and some people at the company were looking into Clojure for a side-project. They brought it to my attention and that was my gateway drug into functional programming.

Although, if you look at my late Ruby code it was very non-idiomatic and was looking more like functional code. Very explicit, with explicit dependencies and immutable. So then I went into Clojure and it was amazing. It opened so many ideas for me and a completely different look at how software can be built. If you spend your whole career doing object-oriented programming, you never touch functional, then you may not know what simplicity means. It was like enlightenment for me. An “aha” moment.

I didn’t want to go back.

So I used Clojure for some projects, like the one I described with the API for Alex. I was aware of Elixir early on, back in 2014 or 2015. José Valim lives near Krakow and had showed up to some Meetups in Krakow and I was at one of those. He was showing off opening two terminals with different IEx sessions and connecting them as nodes.

At that time Phoenix looked like it was trying to please Rails developers and my personal feeling was “Nah, this is just another Rails and I don’t wanna go there”. So I had mixed feelings at that point.

Everything else looked great but I wasn’t sure about the web part and how it would evolve. Today we know it evolved into this amazing space of interactive, real-time solutions like LiveView and LiveBook among others. But as it was I decided to observe from a distance. Around 2016 or so I started playing with it.

Yeah. So I read the Programming Elixir book by Dave Thomas and then the Programming Phoenix by Bruce Tate (Editor’s note: not to forget Chris McCord & José Valim). After that I was like, oh man, this is it. The language, the whole platform, it solves so many problems I’ve observed and dealt with in the past. How is it possible that nobody knows that this gold is here?

Lars So did Vic start with Elixir?

Marcin When I joined Vic.ai there was already a guy working on some AI parts. So we ended up having two areas. The AI parts with all the OCR, document processing and machine learning. That part uses Python.

So Alex started the company with the guy doing the Python work but they needed an API and many other missing pieces. So when I joined we discussed what we should use for the API.

I suggested that Elixir and Phoenix would be a great fit for it. It wasn’t hard to convince them, they trusted me with this choice. We didn’t see a need to rebuild the Python code in Elixir because we didn’t need to closely couple the AI part with the API.

So I built the API from scratch with Phoenix. There was nothing before that.

Lars How long did you spend working on the Elixir side more or less on your own?

Marcin Over two years. I believe I started in 2017 around June and at that time it was still consulting, not as an employee. And late 2019 we hired three Elixir developers, my first Elixir colleagues.

Lars Did they already have Elixir experience when they came on?

Marcin Yeah, yeah. They did. We explicitly looked for people who had experience. I posted on ElixirForum and we had many responses and lots of great candidates. So we spent quite a while doing interviews. There were so many candidates and we could only hire three. With how good the pool was we could have easily hired three more if the company would have been able to at the time.

Lars How was it like to bring people into that Elixir code base? A two-year old code base has a lot of stuff in it already, especially one under startup pressure.

Marcin Right, especially one created by a single person. Well, so the onboarding, I remember, was hard work for me. It was mostly challenging due to the fact that I now had to share this codebase and what is in my mind with the external world. Previously I’d only needed to share relevant details with the Python developer and they weren’t necessarily interested in the Elixir codebase itself.

So I had to go from not documenting anything and knowing how everything connects in my head to be more explicit in every aspect of it. So for me it required a lot of time. But as for those colleagues they were already experienced when we hired them, they didn’t require any training. They just needed many introductions to various pieces of the system and corners of the codebase.

So one thing we did was that they all started during the same week. So we brought them together in our office and we had an onboarding week where we spent a whole week on that. Explaining how things are connected and how all of it works. And then we fixed one or two bugs together. It was great. I think it did wonders to get them up to speed. Just to be able to be in the same room for a week. (Editor’s note: This was before COVID-19, this is being published during the pandemic and in some places bringing people into the same room is still weird so note that this was normal at the time.)

Lars So this was about two years ago. And you’ve kept bringing in people, especially recently. How would you compare bringing people into an Elixir codebase and other language codebases you’ve worked with?

Marcin That’s a good question. This is not a scientific answer but I feel like with Elixir, when they ask questions about why things were done this way where do I find something, I can just point them to the right part of the code. A specific module or area. Then they could just read and understand it due to the simplicity of the language and explicitness. Most issues with understanding have been where we rely too much on macros.

There was less back-and-forth because people can figure things out on their own because the code is more understandable. There’s not a lot of hidden things and no chains of parent classes bringing in new methods and different behaviors.

They can figure out the context on their own easier. It could just be that the guys we hired just did really well. They are fantastic developers. So that helped. But I think the aspect of having less magic and being easier to understand made a difference.

Lars Have you experienced any difference in recruiting and hiring for Elixir? Has it been harder or easier than your other experiences?

Marcin It was much easier. We had so many good candidates. Actually, when we did hire we were also recruiting for the Python team and me and the guy who was in charge of the Python system were doing interviews together. There was no-one else who could pair up and discuss the candidates.

We had so many Python candidates. But the average experience level was lower and we had to spend much more time determining who could be a good candidate.

Lars What have been the best parts of working with Elixir, for the product and the codebase?

Marcin The one I see the most is that Elixir enables much, much easier refactoring of the code. Because everything is a module, which is a bag of functions, nothing is hidden. So when refactoring you just move things, cut and paste functions around. And it always just works. In most cases the compiler can tell you if you broke something. I feel like refactoring in Elixir could not be easier.

When changing configuration, or more in-depth OTP stuff, dynamic runtime concerns and supervision trees then you have to pay more attention. But just moving functions around is easier.

So, a fun example of this, as our codebase was growing I missed a feature from Clojure which was preventing cyclic dependencies. Initially I was bummed about it but later realized that it is probably to be interoperable with Erlang and hot code updates. That would make it hard to enforce.

So to get some of those guarantees I converted the application into an umbrella application which means it is broken up into multiple applications and they can’t have circular dependencies between themselves or the application won’t boot. When the first colleagues came that changed, the umbrella structure was getting in the way of the daily programming and consensus was to change back into a monolithic app. So one of them made a PR for changing it back and it was pretty much just moving things. Moving files, updating the project mix project file. A lot of moved files and folders.

It wasn’t a big struggle, war and suffering. He just opened the PR, moved all the stuff, we merged it and it wasn’t an umbrella app anymore.

Lars Typically moving from a decoupled system into a monolith or vice versa is a horrible process. Often months or potentially even years, right?

Marcin Yes, exactly. So I think this is a testament to how refactoring is simpler in Elixir.

Lars So right now you are doing more ops than regular development, how is Elixir on that end, operating it?

Marcin Yeah, when we hired my three colleagues I started my transition into a more DevOps role. We needed this position at the company. For me it was an opportunity to do something different in a familiar environment.

Our Elixir deployment is fairly similar to other languages. We use Docker containers. We build a release, wrap it in a container and deploy it.

We do use Erlang Distribution for clustering and we use libcluster to form it. This allows us to use Phoenix Channels and pg2 process groups for messaging.

We actually also have a number of separate Elixir projects that we use for integrations with external systems, we call them connectors. They are developed separately but join the same cluster as our main API service. So our cluster is heterogenous. And we call these services via RPC calls. So we didn’t need to implement Phoenix endpoints and JSON serialization. We just send native terms over the wire. It also allows us to use pg2 groups for service discovery without any extra infrastructure.

Lars If you were building another SaaS backend, would you choose Elixir again?

Marcin Yeah. Totally. I would not hesitate to choose it again.

The only issue we’ve had that is worth mentioning that could sometimes be a problem is that you won’t find as many ready-made libraries as you would in Python or Java. More than 80% of the libraries we needed were there and more often than not they’ve been of high quality.

We’ve also worked around that with a Port at one point, where we were already using a Python library on the machine learning side, so we wrapped that up in a Port and it has been working quite well to this day. We are looking at reimplementing it in Elixir now but it has been great.

Once you learn how to use Ports it is tremendously useful because you can run anything, a Python library or a super efficient Rust implementation or anything that you can communicate with over standard input.

If you treat the BEAM, Erlang and Elixir, as a robust orchestrator then you can pretty much integrate with anything. The concurrency of the BEAM allows you to hide anything behind a function call, GenServer calls, HTTP calls. And the caller waiting there doesn’t block any other workloads in the system.

I want to thank Marcin and Vic.ai for letting me do this interview. And I don’t know which take-away I like more, ”trust your developers on their choice of tools” or ”Elixir is a solid choice for your startup”. Both sound good to me.

The thing is, I’ve worked on this code. It has both good parts and rough edges, as all code-bases I’ve ever seen do. I think the fact that I felt comfortable making significant updates to it within a week of starting speaks to what Marcin said, changing code, reworking and refactoring is made easier by Elixir. I spent my first few months deep in the backend and out in some integration services. I never had to care about the GraphQL API while doing that because they just weren’t coupled. And when I got into the GraphQL stuff I didn’t have to spend effort thinking about the deeper parts of invoice processing because I was working at the surface level. Part of this is because someone, probably Marcin, did a good job of layering the design of the system but some of it is just the conceptual simplicity of the language.

I’ll add that one of the things I worked on is something I don’t know how I’d do as well in most other languages and runtimes. The Vic.ai system has integrations to invoicing systems and those systems have published rate limits and constraints that we need to conform to. We want to conform in an efficient way and even if we hit an undocumented limitation we need to handle that. So we built an approach to distributed rate limiting according to particular rules, pooling to constrain concurrency of requests, back-off and circuit breakers for handling problems predictably. Without storing anything in Redis, without relying on a single source of truth to store ephemeral information about our communication. It was fascinating to put together and seems to have worked really well.

This is all at the technical detail level and the talent onboarding level. Perhaps as interesting to a decision-maker should be that Elixir paid off in effectively orchestrating a normal Python-based Machine Learning platform to provide a service to customers that shows enough potential and success that it received $50 million in funding. To me that is a data point supporting that the technical merit of Elixir isn’t just appealing to in-depth nerds like me but actually useful and beneficial in commercial applications. When your customer is an accounting firm you better provide a clear ROI, it’s almost like they look at the numbers.

No one needs to feel too bad about this. Not all projects pan out. But I’ve recently found my footing on putting these devices to use and figured I’d share some thoughts. The idea of Pis gathering dust in a drawer along with all your ambitions is a bit of a meme. As the introduction might have intimated I have a worse Raspberry Pi habit than many others. Going from memory this is my idle inventory right now:

• Pi 1, some revision
• Pi 3, at least 2 of them
• Pi 3 A+
• Pi Zero WH with OLED Bonnet
• Pi 400, swedish layout
• Pi Zero 2 W

Currently in use:

• Pi 4, handed off to the inlaws to provide a computer for their TV, straight Raspbian
• Pi Zero WH with Inky eInk Phat running the calendar_gadget
• Pi Zero WH with Keybow 12-key, running the lightswitch

Let’s not talk about the Arduinos and compatibles. I swear I’ll get to them any day now.

To give myself some additional points I’ll note that both the Pi0 OLED Bonnet and Pi 400 have been frequently featured in recent livestream experiments. They are seeing some use in that way but they are not in service as production devices yet. The Pi Zero 2 W just arrived.

The two Pi Zero I’ve recently put into service for showing my calendar and controlling my workstation camera lights respectively are both set up with the Elixir/Erlang-based Nerves project (less markety Github) instead of your typical Raspbian. Why? Because I prefer it. Why do I prefer it? A few things come to mind. For one thing it makes doing hardware on Linux in Elixir a darned delight. It turns the whole project into a single repo for generating firmware that doesn’t depend on setting up an OS, installing language runtimes or compiling anything on-device. It makes shipping code updates to the device comfortable and flexible.

Nerves has great ergonomics for development and it also has the useful property of constraining your possible approaches a bit. It needs an Erlang language, so you don’t have to decide between all languages you could run on a Pi. It has a way to deploy firmware so you do not have to think about Ansible, Docker or any other deployment approach. It comes with an OS set up to let you make the device an IoT style device, you don’t have to rummage through Raspbian making all of the choices. The world is no longer your oyster so you might get some actual stuff done.

So what is the OS? The Nerves project is built on Buildroot which is a bunch of tooling for building embedded Linux systems. So for kernel, drivers, OS-level packages we get all of that via Buildroot. Sort of like a compile-time apt you get a repository of packages to choose from to include in your OS. Typically I don’t have to touch Buildroot but I’ve customized it when needed. Once a Nerves system has booted Linux it will start the Erlang virtual machine, the BEAM, to start your project. And everything you want to do, you typically implement in, or orchestrate from Erlang or Elixir. So Nerves runs a much leaner install than Raspbian would and you build your project and install on your dev machine, burn the entire thing to SD card or ship it over SSH, its a very nice workflow. You can mock out hardware and iterate quickly on your dev host, or you can paste a bunch of code into your machine over SSH just to try it. Lots of workflow options.

Nerves does a lot of neat stuff for you, such as keeping most of your SD card read-only, protecting you from the dangers of corruption. And if your SD-card craps out, setting up a new environment is just burning the firmware to a new card. No need to set up deployment infrastructure to be able to repeat the build. I think that’s one place where Pi projects go to die. Reproducing the builds or recovering from a screw-up. The Getting Started guide is pretty good.

The Pi Zero W or Pi 3 A+ are ideal devices for all of this since they have everything you need to work conveniently. WiFi, USB Gadget mode, easy to power off of your computer USB. But any WiFi Pi is easy enough and any Pi at all should work.

So once you’ve burned a Nerves system firmware to an SD card you can just plug the device into your computer, it should start up and present itself as nerves.local over mDNS and you can SSH to it. All across the USB umbilical, power and data, the glory of gadget mode. If you gave it WiFi credentials it’ll also announce itself there. If you want the fatest way to try it out you can download and flash Nerves Livebook which will give you a Nerves device which provides the Livebook web UI for trying out code, has lots of examples of fun stuff you can do with the device and shows how to get started with a ton of examples. Livebook is a pretty wild Elixir take on Jupyter Notebooks and Nerves really makes good use of it as an introduction.

Now, for additional hardware.. Most hobbyist RPi hardware comes with Python libraries which is convenient on Raspbian but less useful for us here. Hardware support will vary wildly. But if the device does SPI/I2C/GPIO the Circuits library provides the foundation you need. I’ve ported some stuff, just implemented a subset for some others. There is a list on the Circuits site, I don’t expect that’s exhaustive.

Anyway. I’d suggest starting with something that is known to work. Either something with inputs, buttons or sensors or some output, lights, a motor, a relay.

Make it mostly use skills you already have and let the hardware be the new thing if that is unfamiliar. If you already know web dev for example you could have a device with buttons that talks to an API on the web somewhere to make things happen.

So for example, I had the Pi with an eInk display already. It doesn’t have any fun hardware input connected so I made a Web UI for setting up a calendar URL to import and made pulling that calendar the input. The Web UI was easy enough to make and Phoenix LiveView made it a pleasure. The novel part was rendering a nice visual of the next calendar event to the device and I did do a bit of yak shaving in the form of a live web preview.

Look for controllable things in what you use. Your computer itself, remote servers, simple devices on your network. My other project I got working controls my Elgato Keylights which are normal Web API endpoints. The tricky new part there was mostly around discovery via mDNS and there are good Nerves libraries for that.

You can follow some of the process of me getting familiar with different parts of using these devices from my livestreams where I’ve done both exploration and implementation work on these projects. There is a playlist here on YouTube or you can find them on this blog if you’d rather feel indie.

So what has been the key in making these Pis stop gathering dust for me? I’ve chosen tools I like and care about. I know that if I want to set up these projects in 2 years it should all just work. The work doesn’t feel ephemeral, it feels finished now and trivial to continue if I want to. These kinds of projects can easily turn into unwieldy yak shaves where I make no progress. I feel like Nerves constrains me and saves me from that sprawling complexity by providing solutions I like and don’t have to pour energy into. I also set aside time to do it, mostly around the livestream. That’s an important non-technical aspect, know when you are planning on actually doing it, if you care about it getting done.

If you are curious about Nerves I suggest signing up for the Nerves Newsletter which is bi-weekly and updates you on what is happening in the project as well as neat projects people have put together with it. It is hand-typed by a human (me) that is active in and excited about the project. The entire archive is available here.

I’ve also experimented a fair bit. Some of the most fun stuff I’ve done with these things is to set up Erlang clustering where all of my Pi devices can detect each other on the network as they start and connect up to form and Erlang cluster. This would allow easy communication between the devices if I want to integrate them with each other. I’m not using it for anything currently but it is an interesting piece. I’m considering setting up the OLED Bonnet to be a blinking reminder of meetings that are about to start as it has buttons that would let me also dismiss the meeting. If I set up a node that runs on my desktop computer as well I could even have the button presses trigger opening the calendar event information on the desktop over the cluster. There are a lot of fun possibilities when your devices can connect to each other and communicate via message passing.

The next thing I would want to get started on is a radio thing. I have a number of Arduino-style boards with RFM69 transceivers on them. I also have a transceiver with an RPi-friendly breakout. I’ve had them talking in the past but I want to revisit it with a new approach. I want the Arduinos on battery, reading sensors or operating relays and I want an Rpi on my network being the hub for collecting sensor data and controlling the devices. That’s a bit more involved to do right but it should be a fun project.

If you have projects with Pis that you think are cool or plans that you would like to share, don’t hesitate to reach out over email (lars@underjord.io) or on Twitter (@lawik@hachyderm.io).

We ran into an issue with the expectations we had on what mDNS would give us. Those issues are still under investigation but they seem to be a change or regression in underlying parts of Nerves or Erlang. I’ve since gotten it to work by going back a few versions.

Those frustrations aside we also verified that we can control the lights and detect the keypresses for the keypad we are using. The Keybow 12-key thing with RGB lighting.

We also poke around with an HDMI screen on the Pi400. I want to run Scenic on that one.

With recent support for Erlang distribution in MdnsLite we got the power to help nodes find each other via mDNS instead of some other method. So I wanted to give it a whirl, and I did :)

Special thanks to Isaac for his help in the chat.

Turns out that was a great call. It worked beautifully, almost confusingly well. Then we investigated some of the things we can do with a Pi with input devices but no output connected. It was a fun stream. Lovely little device and that prompted Frank to go and get us ethernet gadget mode. So that might be something for next time.

Enjoy the video and let me know if there’s more hardware you’d like me to play around with.

This triggered a not-yet-working Nerves device project I’m calling my calendar gadget which will be more documented at some point I’m sure. The Phoenix part of that under calendar_app is completely locally runnable. You have to add some fonts and use iex to add your calendar URL though. Not very ergonomic yet. The idea is to show my next calendar event on the eInk display. And since I didn’t want to push firmware for every change to the display I made a silly/naive eInk preview in LiveView. If you are curious about what my just-get-it-working code is like, that project is full of it.

Anyway, I’m really glad to be doing some more hardware stuff and be back with Nerves. The livestream video covers putting text on these displays and some capturing of button presses :)

The official blog announcement for RC 0 has the details and I reference it through the video.

I particularly tried out the LiveView Heex template changes which was great. This seems much more ergonomic and it wasn’t bad to begin with at eex/leex.

Then we actually had time to try the new Esbuild setup as the replacement for Webpack. We even added in TailwindCSS. Surprisingly straight-forward. Look forward to use it.

Thanks to the people who dropped in, especially Marcel Fahle for providing some back-and-forth on the frontend-y features.

And I wanted this to be broadly compatible. That’s what most of this livestream ended up being about from what I remember. Trying things and swearing about internals. All in all a good time right?

You can find the livebooks of the modified lessons in my fork of ElixirSchool.

I’ve outlined some of that in The WordPress merging problem and WordPress & the gross inefficiencies. I’m trying to create a sound technical solution with a sound human solution on top of it. My first efforts were quite top-down in some aspects. I took a small piece of the pie and tried to solve it. This effort is more ground up.

This video is me showing some of the work I’ve done on some foundations of what could turn into an Elixir CMS. I also try to explain some of my requirements and some of my thinking. There is a long way still to go. Clearly. Hope you enjoy my perspective on these challenges. If you want to be involved you can reach out over email or find me on Twitter.

You can find the livebooks of the modified lessons in my fork of ElixirSchool.

I end up doing almost no TextDelta or Operational Transformation in this one as I try to pin down whether I can reliably insert and delete text from the different editor sessions without causing endless loops of updates. Turns out that there is no great facility for handling that. We learn quite a bit. But spoiler, I wrap it up deciding that making it actually work with what is available to me will be too painful.

I do want to revisit Operational Transformation. I’ll probably do it with Quill.js which produces Deltas of changes by default. At some point. Let me know if you do figure out any options for this.

I’m still very keen on doing things with this wxStyledTextCtrl. Live collaboration is a very particular use-case, I still see all sorts of fun possibilities with implementing an editor with it.

You can find the livebooks of the modified lessons in my fork of ElixirSchool.

Hopefully the flailing is interesting :)

You can find the livebooks of the modified lessons in my fork of ElixirSchool.

Before you read this post with your teeth clenched and ready for some hot takes, chill. This isn’t where I throw Elixir under the bus. Rather it is where I try to put my finger on the values that makes Elixir a good fit for me and a language I consistently recommend to people.

So to summarize the values:

• Software should be efficient
• Software should be maintainable
• Software should be suitable for continuous development
• Developers should not burn out from the work
• Teams should be able to deliver consistently without stress
• Technically conservative but extremely progressive

So some of my values are fairly specific to software though they apply in other contexts as well. Some other ones are much more focused on the people. We can start on the software side.

Software should be efficient

I find it immensely frustrating that we have incredibly powerful computers now and pretty much everything feels slower to me. I think back to the days of native desktop software that was snappy and fast. Our browsers are beastly software platforms and everything is now a browser.

Beyond that we have programming languages and paradigms that keep things simple by constraining it to single core operation and making processing that is both concurrent and parallel a second class citizen. In a time where clock-speed is no longer going up but core-count is, this is terrible for efficiently utilizing modern hardware.

Efficiency is not just about hardware utilization. It is also about the experience of using things. If you are crunching the numbers quickly but your software is unresponsive during the whole process that can be just as bad as processing slowly.

In some cases Python is perfectly performant enough and you’ll never see an issue. In others you might want Go, Rust or even Fortran, I don’t make the rules about what’s efficient for your use-case. With Elixir I have the BEAM virtual machine which provides low latency and which typically protects me from screwing up latency too badly. It models applications in primitives that map well to concurrent workloads and they make it much, much easier to not botch it. I find Elixir efficient at utilizing hardware while making it straight-forward to maintain a responsive experience.

Software should be long-term maintainable

I’ve built systems that have been hard to maintain. And then I’ve had to maintain them. I don’t recommend it.

There is some software that through a combination of persistent developers and maybe good design just keep going and stick with us. I’ll try to paint a wide picture here. I wish more software could be like VLC. A piece of software that just keeps working. Since I discovered it I have never had to think about what to use if I want to play media. It has persisted because someone has obviously been able to maintain it.

There is so much software we rely on that some person or group of people maintains for us. Postgres, ffmpeg, Linux, git, bash, vim, Ubuntu, Photoshop, Office. Some of these are maybe driven more by a headstrong business making sure they persist than good architectural choices. With proprietary things that’s hard to know. But if we could reduce the churn of software that launches and then dies from complexity I think that’d be great.

With my consulting work I’ve also dove head-first into other people’s code on a regular basis and I’ve seen approachable code bases and less approachable ones. Unwinding other people’s code to understand it is often challenging and also a big part of the job as a developer.

Some products die as the team decides it needs a full rewrite and the rewrite fails. Or because the only person who knew how it fit together left. Or the team burnt out because every update was a thankless slog of complex drudgery. These are typical costs of unmaintainable systems.

Being able to write parts of the code in isolation, mostly reason about code in isolation and know the scope of changes you make is incredibly important to maintainability. I have never had this experience as strongly as in Elixir. I think this is a combination of the high-level dynamic pragmatism and a general feature of functional programming meeting the Actor model. It is easier to reason about state while the dynamic nature of it keeps the threshold to actual coding low and unceremonious. In Elixir I have been able to contribute code in the first few days of a project in a way that hasn’t happened elsewhere.

I’ve seen some hairy Elixir code. But it has generally been better than hairy PHP, Python or Javascript from my experiences. I think Elixir provides a paradigm that makes it easier to build maintainable systems.

Software should be well suited for continuous development

The development of software is commonly a continuous labor. Every week developers all around the world pick up issues and implement solutions to them. There are a number of ways in which this can work. But important parts are generally around delivering improvements and avoiding the introduction of defects. Usually there are also heavy incentives towards moving fast.

Good tool support, a strong testing story and all of the previously mentioned maintainability concerns feed into what a language is like to do continuous development with. There are also things like productivity, in the sense that Rails is known to be very productive, you can build new features quickly. Rails is not known for maintainability or efficiency. I think this is why a lot of people that have moved from Rails to Phoenix and Ruby to Elixir have done so.

Any later stage software project tends to achieve a certain level of complexity and at a certain point it matters a lot if your tools are good at managing complexity. If the paradigm you work in supports you in keeping things manageable that helps you to move forward. It is quite possible to move forward in something less structured but it places higher demand on each developer to make good choices at every turn.

So aside from software values I try to think in terms of individuals, teams and wider organizations.

No developer should be burning out due to the work

This happens all the time of course.

It is not primarily a tools issue. But as noted in discussing maintainability and continuous development there is pressure to proceed and complexity to manage and each individual developer is under this pressure. How you manage it is partly up to you. However, it also matters what you are dealing with in the work. Is it clear, is it well-constructed. Are there strong conventions to support you? Do the tools help or hinder you?

I’ve heard plenty of complaints from people working in stacks they hate and how the tools cause more problems than they solve or how frustrating it all is.

At the individual level I find developers have an easy time being enthusiastic about Elixir and often delighted by the power and clarity it provides. The practical tooling is very good. Documentation is great. I think this is a tool that supports individual developers in creating software without causing undue stress.

At the end of the day, therapy, experience and a few run-ins with how not to do it has probably done more for me in this area than working with Elixir. But I do feel like the language supports my work personally and that’s a good feeling that contributes to building a good stable self in this line of work.

Teams should be able to deliver consistently without stress

All teams can screw up. That’s not entirely avoidable. But consistency isn’t about never failing, it is about repeatably being able to succeed. Much like with individual devs I find this to be a people challenge more than a tools challenge. But a tool can provide benefit. Having strong conventions around the tools gives some stable expectations for a team.

I would describe the idioms and conventions of Elixir as a language, tool and community as pretty strong but not very strict. There is direction, there are core ideas, recommendations and writing to lean on for how you can do things. But you don’t have to do it that way. In other parts, the language and frameworks actually press you to work within certain constraints.

The Actor model is a particular way of solving particular problems and working with it forces you to think and work within that solution the absolute majority of the time. And those constraints can be very helpful. There are mistakes it simply won’t let you make or where it will make the wrong thing really hard to do.

I’ve found Elixir to enable and support what I want to do and be enjoyable to work with. When working fast on something that needs to be delivered quickly it lets me do that without ruining the future potential of the system. Reworking internals and refactoring code is very doable due to the limits on shared state and mutability. And when I’m working with complex, intricate things and need to slow down and move carefully there is good tooling for creating comprehensive tests. The language and frameworks allow you to build complex and intricate things that can be tested in isolation. It doesn’t force me to always prioritize one way.

Technically conservative but extremely progressive

I don’t think we should avoid making up new stuff. I do however think there is value in tempering what we invent with some of the wisdom in using what already exists. And I think that balance is very present in Elixir as a technical solution and as a community.

The BEAM it runs on is proven with Erlang since the 80’s and that has only improved with time. The Elixir language itself was sort of dubbed “done” in a keynote (re-iterated here) and it doesn’t change very much while still adding nice improvements consistently along with improvements to Erlang/OTP. That’s a fairly conservative stance and I think it has a good chance of preventing large breakages like Python 2/3 or enormous legacy to maintain in the standard library like what I’ve seen in PHP. Only time will tell.

Elixir as a project, community and going concern hasn’t nearly stopped moving though. There is an enthusiasm and ambition in this community that drives forward some remarkable progress on top of a VM that allows us to do things that simply would be unadvisable in other runtimes. Such as Phoenix LiveView. Which allows developers to build reactive HTML-over-the-wire applications with a minimum of frontend/backend ceremony.

The Nerves project, an IoT-ish embedded framework where the BEAM is most of your OS on a Linux kernel, is already sniffing some RISC-V boards. Dashbit and friends is bringing Machine Learning to Elixir with the Nx project and their ludicrously ambitious Livebook project lets us do some very cool things that I don’t think would be feasible outside of Elixir.

A big reservation I have about the JS ecosystem is the rate of change, churn and the breakages I’ve experienced in projects just from time passing. My experience with Elixir has been very stable, some libraries barely move at all, they are essentially done. Some move slowly, some move faster but I haven’t experienced any issues getting older projects going or handling dependencies. Progress, but not very much churn.

Elixir has taken documentation, tooling and developer experience seriously from early on and it shows. Phoenix and LiveView are both great examples of tools that are built to keep you happy and productive when doing the bulk of the work. This is not an ecosystem that is just reaching the baseline, it is pushing forward. You can see Livewire (Laravel) and Hotwire (Rails, etc) picking up what LiveView put forward. I’ve heard many devs from other langs speak well of Hex, the package manager and the hexdocs it provides.

I don’t need new things for no reason, that’s the realm of experiments. I do want new things, actually useful progress, powerful tools built on solid foundations.

Stability and progress.

In Conclusion

It is not about Elixir.

Elixir just happens to be a language, ecosystem and community that suits my values and priorities. That is no small thing. It feels like home.

To address the vacation work-days thing. I’m not being terrible at vacationing, or at least not as far as I can tell. We have a young child in the house, so some time on my own is quite nice and I have established some routines I really enjoy that I didn’t want a break from this summer staycation. Specifically the newsletter and the livestreaming which I both do weekly on fridays. I also really enjoy sitting down with code for a solid session. So spending 1 day per week on this feels pretty good.

Alright. This video covers setting up the project, basing it on another project I’ve made and getting started.

The product idea is about sharing media and parts of life with your loved ones and friends without the social mix of sharing your wild friends with your stodgy relatives, or sharing your wild relatives with your stodgy friends. A small-scale, mindful way of keeping in touch and not needing to work with half a dozen message apps to share a picture of your kid. My kid actually. She kicks ass.

If you want to reach about about anything, email or Twitter is good. lars@underjord.io and @lawik@hachyderm.io respectively.

This is the second installment, consider watching Part 1 before or after this.

So if you enjoy watching people turn blank white nothing into something more visual, this might be for you, or if you want to see someone that is not a Tailwind expert wield it, also accurate.

Let me know what you think at lars@underjord.io or on Twitter at @lawik@hachyderm.io. If you want more of what I do, I suggest the newsletter, signup is at the bottom of the page.

This is the third installment Part 2 and Part 1. I don’t think they are necessary context but it probably helps. There has also been a few hours of work before each stream so the system moves forward off screen a bit.

I’ll say we had very few actual problems in this one, that was nice.

Windows is an Adversary of the Desktop User

Most computer users I know don’t particularly want a Microsoft Account. Microsoft wants them to have one though. None of the people I’ve set up with computers have asked for OneDrive, Microsoft really wants them to know about it though. No one I’ve ever heard of uses Cortana but you bet there’s stuff about it in your toolbars. The experience of installing, setting up and using the operating systems is an exercise in finding new novel ways to decline things or simply capitulating to see if it makes the nagging go away.

And then there’s the ads. I mean, personalized experiences. Smart Cloud Somethings. It is at a foundational level completely unacceptable to me that a commercial operating system that you’ve paid for, or that was included on the computer you paid for, will show you advertising in your primary modes of navigation. Read, the Start menu. It just looks like an app you might have that you never installed. But it’ll take you to the store. It’ll take my mom to the store. It’ll mean she thinks twice about everything her OS shows her, or capitulate entirely and just accept that her computer is not really hers and isn’t knowable to her.

I find it infuriating. I don’t hate Windows. I don’t love using it but I use it for gaming and I help friends and loved ones with it. I’m not precious about using it when it makes sense. There are things about Windows that are immensely impressive. My father-in-law’s old desktop invoicing program still runs, more than a decade later without updates. I respect that. That’s worthy of acknowledgment.

Windows 10, which I liked as much as Windows 7 at the start, has slowly rotted on me, or just ground me down to where I find it reprehensible. Where every new notification, interruption, thing changing under my feet or update that bricks my entire computer, just pokes straight at the raw exposed nerve ending that used to be a thick skin.

Beyond that they’ve not reconciled their old and new control panels which just makes the OS numerous layers of confusion, there are endless screens of paragraphs of text, toggle switches that only barely try to do what you want and any time you actually need something done you need to find a link to Advanced or whatever takes you to the old UI that actually does the damn thing. How does that belong here? Well it isn’t really learnable anymore. The paradigms and concepts are barely cohesive anymore.

And Microsoft has with help from the rest of the industry turned the word “telemetry” into something that makes entire sections of the tech world recoil and rev up the privacy monologue engines. I don’t want to care too much about that but when every goddamn app feels the need to get that sweet data it starts to get old. And that’s not even the tracking stuff, that’s later.

Windows is an amazing platform in many ways. It is commerically successful and viable. It doesn’t operate under Microsoft’s thumb in the way the Apple ecosystem does which there is currently war about. Windows could be the most reasonable operating system. They could just stop all the bullshit. They absolutely don’t need it. Telemetry, maybe they need some of that. But ads? Nah. Upsell? Nah. They could just choose to be better.

Also, people liked Skype once and hate Teams. So with Windows 11 and the pandemic we are building it into the OS. Skype? No. Of course not. Teams! More Teams!

I really hope Windows 11 manages to improve the UI cohesion. I have absolutely zero hopes that the rest will be less egregious. Also, they are partnering with trusted friends Amazon for bringing Android apps to the Microsoft Store. That just gives me all sorts of warm and fuzzy creeps. I like the feature of being able to run Android apps. Great! Cool! But Amazon?

The Smartphone Era

As the smartphone arrived everyone started building apps. Small plucky software packages that did things previously untenable on a phone. Pretty cool. And there was money to be made.

The friction to building, publishing and selling software went from significant to negligable and the potential audience went from large to pretty much every single person. With this came the easy-money-chasers, the scammers, the thieves, the corporate giants with the half-assed ideas. Pretty much everyone. Among them were the gems, the hard-fought stalwarts and a bunch of pretty decent software. But the deluge is more noticeable.

I’m pretty sure the “your device starts randomly asking for shit” thing wasn’t introduced with the smartphone. I feel like OEM bullshit driver software had that capability way back. It did however become ubiquitous. Now my mother has the pleasure of calling me because her phone asked her for something weird. Why? Samsung, maker of the phone, decided they should have access to her Google Drive or something. It’s probably harmless. But we can’t know that. And Samsung isn’t so good at bundled software that I’d necessarily trust them.

If you want to find any simple utility app you have no chance to find a reasonably priced, high quality one without tons of up front research. Because the roulette of an app store search will get you an undiscernable mess. And you know you have to be careful, mindful. Because most apps are traps, trackers or hustles.

Smartphone apps taught us about security in the least satisfying way. It showed us that you can’t trust the software and you need to very, very mindful to not get fucked.

So many I know stick with the big actors. They wouldn’t pick up small indie apps because it is hopeless to know a good small app from an evil small app up front. This is especially true on Android which doesn’t have a cultural ecosystem of app devs that are worth trusting (at least I don’t know of it). iOS in that sense has a bit of a reputation system and a large enthusiast sphere which helps with curation. The App Store is of course still a shitshow. I mean, carefully reviewed and curated.

Right, so the big actors. They have again and again been shown to be worthy of no real trust. Facebook standing head and shoulders above everyone else in being creepy and worthy of all skepticism and distrust. Google has a sort of elemental honesty in being unapologetic trackers. It was always their business and I don’t feel like they were every quite so psychotic as Facebook. That said, we know they are in everything and we absolutely are being tracked by them.

So while the big actors are slightly less likely to completely screw you on security, software quality or straight money. They are pretty much guaranteed to screw you on privacy. As an EU citizen the GDPR does make them disclose and ask me to opt in and limit how much they can require from me. It really does make it more noticeable. But with near-monopolies, network effects and plenty of money to throw at these problems they do what they can to entice, inconvenience and generally prod you towards surrender. You’ll get much better smart somethings.

Hostile Devices

I’ve mostly wrangled my devices into a semblance of being under control. But I don’t trust them, I only like some of them and I don’t trust any of them very far. The one I trust furthest is my Linux workstation. That one is finicky only in being a bit rough here and there, it isn’t hostile or under the control of some alien software. To my knowledge. Maybe System76 will brick Pop or unveil their new ads. I doubt it and if they did I’d fall back to Debian.

Everything else is asking for your permission to do the thing you don’t want them to do. Everything is asking for more than they need. There’s a very cool trick you can pull to avoid showing a cookie notice for your site, don’t use unnecessary cookies. Cookie for login? Pretty much fine with the law. GDPR? If you aren’t collecting personal data you don’t need to ask complicated questions.

Want to improve the UX of your application or SaaS? Don’t do things that require users to opt in to things they don’t want to do. Free UX!

But oh, the metrics and analytics, we so very need them. I don’t think we do. Sure, they can be useful but the most actionable and useful information is already very legal and available. Did people visit the thing? Did people buy the thing? Did they complain about it afterwards? I’m aware that there is signal that is hard to get. But I really don’t think grinding away at the foundations of human-computer interactions is a very good investment.

We are spending our time deciding whether to fight dark patterns or just give up. They are everywhere. Every convenience, every delightful offering comes with a side of “just consent to everything”.

Doing the opposite of this, respecting privacy, providing user control, minding the user’s agency, these are things that only small operations can do today. And it is the differentiator that we can offer. We can build things that don’t suck. Any sufficiently large company is required to start sucking at some point. It’s probably a law.

Each megacorp can grab a shred of something real and important and claim that they do that thing oh so well. Apple with privacy. Google with .. the heck kind of good thing should I be associating with Google these days? I don’t even know. They’ve been crap at marketing themselves for ages now. Facebook with .. political manipulation? Connecting people? Amazon with .. exploitation.. Microsoft with Open Source and bringing Linux to the desktop? Regardless, they can push a very limited set of virtues and we can just assume everything else is corporately mandated to be morally bankrupt.

In Conclusion

The background buzz of permission dialogues is deafening, deadening. Cookie notices, consent forms, allow/reject. Should I trust this? Can I install that? Do you want to try OneDrive? Dropbox needs to update again. You need to restart your browser. Are you sure you don’t want to try Edge? You really shouldn’t be installing untrusted dangerous software. This trusted approved software wants to know your location always. Allow/Decline?

Your privacy is very important to us. We would like to know what you are doing at all times. Accept / Ask me again later.

Yeah. I couldn’t give a good reason why anyone should trust, or like, software the way it typically works these days.

Ask me again later. Not right now. More information. Are you sure?

If you want more of my thoughts and writing I suggest the newsletter .

After this show I primarily have more nuanced feelings about the whole thing. I see advantages to this approach but I would say I still see to much mystery and magic in it for my taste. Things are doing stuff and I have no idea what’s what.

Regardless where you fall on the spectrum I think Gerhard showed us some very thoughtful approaches to using k3s, some absolutely over the top nice uses of make and scratched the surface of ArgoCD.

The video is also available on YouTube.

Let me know if this type of thinky-talky streams are of interest and I just might do some more of them. The video is also on the YouTube channel if you prefer.

You can find the very simplistic Livebook version of the Elixir school content on my forked Elixir School repo for now.

The video is also available on YouTube.

Elixir projects tend to be very consistently laid out. Especially Phoenix-based web projects. The basic Phoenix generators provide a fair number of opinions on where things go. Even without that the general shape of an Elixir project is surprisingly stable.

So across a few different clients I’ve seen a number of Elixir projecs that have a few years on them and while there were variations such as not actually utilizing the module hierarchy and putting all of it in the widest possible namespace or overriding some common conventions, overall even the old projects were very reasonable to explore. They tend to flow from the mix.exs projects file and from there you can logically follow what parts of the system are started as you run the application. That will let you figure out what routes it exposes, what work it does in the background and much more. The Elixir supervision tree (inherited from Erlang) makes the shape of the application more clear to you as a developer.

My first onboarding onto an Elixir project was a project that essentially predated Phoenix having a stable release and so it was built right on top of Cowboy and WebSockets. I was pretty familiar with Elixir already. The guy who introduced me to the system and product on the other hand, was doing some Elixir, but was primarily doing other work. I estimated some work for this client, around a 100 hours, for the initial engagement of adding some API endpoints to provide some new functionality for external parties. And then I built it. Knowing Elixir but with no experience of this particular system and without the strong conventions of Phoenix I successfully over-delivered with some added niceties at a couple of hours shy of the estimate. I like to think I’m competent at what I do but I’m not convinced I would have that turnaround on a similarly sized Python or PHP project. The compounding complexity tends to be much higher.

Comparatively the same work in a PHP client’s code base would likely have taken me around twice the time. Why? Mostly due to clarity. Elixir is a functional programming language with a very strong pragmatic streak. It makes it easy to write clear code, rather than clever code. Comparing two codebases is of course anecdotal but it lines up with all my experiences of dynamic OOP code bases vs Elixir code bases. The consequences of making changes are much clearer, the data you have access to and how it gets passed around is plain to see. It removes a lot of guess-work.

Same Elixir code base, a number of months later I’m the person who will onboard a new developer. He has been doing PHP for a long time, no Elixir experience. No real issues, he certainly spent some time delving in and picking up the syntax and concepts of the language. He is at the time of writing productively the main maintainer of the code base. I think we had two video pairing sessions to introduce the concepts and how the thing was set up. It really didn’t take more.

Another client, a significant fintech code base in Elixir. A few different services, some with Phoenix, some without. I was brought on, given help to get things set up and started. Due to the strong conventions around how Elixir and Phoenix is used that mostly entailed giving me credentials and answering some questions on specifics. I believe I had my first commit the same week. My first major contribution was probably the next week and then we were off to the races.

So bringing on people that know Elixir tends to go well in my experience and I’ve even had good luck bringing on an experienced PHP dev that didn’t know Elixir. How about less experienced folks?

I’ve introduced at least three coding school students to Elixir during their internships or part time work with me. They typically knew some PHP, some Python and some Node.js at that point but hadn’t really used any of them in anger. One is presently working professionally with a production Elixir application and another is currently working with me as an assistant developer and doing well using Elixir on client work.

I haven’t found much difficulty in teaching Elixir or training people into the language. There are some Functional Programming concepts to work through and some habits to break. Thankfully Elixir does not look or feel incredibly different from your typical OOP languages. Most of it will be familiar to any developer. It simply brings some very different powers to the table.

One of the things I find makes onboarding easier is that you can effectively approach Elixir both bottom-up and top-down depending on what you are trying to do. If you want to get a foundational overview of what the application is and how it hangs together, look at the supervision tree, the dependencies and you can absorb the shape of the system. And the moment you need to implement a small change or add something, you can figure out the place where those things happen in the system and work there, in relative isolation and you don’t have to know everything about the surrounding system to be productive. Immutability and no shared state makes your system easier to reason about in separate parts. And that means you need to know less to make useful changes.

I’ve picked up another client’s Phoenix project without any guidance or real introduction to the code and been able to provide changes on the same day. Just by knowing my way around the conventions. Certainly this will vary by person and experience but it isn’t just due to some stroke of brilliance on my part. Strong conventions and a clear language where code is easy to reason about.

All in all, I’ve found onboarding in Elixir to be ridiculously straight-forward compared to a lot of other projects and I think that’s an important factor and I believe it says something about the long-term ease of development as well. A lot of Elixir developers tend to describe “boring” Elixir projects that just keep working and are easy to maintain.

Boring is a rare and powerful property in software systems. Especially when they can achieve exciting outcomes such as near real-time, strong concurrency and parallelism with distributed computing just baked in. Nothing quite like it.

If you want to hear me (and others!) go on about Elixir in further detail I’d suggest BEAM Radio but if you’d rather get more variety in topics I’d suggest the Regular Programming podcast or my newsletter.

You can find the very simplistic Livebook version of the Elixir school content on my forked Elixir School repo for now.

The video is also available on YouTube.

If you enjoy this or have questions you can reach me via email as lars@underjord.io or on Twitter where I’m @lawik@hachyderm.io.

The branch is on Github as lawik/noted:prom_ex and the video is below.

Also available on the YouTube channel.

You can find more by Alex at akoutmos.com, beamrad.io and you can tell him how you really feel @akoutmos on Twitter. PromEx is prom_ex on Hex and that’ll lead you to docs and repo.

Enjoy the things I do? Sign up for the newsletter, I write more about Elixir, my projects and developer experiences there.

Want more voice? I talk about Elixir on Beam Radio and more about programming in general on Regular Programming.

If you want to discuss it with me, you can find me via email as lars@underjord.io or on Twitter where I’m @lawik@hachyderm.io.

You can find the very simplistic Livebook version of the Elixir school content on my forked Elixir School repo for now.

The video is also available on YouTube.

If you enjoy this or have questions you can reach me via email as lars@underjord.io or on Twitter where I’m @lawik@hachyderm.io.

The project repo I built up during the stream is on Github as lawik/caterpillar and the video is below.

Also available on the YouTube channel.

Enjoy the things I do? Sign up for the newsletter, I write more about Elixir, my projects and developer experiences there.

Want more voice? I talk about Elixir on Beam Radio and more about programming in general on Regular Programming.

If you want to discuss it with me, you can find me via email as lars@underjord.io or on Twitter where I’m @lawik@hachyderm.io.

You can find the very simplistic Livebook version of the Elixir school content on my forked Elixir School repo for now.

Today’s goof is to call the whole thing Learning Elixir at the start. Also we had to start over once because I screwed up the entire Livebook. But you don’t have to see that. It wasn’t funny enough.

The video is also available on YouTube.

If you enjoy this or have questions you can reach me via email as lars@underjord.io or on Twitter where I’m @lawik@hachyderm.io.

We grab an HTTP client, and HTML parser and then we go hog wild spawning Task processes and fetching web pages.

The project repo I built up during the stream is on Github as lawik/caterpillar and the video is below.

Also available on the YouTube channel.

Enjoy the things I do? Sign up for the newsletter, I write more about Elixir, my projects and developer experiences there.

Want more voice? I talk about Elixir on Beam Radio and more about programming in general on Regular Programming.

If you want to discuss it with me, you can find me via email as lars@underjord.io or on Twitter where I’m @lawik@hachyderm.io.

You can find the very simplistic Livebook version of the Elixir school content on my forked Elixir School repo for now.

There was an audio issue while recording that couldn’t be isolated and fixed where my friend was quite a bit lower in volume than I was. I’ll be looking at making sure those are separate tracks so I can fix those issues in the future. I’ve tried to with some audio-mangling so compared to other episodes the audio might be a bit rubbish. Very sorry about that.

The video is also available on YouTube.

If you enjoy this or have questions you can reach me via email as lars@underjord.io or on Twitter where I’m @lawik@hachyderm.io.

This stream connects to some previous thoughts I’ve expressed about taking shortcuts in software or prioritizing developer experience over end-user experience, efficiency, consistency and so on. I wrote about that as the idea of artisanal software. wxWidgets is a bit more pragmatic than that lofty post. But I’ve been curious to do more with it for ages.

The project repo I built up during the stream is available here and the video is below.

Also available on the YouTube channel.

Enjoy the things I do? Sign up for the newsletter, I write more about Elixir, my projects and developer experiences there.

If you want to discuss it with me, you can find me via email as lars@underjord.io or on Twitter where I’m @lawik@hachyderm.io.

Then we also spent some time setting up Exqlite so we can use Sqlite instead of Postgres. That worked nicely, we tested it with phx.gen.auth. I will try to find some time to make a post on this later. For now you can mostly look at lawik/noted which has a branch and commit for Sqlite support.

And finally we started looking at setting up SiteEncrypt. We didn’t finish that part, so maybe we do that in the future sometime.

Also available on the YouTube channel.

Enjoy the things I do? Sign up for the newsletter, I write more about Elixir, my projects and developer life there.

If you want to discuss it with me, you can find me via email as lars@underjord.io or on Twitter where I’m @lawik@hachyderm.io.

You can find the very simplistic Livebook version of the Elixir school content on my forked Elixir School repo for now.

The video is also available on YouTube.

If you enjoy this or have questions you can reach me via email as lars@underjord.io or on Twitter where I’m @lawik@hachyderm.io.

This is the first episode and we’ll definitely polish our process as we go. They will be much shorter and probably simpler than the live stream videos. The goal is to work through Elixir School and get my friend properly started with Elixir. So it will be a mix of going through the content and any derailments and asides we end up going through.

You can find the very simplistic Livebook version of the Elixir school content on my forked Elixir School repo for now.

The video is also available on YouTube.

If you enjoy this or have questions you can reach me via email as lars@underjord.io or on Twitter where I’m @lawik@hachyderm.io.

Also available on the YouTube.

If you want to discuss it with me, you can find me via email as lars@underjord.io or on Twitter where I’m @lawik@hachyderm.io.

At the end of the post there is also a video showing you what the thing does. The idea was to do something better for the visuals. I ended up also fixing an overflowing buffer thing.

The project along with commits and my experimental branch for fixing it without actually having a microphone are available at the lawik/media repo.

The first pipeline iteration

With this commit I had an initial working pipeline, this was my baseline going into the live stream.

Let’s see it:

defmodule Media.StreamToFile do
  use Membrane.Pipeline

  alias Membrane.{File, FFmpeg, MP3.MAD, MP3.Lame, PortAudio, Time}
  alias Membrane.Element.Tee
  alias Membrane.Caps.Audio.Raw

  @impl true
  def handle_init(output_directory) do
    Elixir.File.mkdir_p!(output_directory)

    children = [
      mic_input: PortAudio.Source,
      converter: %FFmpeg.SWResample.Converter{
        input_caps: %Raw{channels: 2, format: :s16le, sample_rate: 48_000},
        output_caps: %Raw{channels: 2, format: :s32le, sample_rate: 44_100}
      },
      splitter: Tee.Master,
      encoder: Lame.Encoder,
      raw_output: %File.Sink{location: Path.join(output_directory, "out.raw")},
      encoded_output: %File.Sink{location: Path.join(output_directory, "out.mp3")}
    ]

    links = [
      link(:mic_input) |> to(:converter) |> to(:splitter),
      link(:splitter) |> via_out(:master) |> to(:raw_output),
      link(:splitter) |> via_out(:copy) |> to(:encoder) |> to(:encoded_output)
    ]

    {{:ok, spec: %ParentSpec{children: children, links: links}}, %{}}
  end
end

What it do you say? What it do indeed. It will use the cross-platform audio device wrangler portaudio and grab the microphone input as the first stage. It produces a stream of media, so that’s a source. It will throw it at a converter based on FFMPEG to convert it into a format that later stage can accept. Then it will duplicate the stream to a master copy and a .. copy .. copy. It will feed the master to a raw file output and the copy? The copy goes into the lame encoder to be turned into an MP3 and then on to a file. The files consume a stream as an endpoint, those are sinks. Source and sink.

So we define the different stages as children and then we connect them using the links. Then we can run this. The Media module shows how that can be done. start_link/1 tells me this very much is a GenServer style process and can be handled with your average OTP approaches. I’ll add that to the application for automatic running later.

defmodule Media do
  def record_to_file(path) do
    {:ok, pid} = Media.StreamToFile.start_link(path)
    Media.StreamToFile.play(pid)
    {:ok, pid}
  end

  def stop_to_file(pid) do
    Media.StreamToFile.stop_and_terminate(pid)
  end
end

If you call Media.record_to_file/1 this it will:

• Grab your default input audio device
• Resample it for encoding
• Duplicate it to two streams
• Write a raw PCM format file from the stream
• Encode an MP3 of the audio
• Write the encoded MP3 file

Neat thing, turns out MP3s are quite the space saver, 9-10 Mb of audio turned into ~600 Kb if I recall correctly. The stream VOD likely has the exact numbers.

The second pipeline

Alright. Then I wanted to do more cool stuff, such as acting on the live data as it came in and showing something of it in LiveView. So I ended up ripping most of this out to get a better working example. Including stopping all that file writing as my poor machine was trying to encode video at the time.

So this code was changed into a mic_input, an audiometer and a fake sink. It also introduces handle_notification/4 to deal with messages sent by the pipeline as part of the audiometer.

defmodule Media.StreamToFile do
  use Membrane.Pipeline

  alias Membrane.PortAudio
  alias Membrane.Audiometer.Peakmeter
  alias Membrane.Element.Fake

  @impl true
  def handle_init(output_directory) do
    Elixir.File.mkdir_p!(output_directory)
    Process.register(self(), :default_stream)

    children = [
      mic_input: PortAudio.Source,
      audiometer: %Peakmeter{interval: Membrane.Time.milliseconds(50)},
      sink: Fake.Sink.Buffers
    ]

    links = [
      link(:mic_input) |> to(:audiometer) |> to(:sink)
    ]

    {{:ok, spec: %ParentSpec{children: children, links: links}}, %{}}
  end

  @impl true
  def handle_notification({:amplitudes, channels}, _element, _context, state) do
    IO.inspect(channels, label: "amplitude")
    Phoenix.PubSub.broadcast!(Media.PubSub, "audio", {:amplitudes, channels})
    {:ok, state}
  end

  def handle_notification(_any, _element, _context, state) do
    {:ok, state}
  end
end

The updated module is above and was current as of this commit.

The LiveView is modified from calling mix phx.gen media --live and editing lib/media_web/live/page_live.ex like any good quick hack.

defmodule MediaWeb.PageLive do
  use MediaWeb, :live_view

  @impl true
  def mount(_params, _session, socket) do
    if connected?(socket) do
      Phoenix.PubSub.subscribe(Media.PubSub, "audio")
      Media.ensure_playing()
    end

    {:ok, assign(socket, channels: [])}
  end

  def handle_info({:amplitudes, channels}, socket) do
    channels =
      Enum.map(channels, fn negative_db ->
        case negative_db do
          :infinity ->
            0

          num ->
            100 - round(num * -1.0)
        end
      end)

    {:noreply, assign(socket, channels: channels)}
  end
end

It subscribes to the events we send in the pipeline. It mangles the numbers and it updates the state. I love a good handle_info/2, screw events, who needs em. Users shouldn’t interact with my stuff, my stuff interacts with users! I kid, but my most interesting uses of LiveView rarely require the user to trigger the behavior, stuff just goes. I find it super satisfying.

The template side is very dense and simplistic, the video at the end covers my slight style tweaking.

    <%= for amplitude <- @channels do %>
    <div class="amp-bar" style="width: <%= amplitude %>%;">
    </div>
    <% end %>

That’s it essentially. There is some additional plumbing to just make sure it is started and running, so clone and run the project to try it.

A brief video of how it looks and how I solve problems under unexpected circumstances is available below, and also on my YouTube channel.

The post about making a Twitch clone that was referenced is this one.

So while the big goal is the WebAssembly (WASM) target the compiler is built on top of LLVM and could also be great for making static binaries for x86 and friends. There are still some missing facilities in WASM for handling the kind of light-weight processes and scheduling we enjoy in Erlang land. The team is working with the WASM crowd to make those things happen. While waiting we get an x86 target to play with.

So, to get started we follow the instructions at the Lumen repo. This post will get outdated. The project README is more likely to stay up to date. We’ll cover the steps here anyway.

The basic requirements are the following:

• CMake
• Ninja (recommended)
• CCache (recommended)

Installing these on MacOS is done like so (assuming homebrew is installed):

brew install cmake ninja ccache

Now, let’s get the Lumen repo from Github and cd into it.

git clone git@github.com:lumen/lumen.git
cd lumen

First, we need a modified version of LLVM. Thankfully a precompiled version is provided for Linux and x86 Mac. Below we provide the instructions for MacOS, for Linux, use the previous link. The instructions reference $XDG_DATA_HOME as an environment variable, it is recommended to export XDG variables in general, but if you have not, just replace the usages of $XDG_DATA_HOME below with $HOME/.local/share, which is the usual default for this XDG variable.

mkdir -p $XDG_DATA_HOME/llvm/
cd $XDG_DATA_HOME/llvm/
wget https://github.com/lumen/llvm-project/releases/download/lumen-12.0.0-dev_2020-10-22/clang+llvm-12.0.0-x86_64-apple-darwin19.5.0.tar.gz
tar -xzf clang+llvm-12.0.0-x86_64-apple-darwin19.5.0.tar.gz
rm clang+llvm-12.0.0-x86_64-apple-darwin19.5.0.tar.gz
mv clang+llvm-12.0.0-x86_64-apple-darwin19.5.0 lumen
cd -

Lumen and related tooling such as Eir is built with Rust. So you need rustup which is a Rust installer. Lumen builds off of the Rust nightly due to some features, such as WebAssembly, requiring nightly features. Then we use the Rust package manager cargo to install some additional requirements.

# Install rustup by running this script and follow the onscreen instructions:
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

# Set default Rust version to match Lumen's CI:
rustup default nightly-2021-01-29

# Install the wasm32 targets for the toolchain
rustup target add wasm32-unknown-unknown --toolchain nightly-2021-01-29

# Install cargo-make to run build tasks in project
cargo install cargo-make

Then we need to build our compiler. Replace $XDG_DATA_HOME with $HOME/.local/share as above if needed.

LLVM_PREFIX=$XDG_DATA_HOME/llvm/lumen cargo make

At this stage we encountered a problem. The build failed with a rustup error. This was solved by hard coding the toolchain version in Makefile.toml, line 76.

CARGO_MAKE_TOOLCHAIN = "nightly-2021-01-29"

With this fix the build passed, and we have a compiler. Let’s make sure it runs at all by calling its help (from project root).

bin/lumen --help

Let’s create some of our own code, to start we just set up separate directory:

cd ..
mkdir our_code
cd our_code

To actually try things out we need to write some code. The process for Elixir code is less convenient at the moment so we’ll go with an erlang example. This is our hello world in Erlang:

-module(init).
-export([start/0]).
-import(erlang, [display/1]).
-spec start() -> ok | error.
start() ->
   display("Hello World!").

So now we can compile that:

../lumen/bin/lumen compile --output-dir . hello.erl

That generates a binary for us called hello.out. We can run it. Usually you have to set the executable bit the first time:

chmod +x hello.out
./hello.out

To make things a bit more interesting we can make it do multiple things. Lets spawn a couple of processes that will output things as well as output from the initial process.

-module(init).

-export([start/0]).

-import(erlang, [display/1]).

-spec start() -> ok | error.
start() ->
  spawn(fun() -> server("Foo") end),
  spawn(fun() -> server("Bar") end).

server(Message) ->
  display(Message).

A single static binary running from Erlang code, compiled via Rust. Bakeware achieves something similar, as do certain options in Distillery I believe, but that ships the entire Erlang runtime and has a decompression step on the first run due to this. With Lumen the runtime is compiled in. At later points, due to removing hot code updates, there will be optimizations such as dead code elimination and simply not shipping the entirety of Erlang and OTP with each binary. I think that’s quite a compelling concept. update: this is actually already a thing for NIFs, they aren’t linked in if they aren’t used, I really should run these things by the Lumen team :)

The tradeoffs with the BEAM being a VM are generally size and startup time. And it gives us immense flexibility with absurd capabilities like hot code updates. The downside is only for some very particular use-cases where size and startup time matter more. The web is one of these but far from the only one. I would expect Lumen binaries, even when the project has matured a bit, to still be larger than something like Rust and Go binaries but I honestly don’t know. I would also expect them to start a lot faster than your average BEAM-based release. So I’m keen on using Lumen for two things in the future. Elixir-based CLI and web frontend code. With WASM the world does open up quite wide with WASM poised to being a general target for anything that needs to be sandboxable and fast. It is being considered as a container replacement, edge compute, serverless functions, embedded, so many things.

So that’s a preview of the Lumen compiler. I tried to do some timer:sleep/1 stuff but found out that wasn’t implemented yet. If you are curious to see what you do have I think this gave a decent view if you know your Erlang module names. So going into the timer folder there I can see what functions do exist and there isn’t presently a sleep_1.rs.

So this example outputs x86 binaries. The next step is WASM and while I’ve been out of the loop for a bit a bird just told me that there’s a big PR rolling in that should get us to WASM output or thereabouts. If you are keen on helping Lumen move forward, check the issues on the lumen repo and you can help implement more of Erlang/OTP. Maybe get me that sleep function? You can find the runtime BIFs label in the issues.

Big props to Elin for helping me work through this. She deserves a 75% credit on this blog post. All the annoying work was done by her and I’m very happy to know that you can be unfamiliar with the project and get from start to finish. Also thanks to Luke and Brian for correcting me on some important details.

If you have thoughts or question, don’t hesitate to let me know. I tend to both read and respond via email at lars@underjord.io and on Twitter where I’m @lawik@hachyderm.io.

I’ll show some of the process I went through so you can repeat it for your own needs but the repo is open if you want to see where I’ve taken it since. You can find it as lawik/noted on GitHub. This guide recreates a simpler approach based off of what I learnt setting that up.

Setting up your project

I’m using Elixir and Phoenix with LiveView. You need to have the Phoenix project generator installed to follow along, you can follow the Phoenix installation instructions to get it.

mix phx.new --live noted

It should create your project, you can certainly name it something different than noted. I left Ecto in because you’ll likely want it at some point though I won’t use it in this post.

Some initial setup

In lib/noted/application.ex I comment out the Noted.Repo because we’re not using the DB.

In mix.exs for deps I add:

    ..
      {:telegram, git: "https://github.com/visciang/telegram.git", tag: "0.7.0"},
      # Gun mismatch with telegram and cowboy
      {:cowlib, "~> 2.7", override: true}
    ..

The cowlib thing has to do with some weirdness in the telegram library deps. But adding this should allow you to do mix deps.get and have it work.

Get yourself a Telegram bot

So get situated with a Telegram account and add The Botfather (Telegram link) as someone to talk to and that’s Telegrams bot for creating bots. Very meta. Pretty convenient.

You write /newbot to him and he’ll guide you through the rest. You’ll receive a secret that you should squirrel away and we’ll use it as a an API key for running our bot.

You’re going to want a convenient client to test things as well. I’m using the MacOS desktop client on this machine, it is nice and native.

Minimum Viable Bot

It needs to be self-aware. So let’s make that happen. To protect the bot token I put it in a file in my home dir called .mybotcredentials or something similar. This file is just:

export TELEGRAM_BOT_SECRET="my_secret_goes_here"

And before running the server I do: source ~/.mybotcredentials

This prevents me from accidentally committing the damned things.

Okay, time to create this bot. Create a file in the project named lib/noted/bot.ex. In this file we do this:

defmodule Noted.Bot do
  use GenServer
  require Logger

  def start_link(opts) do
    GenServer.start_link(__MODULE__, opts, opts)
  end

  @impl GenServer
  def init(opts) do
    {key, _opts} = Keyword.pop!(opts, :bot_key)

    case Telegram.Api.request(key, "getMe") do
      {:ok, me} ->
        Logger.info("Bot successfully self-identified: #{me["username"]}")

        state = %{
          bot_key: key,
          me: me,
          last_seen: -2
        }

        {:ok, state}

      error ->
        Logger.error("Bot failed to self-identify: #{inspect(error)}")
        :error
    end
  end
end

This is a GenServer and we want to add it to run when our application starts. So below your Noted.Endpoint, inside the list of children in lib/noted/application.ex add this:

..
{Noted.Bot, bot_key: System.get_env("TELEGRAM_BOT_SECRET")}
..

Now you can make sure you’ve run mix deps.get and then run mix phx.server to try it out. Check your logs, the happy message should be there.

Set yourself up to receive updates

So it still doesn’t really do anything. We will be using long polling which is a neat way of getting updates from a Telegram bot without publishing the server on the web on some real domain and in some systems it might be a bit of a pain. In Elixir, on a GenServer it is comparatively trivial. At least if you know your GenServers.

Just after we set up our state and before we return {:ok, state} in the bot we need to add a call to next_loop(). So the your init/1 looks like this:

..
  def init(opts) do
    {key, _opts} = Keyword.pop!(opts, :bot_key)

    case Telegram.Api.request(key, "getMe") do
      {:ok, me} ->
        Logger.info("Bot successfully self-identified: #{me["username"]}")

        state = %{
          bot_key: key,
          me: me,
          last_seen: -2
        }
        next_loop()

        {:ok, state}

      error ->
        Logger.error("Bot failed to self-identify: #{inspect(error)}")
        :error
    end
  end
..

Now to add the rest. This GenServer will send itself a message saying “ey, check for updates” and then act on that message until the activity is done and then call next_loop/0 again to trigger another message. We’ll start with acting on the message with a handle_info/2 callback. In your bot GenServer module:

..
  @impl GenServer
  def handle_info(:check, %{bot_key: key, last_seen: last_seen} = state) do
    state =
      key
      |> Telegram.Api.request("getUpdates", offset: last_seen + 1, timeout: 30)
      |> case do
        # Empty, typically a timeout. State returned unchanged.
        {:ok, []} ->
          state

        # A response with content, exciting!
        {:ok, updates} ->
          # Process our updates and return the latest update ID
          last_seen = handle_updates(updates, last_seen)

          # Update the last_seen state so we only get new updates on the
          # next check
          %{state | last_seen: last_seen}
      end

    # Re-trigger the looping behavior
    next_loop()
    {:noreply, state}
  end

  defp handle_updates(updates, last_seen) do
    updates
    # Process our updates
    |> Enum.map(fn update ->
      Logger.info("Update received: #{inspect(update)}")
      # Offload the updates to whoever they may concern
      broadcast(update)

      # Return the update ID so we can boil it down to a new last_seen
      update["update_id"]
    end)
    # Get the highest seen id from the new updates or fall back to last_seen
    |> Enum.max(fn -> last_seen end)
  end

  defp broadcast(update) do
    # Send each update to a topic for others to listen to.
    Phoenix.PubSub.broadcast!(Noted.PubSub, "bot_update", {:update, update})
  end

  defp next_loop do
    Process.send_after(self(), :check, 0)
  end
..

Try it out, it should log messages as they are sent to your bot.

Getting real-time with LiveView

We can repurpose the existing LiveView that the Phoenix generator gives us. So just open lib/noted_web/live/page_live.ex and its sibling the template lib/noted_web/live/page_live.html.leex. In the page_live.ex file we change it to this:

defmodule NotedWeb.PageLive do
  use NotedWeb, :live_view

  @impl true
  def mount(_params, _session, socket) do
    Phoenix.PubSub.subscribe(Noted.PubSub, "bot_update")
    {:ok, assign(socket, messages: [])}
  end

  @impl true
  def handle_info({:update, update}, socket) do
    {:noreply, assign(socket, messages: [to_message(update) | socket.assigns.messages])}
  end

  defp to_message(%{"message" => message} = _update) do
    firstname = get_in(message, ["from", "first_name"])
    lastname = get_in(message, ["from", "last_name"])
    username = get_in(message, ["from", "username"])

    from =
      case {firstname, lastname, username} do
        {nil, _, username} -> username
        {firstname, nil, _} -> firstname
        {firstname, lastname, _} -> "#{firstname} #{lastname}"
      end

    text = get_in(message, ["text"])
    %{from: from, text: text}
  end
end

And we change the page_live.html.leex template file to:

<section class="phx-hero">
  <h1><%= gettext "Welcome to my Bot!" %></h1>

  <%= for message <- Enum.reverse(@messages) do %>
  <p><strong><%= message.from %>: </strong><%= message.text %></p>
  <% end %>
</section>

Run this and you should see any message to type in the Telegram bot show up. I find it very gratifying just how instantaneous it can be.

In closing

This is just a start and in the noted repo you will find some of what I’ve done to push it further, structure the bot a little bit more. Add some responses. You can just try it out by cloning and setting it up.

I implemented some authentication. Currently this means that notes are separated by user. It also means that Noted allows new users by default if they find your bot :D

I implemented the database backing to save notes and tags. I added a bunch more LiveView code. I separated the polling for getUpdates from the handling of the updates a bit more and added a Supervisor to the pairing of GenServers.

Currently on my todo-list is to receive files and media in a nice way and integrate those with the markdown support. If you want practice with Tailwind CSS I’d love for someone to make it look better.

But this particular project aside I find the realtime nature of chat bots very satisfying. Telegram has a well-working API, nice native clients and a hilariously large feature set. And with how fast you can get this up and running when it isn’t your first time I really recommend trying it for things. Heck, you could use it for sending you notices about your app catching fire or occasional status updates from your system. There are so many fun options.

Slightly timely update, especially if you are curious about LiveView or want more of me. Me and a wild gang of Elixir community folks just put our new podcast out there. It is called BEAM Radio and you find it at beamrad.io.

I hope you’ve enjoyed the post. Let me know if this is something you would like to see more of as I have quite a few things I could go deeper into from this project. You can let me know via email at lars@underjord.io or on Twitter where I’m @lawik@hachyderm.io.

It also lets me verify that the feed has updated correctly and works fine. If it comes out weird in your feed reader, please let me know.

If you aren’t familiar with RSS. Consider it the open standard that drives podcasting and an exceptionally useful way of reading blogs, news, webcomics in a more intentional way via a dedicated, nice, newsreader application or web UI. It lets you avoid attention-grabbing or remembering to check but it doesn’t have to go “PING!” or anything. More things should be like RSS.

I blame Martin Gausby for reminding me that full articles in RSS is a better experience than needing to click through. So in the interest of pampering my audience with the best text experiences possible. There you go ;)

This continues our dive from Part 2. Part 1 can be found here.

The better system 2.0

The big embarrassment of the system we’d previously built was performance, it wasn’t scaling with a growing customer base and was slowing down. When discussions started for a proper rewrite of the system “scaling” was probably the central word. There was another parallel product in the company that was also experiencing performance issues. This put performance front and center but since it was attributed to growth the focus was more on scalability than specifically efficient performance.

We were tasked with finding a plan. This was when microservices was heavily on the rise and they promised to let us scale horizontally rather than vertically. In what felt like a stroke of pragmatism and wisdom we leaned on the experience of others. We read the blog post In praise of “boring” technology by Spotify and basically followed it. Making plenty of decisions along the way since it is fairly high-level.

I rather like the core of that post still. It basically reinforces the idea that you don’t need to get fancy. You can use well-known, well-proven tools and build a series of simple things that achieve something complex and they are likely to be more reliable than more new-fangled and fancy solutions. Reliable, maintainable and thus, potentially sustainable.

I don’t think leaning on the general idea was a bad call. I have some reservations about leaning on the details of it that we should get into. The basic philosophy is still pretty close to what I subscribe to.

The second priority was to create something that could be re-used for when the other product and any number of future products would be created. That is, re-use what makes sense. Build some things to be general. Sounded good at the time. The impact of this and the way we divided our microservices had an enormous impact down the line.

Meanwhile the product designer and some domain experts got busy with focus groups and exploring what the system should be to take what we’d learnt from the existing version and push that to the next level.

Product team life

I really like the team we had. Good people throughout. Overall I ended up having most of the technical responsibility because I was the only one with a 100% time commitment to this product. Or mostly 100%, it varied over time but during this green field development it was certainly most of my time. We were 2-3 devs off and on. One product designer, UX, sometimes JS person that was learning programming as we went, a pretty hardcore journey and an interesting story on its own.

I really liked being on a team with only the product work to focus on. Agency life was always a balance between serving many masters and delivering enough stuff on time. Here we more or less embarked on grand voyage to build The Better Thing.

In the end we also had deadlines and limitations, so the first part of the work was the thoroughly considered and enjoyable part and then there was a significant period of forced march to get it past the goalposts. Not the best but not the worst I’ve had either.

This Time We’ll Do It Right

So the technical stack was:

• Python
  • Django
    • Django Rest Framework v2 (for our product-specific API)
    • OAuth2 toolkit (for our generalized account service)
  • Flask (for our smaller public APIs)
• ZeroMQ (for RPC, Queue/Worker, PubSub communication)
• Protocol Buffers (for message formats/contracts)
• PostgreSQL (main datastore)
• Redis (cache)
• Elasticsearch (as needed, for some statistics)
• Bind (not used for DNS, only service discovery)
• AngularJS (web frontend, 1.x series, SPA-style app)
• Android app, thin web wrapper
• iOS app, thin web wrapper
• Ionic (1.x, later limited featureset, better UX app)

I don’t really regret the specific technical building blocks. There was a lot of implementation detail that was not right and several architectural turns I wouldn’t take today. And some of these things I wouldn’t pick up now (Angular 1 and Ionic 1 were both evolutionary dead-ends apparently). Many of these things could have been done easier with slightly different tools. But I’m happy with the selection and would definitely consider using a bunch of these tools again. ZeroMQ specifically isn’t something I expect to use again but I learned tons from getting to grips with it. The ZeroMQ online guide was a good read, if a bit hyperbolic at times.

So the microservices thing. Turns out how you divide your services matters a lot. We divided a lot to get that sweet, sweet scalability. We broke it apart into something like this:

The general services

• users (user data, primarily for auth, password hashes, resetting passwords, confirming email, extra security concerns)
• account (OAuth, SSO, public login UI, profile information and public APIs for anything concerning the user)
• relations (hierarchies, organizational units, user roles)
• files (public endpoints and internal, file storage, hooks for access checking, video transcoding)
• media (image sets, embeds, extra data such as captions, files)
• notifications (queueing, separate workers for GCM, Apple Push Notifications and email)

Product-specific services

• public API
• core domain functionality
• calendar functionality
• messaging functionality
• export functionality
• access (used by files API and public API for authorization)

Independent parts of the domain were separated which was generally fine. One slow killer was that we generalized out the organizational structure, and user roles to a service, user information to a different service and user access checking to a third service. Over time I ended up building so much caching around this to make it faster. So much caching.

Managing files and media in a different service worked pretty well and that did need to scale separately as it turned out. Now we didn’t need to make it two services, the line between media and the underlying files did not end up being important and the general case implementation added a lot of overhead that caused frustration anytime anyone wanted to understand how the system uploads a picture.

ZeroMQ and Protocol Buffers was much more hardcore than we needed. The multitude of git repositories (one per service, one per frontend client) were sometimes painful and a monorepo would probably have been easier to keep moving in good order. We should have used python packages or just some scripting for some things where we used git submodules. Git submodules have never been the right choice for me.

What did it do well?

Well it worked. And it could scale horizontally and did so whenever that turned out to be the bottleneck. It did a good job separating concerns between services sometimes.

Once the very painful migration was done and most early bugs worked through customers generally liked the system. Overall it was an improvement. It was definitely visually more professional which some loved and some hated.

It was built to match the government requirements placed on preschools much more closely and I believe it did that well. Including having good support for showing the curriculum and tagging documentation according to the curriculum.

It also did well in separating frontend development and backend development a fair bit. Unfortunately the win was limited there as in the long term all development was me and other full-stackish people. It did allow releasing frontend updates without making a bunch of backend changes. That was good.

What technical flaws contributed to its end?

The weight of the architecture was crushing. It was built on ideas from a much larger company with much larger teams. I spent long periods of time alone in running and developing the system. I got to be pretty fast at it. But it was an incredibly poor choice for a small lean team to spend time updating protobufs for service communication and worrying about scaling calendar and messages separately when both were at trivial usage.

The cost of the number of hosts we spread across was so much more than should have been necessary for a product at this scale.

YAGNI. You Ain’t Gonna Need It. We ended up building one prototype product beyond the preschool product on top of this ecosystem. It was abandoned quite quickly. The idea of sharing services was abandoned but this product bore the weight of that generalization until the very end. Especially with an unecessarily generalized and complex system for handling the customer org structure which affected access checks, authorization, querying for common pieces of data. This was why I needed to add a lot of caching, so many checks. I started efforts to bring this closer to the core and remove the generalization but never had enough time.

No circuit breakers or controls to prevent services just murdering each other during high load. This lead to some quite pathological problems whenever something hit a performance snag or reached a breaking point and it could take a bit for the system to recover. We also had very spikey usage. Everyone posted things around lunch and every parent and guardian would get notifications and check around the same time during their lunch.

Wrapped web apps on mobile. So much futzing with video and bad web view feature sets. This was on the old iOS web view and mostly on the old Android one as well. The later Ionic stuff worked great. Until we hadn’t updated the app in a while and our dependencies rotted and shipping an update turned into NPM dependency hell.

What did you learn from working on it?

So much, so very much. Reading up on ZeroMQ was great fun learning to wield it was wild. Building the request/response protocol we used as well as queue/worker and pub/sub patterns were great learning experiences.

Lots of Ansible. So much Ansible. Our playbooks weren’t perfect but the Ansible part of deployment was generally great. Our vagrant dev setup was, eh, not great but worked.

I learned to dislike the GIL in Python as we actually tried to use threads and later green threads for a bunch of things.

APM can be incredibly useful. We used New Relic which was costly but did so much for letting me figure out bottlenecks and fix things.

I don’t disagree with all our choices at all. They were wrong for the product and the team. But the choices themselves were mostly fine. But with this particular combination it mostly caused resource starvation which wasn’t particularly fun. So I think this has given me a better sense of what can be heavy and what can be light and when one or the other is appropriate.

Invent fewer things if you can. We did too many things basically from scratch. Super fun to build. Could have done something simpler. Thrift would probably have given us all we needed rather than ZMQ. But even so, I don’t think the microservice approach was necessary at all for this product.

Don’t work counter to your tools. We really did not work in alignment with Django for our uses. Our biggest Django application didn’t have a database, it just called out to services. So we lost most of the conveniences offered via Django models. It was still more featureful in some ways than Flask but it wasn’t pretty.

We could have just built on Django, Rails or Laravel and done all of it. We couldn’t have used Phoenix because it didn’t exist at the time. I don’t think Elixir existed when we started, I’d never heard of it at least.

I got to work with a bunch of the tricky parts of large amounts of files and how tricky performant IO can be, before object storage was standard practice.

So I learned a lot of tech throughout this but I probably learned more about tradeoffs during the creation and the subsequent operation of this system.

In reflecting on this system I’m glad for the experience overall. Each iteration pushed me along in building my skillset and challenged me immensely. I’m continuously both proud for the product we provided and how appreciated it was while also being very frustrated that we didn’t make better choices so we could have provided something much simpler and much better. The power of hindsight.

A big take-away is that before you’ve done something it is hard to know what running the system is going to be like. And before you’ve lived the experience or have seen enough of something from afar it is difficult to really determine trade-offs. My current thinking is that keeping things lean and simple works very far. It both scales and performs. I have so far found it easier to break things apart when required rather than needing to shove things back together after the fact.

This ends the saga. For now. I think I’ll write up an epilogue on the end of the product (it isn’t around anymore) and my attempt at creating a spiritual successor.

If you have questions or thoughts on this post feel free to contact me at lars@underjord.io or on Twitter as @lawik@hachyderm.io.

Content warning: Contains poor technical decisions, inexperience and stories of a developer just starting out and up to roughly present day. Be kind to who I was, he gets enough shit from me.

The better system

The team that had formed inside our little division of this larger organisation ended up breaking out to run an agency startup. Apps and web development would be our game. And it was. I did not quite know what a REST API was, every problem could be solved with Drupal. But damn if I wasn’t on the right side of the curve to not know how much I did not know.

So I was the tech lead and most senior web dev. I had my trusty guy from the cheese factory by my side. We had a young, enthusiastic designer who was still a goddamn delight of a person last time I saw him. We had a somewhat grizzled veteran of solo entrepreneurship running the ship and doing all the sales, project management, marketing. A lot.

We also brought on a newly schooled app developer who worked on apps for this system, remotely.

There was a bunch of things going on and a lot of craziness throughout. But to the product point, we got tasked with building a modernized version of the system we had built, with mobile apps and video support. This time for a tech-focused company in the education sector.

I am not sure I have ever pulled a more finicky tooth of technology than the Video Field module that was available for Drupal 6. FFMPEG conversion commands, presets that didn’t work, cron-jobs and JS libraries for display. Abstraction layers for days. It was a whole thing. Today I think I’d pick that thing up quite quickly. But at the time it was absurdly complex.

During this work we started learning how to build things The Drupal Way, there were a lot of interesting flavors of Kool Aid. I won’t dismiss all of The Drupal Way, it has some very interesting properties. But I do not want to revisit that tech. It is all about managing whatever configuration and state you end up having in your database. Because anything that isn’t code ends up in the database.

If you know any Drupal from this era, this was CCK, Views, CTools, Pages. The ecosystem is honestly fascinating and powerful in an extremely odd way that barely transfers to anything else.

Agency life

As you might imagine from our relative inexperience in building services and being very new at building apps and APIs the process was quite fraught. Our fledgling company had trouble balancing different client projects and a bunch of delays and complications lead to a hefty backlog, overtime, expanding the team and the company swelling like a balloon. The system had releases but keeping it moving forward and keeping the client happy was a rough process.

The client was honestly very reasonable and patient. But as a company we were struggling. Mostly because we couldn’t deliver and close things down. This was again largely due to our inexperience but also had do to with scoping projects, pitching with enough margin and generally saying no to things.

It was an extreme learning process for everyone involved. The product was launched, updated and put to use. Bugs and continuous delays aside it was a good product. It had great market fit. It was the first commercial system anyone ever heard of for the preschool segment in Sweden. It had charming design, it had native apps that allowed smartphones to be used for publishing to the system. Parents could see what their children were doing and receive push notifications.

We could never quite finish up our commitments to the client for this product, there was always something else. Always one more bug, always one more promised feature that hadn’t quite landed. Around this time our company folded because of the rapidly expanding payroll and continued limited cashflow.

You could say the way this business was run heavily influences how restrictive I am with my commitments when running Underjord. I am very particular about not committing to more than I can deliver on and my aim has always been to grow organically, in a sustainable fashion, if I am to grow it at all.

I quit just before the company folded actually. Not that it mattered. I got in writing that if I wished I could work for our client that ordered the product because I’d spent some time at their office towards the end and it seemed like a good team, good offices, good company.

Maintenance hell

Oh yes, the product was successful. Probably not profitable, I was not privy to those details at the time. But it was popular, sold well, was appreciated. It was also getting dog slow.

The “node” table grew huge. We had to raise the integer size on the primary key. And we did not have that many customers. In the Drupal 6 days this was the table which gathered every piece of content. In the end, most data aside from user profiles ended up in there.

I ended up freelancing for the company and working on this product, massaging away bottlenecks, figuring out performance issues. Finding a new solution to file storage which was getting wild. I’ll have you know we stored Terabytes of media!

The system was well-liked overall but was often slow at this point. It was hard to maintain, run and develop. Drupal 6 with heaps of custom code and configuration on top of it did not give us a clean path forward. Meanwhile the company had another product in Drupal 7 which also had performance issues and a lot of unwanted complexity. So our team at this company turned our eyes to the horizon. It was time for a new system.

But first. Lets review.

What did it do well?

It delivered something the customers definitely wanted and were excited about. It delivered on an organizational requirement they had that was handed down to them by the government. It mandated a certain systematic documentation of activities in preschool education. It also served a number of practical needs and desires. Such as sending information out to guardians.

In some sense it was built in the simplest way we knew. It was never over-engineered.

It was simple and easy to use. It was an incredibly common piece of feedback about the system. The customers understood it. It modeled their organizations well, it made sense. It largely acted as expected. It was simple and that was the single best thing about it. We had untrained preschool teachers working as effective admins for their orgs. It mostly worked well.

What technical flaws contributed to its end?

So many. File management was done through Drupal 6 and PHP, including a maddening video conversion cron-job that was incredibly flaky. It all ran on one server, a physical one. Not bad in itself but scalability was definitely a concern. After a while files were moved to a mounted file server when adding disks became untenable.

VPS:es were a thing, we didn’t know much about them when we started and there was precious little breathing room for making transitions and non-critical work.

Performance, caching, page loading performance, file serving. So much performance-related stuff. PHP opcode caching was one thing that helped. But there was so much complexity, global state gone wild and so much weirdness. We used a lot of Views and Pages which are Drupal modules that do rather cool things in some of the most complex ways I can imagine. The are incredibly flexible, powerful and hard to not shoot yourself in the foot with. Its an odd ecosystem. I was really into it at the time. But also falling out of it rather quickly.

Our users were change-sensitive. I think we did manage a fairly successful overhaul of the admin backend. But in general, changing anything related to finding your way through the system was heavily disliked. I think that was a good take-away honestly. Don’t change things unless there is good reason.

The LAMP stack wasn’t the problem, Drupal 6 wasn’t the problem, file storage wasn’t the problem. But I will say that our pressure-cooked code-base made by inexperienced and stressed people in this combination of tools was an absolute liability. But it was also a successful and much loved product. Go figure.

What did you learn from working on it?

It don’t gotta be fancy.

We were early on the app side of things. We were fairly old-school on the web side, a major version back which in Drupal terms, that’s like a major version of Debian back (or at least how Debian used to be, glacial).

A good fit for the customer matters a lot.

People genuinely loved this product. It was heart-warming. They could be pissed at it and about things we did. But we had so much positive feedback.

Friendly and helpful customer support is golden.

Our product was largely represented by one person. An upbeat guy with patience for days and a real enthusiasm for helping people figure things out. And when he couldn’t wrangle the issue I’d usually be involved, or one of the other occasional team members.

Fit your purpose and know your people.

I think this was an underserved group. Actually, I know it was. It still is to some extent. And they rallied around their solution. The product was only for preschools. It only prioritized preschools. It didn’t aspire to be anything for primary school or above. The preschools in Sweden run quite differently from the stages of schooling that come after it. They loved the platform for this. It was theirs.

On more personal skillsets. I learned how to run and not run projects. How to get things done under intense pressures as well as under more reasonable work-life balances. I learned how much more productive I can be when I’m not splitting myself over a large number of priorities.

On a technical level I picked up a lot of Apache, Nginx, PHP, Drupal 6, MySQL, some Redis and plenty of Linux admin. All the PHP optimization tips and tricks. I did weird things with .htaccess files. FFMPEG and video conversion. AJAX, which was hot back then and often really bad to work with in Drupal (AHAH, I remember you). I had learned Python in parallel but didn’t get a chance to use it. RabbitMQ for moving away from the video cronjob was a thing. So much random stuff.

I’m not certain it matters how I feel about the total experience. Because it has been incredibly influential on my work and what I do. It is base facts and part of the groundwork for everything I know now. And I have a warm place in my heart for the product and its users, a touch of shame for some of the technical outcomes and plenty of understanding for why we ended up where we did.

Next, we knew we needed to scale…

Part 3 is slated to arrive in two weeks. So join the newsletter or subscribe to the RSS and you’ll find out when it happens.

So I used Phoenix 1.5.6 that I had already installed. The Phoenix docs should cover installation for you. My projects is called pubd, we’ll see if I get it go anywhere in that case that’ll be separate posts. I ran mix phx.new pubd --live to set up the Phoenix boilerplate project and that lead to this commit. The --live flag indicates that it should give us LiveView.

The next part was based on Chris McCord’s article about Alpine on Dockyard.

So I ran these commands to install Alpine.js:

cd assets
npm install alpinejs

// .. after the app.scss import
import Alpine from "alpinejs";
// .. in the let liveSocket configuration, after params
dom: {
    onBeforeElUpdated(from, to) {
        if (from.__x) {
        Alpine.clone(from.__x, to);
        }
    },
},

It lead to this Alpine installation commit and afterwards I decided to verify that I could successfully invoke Alpine and added this other Alpine verification commit.

Alpine was fairly straight-forward. Tailwind CSS was a bit more involved as that has much more dealings with preprocessing and the JS tooling. I followed the Phoenix 1.5 parts from this guide by Pragmatic Studio.

Rather than restate their very nice tutorial here which feels rude I want to offer you two options. Either you work through the above tutorial and gain all that related context they so kindly included. Or you work through my commit diff. Either way you should end up with a working Tailwind CSS install.

My approach included removing the default Phoenix styles which of course made the default application quite ugly. I couldn’t be bothered to fix that and instead I just wanted to make sure Tailwind was doing its thing. A commit for my test is also available.

Now with these installed you should be able to get going with working in this stack. Assuming you know a bit about Elixir and Phoenix you should be quite capable of learning LiveView. And when that won’t suffice you have Alpine.js. If you need visual tools, you should have good use out of Tailwind, or so I hear. I’ll link their docs here:

• The Elixir language guide
• The Elixir documentation on Hex
• The Phoenix documentation on Hex
• Alpine JS GitHub docs
• Tailwind CSS docs

I genuinely look forward to digging into more work with this stack. I’ve heard plenty of hype for Tailwind CSS and the way they use classes looks utterly wrong to me so far. So it should be a novel challenge for my sensibilities. Maybe I’ll hate it? Alpine truly seems to follow the Tailwind concept, so that’s similarly noisy. Maybe that’s all good pragmatism. I hope to find out.

Anyway, I hadn’t seen a guide for setting up the PETAL stack so I figured I’d put one out there. Bare-bones as it is.

In case you’d like to know more about what I do, sign up for the newsletter. I write more about Elixir, my projects and developer life there.

If you have thoughts, questions or want to otherwise get in touch, reach out via email lars@underjord.io or @lawik@hachyderm.io. If you want more of my writing, consider my newsletter.

Content warning: Contains poor technical decisions, inexperience and stories of a developer just starting out and up to roughly present day. Be kind to who I was, he gets enough shit from me.

I will cover some different iterations of the same kind of product/system along the way and I will go back to a few questions as we consider each system.

• What did it do well?
• What technical flaws contributed to its end?
• What did you learn from working on it?

The first system (2009)

During my first job as a professional web developer, not as my first project, a few projects in, I ended up technically responsible for building a web system to support the preschools in an entire municipality. I was a very cheap contractor. I think I pulled my weight overall.

It was a fairly tight budget project, I was at a stage of my work where I had three hammers in my belt, they were named Joomla, WordPress and Drupal. I liked Joomla because it had a lot of GUI for admin and looked sharp by default. I liked WordPress for simpler, bloggier sites. I had just started looking into Drupal and found the way of Drupal 6 a lot clearer than all this intricate MVC-stuff I’d never seen before.

So for building this rather custom system I knew WordPress was a bad fit, Joomla would only work if there was a ready-made extension or something for Swedish preschool systems. I picked Drupal. From this point and for several years to follow Drupal would be the hammer to most nails I encountered. I have feelings about this. But at least its funny to me now.

I started the work on this. We brought in another developer for the project. I recruited him by recommendation from a conversation in a locker room at the Muay Thai club I trained at then. He left a not-so-lucrative gig at a cheese factory to try his hand at actually working with the computer stuff he knew. Great guy. I never regretted bringing him on and I hope he doesn’t miss the cheese because he is still in tech.

I think we delivered pretty well. It was all very exciting. To our advantage, the needs were quite basic. Each preschool needs a blog/newsfeed. Each preschool needs to be able to write portfolio posts for the different children. All posts could have pictures attached. Teachers could set up reminders in the system to remind parents about things. This was fairly simple in Drupal and if we had more experience it could’ve been quite clean. It was not that, but it worked fine and did well enough. There were no apps, no videos, maybe some mail-notifications, I forget. iPhones were not a thing in education and iPads (2012) hadn’t even been conceived of yet.

I don’t think there were any additional development efforts on the system, I don’t recall much beyond some post-launch fixing. But it did introduce me to the audience of preschools. Preschool teachers at the time were not tech savvy. We got a lot of praise for our system being simple to use. Not too complicated. This would be a thing going forward. Customers that were generally pleased and a system that strived for user simplicity.

What did it do well?

It was pitched to provide a blog-style newsfeed and portfolios for the kids and their parents. It did. It worked. They could be exported. It did what it should and was very straight-forward. Simple design, simple execution, probably not particularly performant. I couldn’t really say from here. But I know I didn’t know enough to make Drupal fast at the time.

What technical flaws contributed to its end?

Not really any as far as I know. It was mostly a bunch of custom modules in Drupal along with a bunch of configuration. It ran its course and was probably never updated as the smartphone-age hit. I know the municipality was later a customer in the next version of the system.

What did you learn from working on it?

Tons but all of it somewhat basic. A lot about basic Drupal development, a bunch of polish on the ol' PHP skills (this would be PHP 4 I think, definitely LAMP). Lots of CSS-fighting to get a decent cross-browser look according to the pixel-perfect cartoony design.

Thank you for reading. I expect to publish the next part of this in two weeks. While you wait, if you want more of my writing, my newsletter contains things that the blog does not. You can sign up for that at the bottom of the page. I also provide an RSS feed for this blog. If you have questions or want to get in touch you can reach me via email at lars@underjord.io or on Twitter as @lawik@hachyderm.io.

Underjord is my business and most of the work is me doing things. My wife does the books as that’s conveniently her profession and at times I’ve brought in people I’ve worked with when the budget allows. Now this isn’t the first time I’ve been self-employed but it is the first time I’ve continued to run my business, worked multiple clients, established separate revenue streams and generally buckled in for the long term. The other times I ended up an employee in a year or two after settling in with one large client. That was fine then. Now I’ve been going for two and a half years, I’ve had a number of different clients, some very brief contracts and consultations, some longer term engagements. I’ve weathered most of a pandemic, the birth of our child and moving to a house. So it feels solid. I planned to go independent and I have stayed that course.

Since my wife does work with businesses in her accounting work she gets the world of business very well. And since she and I share our lives and this business is an integral part of our personal finances she’s definitely invested. But even with her special incentives this sometimes frustrating thing holds true. No one cares about your business as much as you do. It’s not weird exactly but it can become painfully obvious when you try to talk about it with people who aren’t interested in business. Most people don’t even care about running a small business in the abstract. Even less about whatever your deal is. That’s fine. But it takes up a significant chunk of my mind and business owners I’ve spoken to share the sentiment. Being able to really get into it with someone who gets it is fantastic.

A good second best is listening to people that talk about it. And thats what I want to push for here in the form of podcasts. If you follow my newsletter I’ve covered a bundle of these there already. If you are curious about the newsletter you can sign up at the bottom of the site.

Focused (formerly Free Agents)

So this podcast has transformed from being about working as a free agent into a productivity podcast. It is a pretty decent productivity podcast but I’ll admit to missing the Free Agents and preferring the olden times.

If you want to run your own business and want to listen to people that do, go back to the older episodes and pick some that sound good to you. They hit on some good topics and I think it was quite helpful to verify and adjust my thinking on a number of things early on.

Hurry Slowly

This podcast is magnificent in a very understated way. It is strangely serene and lovingly crafted podcast. It sits in the productivity space. But for me this was very useful when starting my business as I have a brain that wants to go fast. And this podcast is all about pacing yourself. Spending the time to slow down, give my brain some time to work things through, consider some things carefully. The things I’ve done most deliberately are the things that have paid off the best down the road. I credit this podcast with a lot of that.

I would suggest starting from the beginning and then picking topics you like as you go a bit. It’s fine to skip an episode you don’t care about. Further along as show host Jocelyn K. Glei opens up more it can get a bit uncomfortable if you are a very skeptical person as she opens up about alternative interests. I think it has remained mostly good and while I’m quite skeptical of most of this stuff myself I value her perspective and I think listening to the less out there stuff in the beginning is valuable to anyone but I don’t expect my audience would appreciate the recent stuff without getting to know the more concrete episodes from early on.

There is especially one on transformative gatherings that I found incredibly useful. Because I know people that do these things well and I want to understand how to do it myself.

Cortex

Very private YouTube creator CGP Grey and fairly public podcast network founder Myke Hurley talk productivity, apps and all that stuff. They get into business and independent creation quite a bit and I’ve found their thoughts quite useful. It also helps that the two are quite different and I don’t feel like I align fully with either of them which gives me a good spectrum to consider on many topics.

Lots of Apple as per usual with Relay FM. If you hate Apple you might want to steer clear but while I run a very mixed setup I’m quite curious about the Apple enthusiast space so it suits me fine.

Ditching Hourly

Jonathan Stark has a thing about not billing hourly. I think he has some points, especially for experienced techies there. Beyond that, this podcast repeatedly gets into your relationship with your clients. How you market, sell and execute. How you promise, negotiate, figure out what clients want and all that. There’s lots of good stuff. Unfortunately the podcast occasionally becomes very mostly a vehicle for fairly cheap questions and answers and after a while you’ve heard a lot of what he has to say. But I think the value before you hit that point is quite high. And I still check in on it.

The Business of Authority

Now this is Jonathan Stark again but now with Rachelle Moulton and better topics. So this gets into developing a business in the consulting space. I think its incredibly valuable for programmers that want to be more than just a pair of hired hands.

The Art of Product

This goes into the space of trying to start a product business and I think that’s in mind for a lot of independent developers since generally you have the skillset to do the core building. This podcast covers both that and all the other stuff that goes into running a business. I think they cover it well and they don’t overstay their welcome. Strongly recommended if you think product might be for you.

I will cut this list there and call it short and sweet. I hope this ends up useful to you. Considering they are podcasts you are likely hours away from finding out still, even if any of these speak to you. For me these sorts of podcasts have kept me good company, contributed greatly to my thinking around my business and in general been very entertaining. I hope they serve many of my readers just as well.

So I don’t particularly love the VC-backed model of building businesses. It sort of makes sense for some types of endeavors but generally I’m an organic growth and small teams kind of person. I’ve been in a fast-growing small startup and have the scars to prove it. I’ve also seen good and bad examples of how these kinds of startups can work. Startups that can work at a sustainable pace basically mostly feel like any other company and are fine. Startups that are under fire or in financial panic are generally rough to work inside of. Not my type of business to run and own these days but highly relevant to our sector and Elixir is quite popular in the startup space, as it took on the mantle of Ruby’s successor.

I don’t know either of these companies intimately. I basically just found out about Boulevard, I might have heard the name before but not sure. Discord I know as an occasional gamer and community participant, they’re huge of course. These companies are at different scales and with different levels of name recognition I dare say. But they both build on Elixir and have done so since fairly early on. They both also seem to be killing it as they say in the biz. I think, I’m only tangentially in any biz.

• Discord is close to closing a round with a $7B valuation
• Boulevard closed a $27 million Series B

Both of these are serious money. Different levels of serious but either way, quite serious. I’ve written a little bit about Elixir as a competitive advantage and this just adds some good sauce on top of that.

And I do believe the technology choice has given these companies a genuine advantage. Let’s pontificate.

Discord’s use-case is so very closely mapped to telecom needs that it’s almost absurd. They do calls and chat, presence and soft real-time. So building on something with the backing of Erlang and the BEAM, straight out of Ericsson, probably fit pretty much perfectly.

Boulevard’s case isn’t as clear to me. I think they are closer to the startup that would have defaulted to Rails a bunch of years back. So what this tells me (my sample size is like 6+) is that whatever Rails brought to the table Elixir seems to be matching. They actually had a Sean Stavropoulos on the Elixir Talk podcast that just resurfaced to talk. And he called out the lower memory usage on Heroku and a bunch of other sweet stuff. Good listen I thought. Their use-case is not as clear because they aren’t necessarily connecting calls and sending audio, doing lots of real-time stuff and all that. They might be but their business model would likely not demand it. To me this strengthens the image of Phoenix and Elixir covering the 80-90% use-case. Which is even more important.

As I’ve been prone to claiming. I think Elixir allows you to build better solutions. It gives you an exceedingly powerful baseline if you treat it like any other high-level language with a web framework. But you have enormous headroom to build systems that are a nightmare to build in languages with a similar expressiveness. The trade-offs you currently make with Elixir are things that place the language and runtime squarely in the space of building distributed services. I wouldn’t use it for a CLI application because of startup time (though Lumen might fix that), I wouldn’t use it for a mobile app because it doesn’t really fit (Lumen might help there, but I don’t expect it to). But I mostly build services. And there it just allows me to do more with less. Less hardware, less dependencies, less complexity. And it gives more. More guarantees, more clarity, more consistency and more reliability.

If you have thoughts, feel free to reach out at lars@underjord.io or on Twitter at @lawik@hachyderm.io. If you want more of my writing I have a newsletter where I write on similar topics but perhaps more ephemerally and more experimental stuff. You’ll find a signup thing in the footer.

What is it? Well, a live number of my readership for the entire site for one. If you turn on the magic you should also see the rough positions of other readers on the site that are on the same page as a small hovering geometric shape. I call them wisps.

It’s not often that you can see the footfalls of other people on the web unless they write a comment or something. I don’t want to expose my readers to one another in any privacy-sensitive way or in any fashion that requires moderation. This simply gives a rough indication of activity on the site and whether people are reading the same thing as you. And it moves around in fun ways.

Anyway, I enjoy it and I hope you will too. So how is this done? A small backend in Phoenix using Channels (basically just WebSockets), a small amount of Javascript to render new elements and update positions based on the state updates from the backend. The idle hovering of the symbols is CSS Animation and moving from area to area as the reader reads on is done via CSS Transitions.

I was hoping to default to on, but as I’ve found out with my sweet animated gradient lines, CSS animation can get CPU intensive. Right now my gradient lines pause after a period of inactivity and only run if there is activity. They hit 12% or so on my CPU which is dumb. A better fix to make them more lightweight is on the way, the idle-handling is a mitigation. So for these I wanted to avoid a big performance hit. They perform okay right now from my tests, they use the cheap-to-animate transforms so there’s nothing too heavy. But also, there’s not a known upper bound on the potential number of wisps active at the same time and if that scales poorly on some browsers I don’t want to default to setting someone’s phone on fire.

The number is available regardless since that doesn’t animate. I think showing this sort of thing is a fair way of both measuring concurrent visitors and sharing that with the visitor. I can see it, you can see it. I’ve also been considering a 90’s web visitor counter that is actually live. Could be fun. Just gotta find the right aesthetic.

I like this way of extending my site. I guess in a JAMStack fashion. The site is static but I can build small interactive parts and back them with minimal backends which for stateful applications Elixir does exceedingly well. If these backends go down the site will still work. All the important stuff is there. The fun stuff degrades to a connection error in the console.

Let me know how you like it, I do get and read responses to my posts and I appreciate when y’all reach out so try that via lars@underjord.io or @ me on Twitter @lawik@hachyderm.io. If you want more writing, sign up for the newsletter further down the page.

During my first few days as a free agent I did reach out to one recruiter in particular. This was the one that had been closest to dislodging me from my previous position and I had a feeling he was a sharp one. I had also thrown my cousin at him and he had helped him land his first real ops gig. When I got in touch this recruiter quite swiftly landed me my first client. In parallel I started to accept more recruiter connections and had a lot more conversations with assorted recruitment agencies. It has netted a fair bit of work. But I dare say the hit-rate is mostly low.

The recruiters that I’ve found to give the best results also give recurring results. They are the people that follow up, consider your needs, balance them with client needs and make things happen. It is my feeling that there remains a large cultural gap between the majority of recruiters and developers. I’ve been thinking about how to usefully bridge that. I don’t particularly need it right now but I want to help junior developers find their way into work and more experienced developers find their way to what they actually want. I think recruiters could help there. But I think we’re still quite far off from that.

I reached out about this to my network on LinkedIn (where the recruiters live). I got a response from Emy Wennerberg Kristoffersson who was willing to take a chance and reach some new developers. Emy works mostly in Sweden around Gothenburg and Helsingborg, so while she might not work in your particular area I think the information and exchange is widely applicable. We figured a good first step is to tackle some of the common skepticisms that developers tend to have around recruiters and recruitment. I hope this will be helpful. The post is not sponsored, I asked her to answer a bunch of uncomfortable and nuanced questions which I think she does gracefully. Let’s get into it.

For some background, can you introduce yourself and tell us a little bit about your professional experience?

Emy: My name is Emy Wennerberg Kristoffersson. I was born and raised in Helsingborg (south of Sweden), but moved to Gothenburg back in 2016. I am passionate about tech, human beings and business development. I settled on tech-recruitment because it gives me the opportunity to combine all of these areas. For the last three years, I have been working in the recruitment industry. I work for Bonsai Consulting, a Gothenburg-based company that specializes in tech recruitment.

I have always had a huge tech-interest. Though, this wasn’t something that I seized back in my younger years, at least not to a greater extent (apart from when loved ones encountered technical problems and I wanted to impress – hah!). My father has always been in the IT sector so I’m quite sure that his tech skills have influenced me. I am a people-person at heart, so I eventually decided to study Human Resources in Gothenburg. In time, I got in touch with Bonsai Consulting whereupon I started to work as a researcher, and my main task was to build a network of candidates who were open to new opportunities. After a couple of months, I leveled up to a position as a recruiter and got a bigger responsibility within the company. Back then we worked broadly in recruitment and recruited to many different industries, but due to my tech-interest, the positions that related to IT and tech always ended up on my desk. One and a half years ago, we decided to work exclusively with tech recruitment due to the enormous demand within the industry.

One of the most interesting things in my profession is the potential for improvement in the recruitment industry. Today, I am aware that there is a lot of frustration against the recruitment profession and I do think that this is a misconception. Many jobseekers consider recruiters as an annoying part of the job search. Generally speaking, we have a pretty bad reputation (let’s talk more about this later). But the thing is, in fact, that we are an asset in a candidate’s job search and in a company’s recruiting process. My vision is to get fewer people out there to see us as an annoying piece of the puzzle, and instead see the value of taking our help as a job coach.

Finding and hiring experienced developers has been a challenging proposition for a while due simply to enormous demand, how does this affect your job?

Emy: The first thing that comes to my mind, is the challenge of getting the companies to understand the market and the developers’ situation. It is a bitter pill to swallow for many recruiters and companies, but today many developers have at least 4-5 opportunities available for him or her. Unfortunately, not all companies understand how coveted many developers are, and therefore they don’t understand the necessity of offering a great deal to potential employees. Not just the salary has been rising during the last years, other requirements have changed considerably as well. Today, many developers expect to be able to work remotely, having flexibility in their working day, good opportunities to develop within the company and to be able to develop their own skills (and so on…). Outstanding developers know their value on the market, and if a company’s position doesn’t sound interesting or profitable, they will go on to their next available opportunity. Many companies lack the understanding of how many offers a developer can have on their table and are therefore unable (or even unwilling) to match their needs. This is a tough nut to crack.

Another thing that comes top of mind is the art of standing out as a recruiter. Due to the enormous demand, many developers are likely to get contacted by a countless number of recruiters every day. The old-fashioned way of sending an email to a developer saying “Hi, here’s a job I’d like you to consider” doesn’t work today. Why? Because that developer has probably received multiple requests from other recruiters already, and my message is likely to disappear somewhere in all that noise. Over the years that I have worked as a recruiter, I have come to understand the importance of understanding the developers needs and desires before sending them multiple job descriptions, preferably even before I contact him or her. It is my duty, as a recruiter, to do my research before I expect a developer to take his or her time to talk with me. For example, If I check their Github I may find out that this developer prefers back-end development in C#/.Net, then I know that it won’t be necessary for me to contact him or her in order to talk about a front-end position where your main focus is in React and Typescript. If I don’t do my research, I’m likely to waste the person’s time. If I don’t find anything on Github or similar, then I think it is pure decency of me to first of all ask if they are interested in having a conversation with me and if they are, I can’t just throw a job description in their face without first understanding what this person is interested in.

Has everything changed with the pandemic? Is development work hard to find now?

Emy: A lot has changed with the pandemic. From my experience, I think that the biggest challenge for recruiters right now is that developers in general are unwilling to take on a new job, even though they might know that their current position isn’t exactly what they want. I think it’s a result of the uncertainty with the pandemic, that no one knows how it will develop and what will happen next. Since the pandemic seriously shook the market during spring and summer, many developers are worried that it will put them in a situation where they’ve left a permanent employment and the safety that it entails, to be the “last man/woman in, first out”.

In the beginning of the pandemic the market was disastrous, from March until September it was clear that even the IT-industry (despite the great demand) suffered from the pandemic. Many start-ups had to end their businesses and bigger companies were prohibited from hiring, many were even forced to dismiss employees in order to survive. Since August until today it has eased, and more companies dare to hire today. With that said though, companies take precautions when hiring and the processes might include more steps than normally in order to be really sure that it’s a good fit for the position.

I’d say that there are many opportunities on the market by now, but of course we are far from “normal”. Unfortunately, many companies demand more senior developers today, in order to fill the positions that they dismissed during spring. So, for junior developers it may still be a challenge to find their first or next position. Many companies can hire junior developers as a short-term consultant-assignment, so it is advantageous to be open to these opportunities as a junior developer.

Is the poor reputation of the recruitment profession in tech among developers deserved or overstated?

Emy: Sadly, I do think that it is deserved. I think that many recruiters have the wrong approach when recruiting for developer-positions. I have talked to many, many, many developers about this, and my understanding of the situation is that developers experience that recruiters don’t understand them nor their industry. And above all, many developers think that recruiters are a bit ignorant and uninterested in understanding it.

Recruiters and developers communicate differently, which is natural due to very different professions. Let’s face it, most tech recruiters lack programming skills. That’s understandable but since I am the recruiter and in most cases I am the one who expects them to talk to me, it is in my responsibility to educate myself to the extent so that I get a basic understanding of the developer profession and am able to communicate better with them. As a recruiter, and especially as a tech recruiter specializing in IT recruitment, you must have interest in getting to know the IT industry. I am no expert in IT and definitely not in development, but I quickly realized that I at least had to have a basic understanding in order to be able to communicate with the people whom I am trying to reach out to. Therefore I have spent my free time on reading books about IT and I took some online basic courses and bootcamps in development, just in order to understand the technical language. If I don’t have this basic understanding, I will appear frivolous and ignorant in my professional position, and then why would anyone spend their time talking to me?

There will probably always be a gap in the communication between developers and recruiters, recruiters will always be at disadvantage in terms of technology, but I hope that more recruiters understand the problem and start to educate themselves so that in the long run we can earn a better reputation among developers.

What are the major challenges you and your colleagues face as recruiters?

Emy: From a candidate perspective, the major challenge is what we talked about earlier, regarding the poor reputation of the recruiter profession among developers. Since many developers have preconceptions of recruiters even before we reach out to them, it’s always a challenge to turn it around. If developers have had some bad experiences with recruiters before, it is often quite hard to even get to the point where we have a fair chance to prove that not all recruiters are the same. Nothing makes me happier than being the one who disproves the preconceptions or being the one who gives a developer a better experience – but to do that I must succeed with the first part, to break through the noise of bad experiences to the extent that the developer is willing to have a conversation with me.

Another major challenge is also one that I’ve already mentioned. The one which refers to understanding the IT-industry and the developer profession. I once read that “never hire anyone to do a job until you’ve tried to do it yourself first”. This is, of course, difficult in a recruiter’s position since it is impossible to master every language or skill, but I think that the mindset is very wise. It may not be necessary for a recruiter to be familiar with every detail of all programming skills, but I need to understand everything that is stated in a job description and in a developers CV in order to be successful in my profession.

In terms of challenges with clients, one of our biggest challenges is convincing companies to consider junior developers to fill their positions. Many companies ignore the potential of candidates, and stare blindly into the number of years of experience in their resume. In my opinion, the number of years doesn’t say it all. Many CEOs can testify that some of the best people that they’ve hired had very little programming experience when they took on the job, instead they had raw intelligence and knowledge that outweighed the missing experience. But unfortunately, many companies are afraid of hiring more junior developers – especially if it is a replacement recruitment of a more senior developer.

Are developers especially tricky to recruit or are the challenges similar in other fields?

Emy: With experience in recruiting for positions within different industries, from economics and administration to marketing, logistics/shipping, civil engineering and more, I know for sure that it’s trickier to recruit developers than any other profession. It may be due to the high demand, and the poor reputation of recruiters in addition. The shortage of talent and the infinite war over tech candidates make the job of technical recruiters challenging. When recruiting developers, a recruiter’s job doesn’t end after putting a job description on the career site, sitting back and waiting for the right candidate to drop by and say, “Hi I want this job”. It requires being active, putting effort in looking through coding-related sites and working actively with headhunting, since the person who we are looking for is likely to already have a job. I think that a huge contributing factor is that developers don’t “hang out” in the places where we traditionally look for them. Not all developers spend their time on Linkedin, which is the biggest business and employment-oriented social service available, and the number 1 social media tool for recruitment and hiring. While recruiters search for the right competence on Linkedin, developers hang out on Github, Stack Overflow, CodeProject, Hacker News and so on; i.e platforms where recruiters “don’t belong” and in some cases even are prohibited from contacting developers.

I’ve heard it said that recruitment is a numbers game where finding and reaching out to the largest number of candidates is the way to play. Does this match up with your experience?

Emy: I hear what you’re saying, and once again I must disappoint my fellow recruiters out there by answering yes, in some ways it matches up with my experience. I would not call it a game though. A recruiter’s career often depends on getting in touch with the right candidate. And as sad as it is, the fact remains that our reputation among developers have seen better days. And as a result of this fact, it is likely that we get 1 or 2 positive answers from contacting 30 candidates for a position. From my point of view, and from doing some research by reading and talking to other recruiters, I find that this is partly due to a lack of knowledge. I think that recruiters who give this impression, who unconsciously create this reputation, don’t understand the positions that they are recruiting for. They are fanatically searching broadly, contacting every candidate that they find at least a little bit appropriate for the position, hoping that they will find the right person among all of the people whom they are contacting. This is absolutely the wrong way of doing it, since it put us recruiters in a bad light, appearing unserious and unprofessional.

Then of course, it is not unthinkable that some recruiters do this on purpose in order to challenge a colleague for fun, or gain a bigger network. Many recruiters also receive high demands from managers to achieve different goals (including a number of contacted candidates) which may result in these kinds of “numbers games”. But I find it hard to believe that professional recruiters in general would aim to do this just for fun. I hope that the majority of recruiters understand that these habits destroy the rumor of our profession even more.

I personally care a lot about finding work for junior developers as I know how hard it can be to find that first step into the business. Many are being trained but very few companies are in my experience willing to invest in levelling people up. What do you think we should be doing to improve that situation in this industry?

Emy: First of all, I think that this is a very important question that more recruiters should work with actively. In my profession as a Tech Recruiter, I am actively working with questions about recruiting on potential. I think that it is important for companies to rethink the idea that years of experience translate to performance. Just because a developer has done a certain type of job before, doesn’t implicate that he or she is necessarily good at it. More companies should think about what’s important to their company. Hiring a developer who is highly experienced can mean that they are very skilled at their job, which most companies strive to find. When recruiting for a replacement position, many companies aim to find a developer who possesses the same skills and experience as the person who they are replacing.

What many companies and hiring managers forget, is that an abundance of experience might just as well come with a very high salary requirement, an inflexibility toward change and difficulties in adapting to the corporate culture. A junior developer on the other hand, is likely to offer a fresh outlook for the company and is often more motivated to gain more knowledge and experience in the company’s field. In addition, a shorter career history may indicate that this person won’t have as many negative habits as a more experienced candidate. Moreover, hiring a candidate with less experience and giving them a chance to prove themselves will drive employee loyalty which is extremely important.

Skills can always be taught, but motivation and ambition cannot. Instead of putting all of their focus on the hard skills, companies should take vital soft skills that establish potential, in greater consideration. I am convinced that companies and hiring managers must transition into a new mindset when it comes to talent spotting, one in which an assessment of a candidate is based not on direct experience and competencies, but on soft skills and potential.

What advice do you have for developers to make the most of their relationships with recruiters?

Emy: To be open-minded and to reach out in time. Even if you’re not looking for a new job as of today, it is a good thing to establish a contact. Good recruiters won’t just throw random job descriptions in your face and expect you to grab the first opportunity that comes up. Instead, a good recruiter will take time getting to know you, try to understand what your biggest ambitions are and what you want to do next, and that’s a time consuming process. Some of the best moments I have had as a tech recruiter have been when I’ve had a chance to really get to know the candidate, and thereafter, I’ve done my utmost to help him or her to land the right job, at the right company. The ideal situation isn’t when you contact a recruiter by the time that you’re already in a hurry to find another job, by the time that your previous employment has already ended. Establish a contact today, explain that you’re not interested in a new job right now but that you would like to get in touch so that, in the long run, the two of you can work together in order to find your next challenge. I think that the conclusion is that developers should consider recruiters more as a friendly job coach. Consider us as a friend, one with a very large network that can offer many valuable contacts and opportunities. A friend who can set up a meeting between you and a hiring manager at your dream company, a friend who has inside information about specific companies and positions and who can provide you with career advice that increases your chances of landing the job. Working with a recruiter will get you closer to that dream job you’ve been trying to land. But don’t just work with any recruiter who reaches out, do your research: ask what companies they usually work with, what positions they’ve successfully filled historically. If you are an IT developer, it makes no sense working with a financial recruiter who specializes in accounting. Find a recruiter who is specialized in your field. The candidate/recruiter relationship should be a relationship of understanding what the two of you can do for one another. It is, after all, your future we’re talking about.

Finally, thanks for answering these questions. And where can developers that are curious to get in touch reach you personally and where do you work?

Emy: Thank you for the possibility to do this Q&A! I really hope that I have answered at least some of the most common questions and concerns regarding recruiters. Developers are more than welcome to contact me, either through Linkedin (Emy Wennerberg Kristoffersson) or by email, emy.kristoffersson@bonsai.se.

So, dear reader. Was this useful or interesting to you? Would you be interested in me tracking down more of these proactive and deeply engaged recruiters that you could get in touch with?

Hell, I’ll poll it and pull the result from my logs:

• I’d be interested
• Don’t bother

Thanks for reading. As always, reach out to me on lars@underjord.io via email or on Twitter @lawik@hachyderm.io. I’m very curious to hear what you think. If you want to follow my thinking and what I’m up to you’ll find the newsletter signup further down the page. I write different but related things there.

My history with the Mac

So what was that computer to me? It was an extremely well-built and solid-feeling piece of aluminium. A cool keyboard backlight. The magsafe charger and a bunch of USB ports. The keyboard and touchpad were best in class. The build quality was better than any laptop I had owned before. Partly because I had only bought the cheap ones before but also because they really were a step above at the time.

I got it primarily on recommendation from a friendly hacker who’s recommendations had never lead me wrong before. He spoke well of the underlying UNIX and the experience of using it. Fast, clean and visually pleasing. I think he mostly really liked the backlit keyboard, very hackerly. His recommendation held true, this computer was very good to me.

I’ve also been the admin on an Xserve server. That was wild. Neat UIs but buggy and finicky as all hell.

Since then I’ve used the second generation Macbook Air (I believe 2nd, first wedge version) at 11". That was a cool little machine and did what it did very nicely.

Hardware aside, MacOS in it’s earlier incarnations on these computers was always a snappy and competent experience. A polished surface which did a bunch of stuff under the hood that generally made it work better than the different Ubuntu desktop environments and various Windows versions I’ve had before. Things like Wifi and even Bluetooth felt good in a way they never had before.

A lot of it was the visual polish and the extremely snappy UI. But in total the experience was just great. Spotlight was glorious. I think my first upgrade was Snow Leopard which was generally a very good update as it focused on performance and stability.

As a developer’s machine it was fast enough, competent enough, got out of the way and the UNIX underpinnings meant I didn’t miss Linux at all. I’ve never been able to really connect with the equivalent powershell stuff in Windows. I guess I just like UNIX.

The only bad thing I can say about my early years of MacBook Pro usage was that the trackpad and the Magic Trackpad I got eventually are probably some of the biggest culprits in some of the RSI-style hand pain I’ve been dealing with. Trackpads are just murder on my hands and I worked loads off of that setup for a number of years. Took a while before I realized that it was trackpad-related.

Beyond that I’ve had some refurb Macs for my wife and assorted family, some 13" MBP for work at one point and then a 15" butterfly touchbar MBP for work. I think that was when I started to feel that Apple was diverging from my preferences in the MBP line.

My current experience

That’s the same computer I have and work on now and it is.. fine? Maybe just OK. Not great. I’ve had some keys getting stuck but fixable with canned air. I don’t like the touchbar, it has been between useless and an actual hindrance. The TouchID power button is good though. I don’t like living in dongle-town though I mostly like USB C in the long run.

The reason it has been mostly fine for me is that I keep it on tray mounted on a VESA arm, dangling dongles like a technical octopus and I use external peripherals for input.

It gets really hot and loud and then it performs incredibly poorly. So I guess this is one of the throttliest generations. I think I had the “don’t charge it on the wrong side” problem as well. Some of the CPU shenanigans have calmed down as I installed the Turbo Boost Switcher tool to just disable the Turbo Boost, removing performance for peace and quiet.

As I’ve been using these devices it has become increasingly annoying to figure out how to install “unknown” apps. I need some non-discoverable terminal incantation to get the option to accept installing things that are unsigned. There’s always a new piece getting locked down. And while I think that’s often to the benefit of the average consumer, I’m not that. And I just get more annoyed.

I’ve been frustrated about the uninspiring performance delivered for the incredible brand markup that Apple charges. I don’t mind the computer being expensive if the experience is good and the hardware reasonable. The experience feels like it is slipping, especially for my needs and the hardware has just been getting less impressive to me.

The hardware

My gaming computer has a Ryzen. For a while I did my dev on that as we had just moved to our house and the office wasn’t finalized. Woof, aside from running Windows as a dev environment which I didn’t enjoy there was some serious upside on that machine.

On the Mac my options are very limited. I can’t get a Ryzen, I can’t get anything modern with Intel or meaningfully upgradable at all. The Mac Pro doesn’t count. It comes underspeced at hilarious prices. I like some of the design decisions but the price-point doesn’t make any kind of sense for me and what I do. I can’t buy an interesting Mac from a performance standpoint.

Or can I? Well, they just announced the M1 chip and ARM Macs are now a fact. I think I might get one at some point. For a travel laptop I don’t think the rest of the industry is ready to fight Apple. Battery life and good bang for buck power might actually keep a Mac in my life for that. But I feel like the general trend is away from what I want. Or I might just use my iPad Pro for that use-case.

I think the M1 will be quite impressive when the benchmarks roll in. I’m sure it will suit many people for real-world use-cases as well. However, from the first presentation on it and the first batch of Macs I don’t feel like the direction is for me. IO was heavily sacrificed. Upgradability is pretty much out the window. These things can be fine for a travel device for me where battery and weight are primary concerns. In that regard the new Air looks pretty good.

The software

Beyond that the coming OS, Big Sur, is taking MacOS in a direction I dislike. Catalina was quite messy and felt like it took steps toward walling off the Mac. Big Sur seems even more heavy-handed in that area and finally the M1 can push that even further if Apple feels like it. My trust is eroding on letting Apple set the tone for my computing life.

Don’t get me wrong, I think their approach to this transition is incredibly neat with Rosetta2 and how they are using the bytecode stuff with the App Store and whatnot. And the possibility to run iOS and iPad apps natively could be very useful. But none of this really moves the needle for me.

So what’s next?

With my office in the garage as my primary work location I’m looking to transition to a desktop computer with lots of power. It will run Linux. Marking my first major return to desktop linux as a daily driver in a bundle of years. And it will run as light a desktop environment as I can stomach. I just wanted a stupid amount of performance to offset som of the UX niceties I know I will miss or have to customize on my own.

I’m excited about exercising my development tools on a strong modern CPU rather than the throttled mess my current laptop offers. But I’m not thrilled about this move beyond the hardware aspect. I liked MacOS but I just don’t feel like Apple gives much of a care for the things I care about. And I feel like the software side on the Mac is slipping, consistently.

If they released an expandable Mac that wasn’t ridiculously expensive they would really make me think twice. But that feels unlikely.

So I’ll build myself a monstrous machine that can compensate in raw power for the potential lack of elegance and which offers unbounded flexibility rather than a poorly tended garden that someone keeps trying to wall in.

I’m not happy about it. I’ve generally enjoyed using my Macs. But when someone says “we’ll give you the full experience”, settling into that requires trust. And my trust that Apple and me are in alignment keeps fading.

Also, I’m developing a lot with heavily concurrent workloads. So I really look forward to exercising more cores. 2021, year of the Linux desktop (for me).

If you have thoughts, comments or a hell yeah you want to share about this topic or maybe you want me to cover some specific part of my transition here, let me know either via lars@underjord.io or on Twitter @lawik@hachyderm.io. My first thoughts and my build might just show up first on my newsletter, so consider signing up for that below. It don’t track. Thanks for reading.

The JIT is here (soon, OTP 24)

With OTP 24 landing sometime next year we are going to get the a JIT for the BEAM. Based on the project AsmJit this will mean that some BEAM code will be translated to native instructions. It will not be the kind of warm-up-for-performance-gains JIT that I’ve heard of in PyPy but rather significantly simpler. The goal of the project was to introduce a JIT that could give performance gains for some cases but would not cause any performance regressions. A pragmatic and laudable approach. Considering this made the Jason JSON-library (written in Elixir) beat the Jiffy JSON-library (written as a C NIF) in some tests I think this has the potential to obviate the need for some NIF implementations. Avoiding reaching out to the lower level code that is more capable but more dangerous is a good win.

Anyone running RabbitMQ should look forward to the update as measurements indicate 30-50% increased message throughput. Which is a nice thing to get for no code changes at all.

Pushing the performance of the BEAM closer to native is magnificent. To be clear the BEAM is already quite a good performer. I would put Erlang and Elixir at the abstraction level of languages like Python/Ruby/Node.js. Python and Ruby are poor performers. The Python ML stuff all goes into C++ or similar for performance. I’ve worked a bunch with Python and the things I hear from the Ruby world makes them sound quite equivalent in performance. They are a bit slow and can only do one thing at a time. Node.js is a bit different. It can do multiple things at a time, if you append asterisks and squint. It does it largely the same way Python + Gevent does it. This approach is incredibly susceptible to CPU-bound work causing head-of-line blocking. It becomes the single most important consideration for building a performant application “get to IO, don’t compute”. V8 that Node.js runs on is heavily optimized and fast for such a dynamic language. I think the BEAM provides a better approach that can deliver comparable results without as many footguns (opportunities for shooting yourself in the foot). But getting better at the raw crunching is a big gain with this JIT implementation and I look forward to the release.

Lumen - Static compilation & WASM

The Lumen project is a huge effort by a gang of open source developers and DockYard to implement a compiler (and more) that can take Elixir and Erlang into the browser. By solving that they end up solving static compilation for Erlang and Elixir as well. So this isn’t compiling and shipping the BEAM to the browser. This is a faithful reimplementation of the BEAM functionality in a way that allows it to be compiled statically. It uses LLVM and requires quite a bit of effort both in development and in wrangling the Web Assembly work group process stuff to make sure that the standard is not entirely run by Object-Oriented Programming needs.

I don’t think Lumen will replace the BEAM. The BEAM has a brilliant track record for long-running services and distributed computing that the Lumen project do not even attempt to achieve right now. Instead the Lumen project will allow Elixir and Erlang to move into spaces where the BEAM might be a bit too heavy and still provide the same guarantees. Typically I see it being good for command line tools, web frontends (super interesting to consider the Actor model going there), serverless/edge computing and potentially with WASM competing with Docker as a delivery mechanism for code in Kubernetes, using something like Krustlet (good podcast episode on WASM/Krustlet). It’s probably Cloud Native or something. Who knows.

What gives Lumen the potential to be a better fit in these circumstances is that it can optimize for filesize (by cutting out hot code updates) and it is likely able to start much faster. Lumen is written in Rust. Which seems to be the popular choice around Web Assembly from what I’ve seen. Lumen is still an early release project and not fit for production. But it is beeing actively pushed forward.

Nerves - An IoT platform with minimal suck

The Nerves project is fantastic. I’m a hardware hobbyist, not an IoT dev but I’ve worked a fair bit with Nerves and it is so, so good. What Nerves gives you when working with a Raspberry Pi for example is a way to let your code run all of the device. The BEAM is basically your operating system on top of a minimal Linux installation. The Linux you have is based on the solid foundation of Buildroot so it is quite feasible to modify it as you see fit. The big idea is that if you are running a Linux-level SBC already you might as well build on something that gives you the guarantees of the BEAM.

Beyond that the default setup encodes a lot of good embedded practices by default so that you avoid bricking devices with firmware updates, you get easy support for pushing firmware over the network or USB and much, much more.

There are a ton of good libraries for sensors and assorted hardware, as well as the common protocols like GPIO/SPI/I2C/UART. Networking support is well considered and has been reworked since I first started using Nerves a few years back (and it worked well then too). BLE is getting more and more good support recently.

The project also created NervesHub which is a solution for managing a fleet of devices by securely providing firmware updates, allowing the switching on of a remote console on devices if that’s a need on your product. I think the most recent stuff is a UI revamp and some serious work on binary diffed patches to minimize firmware update sizes for data-constrained deployments.

This is very much a production project and people are shipping hardware with Nerves. It keeps marching forward.

The BEAM can be your entire application

Saša Jurić, author of the much-acclaimed Elixir in Action book has produced a library called site_encrypt. It allows you to handle LetsEncrypt configuration without a separate webserver or actually using certbot.

Now this library is good and meaningful in its own right but the underlying idea is why I bring it up. The BEAM can be your entire application. This is something I’ve realized over time. Where in Python you would reach for Gunicorn to run you Django app and Nginx to protect Gunicorn from the big bad world.. The BEAM is made for this. Introducing an intermediate layer of Nginx (or another HTTP server) might actually be detrimental in that you now have two things you need to configure correctly and two pools of multi-core processing workers that care about this request/response cycle and can independently screw it up.

The BEAM was always built for this. OTP has a lot weird corners where you find interesting libraries such as wx for WxWidgets (window management) and ssh for both SSH client and server work I believe. Because it is meant to be delivered as a full solution. It can run and manage multiple different types of work inside of it. Gracefully. It doesn’t replace Kubernetes for the large deployment or polyglot environments. But it might very well mean you don’t actually need to go there early. Or you can reduce how much Ops you need in your Dev. If your entire stack is Elixir or Erlang front to back I think you have empowered your developers significantly.

There is already a move towards this where tools are converging that give us a lot of things out of the box that we’d otherwise need to move outside our application for. These are pragmatic 80-90% solutions. The normal solutions are still all there if you need to reach for them. But maybe you don’t. I see these as moves in the same vein:

• LiveView - We can reduce the amount of frontend we need to build that isn’t BEAM code (Elixir or Erlang), in some cases get rid of it entirely.
• Live Dashboard - Application insights right in your application stack instead of pushing them out to another solution.
• Phoenix PubSub - Distributed PubSub without requiring coordination via something like Redis.
• Phoenix Channels - Distributed PubSub over WebSockets using the above PubSub to coordinate delivery.
• Phoenix Presence - Distribute Presence. A CRDT-powered thing for maintaining information about if someone is connected to a channel or not, like chatrooms and online/offline. Using Channels.

Lowering complexity by keeping the solutions in a system you understand well is potentially very powerful. At some point many projects will need to pick up external dependencies such as Nginx, Redis or whatever. But I think there is something compelling about building your application inside a system that can do all of it quite well. Elixir and Phoenix already have significant mind-share in the startup world. I wouldn’t be surprised if this ends up being a very popular solution for startups. No frontend-specific code for the MVP, no New Relic or Mixpanel bill we make do with the Live Dashboard. Distribution is Erlang distribution + Swarm/Horde/Libcluster or something like that when we need it. And if we need “real-time” functionality we are well and truly covered.

I think this is a pragmatic approach that aligns with what Erlang has always strived to be, what Elixir has turned out to be and what many in the community want. I look forward to seeing if it grows the way I hope. We discuss this further in an upcoming episode of Elixir Mix (the podcast) with Saša Jurić. So keep an eye out for that.

I think this might be a future topic on its own. I am not done with this.

As easy as anything else

While the BEAM is incredibly capable, Elixir and Erlang are not difficult. There is usually a bit of a hump going from Object-oriented programming to Functional programming. But beyond that you are mostly working at the abstaction level of something like Python while not needing to be as mindful of class hierarchies. I find it easier to reason about. It isn’t trivial. No programming is at the start. But you aren’t working in some mad realm of math and abstract thought. At least not that you need to care about. Much like “being good at math” is not required for programming. You don’t need to be some kind of hardcore whizz to get stuff done in Elixir or Erlang. Elixir looks like Ruby if that helps. Or so they say. I never did Ruby.

It is definitely easier to reason about than Node.js and the event loop ever was to me.

Conclusion

I have a strong confidence in the BEAM as a runtime/VM to build my software on. There are certainly places where it fits poorly. I primarily work in web service development and there it fits perfectly.

If you have thoughts, feelings or even “more of a comment really” feel free to reach out at lars@underjord.io or via my Twitter @lawik@hachyderm.io. My newsletter is available below if you have interest in further writing.

Since that method was deprecated I figured it was time I bite the bullet and get comfortable with child specs and the way they work and figure out if I can avoid creating a module for normal use of a simple bog standard Supervisor. Spoiler: I could.

This repo has all the code for what I built. So if we dive into lib/supervisor_sample/application.ex we find the following:

  # ..
  def start(_type, _args) do
    children = [
      worker(:root_worker),
      supervisor(
        :one_for_one,
        [
          worker(:worker_1),
          worker(:worker_2)
        ],
        name: :supervisor_1
      ),
      supervisor(
        :rest_for_one,
        [
          worker(:worker_3),
          worker(:worker_4),
          worker(:worker_5),
          supervisor(
            :one_for_one,
            [worker(:subworker_1)],
            name: :subsupervisor_1
          )
        ],
        name: :supervisor_2
      ),
      supervisor(
        :one_for_all,
        [
          worker(:worker_6),
          worker(:worker_7),
          worker(:worker_8)
        ],
        name: :supervisor_3
      ),
      worker(:transient_root_worker, :transient)
    ]

    # The root of the tree is a supervisor that runs everything we defined above
    opts = [strategy: :one_for_one, name: SupervisorSample.Supervisor]
    Supervisor.start_link(children, opts)
  end

  # ..

GitHub link

This is my supervision tree definition. It uses utility functions supervisor and worker to make it easier to get an overview. These functions generate a child spec with each call. Even with some experience using OTP I never really spent any time understanding child specs. I won’t go too deep into them here, the docs above honestly cover it but I’ll try to make it digestible in my own way.

This is what the supervisor child spec looks like if I call supervisor(:one_for_one, [], name: :my_supervisor):

%{
  id: -576460752303423326,
  start: {Supervisor, :start_link,
   [[], [strategy: :one_for_one, name: :my_supervisor]]}
}

So the :id is fairly arbitrary. According to my sources, I asked on Twitter, more knowledgeable people responded, it is used for restarts and whatnot. It can also be used when doing interesting things with your supervisor implementation. But it is not important aside from needing to be unique, unless you have specific plans for it.

The :start key gives what we are actually starting. The format might become familiar to you. A tuple with a module atom, a function atom and a list of args to pass into the function. This matches the signature of apply/3. In this case the args are a list of children and some options. Because that is what the Supervisor module takes for the function start_link.

That’s all that is necessary. The child spec docs cover the other options. We can check our worker example as well:

%{
  id: -576460752303423294,
  restart: :permanent,
  start: {SupervisorSample.Worker, :start_link,
   [[label: :my_worker, name: :my_worker]]}
}

Here we also use the :restart key because I have an example with :transient and so I set it explicitly.

These maps aren’t complicated to create and they shouldn’t be intimidating. But they are visually bulky and I think there are many ways of building the tree that could be done in a visually pleasing and less noisy way. That’s what I use the utility functions in the sample project for.

GenServer and child_spec/1

In many cases you don’t actually have to create the child spec yourself. Anything that is a GenServer will have a child_spec/1 already included. So then we can reduce the above to {SupervisorSample.Worker, name: :my_worker}. Or without a name it could be SupervisorSample.Worker. Very clean. A bit of convention saving you a bunch of repetitive detail. But the Supervisor module doesn’t offer child_spec/1. It offers child_spec/2 which is used to modify child_specs for module’s that already have them. Usually because you want to override something in the default child spec. Such as the :id.

Most libraries you’d use where you need to start an instance of them as part of your supervision tree would already provide you with a child_spec. If they don’t, you can create one yourself quite easily just as we did for Supervisor.

Another thing you can do is create a module that provides a child_spec/1 for starting a supervisor as detailed in this converstation on the forum. The code is partial, but I think the idea is complete. Then you could use that module instead of Supervisor.

The tests & strategies

So the supervision tree above showcases the different strategies available. It also shows that we can supervise a supervisor, that’s how you build a bigger tree with processes that depend on one another.

To demonstrate how these work I’ll direct you to test/supervisor_sample_test.exs. Every test looks something like this:

  # ..
  test "restart root worker" do
    Worker.stop(:root_worker)

    # Should restart
    assert_receive {:stopped, :root_worker}
    assert_receive {:started, :root_worker}
    # Shouldn't restart anything else
    refute_received {:stopped, _}
    refute_received {:started, _}
  end
  # ..

They follow this model of, okay, let’s tell a worker process to stop and assert that we receive the messages we expect. I expect this one to be stopped and then restarted. These messages are sent in lib/supervisor_Sample/worker.ex and we register for listening in the setup hook in the test module.

The different strategies are succinctly explained in the Elixir docs. I’ll briefly restate it here:

• :one_for_one, the supervisor will restart each child separately if they terminate. Only if the supervisor goes down, does it affect the whole group.
• :one_for_all, if a single child process terminates the whole set of child processes will be restarted.
• :rest_for_one, this one is interesting. If a child process terminates all the children started after it in the list will be restarted. This might seem odd but has some uses.

So you can look at the tests to see examples of these behaviors.

This doesn’t really cover DynamicSupervisor at all. But that has a lot of its own considerations. It is very useful and maybe I should cover it at some point. Alex Koutmos one of my co-hosts on Elixir Mix has a good piece on DynamicSupervisor.

I hope this is helpful to people. Thank you for your attention.

If you have questions, thoughts or more of a comment, really, you can find me on twitter {{ lars_twitter }} or reach me via email {{ lars_email }}.

And then it hit me. From some old Nerves projects I'd seen, that there is a standard for just sending a stream of JPEG frames as a video. MJPEG. Did you know about MJPEG? Lots of people don't. It is used by lots of webcams and security cameras. Common option for Raspberry Pi hacks as well. MJPEG is super simple which is its big advantage.

But video is what we expect. I was going for something else. So this could be used for a CI status light, showing any amount of visual status information. I use it for this:

That's live. Or dead if my server falls over.

So how does MJPEG work. Well, you take an <img/> tag and you shove an MJPEG URL into it. Done.

Okay, that's how you use it. Not how it works. I implemented it in Elixir, Elixir is quite good at keeping state and serving updates. Links are below. But basically the browser opens the connection, receives some headers and some chunks of data and then realizes it is dealing with MJPEG. It wil then just expect the chunks to keep coming. Indefinitely. Because this is live video. Frame by frame of JPEG.

The basic code for the MJPEG headers and chunking was lifted from a pi camera repo made by the Nerves team. It had a lot of Frank Hunleth and Connor Rigby in it so kudos to those guys as always. This is what I did with it: lawik/mjpeg

My server implementation is here and uses the above code: lawik/mjpeg_example

So I receive the connection and then that calls my MjpegExample GenServer to persist the connection and keep track of how to notify that connection about new data. It also triggers an update to notify everyone already connected.

This is not polished, it is hammered together and I'm curious to see if it falls over the next time I get a decent amount of traffic.

I really like this approach because it is a fun hack that simply happens to work across browsers and quite well at that. I like how it is just an img element and no frills. I added lazy loading because that works more nicely with things like Google Lighthouse scores and the loading experience (your browser doesn't spend a few seconds thinking about loading the image).

Unfortunately it is absolutely a poor choice to actually use aside from fun and hacky stuff. There is no good way of doing accessibility with it. You can update the pixels and that is it. Unless you can chunk-stream a txt-file in an iframe. Haven't tried that yet...

So, don't use it. But isn't it pretty neat?

Of course, much like the CSS tracking, this can be used for evilish things. You can absolutely keep track of how long someone keeps receiving your frames and use that for your analytics. I also find it neat that it lets me know concurrent users. I just log the number along with sending it out because I want to know at what number it breaks and out of curiosity but you can probably do some mildly untoward things with this. Oh.. Wait. You could serve rotating ads with this. You know what the last frame you sent was so if you get a click on it you can direct that particular user to the right thing. New title: Ad placement entirely without JS, ugh, no thanks. Moving on.

If you know how to make this more dirty, hacky and fun or even more useful or accessible feel free to get in touch at lars@underjord.io or on Twitter where I'm @lawik@hachyderm.io.