Guest post: Cross-cutting Elixir in Teams

Underjord is a tiny, wholesome team doing Elixir consulting and contract work. If you like the writing you should really try the code. See our services for more information.

Release day. Although the product team was used to a bit of a hustle to wrap up open features, perform final testing, and prepare for deployment, today presented some additional challenges. This morning Jerry, the team’s product manager, discovered that one of the major features slated for this release was implemented in a way that would cause problems for some key users on the platform. Jerry notified Mark, the backend engineer, who said he might be able to fix the issue by modifying the data sent to the frontend. Jerry thought he might be out of the woods and back on track for the release. A couple hours later Mark returned with bad news: The fix would cause a breaking API change. Although Mark completed the fix on his end, the issue would now need to be handed over to Kelly, the team’s frontend engineer. Kelly had taken the day off for a personal emergency. Jerry was now left with a difficult decision: delay the release of a feature that had been publicly announced to the users, or proceed with the release knowing that some key users were going to have a bad experience. How could Jerry and his company’s leadership set themselves up with more options in this scenario?

This is a guest post to Underjord where Alex McLain, who I know from the Nerves community, lays out his ideas about how Elixir can fill a special role in software teams. Thanks to DockYard for supporting our work on this piece. They are a digital product consultancy with deep roots in the Elixir community and ecosystem. You’ve seen them support Phoenix, ElixirConf as well as this blog and several others. Let’s hear from Alex.

I started my professional software engineering career working with low-level control system hardware for commercial audio/video system automation. These systems contained a central controller unit that orchestrated the other devices in the system, therefore uptime and resiliency were important properties. That led me to search for programming languages that were designed with an emphasis on these properties, in which Elixir was one that rose to the top.

During this time I had also been using Ruby in areas not sensitive to system performance, due to its developer friendliness. This reduced the time it took to deliver a control system project. This led to an opportunity to do web development in Ruby, so I made the transition. While working on an enterprise IoT platform built with Ruby, we started running into performance and cost challenges at scale. Recalling that Elixir could be a good solution to these problems while maintaining the developer friendliness of Ruby, we transitioned part of the system to Elixir and were able to reduce the compute resources required for the platform. The culmination of my experience at this point drew me to IoT consulting, with a focus on utilizing Elixir across the stack for developing both software and hardware.

It’s probably no surprise that web-based platforms running at scale are complex. Today more and more devices that were once standalone are now “connected”, utilizing cloud and/or edge computing to supplement their functionality. This means that products are no longer confined to the bounds of a website or a widget. A modern product can consist of hardware, a cloud platform, data science, and web/mobile/desktop applications to deliver the full user experience. Product complexity is increasing substantially. The obvious approach is to staff engineers from this wide range of disciplines onto the product team. However, the budgets for these connected products have not necessarily grown proportionally to their complexity, which can result in fewer resources than ideal to create the product. In practice this can manifest into situations like in the opening story: The critical path for feature delivery can depend on specific team members more frequently, which can not only affect their work/life balance, but also increases overall project risk. So what are some other approaches to this situation?

If you practice Agile you’re most likely familiar with building cross-functional teams, in which each member of the team is able to perform more than one role. If Jerry’s team had been set up this way, the release wouldn’t hinge on a single team member. Mark would have been able to complete the changes to the application without being dependent on Kelly. For projects that span a small number of disciplines, like frontend and backend, this solution may be enough. However, for more complex projects that include more disciplines like DevOps, firmware, native mobile, data science, etc., then it’s likely not feasible for every team member to be able to work proficiently across every discipline. There are just too many skills for any one person to learn. To overcome this challenge we need to look at cross-functional teams from another angle: the barrier to entry of each discipline, the amounts in which these disciplines contribute to the project’s complexity, and team size.

Barriers to entry can be created when drastically different tooling is used across disciplines. One notable example is the choice of programming languages used, each which can have a substantial learning curve to reach proficiency. The greater the number of languages in a project, the more complex the project becomes, and the more difficult it can be for someone to work in different areas of the project. For example, a typical IoT project could utilize at least four different programming languages: Elixir, JavaScript, C, and Bash.

A strategy to reduce this barrier and reduce complexity is to find tooling that can be utilized by multiple disciplines. In this case, let’s say the team decided to use Elixir as their primary programming language across disciplines. Elixir’s Phoenix web framework can be used for the frontend and backend, Mix tasks for DevOps, and the Nerves Project for firmware. Given that the community is actively expanding Elixir’s capabilities, Nx and Livebook are relatively new tools that could start bringing data science into the stack as well. Now all of the disciplines share a certain amount of baseline knowledge with each other, and can understand each other’s code. There may still be a need for traces of other technologies, but stepping into another discipline is more fluid.

Fluidity comes with other benefits as well. If team members can move across parts of the project easier, this breaks down silos within the team that could have a tendency to form otherwise. Each section of code can now have more eyes on it, increasing the opportunity for engineers to help each other, and improving code quality. This also spreads project knowledge across more of the team, reducing the amount of information that could get lost in translation, and enabling more team members to be able to help solve issues that arise. If your product management methodology is to deliver features in vertical slices, this can also enable engineers to work across more or all of the slice. From a resourcing perspective, if more team members are cross-discipline, the project may require fewer resources to achieve the same output compared to a project resourced traditionally. This can also be beneficial for startups or those trying to get to MVP quickly with a small team.

For companies that maintain more than one product or are involved in consulting, not only can it be beneficial to find tooling that works across multiple areas of a given project, but that also works across multiple projects in the company. From a technical standpoint this can be an opportunity to leverage code reuse, allowing new projects to start on second base. From a staffing perspective it can reduce the friction of rotations between projects.

Although we’ve covered many advantages of this philosophy, there are situations it may not apply well to. For example, if a mobile app is required, there is currently no well-established way to do this in Elixir. Inevitably other technologies like React Native and JavaScript would need to be brought in, resulting in less reuse across disciplines. Another challenge is that engineers who are fluid across disciplines need to have a broader skill set and be able to keep more context in their head. One way to help less experienced engineers in this environment is to start them off in an area that suits their strengths, and support them in expanding throughout the project from there. However, some engineers prefer to narrow their focus rather than to expand more broadly. Finally, diminishing returns can be encountered with larger teams, especially when each discipline has its own squad. At this scale it can be more effective to run focused squads rather than having them be jacks of all trades.

In conclusion, modern products are increasing in complexity and requiring more engineering disciplines to be involved in creating and maintaining them. Cross-functional teams that find technological reuse across disciplines can maintain an advantage in this environment. I’ve personally found Elixir to be a great programming language for this due to its solid foundation, versatility across several disciplines, and the community’s drive to keep expanding it into new areas.

Big thanks to Alex for writing this up and letting me share it. If you want to reach him you can find the user amclain on the Elixir Slack or reach out to alex (at) alexmclain.com.

If you have thoughts about guest posts as far as Underjord is concerned you’ll find me as usual at lars@underjord.io and on Twitter as @lawik. Please note, I don’t take random solicitations about guest posts. That’s weird.

Underjord is a 4 people team doing Elixir consulting and contract work. If you like the writing you should really try the code. See our services for more information.

Note: Or try the videos on the YouTube channel.