Multitier and scale vs a world of singletons

People often ask about architecting Elixir projects. And I usually don't know what to say. I remember the feeling though. You have all these awesome OTP tools and patterns, clustering and all that. So you gotta make different choices right? What does it look like?

Usually it looks like good fundamentals. Multitier or n-tier architecture. You have the outer layer of presentation, whether that's your API, your LiveView, your server-side rendered web pages, your SPA React app depends on what you are doing. You have the application/logic layer where all your implementation happens. And then you have the data layer where most of the hard work actually happens but you pretend it happens in the logic layer because you didn't build Postgres and S3.

This architecture works great for Elixir and can take you far. Because it works great in way worse runtimes and can take them far. What it does brilliantly is to prevent you from making things very hard for yourself. It is a design intended for horizontal scaling. Meaning, you can add more servers/nodes/pods/gloopers and not get too many novel systematic issues.

By keeping a centralized data store and generally making logic and presentation stateless you are deferring the hard distributed systems problems. You opt out of eventual consistency, split brains, ensuring quorum and all that fun costly stuff in favor of a single writer database, maybe read replicas if you feel spicy. And you usually don't notice the limitations of Postgres.

Phoenix and Ecto contain a bunch of nice OTP thinking and architecture that is local to the node. And if you cluster up your nodes you can do a lot of nice-to-have stuff with Phoenix PubSub for messaging between nodes. This is barely a change in the architecture. The hard parts are about state, where it lives and how you handle it over time. If you have a Postgres db as a single source of truth you are on easy mode and that's a good place to be.

Things get really fun but really spicy when you add nodes. Maybe you start looking at Khepri for something relatively simple. Maybe you grab Cockroach DB so you can pretend you are dealing with Postgres. Maybe you grab something like Foundation DB. Cockroach and Foundation try to tackle the hard problems. Multiple writers and massive scale. They are way more complex than your typical database. Discord went from MongoDB to Cassandra and then from Cassandra to the cooler sibling Scylla.

At scale the database is the hard part. Okay, not the only hard part. Other things will mess you up too but it is typical for it to be the hardest thing to scale. A single writer Postgres will scale pretty far but the step to get more than one writer is immense.

This is why most people stick to multitier for an Elixir system. Distributed state in your app nodes invites these problems. This is also why making a singleton GenServer that holds state is an anti-pattern in many cases. The moment you add another node that is distributed state and will cause headaches

On the embedded side things are very different. Almost everything is n of one. We have lots of singular resources. We have lots of state. We may be talking to lots of other devices but typically not for uniform undifferentiated scale.

Devices tend to have an identity and a job to do. You can't tear it down and start a new equivalent one from the cloud UI because the darned thing is attached to someone's wall.

We use GenServers all the time to ensure access to a piece of hardware goes through a singleton. We don't want fights on the wire about how bright the backlight should be. We might not use Phoenix PubSub (though we can) but Nerves ships PropertyTable by default (used by VintageNet). And it provides a key value store with a publish subscribe pattern, optional disk persistence. You can get really good mileage out of this.

PubSub is a great pattern in an embedded device. You read a sensor like the temperature and there are many parties interested in that. Some UI wants to display it so it needs to go to whatever composits the state of the system for display. The cloud wants to know so it needs to go to that uplink. Some Thread radio dealio wants to broadcast it for smart home integration so it has to go there. They can all subscribe. None of them talk directly to hardware. They might have an API to pull the latest value or backfill.

And the state proliferates through the device. The temperature sensor holds on to the temperature but so does the UI compositor because in case it doesn't have a new value it probably wants to show the most recent one. The internals of the device is a little distributed system but it is almost entirely heterogenous, meaning, there aren't 3 database nodes trying to agree on a thing. There are 3 different things doing different pieces of work. And the system is probably in continuous flux on the spectrum of mild to wild. Maybe a lot of change, maybe very little. Eventually consistent by default.

And you definitely have to be mindful. Maybe you have a sensor that throws out bad values under certain electrical circumstances. Filtering becomes important. Tolerances. Limiting the potential for bad behavior. Say a relay is controlled by some value. If the value starts oscillating across the threshold, back and forth, you put wear on the relay and the device starts making a lot of clicky sounds.

You mitigate bad behavior.  You filter inputs. You set sane bounds for values. You limit how often you change things. The goal being a device that moves through states in a continuous stable manner.

It might seem like a single process for everything would be the simpler thing. You can determine everything very clearly and control behavior in detail. That gets very complex. But also, considering the number of ways things can fail we benefit immensely from the isolation of GenServers. The temperature sensor crashing means the UI stops changing the temperature display. It doesn't mean the UI stops updating overall. The device can still be responsive, still connected even with relatively serious hardware issues.

I should spend the time to cook this thought more, it is a bit longer than it needs to be. But cloud systems tend to try and avoid distributed state to scale. Embedded devices don't need to "scale" and thrive on internally distributed state.

Hope you'll share your thoughts. I'm here on email or on Bluesky and the Fediverse.

Thank you for reading. I appreciate it.