CMS project is moving forward and this newsletter will actually go into some detail on some of the challenges there. If you want to see what I've been up to you can look at this repo for now. We have a real name in mind but we haven't taken that public yet.

I will be streaming live today on my Twitch at 13.30 CET, so that's stupid early in the US. Unfortunately timing forced my scheduling a bit. This one will be on using LiveView to set up a  stream overlay, visual elements and pretty things. If I get far enough it might even end up interactive.

The previous stream on Membrane was a good time. Thanks to Marcel Fahle for coming on to keep me company and help me scour the docs. I asked Marcel on since he has a fair bit of experience fighting media formats and is generally a chill guy to talk to. The VoD from the stream is on the blog and the Underjord YouTube. Since I didn't achieve my end goal on the stream I wrapped the completion up in an extra blog post and a small video segment. They are also on the blog. So if you want something really real-time, try it out. Microphone input, straight to your HTML.

I also have a new project in the wings which will be more of my thoughts and opinions in a nice casual format. But it is not quite green-lit by all parties yet so I'll hold off a little bit.

Oh, actually, mini job board. If you have around a year of experience with Elixir and previous programming experience, or multiple years with Elixir and are looking to get Elixir work I might have a place to put you in touch with. Just respond to this email if you have interest in interviewing and we'll talk. 

When the easy way has proven to suck

Ecto tackles a large problem space. It is a functional programming replacement for an ORM (Object Relational Mapper). It is a database connection manager. It is a query builder. It also does input validation and sanitation with proper friendly error reporting. And it does this in a fairly general way. It doesn't impose tons of opinions while still allowing the common case to be succinct  and avoiding exploding complexity whenever you want apply some more nuanced operations.

Ecto balances a ton of trade-offs.

One interesting thing that Ecto does is that it attempts to, and probably succeeds at, eliminating SQL injection security issues at compile time. You can only use field names that are atoms, typically compile-time constants. Schemas are built at compile-time, if you build schema modules at runtime you already know you are pushing outside the common case. Changesets and Queries, even the schemaless uses, expect atoms for all fields. Why? Partially because it indicates you are setting this stuff up with known names at compile time. What else forces compile-time use? Macros. So many macros. If you look through the Ecto docs I think there are almost as many macros as functions.

I've run into this a lot recently. It has been frustrating because I want the CMS stuff to be very friendly with Ecto, very familiar if you know Ecto and ideally use as much as possible of Ecto. That goal remains but the level to which it is achievable has shifted. Or rather, the level to which I think it should be done has shifted.

So take schemas, but it applies roughly the same to queries and changesets. If we allow untrusted end-users to create a "content type" such as BlogPost and we generate a schema for that. We allow them to add fields to that schema, such as title and body. And then we generate an Ecto.Schema-derived module from that? Well then we are going to be generating atoms. And that's dangerous. Why is it dangerous? Because the BEAM allows around a million of those before things break. They are not meant to be generated en masse.

What CMS user is likely to create millions of fields? None, not a single one. But it opens the door for a single user to abuse the system and generate a million fields automatically and blammo the system is down or in trouble. Or maybe you are building a system which uses the CMS and opens it to a ton of users and they start generating their fields and indexes and such and suddenly you are going to need to watch that limit. While it is unlikely to be reached, because people will only really call a field so many things (title, subject, label, pretty_name), the concern will be there.

So I and my co-conspirators have considered just keeping track of the number of atoms we generate and enforcing a limit in config so that we can make our code gracefully fail before it makes the system fail. And that's likely doable. But I don't find it a compelling design.

We would then be choosing to introduce a non-obvious limitation and concern that can bite you a few years down the line to retain the full use of Ecto. Ecto expects compile-time constructs and we are dealing with runtime constructs (dynamically defined entity types that can be edited in a UI and persisted to essentially anything). There is a fundamental mismatch and there will be a trade-off.

I don't want this hypothetical Achilles' heel be a constant topic of discussion throughout the life of the CMS. I think most CMS:es suffer from some very rough compromises and I want to try to avoid compromising with the system design for our convenience at this early stage.

I am leaning heavily towards maintaining the clear design, only using the runtime-friendly parts of Ecto. So Ecto.Adapter.query functions for example. We can also still use all the basic Ecto.Repo stuff from what I've seen. But this will mean that we need to own more code. We want something equivalent to Ecto.Changeset and at least parts of what Ecto.Query offers, but we need to be able to pass it string fields. Time will tell if this ends up making it too hard to ship, I don't think it has to. We don't need everything Ecto offers. And we can rip implementation from Ecto for a lot of this  ;)

To be clear, this is mostly relevant to the CMS code itself. If you are developing a system and interacting with these types you might not have any of these issues. We can provide a fair bit of Ecto compatibility so you can use Ecto's schemaless queries and schemaless changesets, as long as you know that you aren't going to have the atom proliferation problems. But you need to opt in to that by using specific APIs to turn string fields into atoms. You as a dev should be able to mostly use Ecto but we probably shouldn't.

So should Ecto become more dynamic. No, I don't believe so. Removing problems at compile-time is very powerful and forcing people to not make certain critical security mistakes is great. I think they have a good design for what they want to do. It just so happens that it does not line up well with what I need to do and is in direct conflict with what a good design for my use-case is.

Sometimes you are outside the 90% use-case and that can lead to a lot of friction.

I hope this provided some novel insights into Ecto or ways of thinking about designing systems. We haven't entirely decided the direction but I think this is where we are headed with that.

If you have questions you can email lars@underjord.io or start a conversation on Twitter where I'm @lawik.

I appreciate you reading this. Thank you for your attention.

- Lars Wikman