I would like your thoughts on something. If you were to pick one of my newsletter issues and just grab the essay part as a good sample. Which one would it be?

Is there one that has stuck with you or jumps out? I want to give people a taste of what I do and I have no idea what has been my best.

Help! :)

CMS Project

I have reached a point of writing data with an Ecto repo. The best part is that I've built a test Ecto SQL Adapter that allows me to code without a specific DB implementation in mind. And then write tests for it. We should also write tests for actual adaptors, but that's more integration testing. Jason has shown some neat UI work. Charles is lost deep in  migration land and I bet he will emerge with something good.

Movement in the Underjord

I did a livestream that was mostly me talking and drawing for an hour about the whole photo-sharing app thing I want to build. So that was systems design and architecture in my small way. It is on the YouTube channel because apparently I haven't put it on the site yet. I know what I'm doing.

Another entry in the Teaching Elixir series, it covers Functions. Putting the F in FP.

BEAM Radio 12 with Bryan Hunter talking about punking the servers is out there and I've already heard some really positive responses to that. I even went back and listened to it because it is quite dense and I wanted to make sure I didn't miss much. And I was there. He is really using the BEAM in that work and I find it exciting. Making the case for hot code updates.

Livestream today

Today's livestream is 14.30 CEST / 08.30 AM Eastern on the YouTube channel. Special guest Gerhard Lazu who runs the Changelog.com operations will come on to show me how things are really done with k3s. Soon you can hear him and me argue about Kubernetes on his new show Ship It! which is a Changelog podcast.

I actually find k3s to be a potentially compelling way of doing the delivery part of the chain. He knows this stuff and likes these tools, can't think of a better way to get familiar with them. I have some alternate ideas I will likely try later as well. Try to contrast and compare.

The Vacation

Everything keeps rolling. July will be my vacation. I don't know that you'll notice much difference because I have plans to keep doing the newsletter and the livestreams since I enjoy them a lot. I'm also unlikely to travel much this summer so might as well have some fun on the Internet.

I'm considering putting my fridays into the photo thing, if that's the case would it be interesting to have longer livestreams of me just plugging away at building that or would you prefer the present ~1 hour density?

You probably shouldn't just dismiss it

It's rotting down, this carcass maggotropolis
Interdependent as worms to the grave
...
- Death Magick for Adepts, Cradle of Filth

I like peeling back the layers and seeing if things can be done with less. I find that quite often a bit more effort into a solution that is explicitly stated can be easier to deal with long term than something that abstracts away all the things.

Poking at that impulse though you can quickly see how that breaks down if you take it towards an extreme. You'd be writing Assembler for you home-built processors in no time. You would also get absolutely no buttons put into production that people can click. It would take forever and achieve nothing.

When that's the goal, putting things out there, you have a balance to strike. And I think that involves looking at each layer and deciding whether you, or your users/customers/teammates/organization, need the abstractions it provides. There are some layers you can often take as given such as: we will run this on computer hardware, we will put an OS on that.

I don't particularly enjoy Docker. I find it a bit too much work for what I use it for. Like a weirdly stilted bash-script. One cool thing though is that it is quite useful for cross-compiling from MacOS to deploy on Linux (as the BEAM wants compilation on a host that matches the target environment). Nerves also does this for compiling custom systems on non-Linux hosts. Also good for quickly setting up some services you depend on. I just never really enjoy working with it.

For some client work right now I'm choosing deployment via Docker. Why? Because it will be delivered for on-prem and Docker Compose is more well-understood and documented than whatever suite of scripts I'd create. The abstraction actually makes a fair bit of sense in this use case.

Docker/containers also make a ton of sense for larger scale operations where you absolutely must be able to look at the system in a wider scope and have the details fade to irrelevance. Unifying the interfaces, generalizing. General abstractions are in my estimate the most costly or least powerful.

So how about the cloud? Proprietary cloud things like Lambda, SQS, DynamoDB and such don't really appeal much to me. They are usually opinionated and powerful so they have their moments. I see a lot of useful power in Cloudflare Workers for example. On the other hand they are proprietary and the transferability of knowing how to wield them varies wildly.

I tend to prefer virtual servers, traditional VPS:es, EC2 instances, et.al. Usually the financials make more sense too though that varies by workload.

So if my baseline for cloud usage is the VPS, where do I peel that back and where do I welcome more abstractions?

File storage, I'm happy to use an S3-compatible object store. Why? Because operating a file store is a pain in the neck. I've done enough NFS-shared-drive things that I'd rather not, ever again. Some redundancy is usually baked in or an option. And the abstraction is very opinionated. S3 is not a normal file system, it isn't FTP. It is a particular tool for a particular job with some good features for it and a number of limitations. I think the API helps you solve some problems. Direct client-to-storage uploads is a good one. Signed URLs for file access to avoid serving via your application is also strong. Pricing is usually good. You can also self-host an S3-compatible. Then you are operating a file store which is somewhere between fine and hell.

Physical servers. The other direction from a VPS. I haven't done this in years but I really want to see how far this would take me today. There are aspects of it I don't miss. RAID and failing drives particularly. Failing hardware overall. I think this is the strength of the VPS, it abstracts away a costly, complicated and potentially very frustrating source of sudden errors with something that allows you to replace a server in a few seconds. When file storage is moved off to an S3-compatible you get less of the "can we fit more drives" and any hardware failures are more manageable.

So there is a balance. I'm not big on serverless. Cloud hype mostly makes me tired. But there is merit to some of this.

Docker straddles that line for me. So far Terraform, k8s and most of the cloud native world is still more than I reasonably need. And I think bringing them in would raise the bar to entry into the projects for less experienced people. If you've never built for production you don't know in your bones what problems these tools intend to solve. And now you need to understand that abstraction layer, something of the actual thing that needs to be done (that we are paving over with abstraction).

You may know everything you need for it to make sense. Does that mean you've set it up in a way that is easy to understand? It is very easy to overlook why something makes sense to you and why it feels easy.

For large orgs this is fine-ish, if you have dedicated ops/SRE folks doing all that work you can bring in devs that just do backend code, package with Docker and don't look further. You have frontend folks that never look at the backend. That's not what I do or how I work and its not really what I encourage people to do. I think it is beneficial to any dev, front to back, to be learn to stand up a server of some kind, with an OS, shove some code onto it and get it to work.

Layers of abstraction are necessary to manage complexity. But they grow the absolute size of the machine and I think a decent heuristic is to question any layer you add where you still need to know the underlying thing well to deal with the abstraction. You still may need it. Database ORM libraries are a very common culprit here. So is Docker in my book. Docker doesn't obviate the need for shell scripts. I find myself writing more of them to enshrine the knowledge about any given Docker invocation. On the other hand it would be absurd to say that Docker doesn't provide any utility.

Case by case. It depends. Always thoughtful consideration. Super annoying.

Have you peeled back the layers? Or poured on abstractions? How did that go?
I'm curious to hear your thoughts at lars@underjord.io or you can hit me up on Twitter where I'm @lawik.

Thank you for your attention, I appreciate you reading my words.

- Lars Wikman