Fundamentals & Deployment

Underjord is a tiny, wholesome team doing Elixir consulting and contract work. If you like the writing you should really try the code. See our services for more information.

I’ve had Gerhard Lazu on my livestream once where he showed me a way of doing CI/CD with k3s and ArgoCD. The podcast Ship It! that he hosts on Changelog recently had Kelsey Hightower on. Kelsey is to many this guru of the cloud native and Kubernetes space. I’ve had good value from what I’ve heard him say in the past. In this episode he really spoke to me.

So he essentially said, I’m interpreting here, that when it comes to deploying software to servers the documented manual steps for deploying something need to be the canonical reference. Then whether you build bash scripts, Ansible playbooks, Makefiles, Dockerfiles, Terraform, Kubernetes or something else to encode that procedure into something repeatable and scalable that’s a separate step. Having documented the process required to set it up means that there is an answer to the question: How do I get this running? An answer that doesn’t require you to parse the .yml files or grok Ansible roles and groups.

I feel like I can spring onward from that. I will also very likely not land on Kubernetes as the answer for anything that I’m trying to do with deployments. This still seems applicable to me. All of these deployment tools are abstractions for systematizing and automating procedures. How much abstraction you need should reflect who you are and what you are trying to do. Given that we create this documentation for the manual steps to set something up. Then when we are trying to work with a script of some sort that automates the process we can at any time return to that source of truth and see why any particular part of that process is there.

Some might say that a Dockerfile can be that step-by-step description. Typically I rarely see a Dockerfile capture the “why” in the way documentation typically does. There are also concerns that are mostly about Docker that get enshrined in a Dockerfile. There are things that are done a certain way in a Makefile because that’s the most convenient way of describing it in that DSL. An Ansible playbook has a ton of potential concerns that are not about describing how to deploy your application but rather about how to deploy it on particular infrastructure.

So if I write up what steps it takes to get an application running for production in the README and make sure those steps are working, by testing them myself, I can then take whatever automation tool I prefer for simplifying things and translate that documentation into that tool.

One of the big points he made in that conversation is that if you don’t figure out the manual process you won’t have a clear explanation of the steps involved and if you don’t understand the thing you are doing you are going to end up in a mess. You’ll may even end up in the situation of “I don’t want to touch it, I don’t know why it works”. Kubernetes is that to me and I have no compelling reason to demystify it. I don’t operate at immense scale with a massive team. Regardless of the deployment system and underpinnings you work with, someone needs to know how to deploy your application, it needs to be well understood somewhere. At the human scale. Documentation is a good way of making that information transmissible. It is not a solution for immense scale, it is a solution for the human that needs to implement it at scale.

I’d be happy to learn some k8s if I have a good reason to. One day I’m sure I’ll be paid for to work on something involving it. Hearing this view of starting with a solid foundation and only really building as high as necessary off of it was good. It was especially good to hear from someone that’s both highly regarded and deeply connected to these massively complex deployment solutions. Seems to me that he deals with massively complex deployment solutions because he deals with massively complex deployments.

Now I don’t want to perform all the manual steps myself. I certainly want to automate my deployments. I’d rather not introduce a ton of complexity to do it though. I think my ideal level of complexity is just above a bash-script. Somewhere in the range of a bash-script and an Ansible playbook. Maybe it’s time to get into Makefiles?

CI/CD isn’t exactly a thing that sparks joy for me so far. I’ve had a decent run at my current client with GitLab CI. It all feels like a Dockerfile which is sort of not offensively bad but not delightful either. I find them both easier to understand than GitHub Actions which are either extremely straight-forward or mind-bendingly weird depending on what I try to make them do. Once I have it set up it is very satisfying to see all the things I don’t have to do happen. The tools often leave something to be desired.

I’m currently considering setting up some boxes and running self-hosted things. That’s not in line with what Kelsey recommended. He spoke a fair bit about managed services. For me it is more about idealism, enthusiasm & Schrems II. I think I’ll go for GitLab because I find Git + CI tooling living together convenient. I think it could be more elegant with smaller tools, Gitea and some other CI tool perhaps, but the integration might just save me a ton of steps. If all my Git repos want CI anyway, might be a win to keep them coupled a bit more tightly.

What this means to me is, I should document the production deployment steps of my various Elixir apps properly, maybe even standardize a bit. Then I can consider automating a way to deploy them well. Right now I’m mostly doing a bunch of manual BS because I haven’t invested time in automating it and don’t love any of the options.

If you have suggestions on lightweight no-nonsense ways to ship things to a server and get them set up I’d be curious to hear and synthesize that input into whatever I end up doing. You can share your thoughts via email lars@underjord.io or on Twitter @lawik.

My writing is supported by a couple of nifty sponsors and they are looking for Elixir developers. If you are an Elixir developer or aspire to be one, please do take a look at the jobs for good positions with companies I’ve vetted.

Thanks for reading.

Underjord is a 4 people team doing Elixir consulting and contract work. If you like the writing you should really try the code. See our services for more information.

Note: Or try the videos on the YouTube channel.