Where are all the videos?

Well, I've started back up on video production. My lovely editor got a real job but introduced me to a good friendly replacement. My plan is to do one video a month for the moment and that means I'm currently doing script/research and development for the February video which likely releases in March. That's the plan. Keep an eye out :)

We decided some things are required

What do you think when you see that a service announces that the system will be down for maintenance on X date? Maybe you scoff about their dated approach and obvious technical incompetence. Maybe you are just happy they told you. Maybe you wonder about what major technical migration they are doing that they couldn't do without disrupting service.

There is an assumption today that an app doesn't need to be down unless catastrophy strikes. Some degree of High Availability is expected. This is probably the biggest reason that people have such a strong preference for paying their way into a managed database service. Most of the complexity of HA lies in the database and most databases becomes much more complex the moment you want them to be resilient to hardware failures without downtime.

Let's back up. We bring in complexity by default for many things in tech and often with good reason. Source control, version control. This is generally a great thing. I worked a lot of PHP before I got comfortable with git and it was simpler. It was not better. The edges were sharp. You might lose entire days or weeks of work to some bad copying and pasting. Collaboration was pure pain. It is a solution that introduces complexity but enables immense gains.

I've often been critical of Kubernetes/k8s as an emerging default. This is because I don't think most small-to-medium operations, and many large ones, necessarily reap those immense gains for the additional complexity introduced in the system.

What else? Not manually deploying. Complicated systems, artifacts, scripts and infrastructure. The gains? Significant. And also easier to avoid problems that disrupt uptime. Beyond reducing stress and mistakes during deployments it also serves to make things like High Availability more achieveable.

But is High Availability a good default? I think it has a large complexity trade-off. Especially at the database. For about 5-6 years I was in charge of the tech for a SaaS product. It ran a Postgres database. We ran it on a single VPS with nightly backups. I think the only interventions we ever had to do was bumping the max connections and bumping the RAM on the instance at some point. It did not go down. It performed well if our queries were reasonable.

Postgres is a mighty fine database. The moment you try to make it more highly available you introduce some uncomfortable complexities. The simplest case is probably replicating the database to have a hot spare available if the first one falls over. It increases availability but from what I've seen it makes the overall database more sensitive. Now it can be negatively affected by network weather and failing to ship it's replication log as it should. The hot spare becoming unresponsive can negatively affect your main node.

There are defaults that should be standard practices. There are aspiring and existing defaults that have significant complexity trade-offs.

What are the costs and consequences if your database is down? If it is only down very rarely and it saves you a ton of maintenance, monitoring and tuning?

If we step back another step. Is this applicable to your application server? Those are far easier to have multiple of and then load balance between them. It is not without complexity either though.

High Availability is not bad. Of course not. But some recent spelunking in Postgres-options puts me square in considering whether HA is needed and what the tradeoffs are.

What is your standard for how an app should be set up. What guarantees do you think belong in the default these days? Holler at lars@underjord.io or as @lawik@fosstodon.org.

Thanks for reading, I really appreciate your time and attention.