Tradition and Ritual in Software, or The Unwritten Code of Code

Underjord is a tiny, wholesome team doing Elixir consulting and contract work. If you like the writing you should really try the code. See our services for more information.

I’ve been thinking about shared experiences: culture and tradition. As the seed of this was written close to the holidays, it might be that they are to blame. But this post isn’t about Christmas or the new year; it’s about my reflections on these human matters and how they apply in the software development world.

How we run dangerous commands

Have you ever had to run a potentially dangerous command on a production server? A destructive piece of SQL? A batch update/data migration script? A script that cleans up accumulated garbage files in a recursive manner?

How did you approach it?

Do you have a small alarm bell that goes off in your head when you are close to doing something that would be really painful (or impossible) to revert? Do you adjust your approach because of this signal? I know I do.

A cluster of synapses got together in your brain just to warn you. They strive to prevent me from shooting myself in the foot at the terminal. They probably first formed that time I blew away my entire music collection with a bad shell script. They hardened the first time I rm -rf:ed something that ended up resolving to /. I haven’t had this kind of catastrophic failure in a while. Thanks, brain.

When I feel that tingle of “are you sure this is the query you want to run?” I usually end up taking the SQL query, changing it to a SELECT and just eyeballing the output a bit. Does it hit the expected number of rows? Do the results look reasonable? Did I, in fact, just run a fresh backup?

Similarly, I ask myself before I run any recursive rm command on UNIX: “What files would this hit?”

If possible, I dry-run the command.

I’m also well aware that some would snort and say that this is not how it should be done. I wager that’s correct but not particularly useful feedback. If you can run testable migrations that can be rolled forward and back, that’s better. If you can do it the better way and the trade-offs make sense, do it the better way. But don’t feel bad if you don’t have those opportunities or resources. We do what we can with what we have. You can get far by just developing a good instinct for when you’re about to screw yourself over.

Is there a word for this alarm bell? This instinct? This moment of hesitation and the practices you put in place to make sure you’re not screwing up? I think it is a shared experience of many server-wrangling developers out there. I know an old colleague that always commented out any impactful command in case he’d accidentally fire it off before it was ready. Another colleague would type it out and Ctrl+C to cancel it but keep the command on screen while checking things.

It’s an interesting thing to me. A small moment of recognizing that you are fallible wetware and that the machine, software and hardware, does not care. Is this the central friction of human-computer interface design?

A moment of humility, perhaps.

Rituals around deployment

When the new thing is going out, what do you do? Do you run a specific command? Do you press a button in a Cloud console? Do you have a friend or colleague double-check your input values? What leads up to this point?

There are processes you can adopt where CI and CD will completely eliminate the final button press. I’ve found these to be rare in actual reality. I hear it’s a strong idea. Even with a very comprehensive CI/CD solution in place you aren’t really free of this. It’s not all about the tech. What does your CI/CD pipeline test for and why does it do that? Who’s notified, and what information is broadcast when an update goes out? What rules does it use for determining success and failure?

There seems to be some level of ritual around updates/releases of software wherever I’ve worked. In my own projects, I’ve created a ritual based on my habits and lessons learned. In an existing organization, I try to understand the things they already do. If I find their ritual lacking, I’ll usually try to leverage my understanding of it to improve it. Trying to change a ritual without understanding it is a fraught process.

Why do I call it a ritual rather than just a process? I’m not convinced it is only a technical, structural matter. Often we do things that aren’t strictly necessary but are reassuring.

You make sure Alex in Marketing knows the update is going out because they get pissed if they don’t feel included. You have a colleague check your thinking on particular things while you let other things right through. You have QA give the OK on the staging environment. Maybe you review the commit messages or the file changes that go into the update in full. Maybe you only check if there are migrations to run. Do you tag a release? Do you push it to a specific branch?

Much like some specific flavor of a burial ceremony, you have things that can seem arbitrary or strange to an outsider. These things are usually part of the culture (colors, songs, phrases), things that serve a clearly social purpose (saying a final goodbye to the deceased, closure) and things that are fair to consider a technical necessity (burying/lowering the casket/incinerating/putting the deceased somewhere).

In a release process some checks, steps and sign-offs can seem arbitrary but are ingrained in the culture and may have an unclear purpose or be entirely vestigial. In any release process there are checks, steps and maybe even sign-offs. These can seem arbitrary, but I believe that is because they are cultural in nature. They exclusively serve a social purpose while others are a technical necessity or clearly technical choices.

I think most dev shops end up with some kind of ritual. Sometimes, it’s only for communicating the changes to the surrounding organization. Other times it can be a technically intricate ritual with many mysterious incantations to get that particular system up and running with a new version. The latter type is usually a good target for significant scripting and automation.

As humans, I think we always attach some level of importance to ritual. Some people feel stronger about it than others. To some, it’s about safety, and changes to the ritual can feel frustrating. To some, finding an ideal ritual is the goal, and changes are inevitable. Some desire a minimal ritual, and others want the iron-clad, double-verify, none-shall-pass sort of approach. Sometimes the ritual is dictated by culture, sometimes by actual regulations, sometimes by resources.

I think some people rely on the stability of the ritual, while some view it more as a means to an end. A tool. But that might also be why it works: it is something in which many different types of people can see value. It is a point where the lines cross, where Venn diagrams overlap and where the mixed purposes of the ritual can satisfy the differences of the participants in a shared procedure. We come to it for different reasons but we share the feeling that it is necessary.

The longer a ritual has lived, the harder I think it is to maintain the protective abstractions. Some will want the ritual to remain as it always was, and some will find it stifling and want to change it. I think a lot of people experience this with Christmas, for example. For many people, the question of how to celebrate with your family is a source of potential conflict. While the general idea of the ritual is to come together with loved ones, the particular ritual is loaded with years of tradition that some love, some endure, and others try to avoid.

My family is very progressive. Our Christmas celebration requirements have scaled back quite far. So far, in fact, that I’ve had to start re-introducing some Christmas traditions for me, my wife and my daughter because I had ended up missing them. I’m also very glad we aren’t gripped by a tradition set in stone by our grandparents. I just really want a Christmas tree. Because I like it. The tree awakens fond memories.

So, what’s my point? Probably that our process is not exclusively based on cold logic and business realities— and that’s fine. We are fallible wetware and we need our creature comforts. Framing a process as a ritual changes my expectation of what it should achieve from the raw technical output to a broader consideration of the people involved. I find that useful.

UNIX as a tradition

I’m not sure whether I should say UNIX, or hacking, or what, to center the culture around. That debate is a rich source of “well, actually,” which I don’t really care about. It is not only free software, or open source; it is not just MIT, and it is definitely not exclusive to Silicon Valley startup culture. As a developer on Linux with mostly open source tech, you are working in a world wrought from a mix of the cultures I mentioned— even though I’m surely missing some. I’ll center it around UNIX, because I think that’s largely accurate.

When you use a term such as grok or borked, you’re leaning on that tradition. There is a lot of useful shorthand that stems from this common traditional lexicon. Some of this stuff is also quite the shibboleth, a way to identify if people are part of this tradition.

Tech and software development as a culture is widening, and I’m sure there are plenty of developers who have minimal exposure to this particular culture. That’s fine. Years in corporate dev might not expose you to this; I don’t know, I haven’t tried it. If you’re going through a software dev bootcamp at 30 you might have no previous interest, and you won’t absorb a culture from an X weeks long intensive course.

So while the traditional culture of UNIX is not the only culture of software development and tech, its influence can probably be found all over. And it is my culture. It is where I found my footing. A lot of the culture is also a bit old and outdated. Most of the terminology in the Jargon file is not in use or generally recognized. A lot of people who are in open source now have no particular connection to the wild 80’s hackers covered in The Underground Book (free download by the way). A lot of them have never heard of The Hacker’s Manifesto, which burned bright to me as a teenager.

That’s fine. The culture has its traditions and history, and I don’t think this culture will be forgotten. It is also fine to leave some of it behind. Some of it was only funny back then, some stuff seemed dated even when I got into it. Some of it is irrelevant. Some of it should be brought with us as good, useful, relevant ideas, interesting thoughts or inspiring history.

I am becoming increasingly aware of the fact that my way of knowing about computers, my way of getting into programming, is not the only “real” way. I continue to practice the UNIX tradition; I rather like it. I’m very fond of its history. I appreciate a lot of its culture. But it doesn’t live or die with whether newcomers know the heritage of grok. I’ll happily put a new-formed hacker in touch with these roots. But if they don’t want it, we still have plenty to share.

It is easier for me not to expect new people to know about these things and relate to them if I recognize that it is a tradition I was brought up in. To me, understanding tech and knowing programming are intimately tied to this cultural movement, the history and its current shape. This is true for me, very likely true for many others, yet absolutely not true for everyone. And it would be absurd if I didn’t acknowledge that people can come to the same skill set from entirely different cultural traditions.

Shared experiences, rituals and traditions in software

I’ve found the trope “it isn’t about $TOPIC, it is about people” pretty tiresome in the past. In the back of my mind I’d go, “sure, but really, $TOPIC is pretty important.” Whether $TOPIC concerns technology, a web framework, software quality or open source software, I still find it a bit disingenuous to claim that it is all about people when the actual thing is what brought them all here.

The trope can still feel tired but I have changed my perspective a fair bit. The people situation has to be in decent working order for anything to happen, for things to get built, delivered and used. And people come in a large variety, with many different opinions, fears, hopes and ambitions.

I think we can most quickly find common ground in shared tradition and we can build an understanding of differences more quickly by recognizing if we come from different places. This applies whether you are from enterprise .Net life in an MS-focused stack, straight UNIX with lots of dynamic scripting on top or career-switch-focused bootcamp graduation. All these places will put you into the world with different tools, priorities and often value systems.

I think the shared experiences often help bridge these differences. We all know how dumb computers are. We all know how dumb we programmers can be. This is the common ground.

Tradition is about where you come from and what feels familiar to you, where you find comfort and meet discomfort. To me, rituals are the recurring patterns in how you perform your practice. It could be the practice of your tradition, of your craft, or of your organization. Lots of ritual practice stems from your culture.

The Github Pull Request is an evolution of the patch sent via email and has firm roots in open source hacker culture and tradition. Those roots have deep implications for what a PR is and how it works. They are asynchronous, require no trust and imply low trust. It is a ritual of checking someone’s work before letting it into your project.

You likely have rituals that you built based on your experience and that are your preferred ways of working. Do you subscribe to a TDD approach in everything you do? Do you always start a piece of work by thinking it through? Or maybe you build the dumbest thing you think could work? With practice, you build up more of these patterns and tear down old ones.

Your organization forms rituals from culture and experience in a similar way. Whatever the driving values are, whatever decision-makers hold dear and press for, whatever the organization enshrines as important by its actions: these will shape the pattern of practice and turn into the rituals of that workplace. A lot of them are accidental, incidental and unintentional— the “long-tail consequences” of choices.

Shared ritual becomes a mix of habit, useful shorthand and practical routine. And the more similar your personal rituals are to your organization’s, team’s, or co-workers’, the less effort it is to settle into working with them. If the rituals are too far apart, you are very likely to experience friction. Rituals that you unknowingly share can turn into moments of strong common understanding with a colleague: “Oh, you do that too!” or “Yeah, makes total sense to me; I do a similar thing.”

There is a grounding in recognizing that you are not a technical optimization machine. You are not a robot making all your choices according to some absolute best. Likely, you are navigating according to your values and experiences. Your technical tradition is your foundation and a place of origin. Your rituals are the tools you’ve built and the experiences you’ve earned. The practice of making software is a complex human endeavor.

If you made through to here you might have reflections or feedback. I’m happy to take those in via email at lars@underjord.io or on Twitter @lawik. Thanks for reading.

Underjord is a 4 people team doing Elixir consulting and contract work. If you like the writing you should really try the code. See our services for more information.

Note: Or try the videos on the YouTube channel.