• BEAM Radio, episode 80: Bryan Hunter on Poka yoke
• Regular Programming, episode 61: About C
• Underjord blog: Anatomy of Embedded Elixir
• Project thread: RevPi Core & PiBridge
• Project thread: Clockwork Pi uConsole, handheld Nerves
• Project thread: The Grand Kiosk, Seeed Studio ReTerminal DM

Shared static

I am going from someone who uses libraries written in C via bindings someone else wrote, to being someone who works with the libraries directly or maybe even writes bindings hopefully.

As part of this I bothered Cocoa Xu yesterday to help me better grok her setup for various projects where she precompiles binaries and shared libraries that are needed for Elixir packages. This was a superb run-down to get me a much clearer idea of how she fits this stuff together. Cocoa is a beast, just an absolute inspiration as a programmer. Also very nice and helpful.

This to say. I'm poking at the details of this a lot and the details of autotools, autoconf, make, linker flags, compiler flags is all kind of messy and frustrating and takes time and practice to get comfortable with. It is not one ecosystem with one set of tools. Unless you are talking about the compilers in which case you generally mostly see a few. But the bigger picture is not so messy and quite helpful to understand. Much like I tried to outline cryptography primitives previously I don't know if people have intuition about this stuff.

I built my intuition and big picture understanding from many years of general Linux use and professional programming work. If I can short-circuit parts of it for anyone that seem wortwhile.

Go is famously good at self-contained binaries. Elixir is famously not. A release is well self-contained so it goes a long way and you can statically compile stuff in there but you will not get one binary. You can kind of do it as a cool and useful hack using Elixir's burrito library. Go does it natively. Static compilation is what that's called. Any code you are using, even if it is a C library you just compile that sucker right in there. I think people try to avoid mixing in non-Go code though as it may affect portability and so on. Go is a bit special.

So let's go to C. You write a small C program. You run the compiler on it. It spits out a binary that you can run. Cool. You can configure your compiler to also spit it out for another architeture (x86 vs ARM vs RISC-V and 32 vs 64 bit) and for another "OS" (linux, MacOS/darwin, BSDs, Windows). Now you want to parse JSON. So you find a library that does that. You download the source and you include it in your program. Then you try to build again. Okay. I will not distinguish compiler vs linker here (because I don't think it matters) so let's say the compiler will complain about not finding your library. You then add a flag about where to find the library and it should be able to build it.

There is a choice when doing this between static and dynamic linking. Static means that the code of the dependency becomes one with you software and the binary slightly larger. It has no external dependencies. But if you use a library that was installed by your package manager (brew, apt, etc) or you got a proprietary pre-compiled library or if you just felt like building that way you can get an object file. On linux a shared object file or library is usually a .so file. On Windows you might have seen a .dll? A dynamically loaded library.

These are not made a part of your application. Your application knows how to loaded their code, call it and use it but it needs to exist on the system the application will run. This was a big deal back in the day because file size mattered and Linux distributions put in a lot of work to ensure we could us a single version of libusb, libjpeg or libmad instead of each app that needed it being bloated by it. This can still be desirable today as you can apply security fixes to a shared library and it protects all applications that use it, without needing to update the application itself.

So this is the trade-off between static linking with larger files but less trouble with dependencies and dynamic linking which gives smaller files, easier to update separate parts but lots of trouble with dependencies.

If you use Debian-derived linuxes you've surely seen libfoo-dev packages being needed when trying to build something weird. Because the libfoo package just installs the .so shared libraries so software can use them. But you are trying to compile something and then you need to know more. libfoo-dev contains headers describing the library interface and that is required to build against the library but not to run something that links against the library.

Static linking gets more love these days because we deliver software differently now and total system disk size is among the smallest concerns in software development and even in OS-level application software. Isolation and packaging is much more desirable typically. Though it does make it hard to make sure xz or liblzma is updated everywhere whenever a state actor hits it again. But it is usually a moot point because we wrap a small application in a container that packs in all its deps and that container is freeze-dried at generation time and whatever random versions of apt-packages it pulled are frozen in with it so you need to update that sucker anyway.

In the Elixir space I know Wojtek Mach is working on delivering statically compiled pre-built Erlang for ease of getting started. This is primarily because OpenSSL and WxWidgets are two absolutely heinous dependencies to wrangle when they feel like it. So statically compiling those is really nice for development. This is also why Nx and the exla library provides pre-compiled XLA. It is big, slow to build, painful build tools.

I hope you groked something previously ungroked. If you didn't you might be able to tell me something I don't know. Feel free :)

Have a good weekend now. You can reach me on the Fediverse where I'm @lawik@fosstodon.org or by responding to this email to lars@underjord.io.

Thank you for reading. I appreciate you.