Delta support - A tale of two firmware versions

Underjord is an artisanal consultancy doing consulting in Elixir, Nerves with an accidental speciality in marketing and outreach. If you like the writing you should really try the pro version.

Binary deltas of firmware via fwup has been a supported feature for a while. There has also been a kind of rudimentary support for it in NervesHub. We can now remove the word rudimentary. There is now proper delta support in NervesHub. It is not in a tagged release quite yet but NervesCloud runs off of main and so can you if you would benefit from this.

(I initially wrote this up for the Nerves Newsletter but wanted it shareable in some other way)

One thing that prompted finalizing this work was that Frank Hunleth shipped the support for disk encryption combined with deltas for fwup. And since they use encryption this enabled us to make deltas work for SmartRent. I am very excited to see what the results are there as we might get some numbers on the savings at significant scale. Also, I learned a neat trick where I can write things for Frank in english and he’ll write things for me in C. I wonder if I can scale that.

The PRs that enabled this:

Enable firmware delta delivery This included making a new library called confuse to parse the libconfuse style of config file that fwup uses. We needed to extract some information from the meta.conf files provided in .fw archives. This let us determine which resources have deltas enabled and which do not and only calculate deltas for the right ones.

This also includes a bag of tests that took a minute to get right to make sure that we produced the expected results with the deltas we generate.

This made deltas work.

If you created a deployment group on a product with deltas enabled it would immediately start making deltas for the firmwares that are in use within that deployment group. And with a bit of luck it all works out. But the control is very coarse-grained.

Coded up and worked through by me with some good review by Josh Kalderimis and Elin Olsson. Lots of help on fwup-related know-how from Frank.

Move delta setting to deployment group This lets you try deltas with a subset of devices. Thanks Elin for that one and Josh for suggesting it. And subsequently we fixed a bug in that update, thanks Nate Shoemaker.

Add information about available deltas to deployment summary A first step towards making deltas a first-class citizen is to make it visible that they exist. This makes them show up on the deployment group page.

There is more work to be done.

In progress: Safer delta updates This is a large-ish PR because it also generalizes a few fwup-specific things and tries to make for a maintainable future.

The important parts are that the Confuse library is now updated to pull out more and better information and to understand which fwup versions allow which features. This means we can disqualify a device from delta updates because it has a fwup version on device that is too old to allow the update to succeed. Then we deliver the full update.

fwup first got support for deltas for “raw” image writes. Then for FAT filesystems. And very recently for encrypted filesystems. When this lands it should mean deltas become an option for pretty much any user of Nerves.

It also add the choice between delta and full update when pushing an update manually. Which is nice.

In progress: Log transfer savings from deltas Knowing if deltas matter for you and how much they matter is kind of important. And this PR will introduce us tracking that information in and surfacing it in the UI. Showing the savings per transfer and in total.

This can be exciting for a couple of reasons. Maybe you run NervesHub and need smaller updates. Or maybe you don’t run NervesHub but want to shrink your updates using deltas. You can use this as a reference implementation for generating valid delta firmware images which is a bit finicky. In NervesHub it is just a checkbox telling the system whether it should generate deltas for you.

And when the PR for additional safety lands it will do a pretty solid job of figuring out cases where a delta could not work and just ship the full version instead.

There is also a quirk we need to look at in fwup. A delta update causes more reads than writes and this makes the progress bar kind of weird. We know :)

This has been awesome to work on. I expect it to make decent savings for companies that use it and I now understand the delta updates way better.

If you have questions or want to reach me you can use email lars@underjord.io or the fediverse @lawik@hachyderm.io.

Underjord is an artisanal consultancy doing consulting in Elixir, Nerves with an accidental speciality in marketing and outreach. If you like the writing you should really try the pro version.

Note: Or try the videos on the YouTube channel.