U7D - The podcast

Sometimes I keep things short. This is that pod.

New podcast: pod.u7d.io

Alternative methods

Technical idea time. Since I am spending a lot of time thinking about uses and abuses of Object Storage (if you missed it, I'm working with Tigris) I have a lot of weirdness in my head.

If you are not familiar with CRDTs, these are Conflict-free Replicated Data Types. Essentially data structures that even if they are edited separately can converge to a common consistent state. These are a key to a good local-first experience as other techniques require significantly more hand-holding or special-purpose code. The implementation I typically look at is Automerge 2.0 which is a Rust implementation with a WASM output and a JS wrapper library. Meaning you can use it in a variety of places.

As a Phoenix enthusiast and unrepenting idealist it does irk me that LiveView does not give you a local-first story. But that's an intentional trade-off and something else could tackle that more complex use-case.

If we want to build a web-app primarily for local-first we want to do that in JavaScript or something that runs in the browser. We can assume Vanilla JS but it doesn't matter for the larger idea. The app begins online, to actually fetch the app. Then you do Service Worker things to be able to operate entirely offline. You build up state, data, things the user has done and would like to save. So we build up that data with Automerge.

If the user wants "cloud sync" they can do something to identify themselves, github login, email, enter a secret key. Whatever. This happens online, in communication with our service because that is the only way it can happen.

This is where I think I have an interesting idea. What does Auth give you? I'm thinking that it gives you a unique-for-that-service-worker path in object storage where you save your state. Say or new namespace is "github:lawik/browser-f8ea01/".

We presign PUT for any object storage paths for that namespace we want the browser to be able to write. For example:

• user_profile.bin
• todo_list.bin

We can presign GET for other prefixes on the same user so they can integrate changes from other browser instances. Now the client can put new updates to replace the ones it has previously done. And it can get updates that other users have made. To complement this, if we want liveness we can let the online service keep tabs on changes to the object storage and clients can be told what prefixes have changed and when so that they can get up to date more quickly. If you want to get really fun about it you could try doing this over a WebRTC data channel to get peer-to-peer notifications. As the clients are really the ones that know when they push an update.

Max expiry you can set for a presign is typically 2 weeks. That is a pretty solid period to not have to talk to a the service. Meaning aside from niceties such as delivering live updates the user and the online service do not have to coordinate at all. The user can just offload their precious changes for two weeks on a single auth. And if it runs out they just need the auth and then can keep going.

The big reason to use CRDTs for this type of stuff is that we end up with a bunch of diverging data structures that should converge at some point but the timeline is completely unknown. We have the local version in any given browser. This may or may not ever be synced to object storage. If it is, now we have two versions. Any new browser instance would have a new local one and eventually get itself a remote one on object storage. And the Automerge promise is that we can converge these when we need to.

I have thrown a few suggestions towards Tigris that would serve this purpose that I hope they implement. And they have some ideas of their own that will arrive soon that will greatly serve this type of use-case. They also encourage this kind of shenanigans as they consider their object storage a very capable KV store for any data. And we will see more around retrieval there, I know that much.

Online sync for local first without a lot of coordinating server involved is an interesting premise and I'm curious just how far we could take it on just object storage.

Do you think about these things or is it just me? You can reach me on the Fediverse where I'm @lawik@fosstodon.org or by responding to this email to lars@underjord.io.

Thanks for reading. I appreciate it.