Streaming lazily

This actually relates directly to someone answering me previous question. Whether I got it from there or have just been poking around with Streams off and on I don't know.

Someone mentioned a dreaded workshop on Value Streams they were going to and I made a  quip along the lines of: No-no-no. Streams are great, especially for business value. Remember that they should really be lazily evaluated so until someone actually attempts to use whatever you were supposed to build you don't start building it. Don't build the API until AFTER someone tries to use it.

I was kidding and offering a lazy life by juxtaposing the data structures and functionality of computing with the management speak definition. But it is also kind of true. If you hide features behind buttons that do nothing but show a "Thank you for your interest in Export to CSV. It has now been added to our roadmap." you are really being quite customer driven.

The joke only works if you understand lazy evaluation. And streams. So let me make my first prototype of explaining streams. It includes unpacking and dismantling things a bit.

A list in programming is a bounded thing. It has a beginning and an end. Streams practically have a beginning and are similar in nature to lists. But they are unbounded. That is, you don't know that they will end or when. Often streams do end and many uses of streams in computing do assume they will eventually end. Stream processing in data engineerings assumes eternal streams typically. Streaming downloads or uploads mostly assume that there is an end. You can also have streams where you are told up front how long it will be if the source system knows that.

A stream is often not actually more continuous or analog than a list. You are still dealing with discrete items. Chunks, messages or parts. The terminology tends to change with the domain. If your source of data is electronic, such as a sensor, that might be analog. That is, there is no discerete separation, just a continuous indicator for the measurement. But the moment you want to bring it into the computer system for programming you don Analog to Digital conversion of some sort. It ends up being samples. This is also how we do audio. 44khz means around 44 thousand measurements of the audio wave every second and that's how we build a "continuous" stream of audio. This is the fundamentals of digital.

A single-shot HTTP request is one chunk. A file upload will stream as many chunks. Slicing things up across packets is how we do networking. So streams are everywhere. But not all devs have spent time hands on with them and they are incredible tools.

Traditionally (or naively, MVP style) if we are receiving an upload in a web service we might wait for the whole file. Write that to disk. Then we take that from disk and upload it to a file storage thing (yay! object storage!). This is very inefficient. First we build up a potentially quite big file in memory. Then we store this file on disk. And then we read the whole thing again to upload it. At high scale this will eat all your memory, requires fast and large disks and just an immense amount of IO.

If we take a streaming approach we receive the first chunk and then start passing that on to our upload service immediately. In a typical HTTP case we can determine how fast we allow the client to go by how fast we respond to their sent requests (within reason). This is back pressure. So we could hold as little as one chunk in memory to pass on to the upstream file store. Or we could allow more buffering with us in case the file store is slower at the cost of memory. This is useful and applicable in both directions and in many additional cases. Just keeping memory usage lower can be massive in backend services when slinging files around or processing data.

Laziness is particularly interesting as a feature for avoiding unnecessary work. Take a download using the Elixir library Packmatic for instance. Your user selects a bunch of files in your UI and asks for a download. You use Packmatic to do a streaming ZIP of them from upstream sources. An eager approach would say: fetch from all sources, immediately, buffer all we can in our application so we can push it out ASAP. This means we might go faster than the client and if so saturate their pipe and hold a buffer in memory while waiting for them.

Packmatic operates lazily, as the client receives chunks of the archive and acks that you send the next part. Your application requests ranges of bytes from your backing stores to keep a buffer, I believe 10 Mb is default. But if the client goes away the stream will stop and eventually terminate. If the client is very fast, things will be fetched at or near whichever capacity is the limiting factor, yours or theirs. But you have reduced your memory usage to ~10Mb. Data that never gets request also never gets fetched. This is the efficiency of laziness. The opposite of busy-work.

It is also a bastard of a gotcha. Ever used Task.async_stream/2 in Elixir? I love it for instantly maxing a thing's parallelism. But I also frequently forget to actually run or enumerate the stream and then my code just stops working. Literally. It does no work. It is a mysterious thing to trouble-shoot until your palm meets your face in recognition.

So streams. Pretty rad. Also this all applies to feeding audio into Whisper or the entirety of the Membrane Framework. Broadway. Flow. GenStage. So many things.

Any favorite stream stories? Was this clarifying for your? You can reach me on the Fediverse where I'm @lawik@fosstodon.org or by responding to this email to lars@underjord.io.

Thanks for reading. I appreciate it.