Too slow

My client work currently involves a collaborative white board style app that was conceived as an Svg-based app, built primarily in Elm with an Elixir backend. We started getting reports from our internal users about some performance issues a while back and we've been hunting them down ever since.

Initially there was low-hanging fruit. Sending too many updates, not rejecting ephemeral updates if the server was under heavy load, some slow operations in critical paths. Some accidental data loading when it wasn't necessary. All kinds of fun. But while our work improved things, we could tell when measuring it didn't iron out those issues or folks were reporting.

So we keep digging. We find the heavier parts of the rendering and pre-processing. Things that are fine in all normal tests but start to get really slow at a thousand widgets on the board. One part is browser performance with a complex SVG. One part is way too heavy pre-processing between the data and some rendering steps.

Click an element, wait literal seconds, it gets selected. Simplifying the processing that happened there was quite straightforward and made all the difference. The simple approach was the right as it was being built but the code called has grown since and things must be made more nuanced and more granular.

This pre-processing also had some expensive text metrics calculations. Caching those (memoization) made an immense difference.

The most satisfying thing was reducing the number of SVG elements when we can. Culling. We have a camera/viewport concept and we can calculate which elements are actually in that thing before rendering. Having that idea and then executing it and just seeing it work felt wild. I really do like "graphics" programming to the extent I get to do it.

This also led me to a research rabbit hole in case we need to abandon SVG down the road. If you check Mural and Figma they do Canvas with either WebGL or just accelerated 2D. We could take such an avenue eventually too. Probably using CanvasKit based on Skia or something in that vein. We already have the abstraction of geometries and gestures. The rendering layer is mostly separated from the interactivity. Unfortunately it probably isn't currently worth the investment because it seems like a ton of fun.

I also noted that our JSON representation going over WebSockets got a bit heavy where a larger board could easily be 3+ Mb without image data. There is not a standard compression over WebSockets. Text compresses really well. For heavily repetitive data (pasting the same element, oh, about a 1000 times) a gzip compression would boil it down to 58Kb. Not a realistic typical outcome but we would certainly have some gains by compressing large boards for transfer. So that's something I implemented. I really don't like dealing with JS streams APIs. They work well enough but feel incredibly clunky. Also, turns out Safari just added DecompressionStream. So a bit of feature detection was in order.

Overall the app will be way more snappy with these changes. The big one was about being way more granular about when and where processing needed to happen. And caching expensive work in memory. I think we've bought a bunch of headroom for most normal users. We've made it good again for heavy users. We will probably keep finding interesting challenges with the really ridiculous users.

Is there a lesson? Probably that over time software will grow and things that were plenty fast will accumulate friction and cause problems. And sometimes it takes focused effort to dislodge those performance bottlenecks. I personally find that kind of work very satisfying. When it works.

Do you have a favorite performance story? You can reply to this email or poke me on the fedi @lawik@fosstodon.org.

Thanks for reading. I appreciate your attention.