I have a couple of things to ask for:

• If you or a business you know needs software work done. Let me know. Primarily Elixir but I'll happily do data work and sling some Python. My colleagues are even less persnickety than I am and we have some spair capacity that I'd love to put into new relationships. Please make the connection if you have it. It really helps :)
   
• If you know a business that should collaborate with me in the more creative ends. Videos, blog posts or open source projects. I would love to make their acquaintance. So help make the connection if you can.

Much love. On to today's stuff.

Mixed bits of brain

My current client workload is kind of odd. I have one client at about 10 hours per week and two at about 5 hours each. This makes for a very decent mix as none of them are in the same space. The trouble is that I don't like just dipping a toe anywhere. So the 10 hour per week is pretty straightforward backend development and my job there is mostly improving system fundamentals to help others continue development. Elixir/Phoenix and Elm. 3 years in. Nothing weird about it.

First 5 hour client is all about data. Data pipelines, data engineering and some ML workflows. The requirements of the project are not particularly wild or novel. Having boiled away all the wild ideas we are left with some very straightforward data engineering and a web app to build. But having not worked in data engineering in this way I am doing full immersion therapy and hitting the podcasts, websites and I'm forming my own opinion as I go. It intersected funny with me ending up at a speaker's dinner with Matt Topol who maintains a bunch of Apache Arrow stuff and that exposed me to the Arrow ecosystem which promises a better future for data pipelines. This is a rabbit hole of things I now want to build.

Other 5 hour client is a focus on local-first, databases and I am building out a mobile app experiment. There is some AI/ML for flavor. I think local-first data and local-first ML are actually things well worth exploring in the near term. This is some hands-on experience for me to see how annoying it might be to do. This is making me use React Native which has not sparked a flurry of curiosity. But running ML tools on mobile devices does require a decent bit of digging and understanding so that's quite interesting.

Meanwhile my mind is in a very high gear and I have for whatever reason been disinclined to try and shift it down. Things I've poked around with doing from Elixir recently:

Library for speaking gRPC to Gazette
Gazette is a broker-type of deal for stream processing. Very interesting approach. Probably won't finish it because I think Gazette won't make it into my final tool selection for Fantasy Data Pipelines 2024.

Fediverse library
There is a very workable Mastodon client called hunter already. But Mastodon is not the Fediverse. Activitypub and all that is. So I started looking at some guides (again) and following this post and then fixing the issues as it is slightly outdated wrt to POST signing. For now the library has three Plugs related to Webfinger and Activitypub. And some fun ways to build up requests. It is chaotic and unfinished. Has been super fun+frustrating to work on which is bad for my bed-time.

Tracing library
Voted most likely to get released on hex eventually by my brain. This is actually an effort where the goal is usage. It has been mostly "done" enough for a while but some ideas made me revisit it and I'll be talking about tracing at a BEAM meetup in Stockholm in January.  You can poke at it here if you are curious. Tests exist and show some usage.

Apache Arrow Flight RPC
I want to build a Rustler NIF to enable using Apache Arrow Flight RPC from Elixir. The reason anything Arrow-related should be a NIF is that one of the good things about Arrow is the ability to do zero-copy as far as possible. Elixir is not the tool for that. Elixir would be great for orchestrating that. This is what NIFs are for.

Arrow Flight is actually not used for anything in the data space currently from what I can tell. And it seems like it should be used for everything. I need to not start this.

If you want to explore a deep and profoundly interesting rabbit hole go to https://arrow.apache.org and hit Subprojects in the menu. ADBC is super interesting, Arrow Flight should be a perfect companion. I think Arrow Flight SQL is intended for future ideal databases while ADBC attempts to address existing ones as well. DataFusion is incredibly interesting and I can't wait to see what it will cause. There are things not directly tied in such as Python's Ibis, Substrait and more as well. It makes me want to build things.

New website stuff?
I've been wanting to port my hugo-based site to something Elixir-based for some time. It has not quite panned out because I have deeply inconsistent formats in use. Any mixture of JSON and Yaml frontmatter with pure HTML or Markdown content. And also template-shortcode-things. It becomes a mess rather quickly.

But I want to experiment with more forms of expression and publishing for Underjord than just blog posts. This might be a new site that is kind of separate from the blog. And this is probably one place I would use the Fediverse library. I've been wanting to do Indie Web stuff for ages as well. My inconsistently used Obsidian notes sync up to github publicly for anyone curious to read. I think that could be part of it.

Enough of that. I'm probably forgetting some as well. Suffice to say I have a bit of tricky time focusing on one thing right now. But to be fair there is no-one paying me to focus on one thing either. I am currently a fullstack data pipeline/mobile app-developer with a side of publishing. I like that fine but it is not very coherent. 

Do you focus? Can you focus? Do you lack variety in your work? You can reply to this email or poke me on the fedi @lawik@fosstodon.org, I enjoy hearing your thoughts.

Thanks for reading. I appreciate your attention.