Building a Startup on Elixir to a $50 mil round

Underjord is a tiny, wholesome team doing Elixir consulting and contract work. If you like the writing you should really try the code. See our services for more information.

I became familiar with Marcin Kulik when I was brought in to work with a company called Vic.ai. A fintech startup. Marcin is the guy who started building their entire Elixir system and was the person to primarily onboard me to the work. When I was brought in they were a small team of 4 devs, looking to grow up to around 7 at the time.

Special thanks to DockYard for supporting my work on this piece. They are a digital product consultancy with deep roots in the Elixir community and ecosystem. You’ve seen their efforts supporting Phoenix and ElixirConfs and with their support I can bring you more stories from companies and people using Elixir. Let’s get to it.

Vic.ai has a product that makes a lot of sense to me. My wife is an accountant (when she isn’t editing podcasts, growing vegetables, etc) and I’ve run my own small business at least three times. So I’m familiar with the domain. I know how disjointed invoicing can be. It is a field which lives and breathes PDF files sent over email. The grand interchange format of a massive industry.

PDFs and photos of documents are not a standardized format as far as what data they contain while invoices are themselves fairly structured and formal documents. Vic trains machine learning models to automate the interpretation of these document files into a structured form and offers ways to verify, make corrections and pass the results on to your ledger. I can see how this can save a lot of time. For a startup this is an unusually lucid value proposition.

I think it is a remarkably good fit for ML. The documents are somewhat formalized and fit in a certain mold. An invoice for a particular region typically contains a limited set of features to look for. Your due date, invoice number, totals, line items, vendor information and such. But it is formalized for humans not machines, there isn’t a single template being used, so it is hard to automate as is. And here ML becomes a missing piece translating human-readable to machine-readable and feeding it into whatever your process is. A mix of OCR and ML.

Vic.ai hasn’t been the most visible company in the Elixir space so far. You are perhaps more likely to know of Marcin from his work with Asciinema which is a very cool tool for recording a terminal and replaying it. You should check it out.

I sat down to talk to Marcin about the use of Elixir in the company and before I had transcribed and edited our conversation the company closed a $50 million round of investment. So that little team is already growing significantly and will likely continue to do so. If you are interested in applying for a job with Vic you can reach out to me, we can talk and if it seems like a good fit I can give a warm intro.

On to the interview, which lands us smack in the middle of going from small-talk into relevant topics.

—

Marcin I know how it is. I used to run my own business as well.

Lars Oh, really?

Marcin Yeah. Before this. So this will transition us into the history of Vic because I knew Alex before, for a couple of years. I met him in 2015 I think. I left a Ruby on Rails shop in 2013 and I started my own company doing consulting gigs.

_(editor’s note: Alex refers to Alexander Hagerup, CEO at Vic.ai) _

From a colleague at that previous company I got in touch with some people in Norway and then did some projects for them. Then I met the guys behind the Gitorious project. A project similar to Github but self-hosted and open source. That project was later acquired by Gitlab.

It was through that chain of people I got to know Alex.

Alex had this idea for a mobile app he was building and he needed an API. So that’s how we got connected. We met and I did two projects for him. During this time I still had my own company in Krakow. So like you I was managing my own time, paying my own taxes and all of that. Until 2017 I think, when I had already done a couple of months of work on Vic.

Lars Was this app Alex was building the foundation for Vic?

Marcin No, so in 2015 he had an idea for an app that was supposed to be for sharing images with your closer circles. Something like Instagram but with friends and family rather than public.

I was building the backend. We worked on this for two years but nothing came of it.

Then around the time I was getting into Elixir and Phoenix, Alex asked if I wanted to build a site for a US startup he was building. It was for booking places at restaurants and venues to use as workspaces during off hours. Essentially providing co-working space when the business is mostly empty.

So this was how Elixir was introduced to Alex. He didn’t care much about the specific technology. That has been a nice thing working with him, he trusted my tech choices. He was never going “Let’s use PHP or Python”. I built the photo app backend in Clojure.

(Editor’s note: Alex would let the record show that he did care about the choice but trusted the suggestion :)

Lars So backing up a bit, and looking at your background, how did you get started with programming?

Marcin I was in seventh grade of primary school and there was a computer shop in my hometown. I already had an Amiga 600 that I was mostly playing games on. I went to that shop one day and they had a book, with a disc, on their shelf which said AMOS and it was basically the handbook for the AMOS programming language. So that was the Amiga version of BASIC.

So I went home and asked my parents if I could have it. They saw that it was programming, figured it was too difficult for me and didn’t want to waste money on it. They assumed it would end up unused on a shelf. So I went to bed crying because my career was in ruins already.

I don’t remember how I finally got it but I think it was a year later, so in eighth grade. So I started programming on the Amiga. After eighth grade they bought me a PC because I was going to high school with a focus on mathematics. So they got me this nice PC, an Intel Pentium at 100 Mhz or so.

So after initially just playing PC games. This was the summer of 95 or 96 and Quake was out, Duke Nukem was already there so I was playing all of these shooters. But after a year I got bored and wanted to do some stuff myself.

I was looking into Turbo Pascal. The IDE was super nice, F5 to compile, F6 to run or something like that. Everything was integrated nicely and it was simple. So I learned Pascal. I was also observing the demo scene, all these crazy people who do magic things with programming. I was hooked. I collected the demos and started reading the zines of the scene. There was a lot of good insight there, in the tech corner they would cover things like doing plasma effects in assembly or casting 3D points into screen pixels and such.

So I was reading that and then I started writing my own silly demos in Turbo Pascal. There were some pieces where that was too slow because I was trying to calculate a lot of pixels, 320 by 240 so over 60.000 of them and Pascal was not ideal for some of this stuff. So I needed to go lower.

I went into C and then hit similar problems and had to do some things in Assembly. And C has this nice ability to just write some assembly in-line with some macro. So I did that and then I was just jumping between languages as I needed them. C, C++ and my first professional job was PHP.

Lars So do you have formal training in computer science?

Marcin Yeah, I do. I have a master’s degree in computer science. So that happened in parallel to my explorations. I wouldn’t say I didn’t learn anything at the university but it was mostly complimentary.

Lars It sounds like you were learning quite actively outside of school.

Marcin Yeah, exactly. I didn’t have much trouble going through university, to put it lightly.

So then around 2003 I was doing PHP and making some websites. Then I learned Python and some other things. In 2006 I got into Ruby a lot and worked at a Ruby shop in Krakow where I spent 6 years. At the end of that period I was kind of burnt out by the complexity that’s hidden under the surface. The magic of that brings you misery while you are not paying attention.

So I was looking into various languages and some people at the company were looking into Clojure for a side-project. They brought it to my attention and that was my gateway drug into functional programming.

Although, if you look at my late Ruby code it was very non-idiomatic and was looking more like functional code. Very explicit, with explicit dependencies and immutable. So then I went into Clojure and it was amazing. It opened so many ideas for me and a completely different look at how software can be built. If you spend your whole career doing object-oriented programming, you never touch functional, then you may not know what simplicity means. It was like enlightenment for me. An “aha” moment.

I didn’t want to go back.

So I used Clojure for some projects, like the one I described with the API for Alex. I was aware of Elixir early on, back in 2014 or 2015. José Valim lives near Krakow and had showed up to some Meetups in Krakow and I was at one of those. He was showing off opening two terminals with different IEx sessions and connecting them as nodes.

At that time Phoenix looked like it was trying to please Rails developers and my personal feeling was “Nah, this is just another Rails and I don’t wanna go there”. So I had mixed feelings at that point.

Everything else looked great but I wasn’t sure about the web part and how it would evolve. Today we know it evolved into this amazing space of interactive, real-time solutions like LiveView and LiveBook among others. But as it was I decided to observe from a distance. Around 2016 or so I started playing with it.

Yeah. So I read the Programming Elixir book by Dave Thomas and then the Programming Phoenix by Bruce Tate (Editor’s note: not to forget Chris McCord & José Valim). After that I was like, oh man, this is it. The language, the whole platform, it solves so many problems I’ve observed and dealt with in the past. How is it possible that nobody knows that this gold is here?

Lars So did Vic start with Elixir?

Marcin When I joined Vic.ai there was already a guy working on some AI parts. So we ended up having two areas. The AI parts with all the OCR, document processing and machine learning. That part uses Python.

So Alex started the company with the guy doing the Python work but they needed an API and many other missing pieces. So when I joined we discussed what we should use for the API.

I suggested that Elixir and Phoenix would be a great fit for it. It wasn’t hard to convince them, they trusted me with this choice. We didn’t see a need to rebuild the Python code in Elixir because we didn’t need to closely couple the AI part with the API.

So I built the API from scratch with Phoenix. There was nothing before that.

Lars How long did you spend working on the Elixir side more or less on your own?

Marcin Over two years. I believe I started in 2017 around June and at that time it was still consulting, not as an employee. And late 2019 we hired three Elixir developers, my first Elixir colleagues.

Lars Did they already have Elixir experience when they came on?

Marcin Yeah, yeah. They did. We explicitly looked for people who had experience. I posted on ElixirForum and we had many responses and lots of great candidates. So we spent quite a while doing interviews. There were so many candidates and we could only hire three. With how good the pool was we could have easily hired three more if the company would have been able to at the time.

Lars How was it like to bring people into that Elixir code base? A two-year old code base has a lot of stuff in it already, especially one under startup pressure.

Marcin Right, especially one created by a single person. Well, so the onboarding, I remember, was hard work for me. It was mostly challenging due to the fact that I now had to share this codebase and what is in my mind with the external world. Previously I’d only needed to share relevant details with the Python developer and they weren’t necessarily interested in the Elixir codebase itself.

So I had to go from not documenting anything and knowing how everything connects in my head to be more explicit in every aspect of it. So for me it required a lot of time. But as for those colleagues they were already experienced when we hired them, they didn’t require any training. They just needed many introductions to various pieces of the system and corners of the codebase.

So one thing we did was that they all started during the same week. So we brought them together in our office and we had an onboarding week where we spent a whole week on that. Explaining how things are connected and how all of it works. And then we fixed one or two bugs together. It was great. I think it did wonders to get them up to speed. Just to be able to be in the same room for a week. (Editor’s note: This was before COVID-19, this is being published during the pandemic and in some places bringing people into the same room is still weird so note that this was normal at the time.)

Lars So this was about two years ago. And you’ve kept bringing in people, especially recently. How would you compare bringing people into an Elixir codebase and other language codebases you’ve worked with?

Marcin That’s a good question. This is not a scientific answer but I feel like with Elixir, when they ask questions about why things were done this way where do I find something, I can just point them to the right part of the code. A specific module or area. Then they could just read and understand it due to the simplicity of the language and explicitness. Most issues with understanding have been where we rely too much on macros.

There was less back-and-forth because people can figure things out on their own because the code is more understandable. There’s not a lot of hidden things and no chains of parent classes bringing in new methods and different behaviors.

They can figure out the context on their own easier. It could just be that the guys we hired just did really well. They are fantastic developers. So that helped. But I think the aspect of having less magic and being easier to understand made a difference.

Lars Have you experienced any difference in recruiting and hiring for Elixir? Has it been harder or easier than your other experiences?

Marcin It was much easier. We had so many good candidates. Actually, when we did hire we were also recruiting for the Python team and me and the guy who was in charge of the Python system were doing interviews together. There was no-one else who could pair up and discuss the candidates.

We had so many Python candidates. But the average experience level was lower and we had to spend much more time determining who could be a good candidate.

Lars What have been the best parts of working with Elixir, for the product and the codebase?

Marcin The one I see the most is that Elixir enables much, much easier refactoring of the code. Because everything is a module, which is a bag of functions, nothing is hidden. So when refactoring you just move things, cut and paste functions around. And it always just works. In most cases the compiler can tell you if you broke something. I feel like refactoring in Elixir could not be easier.

When changing configuration, or more in-depth OTP stuff, dynamic runtime concerns and supervision trees then you have to pay more attention. But just moving functions around is easier.

So, a fun example of this, as our codebase was growing I missed a feature from Clojure which was preventing cyclic dependencies. Initially I was bummed about it but later realized that it is probably to be interoperable with Erlang and hot code updates. That would make it hard to enforce.

So to get some of those guarantees I converted the application into an umbrella application which means it is broken up into multiple applications and they can’t have circular dependencies between themselves or the application won’t boot. When the first colleagues came that changed, the umbrella structure was getting in the way of the daily programming and consensus was to change back into a monolithic app. So one of them made a PR for changing it back and it was pretty much just moving things. Moving files, updating the project mix project file. A lot of moved files and folders.

It wasn’t a big struggle, war and suffering. He just opened the PR, moved all the stuff, we merged it and it wasn’t an umbrella app anymore.

Lars Typically moving from a decoupled system into a monolith or vice versa is a horrible process. Often months or potentially even years, right?

Marcin Yes, exactly. So I think this is a testament to how refactoring is simpler in Elixir.

Lars So right now you are doing more ops than regular development, how is Elixir on that end, operating it?

Marcin Yeah, when we hired my three colleagues I started my transition into a more DevOps role. We needed this position at the company. For me it was an opportunity to do something different in a familiar environment.

Our Elixir deployment is fairly similar to other languages. We use Docker containers. We build a release, wrap it in a container and deploy it.

We do use Erlang Distribution for clustering and we use libcluster to form it. This allows us to use Phoenix Channels and pg2 process groups for messaging.

We actually also have a number of separate Elixir projects that we use for integrations with external systems, we call them connectors. They are developed separately but join the same cluster as our main API service. So our cluster is heterogenous. And we call these services via RPC calls. So we didn’t need to implement Phoenix endpoints and JSON serialization. We just send native terms over the wire. It also allows us to use pg2 groups for service discovery without any extra infrastructure.

Lars If you were building another SaaS backend, would you choose Elixir again?

Marcin Yeah. Totally. I would not hesitate to choose it again.

The only issue we’ve had that is worth mentioning that could sometimes be a problem is that you won’t find as many ready-made libraries as you would in Python or Java. More than 80% of the libraries we needed were there and more often than not they’ve been of high quality.

We’ve also worked around that with a Port at one point, where we were already using a Python library on the machine learning side, so we wrapped that up in a Port and it has been working quite well to this day. We are looking at reimplementing it in Elixir now but it has been great.

Once you learn how to use Ports it is tremendously useful because you can run anything, a Python library or a super efficient Rust implementation or anything that you can communicate with over standard input.

If you treat the BEAM, Erlang and Elixir, as a robust orchestrator then you can pretty much integrate with anything. The concurrency of the BEAM allows you to hide anything behind a function call, GenServer calls, HTTP calls. And the caller waiting there doesn’t block any other workloads in the system.

I want to thank Marcin and Vic.ai for letting me do this interview. And I don’t know which take-away I like more, ”trust your developers on their choice of tools” or ”Elixir is a solid choice for your startup”. Both sound good to me.

The thing is, I’ve worked on this code. It has both good parts and rough edges, as all code-bases I’ve ever seen do. I think the fact that I felt comfortable making significant updates to it within a week of starting speaks to what Marcin said, changing code, reworking and refactoring is made easier by Elixir. I spent my first few months deep in the backend and out in some integration services. I never had to care about the GraphQL API while doing that because they just weren’t coupled. And when I got into the GraphQL stuff I didn’t have to spend effort thinking about the deeper parts of invoice processing because I was working at the surface level. Part of this is because someone, probably Marcin, did a good job of layering the design of the system but some of it is just the conceptual simplicity of the language.

I’ll add that one of the things I worked on is something I don’t know how I’d do as well in most other languages and runtimes. The Vic.ai system has integrations to invoicing systems and those systems have published rate limits and constraints that we need to conform to. We want to conform in an efficient way and even if we hit an undocumented limitation we need to handle that. So we built an approach to distributed rate limiting according to particular rules, pooling to constrain concurrency of requests, back-off and circuit breakers for handling problems predictably. Without storing anything in Redis, without relying on a single source of truth to store ephemeral information about our communication. It was fascinating to put together and seems to have worked really well.

This is all at the technical detail level and the talent onboarding level. Perhaps as interesting to a decision-maker should be that Elixir paid off in effectively orchestrating a normal Python-based Machine Learning platform to provide a service to customers that shows enough potential and success that it received $50 million in funding. To me that is a data point supporting that the technical merit of Elixir isn’t just appealing to in-depth nerds like me but actually useful and beneficial in commercial applications. When your customer is an accounting firm you better provide a clear ROI, it’s almost like they look at the numbers.

Underjord is a 4 people team doing Elixir consulting and contract work. If you like the writing you should really try the code. See our services for more information.

Note: Or try the videos on the YouTube channel.