Video - Building a web crawler

Had some great fun building on this webcrawler on the stream. Plan to continue with this in coming livestreams, so watch this to get up to speed on what we are doing.

We grab an HTTP client, and HTML parser and then we go hog wild spawning Task processes and fetching web pages.

The project repo I built up during the stream is on Github as lawik/caterpillar and the video is below.

Also available on the YouTube channel.

Enjoy the things I do? Sign up for the newsletter, I write more about Elixir, my projects and developer experiences there.

Want more voice? I talk about Elixir on Beam Radio and more about programming in general on Regular Programming.

If you want to discuss it with me, you can find me via email as or on Twitter where I’m @lawik.