Carlos Fenollosa — Blog

Thoughts on science and tips for researchers who use computers

The Elixir of concurrency

May 23, 2016 — Carlos Fenollosa

Elixir is a fairly young language that was born when José and a few Rails developers tried to create a modern language optimized for concurrent, distributed, lightweight processes

They wanted a modern Ruby-like syntax with a well-tested process manager, the Erlang VM. The result is Elixir, defined as a dynamic, functional language designed for building scalable and maintainable applications, a correct but vague affirmation which doesn't do justice to its power and elegancy.

I recently compared the move to Elixir from Python as a similar leapfrog to moving to Python from Java. It feels like something new, modern, powerful, with killer features that you don't want to renounce to.

In Python I found a REPL, list comprehensions, a super clean syntax and decorators. Elixir brings lightweight supervised processes, pattern matching, a fully functional programming language, pipes and a terrific build tool: mix

If you've never written functional code, the jump is significant. I took a Scala course a couple years ago and I've needed almost two full weeks to write production code in Elixir. The language is young, Stack Overflow is of no help —no kidding, that is a big deal—, and there are few libraries in Github.

A small community also comes with some upsides: people are more motivated and willing to help, centralized tools like forums and IRC channels are still manageable, and you may even suggest changes to the language for upcoming versions.

What is Elixir for?

I had a middle school teacher who said that you can't define something by stating what is't not. However, in programming, mentioning use cases which are not suitable for the language is a good way to start.

Elixir is probably not the first choice for single core software: math calculus, CPU-intensive apps or desktop applications. Since it's very high level, systems programming is also out of the picture.

Elixir is great for web applications, standalone or using the Phoenix framework —Elixir's Rails—. It really shines for building highly scalable, fault-tolerant network applications, like chats, telecommunications or generic web services.

Why is that? Thanks to the Erlang VM, processes are really tiny, each one is garbage collected with a low latency, they communicate by sending location-independent messages over the network using the VMs (you can run result = Machine2.Module.function(params) on Machine1), and spawning and managing these processes is effortless thanks to some of its abstractions.

Finally, Elixir's basic modules also shine: Plug and Router for managing HTTP requests, Ecto for relational databases and ETS and Mnesia for distributed in-memory databases.

Many recommend Elixir if only for Phoenix, but I found that for most backend applications it is enough to use Plug and Router. Phoenix is impressive but I believe it's a mistake to jump right into it without trying the base modules first, so my recommendation for beginners is to hold on Phoenix until you really need it.

Elixir's novelty, the pipe operator, is a fantastic approach to working with state in a functional manner. Instead of running readlines(fopen(user_input(), "r")).uppercase().split(), try the more readable user_input |> fopen("r") |> readlines |> uppercase |> split.

It is a language which was clearly designed to stand on the shoulders of giants, while providing modern capabilities for developers.

Elixir's abstractions

To store centralized <key, value>-like data, instead of a Singleton, Elixir's provides an Agent. It keeps state in memory and many processes can access and modify it without concurrency issues.

The language can spawn processes much like threads, using spawn_link, but you probably don't want to do that. You'd rather use a Task, which is basically async/await, or a Gen(eric)Server, a very cool abstraction that receives requests from other processes, spawns helper mini-servers and processes the results in parallel, for free.

All tasks can be controlled using the Supervisor, which holds other abstractions as its "children" and automatically restarts them when they crash.

Finally, your code is contained inside a single project which can manage different apps, with modules that hold functions. No packages, no classes, no objects. Modules, functions, structs and basic data types.

Dependency management is straightforward thanks to mix; builds and testing are handled by mix too. As opposed to other multi-tools like gradle, this one is really fast.

Is that too much to process? I felt that at first, too. Give it some time and your brain will eventually think in terms of Supervisors which manage GenServers which spawn Agents and Tasks when needed.

Let it crash

Elixir's mantra is to let processes crash. I found it shocking and counter-intuitive, but with some explanation it makes a lot of sense.

Neither developers want their code to crash nor Elixir promotes writing bad code. However, let's agree that there are many reasons besides bad programming which can make a software crash. If we have a server which runs stuff and at some point we have, say, 100 connections every second, one might crash eventually because of a bug in any component, hardware issues, a cosmic ray, or Murphy's law.

The question is: in the event of an unfortunate, unavoidable crash, how will your system react?

  1. Bring everything down?
  2. Try to capture the error and recover?
  3. Kill the crashed process and launch another one in its place?

For example, C uses approach 1. Most modern languages with Exceptions like Java and Python use 2. Elixir uses 3. This is not suitable for all environments, but it is perfect for those use cases which fit Elixir: concurrent network processes.

With Elixir, a single failure never brings the system down. What's more, it automatically restarts the crashed process, so the client can instantly retry and, unless there is a reproducible bug in your code, the fresh process will finish without an issue.

The bottom line is: a single client may be unlucky and crash at some point, but the rest of the system will never notice.

How to start?

Let's get our hands dirty. After reading many sites, watching hours of video and following a dozen tutorials, here are the resources I found the most valuable. I'd suggest following this order.

Getting started

  1. Madrid Elixir Meetup 2016-03. If you understand Spanish, this is the best intro to Elixir. Otherwise, watch All aboard the Elixir Express! which is a bit outdated but very comprehensive.
  2. Official "Getting Started" guide. It's the best and the most current. Follow it from start to finish, including the advanced chapters.
  3. Elixir School. A nice complement to the official guide. Most things are very similar, but the different approach on OTP will help you understand it better.
  4. Understanding Elixir's GenServer and Elixir's supervisors, a conceptual understanding are two short reads with yet another explanation of OTP features.
  5. Elixir Cheat Sheet. The best one out there

First projects

  1. vim-elixir-ide. Elixir support for vim, not the best plugin but suitable for beginners.
  2. Elixir examples. The Elixir guide covers all these, but it's handy to have common idioms on a single page: "string to list", "concatenate list", "optional function parameters", etc.
  3. Portal Game by José Valim. A complement to the sample project on the official guide.
  4. Elixir Koans and Exercism are mini exercises that you can use to improve your Elixir agility. On the same line, Elixir Golf proposes weekly puzzles to solve.
  5. Learning Elixir. Joseph Kain has a ton of content with mini projects and examples you can follow. Top quality.
  6. Excasts and Elixir sips have short screencasts that you can check out for reference
  7. ElixirConf videos contain very interesting talks which may be overwhelming for beginners, but are worth a look later on.
  8. Install Elixir and Phoenix on OSX. If you want to use Phoenix on OSX, you may need this help
  9. Phoenix Official Guide. Phoenix isn't necessary for simple web services, you can use Plug. But for large projects you'll need a framework. Nothing like the official guide.

Getting help

  1. Awesome Elixir. A list of Elixir resources, where I found many of these.
  2. Elixir Tip and Elixir Status regularly link to Elixir-related articles and videos, and Plataformatec Elixir posts is where the language authors share news and tips.
  3. If you have questions about code, try the Elixir forum first, the IRC channel or Slack. The developers would like to transition all help requests out of the Mailing list, which you can use for language-related discussions.
  4. /r/elixir if you're into Reddit

Closing thoughts

I think that's all for the moment. I hope this post can help some beginners to get their hands on the language and start writing production code as soon as possible.

For anyone who wants to know what's all the Elixir fuss about, it's difficult to explain, especially for somebody like me who has been programming in imperative languages all his life.

When I recommended Elixir to a friend, he replied, "A highly concurrent, functional language using the Erlang VM? Don't you have something more exotic?". That's right. Elixir is exotic and use-case specific.

Unlike Python, which is my favorite imperative language and ecosystem, I can't recommend Elixir for everyone. Not everybody can spare a couple weeks to get started. Many libraries for common use cases are missing: there is nothing equivalent to Numpy or Matplotlib, and modern applications are built on top of dozens of libs, not everyone has the time or will to write library code. Fortunately, at Paradoxa I am my own boss and I make the tech decisions :)

For hackers or tinkerers it's definitely worth a look, it "won't change your perspective" like Lisp, but it will make you see that writing concurrent code doesn't need to be difficult, and that better tooling is definitely possible.

I bet Elixir will be the foundation of most devops stacks in a few years, when developers realize that the future's bottleneck won't be the CPU, but rather the number of concurrent processes and connections your backend can manage. With Elixir you only need to boot another machine in your network and let the exotic Erlang VM handle the rest.

Tags: programming, learning

Comments? Tweet