rdma_dist

rdma_dist probably represents my most technically demanding, complete, standalone project. I got the idea for it at the SC high-performance computing conference where I saw example after example of people and projects struggling to make good use of OpenMPI for distributed computing, but putting up with it because of its native RDMA support for high-bandwidth, low-latency networking. My own experience had found that Erlang largely solved the complication of distributed computing with its novel OTP framework, but the VM on which Erlang runs did not support RDMA.

I studied other RDMA implementations and other Erlang VM network drivers and began work on the low-level glue between the two subsystems. Along the way, I wrote unit tests to define and check the behavior of the driver. The test suite made it easy to simulate edge conditions, validate features like flow control, and have confidence that the completed driver was not losing any data.

And while that might have been good enough, I wanted to know if my new rdma_dist driver actually worked: did it perform better than the traditional network driver? To that end, I wrote a distributed benchmark app in the Elixir language that also runs on the Erlang VM, and ran it on a real HPC cluster. The data showed an increase in RPCs of about 50% with my driver.

This project is important to me because I identified a need, set an ambitious goal, did the work the right way, and was rewarded with a true improvement over the state of the art. I did it in my spare time from concept to blog post in 45 days.

Incidentally, this is not the only low-level network driver I’ve worked on. I also developed a Solaris driver for wpa_supplicant to support enterprise Wi-Fi authentication, which I wrote about in this blog post.