Skip to content

Working with RDMA in Ubuntu

4.00 avg. rating (83% score) - 3 votes

Ubuntu has integrated RDMA support. In This post we'll discuss how to manage and work with the inbox RDMA packages in this distribution.

Installing RDMA packages

One can install all the RDMA packages manually one by one and resolve the dependency by himself. However, apt-get provides us an easy way to install all the needed packages for working with RDMA and resolve the dependencies in other packages automatically.

The following command line will install the needed packages that are needed to work with RDMA:

sudo apt-get -y --force-yes install libibcm1 libibverbs1 ibverbs-utils librdmacm1 rdmacm-utils libdapl2 ibsim-utils ibutils libcxgb3-1 libibmad5 libibumad3 libmlx4-1 libmthca1 libnes1 rds-tools infiniband-diags libibcommon1 mstflint opensm libopensm5 perftest srptools

Note: In different versions of the Ubuntu distributions, some packages may not be available. For example: in Ubuntu 14.10 the following packages are unavailable anymore: libdapl2, rds-tools and libibcommon1. Trying to install them will lead to an error.

Uninstalling RDMA packages

Just like we used apt-get install the packages group, we'll use it to uninstall those packages, if they aren't needed anymore. The following command line will uninstall the RDMA packages:

sudo apt-get -y --force-yes remove libibcm1 libibverbs1 ibverbs-utils librdmacm1 rdmacm-utils libdapl2 ibsim-utils ibutils libcxgb3-1 libibmad5 libibumad3 libmlx4-1 libmthca1 libnes1 rds-tools infiniband-diags libibcommon1 mstflint opensm libopensm5 perftest srptools

Starting the RDMA services

In Ubuntu there isn't any service file to load and unload the RDMA drivers; this needs to be done manually. Load the RDMA drivers using the following command line(s):

# RDMA stack modules
sudo modprobe rdma_cm
sudo modprobe ib_uverbs
sudo modprobe rdma_ucm
sudo modprobe ib_ucm
sudo modprobe ib_umad
sudo modprobe ib_ipoib
# RDMA devices low-level drivers
sudo modprobe mlx4_ib
sudo modprobe mlx4_en
sudo modprobe iw_cxgb3
sudo modprobe iw_cxgb4
sudo modprobe iw_nes
sudo modprobe iw_c2

If one is using the InfiniBand transport and he doesn't have a managed switch in the subnet, he has to start the Subnet Manager (SM). Doing this in one of the machines in the subnet is enough, this can be done with the following command line:

sudo opensm

Stopping the RDMA services

If the SM is running, then it must be stopped before unloading the drivers. Stop the SM using the following command line(s):

sudo pkill opensm

Unload the RDMA drivers using the following command line:

# RDMA devices low-level drivers
sudo modprobe -r iw_c2
sudo modprobe -r iw_nes
sudo modprobe -r iw_cxgb4
sudo modprobe -r iw_cxgb3
sudo modprobe -r mlx4_en
sudo modprobe -r mlx4_ib
# RDMA stack modules
sudo modprobe -r ib_ipoib
sudo modprobe -r ib_umad
sudo modprobe -r ib_ucm
sudo modprobe -r rdma_ucm
sudo modprobe -r ib_uverbs
sudo modprobe -r rdma_cm

RDMA configuration file(s)

1. If the file /etc/udev/rules.d/40-rdma.rules doesn't exist, create it with the following content:

# udev rules for RDMA
KERNEL=="umad*", NAME="infiniband/%k"
KERNEL=="issm*", NAME="infiniband/%k"
KERNEL=="ucm*", NAME="infiniband/%k", MODE="0666"
KERNEL=="uverbs*", NAME="infiniband/%k", MODE="0666"
KERNEL=="ucma", NAME="infiniband/%k", MODE="0666"
KERNEL=="rdma_cm", NAME="infiniband/%k", MODE="0666"

2. RDMA needs to work with pinned memory, i.e. memory which cannot be swapped out by the kernel. By default, every process that is running as a non-root user is allowed to pin a low amount of memory (64KB). In order to work properly as a non-root user, it is highly recommended to increase the size of memory which can be locked. Edit the file /etc/security/limits.conf and add the following lines:


* soft memlock unlimited
* hard memlock unlimited
root soft memlock unlimited
root hard memlock unlimited

This will allow process that is running as any user to pin unlimited amount of memory. Changing this line will become effective for new login sessions.

After login again, executing the following command line will print how much memory (in KB) can be locked:

ulimit -l

(the expected output is: "unlimited").

If one wishes to allow better control on this configuration: e.g. less memory to be pinned, or allow only specific user(s) to pin more memory - please refer to the Linux distribution manual.

Share Our Posts

Share this post through social bookmarks.

  • Delicious
  • Digg
  • Newsvine
  • RSS
  • StumbleUpon
  • Technorati

Comments

Tell us what do you think.

  1. Jack says: December 4, 2014

    Hello Dotan,
    I have a question, I want to do RDMA programming on my desktop. I only have one machine that has mellanox connect-3 installed. Is it possible that I do RDMA experiment on single node(for example, in the client/server file transfer test, can I have my client and server on the same physical node and communicate via local connectx-3 NIC?

    Thanks a lot for your time!

    All the best
    Jack

    • Jack says: December 4, 2014

      A follow up, I only need a computer that supports PCIe? Is there any other special requirement for connecting mellanox NIC?

      Thanks in advanced!

      • Dotan Barak says: December 4, 2014

        Hi Jack.

        Don't take me as Mellanox representative (although I'm working there :) )

        I can tell you from my experience, I've bought few years ago an InfiniBand HCA (form ebay),
        and installed it in my desktop. A *very* standard desktop, which not even branded.

        All I needed was PCIe slot (some minor manipulation in order to enable the PCIe slot),
        and that's it!

        For InfiniBand this was great, since you only need one HCA (with 2 ports),
        and a cable which is connected to both of them and that's it..

        So, bottom line: yes.

        A simple computer (even a desktop) with a PCIe slot will be enough.

        Thanks
        Dotan

  2. Bill Broadley says: February 21, 2015

    BTW, when I added the limits.conf lines it didn't change the ulimit -l for root. Even after a reboot. But adding:
    root soft memlock unlimited
    root hard memlock unlimited

    Did. Still trying to debug why RDMA + NFS will mount, but I get errors reading files.

    • Dotan Barak says: February 21, 2015

      Hi Bill.

      Thanks for your comment!
      I've updated the post accordingly.

      Good luck in your debug.
      If you think that there is another info that worth be added here, please feel free to give me a comment on this.

      Thanks!
      Dotan

    • June says: June 4, 2015

      Hi Bill.
      I'm am having the same issue, with Input/Output errors when reading files.
      Did you by any chance manage to get it sorted out?

Add a Comment

Fill in the form and submit.

Time limit is exhausted. Please reload CAPTCHA.