Skip to content

Working with RDMA in RedHat/CentOS 5.*

0.00 avg. rating (0% score) - 0 votes

RedHat and CentOS 5.* have integrated RDMA support. In This post we'll discuss how to manage and work with the inbox RDMA packages in those distributions.

Installing RDMA packages

One can install all the RDMA packages manually one by one and resolve the dependency by himself. However, yum provides us an easy way to install all the needed packages for working with RDMA and resolve the dependencies in other packages automatically.

yum allows installation of multiple packages according to a specific area. The group "OpenFabrics Enterprise Distribution" has all the relevant packages for RDMA support, i.e. InfiniBand, RoCE and iWARP. The following command will show which packages are part of the group "OpenFabrics Enterprise Distribution":

[root@localhost]# yum groupinfo "OpenFabrics Enterprise Distribution"
Loaded plugins: fastestmirror
Setting up Group Process
Loading mirror speeds from cached hostfile
* base: centos.spd.co.il
* epel: mirror.vorboss.net
* extras: centos.spd.co.il
* updates: centos.spd.co.ilGroup: OpenFabrics Enterprise Distribution
Description: Components used for high performance networking and clustering, such as Infiniband and RDMA.
Default Packages:

compat-dapl
compat-dapl-devel
compat-dapl-utils
dapl
dapl-devel
dapl-utils
ibutils
infiniband-diags
libcxgb3
libcxgb4
libibcm
libibverbs
libibverbs-utils
libmlx4
libmthca
libnes
librdmacm
librdmacm-utils
libsdp
mpitests-openmpi
ofed-docs
openib
openmpi
perftest
qperf
rds-tools

Optional Packages:

compat-dapl-static
dapl-static
ibsim
mpitests-mvapich
mpitests-mvapich2
mstflint
mvapich
mvapich2
opensm
qlvnictools
srptools
tvflash

As one can see, there are several classifications for packages in this group: "default" and "optional". In RedHat/CentOS 5.* distributions (at least, for now), by default only the "default" packages will be installed. The following command line will install the needed packages that are needed to work with RDMA:

[root@localhost]# yum -y groupinstall "OpenFabrics Enterprise Distribution"

The "optional" packages need to be installed manually. The following command line will install them:

[root@localhost]# yum -y install compat-dapl-static dapl-static ibsim mpitests-mvapich mpitests-mvapich2 mstflint mvapich mvapich2
opensm qlvnictools srptools tvflash

Uninstalling RDMA packages

Just like we used yum to install the package group, we'll use it to uninstall those packages, if they aren't needed anymore. The following command line will uninstall the RDMA packages:

[root@localhost]# yum -y groupremove "OpenFabrics Enterprise Distribution"

Starting the RDMA services

Load the RDMA drivers using the following command line:

[root@localhost]# /etc/init.d/openibd start

If one is using the InfiniBand transport and he doesn't have a managed switch in the subnet, he has to start the Subnet Manager (SM). Doing this in one of the machines in the subnet is enough, this can be done with the following command line:

[root@localhost]# /etc/init.d/opensmd start

If one wishes to start the RDMA service automatically when the operating system is loaded, the following command line will do the trick:

[root@localhost]# chkconfig --level 235 openibd on

Stopping the RDMA services

If the SM is running, then it must be stopped before unloading the drivers. Stop the SM using the following command line:

[root@localhost]# /etc/init.d/opensmd stop

Unload the RDMA drivers using the following command line:

[root@localhost]# /etc/init.d/openibd stop

RDMA configuration file(s)

1. The openibd service loads the configuration file: /etc/ofed/openib.conf. This file controls which modules will be loaded during the service startup and some attributes about the RDMA modules. The following parameters are supported:

Parameter name Description Supported values
IPOIB_LOAD Load IPoIB module yes/no
RDS_LOAD Load RDS module yes/no
SDP_LOAD Load SDP module yes/no
SRP_LOAD Load SRP initiator module yes/no
ISER_LOAD Load ISER initiator module yes/no
FIXUP_MTRR_REGS Modify the system mtrr registers yes/no

2. RDMA needs to work with pinned memory, i.e. memory which cannot be swapped out by the kernel. By default, every process that is running as a non-root user is allowed to pin a low amount of memory (64KB). In order to work properly as a non-root user, it is highly recommended to increase the size of memory which can be locked. Edit the file /etc/security/limits.conf and add the following lines:


* soft memlock unlimited
* hard memlock unlimited

This will allow process that is running as any user to pin unlimited amount of memory. Changing this line will become effective for new login sessions.

After login again, executing the following command line will print how much memory (in KB) can be locked:

ulimit -l

(the expected output is: "unlimited").

If one wishes to allow better control on this configuration: e.g. less memory to be pinned, or allow only specific user(s) to pin more memory - please refer to the Linux distribution manual.

Share Our Posts

Share this post through social bookmarks.

  • Delicious
  • Digg
  • Newsvine
  • RSS
  • StumbleUpon
  • Technorati

Comments

Tell us what do you think.

There are no comments on this entry.

Add a Comment

Fill in the form and submit.

Time limit is exhausted. Please reload CAPTCHA.