Skip to content

Working with RDMA in SLES 11 SP*

5.00 avg. rating (97% score) - 1 vote

SUSE Linux Enterprise Server (SLES) 11.* have integrated RDMA support. In This post we’ll discuss how to manage and work with the inbox RDMA packages in those distributions.

Installing RDMA packages

One can install all the RDMA packages manually one by one and resolve the dependency by himself. However, yast2 provides us an easy way to install all the needed packages for working with RDMA and resolve the dependencies in other packages and packages conflicts automatically.

yast2, the SLES installer, allows installation of packages from the command line:

[root@localhost]# yast2 --install libibverbs libipathverbs compat-dapl dapl libamso-rdmav2 libcxgb3-rdmav2 libcxgb4 -rdmav2 libmlx4-rdmav2 libmthca-rdmav2 libnes-rdmav2 librdmacm ib-bonding ibsim ibutils ibvexdmtools infiniband-diags libibcm libibcommon1 libibmad5 libibumad3 libsdp ofed ofed-doc ofed-kmp-default ofed-kmp-pae ofed-kmp-trace opensm mstflint rds-tools ibutils libibcommon libibcommon1 srptools

Uninstalling RDMA packages

Just like we used yast2 to install the packages, we'll use it to uninstall those packages, if they aren't needed anymore. The following command line will uninstall the RDMA packages:

[root@localhost]# yast2 --remove libibverbs libipathverbs compat-dapl dapl libamso-rdmav2 libcxgb3-rdmav2 libcxgb4 -rdmav2 libmlx4-rdmav2 libmthca-rdmav2 libnes-rdmav2 librdmacm ib-bonding ibsim ibutils ibvexdmtools infiniband-diags libibcm libibcommon1 libibmad5 libibumad3 libsdp ofed ofed-doc ofed-kmp-default ofed-kmp-pae ofed-kmp-trace opensm mstflint rds-tools ibutils libibcommon libibcommon1 srptools

And choose "Accept" in the GUI that will be opened.

Starting the RDMA services

Load the RDMA drivers using the following command line:

[root@localhost]# /etc/init.d/openibd start

If one is using the InfiniBand transport and he doesn't have a managed switch in the subnet, he has to start the Subnet Manager (SM). Doing this in one of the machines in the subnet is enough, this can be done with the following command line:

[root@localhost]# /etc/init.d/opensmd start

By default, when installing the RDMA packages, the RDMA service will be loaded automatically when the operating system is loaded. If this doesn't happen, the following command line will do the trick:

[root@localhost]# chkconfig --level 235 openibd on

Stopping the RDMA services

If the SM is running, then it must be stopped before unloading the drivers. Stop the SM using the following command line:

[root@localhost]# /etc/init.d/opensmd stop

Unload the RDMA drivers using the following command line:

[root@localhost]# /etc/init.d/openibd stop

RDMA configuration file(s)

1. The openibd service loads the configuration file: /etc/infiniband/openib.conf. This file controls which modules will be loaded during the service startup and some attributes about the RDMA modules. The following parameters are supported:

Parameter name Description Supported values
ONBOOT Start RDMA modules/drivers upon machine boot yes/no
RDMA_CM_LOAD Load RDMA_CM module yes/no
RDMA_UCM_LOAD Load RDMA_UCM module yes/no
UCM_LOAD Load UCM module yes/no
RENICE_IB_MAD Increase ib_mad thread priority yes/no
IPOIB_LOAD Load IPoIB module yes/no
SET_IPOIB_CM Enable IPoIB Connected Mode yes/no
IPOIBBOND_ENABLE Enable the bonding driver on machine startup yes/no
IPOIB_BONDS Set bond interface names Interface names, separated by comma (for example: "bond0,bond1")
<bond interface name>_IP(one line for every interface) Set specific bond interface IP address IP address
<bond interface name>_SLAVES(one line for every interface) Set specific bond slaves Interface names, separated by comma (for example: "ib0,ib1")
RDS_LOAD Load RDS module yes/no
SDP_LOAD Load SDP module yes/no
SRP_LOAD Load SRP initiator module yes/no
SRP_DAEMON_ENABLE Enable SRP daemon yes/no
SRPHA_ENABLE Enable SRP High Availability daemon yes/no
SRPT_LOAD Load SRPT module yes/no
SRP_TARGET_LOAD Load SRP Target module yes/no
ISER_LOAD Load ISER initiator module yes/no
MLX4_LOAD Load MLX4 driver yes/no
MLX4_EN_LOAD Load MLX4_EN driver yes/no
MTHCA_LOAD Load MTHCA driver yes/no
CXGB3_LOAD Load CXGB3 driver yes/no
QLGC_VNIC_LOAD Load QLogic VNIC module yes/no
IPATH_LOAD Load IPATH driver yes/no
EHCA_LOAD Load EHCA driver yes/no
QIB_LOAD Load QLogic QIB driver yes/no
NES_LOAD Load iw_nes driver yes/no

2. RDMA needs to work with pinned memory, i.e. memory which cannot be swapped out by the kernel. By default, every process that is running as a non-root user is allowed to pin a low amount of memory (64KB). In order to work properly as a non-root user, it is highly recommended to increase the size of memory which can be locked. Edit the file /etc/security/limits.conf and add the following lines:


* soft memlock unlimited
* hard memlock unlimited

This will allow process that is running as any user to pin unlimited amount of memory. Changing this line will become effective for new login sessions.

After login again, executing the following command line will print how much memory (in KB) can be locked:

ulimit -l

(the expected output is: "unlimited").

If one wishes to allow better control on this configuration: e.g. less memory to be pinned, or allow only specific user(s) to pin more memory - please refer to the Linux distribution manual.

Share Our Posts

Share this post through social bookmarks.

  • Delicious
  • Digg
  • Newsvine
  • RSS
  • StumbleUpon
  • Technorati

Comments

Tell us what do you think.

  1. Andrew M. Sheppard says: June 14, 2015

    Oh man, this is brilliant!

    My objective is to get RDMA/{iSER or SRP}+Xen Project 4.x hypervisor online on my dev metal.

    I picked a pair of Mellanox MHGA28-XTC's (InfiniHost III Ex, dual port 20Gb's) fr fleabay this week. Now knowing mthca has been dropped fr Mellanox's later versions of OFED, & OpenFabric OFED /long/ dropped supported modern distros (eg: past RHEL 6.3 & its 2.6.x kernel) on this HCA, it's my hypothesis I should be able to get SLES 11 SP3 x86_64 up.

    I know I'll be using SLES's "inbox" version of OFED 1.5.4.1-0.11.4, so iSER seems to be a 'no go' (per Mellanox's notes on this OFED). SRP is my only option, it seems. I'm using SuSEStudio to build the iso. Currently, I'm focused on getting all the RDMA components into it before tackling inclusion of Xen.

    A few of questions:
    1.) libibcommon doesn't seem to be available, even in the SP3 {+SKD} Updates. libibcommon1 is. Should I be concerned?
    2.) ofed-kmp-pae doesn't seem available.
    2.b.) Should I include ofed-kmp-bigsmp instead?
    3.) The MHGA28-XTC HCAs are advertised as "[...] the cards include internal Subnet Management Agent (SMA) and General Service Agents [...]" (per Mellanox's InfiniHost III Ex Product Brief). Do I _really_ need to include OpenSM?

    I can share my build on susestudio.com if you'd like.

    Thanks _so much_ for giving a RDMA newbie such valuable resource!

    • Dotan Barak says: June 21, 2015

      Hi.

      Let me answer:
      1) No, you shouldn't be concerned, this package (i.e. libibcommon) was dropped starting a specific point.
      2) I don't have enough information on this; I *think* that the difference is the target kernel that this packet supports.
      3) What do you mean? if you really need to include the OpenSM?
      The SM manages the subnet.
      The SMA that exists in the card description is an agent that the SM communicate with.

      You must have an SM in the subnet, in a switch or in the host.
      So, you think that the answer is "yes, you should have it".

      Sure, glad to help
      :)
      Dotan

      • Andrew M. Sheppard says: July 15, 2015

        Hello Dotan -

        Thanks for the insights. It was my mistaken impression the HCA's SMA was some sort of onboard SM, therefore negating the need for OpenSM.

        I'm 'chomping @ the bit' to get this up... though I'm still waiting for my CX4 cable to come in so I'd like to reserve the right to report back (eg: my second question) . I'm currently reading up on your post on IPoIB & partitioning. Again, brilliant resource!

        Thanks!

      • Dotan Barak says: July 15, 2015

        Thanks for the feedback
        :)

        Dotan

Add a Comment

Fill in the form and submit.

Time limit is exhausted. Please reload CAPTCHA.