Working with RDMA in SLES 11 SP*
SUSE Linux Enterprise Server (SLES) 11.* have integrated RDMA support. In This post we’ll discuss how to manage and work with the inbox RDMA packages in those distributions.
Installing RDMA packages
One can install all the RDMA packages manually one by one and resolve the dependency by himself. However, yast2 provides us an easy way to install all the needed packages for working with RDMA and resolve the dependencies in other packages and packages conflicts automatically.
yast2, the SLES installer, allows installation of packages from the command line:
Uninstalling RDMA packages
Just like we used yast2 to install the packages, we'll use it to uninstall those packages, if they aren't needed anymore. The following command line will uninstall the RDMA packages:
And choose "Accept" in the GUI that will be opened.
Starting the RDMA services
Load the RDMA drivers using the following command line:
If one is using the InfiniBand transport and he doesn't have a managed switch in the subnet, he has to start the Subnet Manager (SM). Doing this in one of the machines in the subnet is enough, this can be done with the following command line:
By default, when installing the RDMA packages, the RDMA service will be loaded automatically when the operating system is loaded. If this doesn't happen, the following command line will do the trick:
Stopping the RDMA services
If the SM is running, then it must be stopped before unloading the drivers. Stop the SM using the following command line:
Unload the RDMA drivers using the following command line:
RDMA configuration file(s)
1. The openibd service loads the configuration file: /etc/infiniband/openib.conf. This file controls which modules will be loaded during the service startup and some attributes about the RDMA modules. The following parameters are supported:
|Parameter name||Description||Supported values|
|ONBOOT||Start RDMA modules/drivers upon machine boot||yes/no|
|RDMA_CM_LOAD||Load RDMA_CM module||yes/no|
|RDMA_UCM_LOAD||Load RDMA_UCM module||yes/no|
|UCM_LOAD||Load UCM module||yes/no|
|RENICE_IB_MAD||Increase ib_mad thread priority||yes/no|
|IPOIB_LOAD||Load IPoIB module||yes/no|
|SET_IPOIB_CM||Enable IPoIB Connected Mode||yes/no|
|IPOIBBOND_ENABLE||Enable the bonding driver on machine startup||yes/no|
|IPOIB_BONDS||Set bond interface names||Interface names, separated by comma (for example: "bond0,bond1")|
|<bond interface name>_IP(one line for every interface)||Set specific bond interface IP address||IP address|
|<bond interface name>_SLAVES(one line for every interface)||Set specific bond slaves||Interface names, separated by comma (for example: "ib0,ib1")|
|RDS_LOAD||Load RDS module||yes/no|
|SDP_LOAD||Load SDP module||yes/no|
|SRP_LOAD||Load SRP initiator module||yes/no|
|SRP_DAEMON_ENABLE||Enable SRP daemon||yes/no|
|SRPHA_ENABLE||Enable SRP High Availability daemon||yes/no|
|SRPT_LOAD||Load SRPT module||yes/no|
|SRP_TARGET_LOAD||Load SRP Target module||yes/no|
|ISER_LOAD||Load ISER initiator module||yes/no|
|MLX4_LOAD||Load MLX4 driver||yes/no|
|MLX4_EN_LOAD||Load MLX4_EN driver||yes/no|
|MTHCA_LOAD||Load MTHCA driver||yes/no|
|CXGB3_LOAD||Load CXGB3 driver||yes/no|
|QLGC_VNIC_LOAD||Load QLogic VNIC module||yes/no|
|IPATH_LOAD||Load IPATH driver||yes/no|
|EHCA_LOAD||Load EHCA driver||yes/no|
|QIB_LOAD||Load QLogic QIB driver||yes/no|
|NES_LOAD||Load iw_nes driver||yes/no|
2. RDMA needs to work with pinned memory, i.e. memory which cannot be swapped out by the kernel. By default, every process that is running as a non-root user is allowed to pin a low amount of memory (64KB). In order to work properly as a non-root user, it is highly recommended to increase the size of memory which can be locked. Edit the file /etc/security/limits.conf and add the following lines:
* soft memlock unlimited
* hard memlock unlimited
This will allow process that is running as any user to pin unlimited amount of memory. Changing this line will become effective for new login sessions.
After login again, executing the following command line will print how much memory (in KB) can be locked:
(the expected output is: "unlimited").
If one wishes to allow better control on this configuration: e.g. less memory to be pinned, or allow only specific user(s) to pin more memory - please refer to the Linux distribution manual.
Tell us what do you think.