Skip to content

Working with RDMA using Mellanox OFED

Contents

5.00 avg. rating (97% score) - 1 vote

Mellanox OFED (MLNX-OFED) is a package that developed and released by Mellanox Technologies. It contains the latest software packages (both kernel modules and userspace code) to work with RDMA. It supports InfiniBand, Ethernet and RoCE transports and during the installation it updates the Firmware of Mellanox adapters.

This package is available for most major Linux distributions and CPU architectures.

Installing RDMA packages

Prerequisites

Make sure that there is at least 1 GB free space in your disk.

Download the package

Open the following URL in a web browser:
http://www.mellanox.com > Products > Software> InfiniBand
Drivers -> Linux SW/Drivers

Download the relevant package for your computer according to the Operating system and CPU architecture. There are two available formats for the package to download:

  1. ISO image
  2. tgz file

Both formats have the same content, and one should choose the best format for his needs.

Only the next pre-install/post-install sections depend on the chosen package format. The installation process is the same.

Note: it is highly recommended to compare the MD5sum of the downloaded package with the value that is published at Mellanox website (near the package download link); to verify that the downloaded package isn't corrupted.

1. ISO image

The ISO image can be mounted and used directly, without the need to extract it and use an extra space from the disk.

Pre-install

After downloading the ISO image, mount it and enter the mounted directory:

[root@localhost]# mkdir /mnt/MLNX_OFED
[root@localhost]# mount -o ro,loop MLNX_OFED_LINUX-<ver>-<OS label><CPU arch>.iso /mnt/MLNX_OFED
[root@localhost]# cd /mnt/MLNX_OFED

Post-install

Unmount the iso file and delete the created directory:

[root@localhost]# cd
[root@localhost]# umount /mnt/MLNX_OFED
[root@localhost]# rmdir /mnt/MLNX_OFED

2. tgz file

This format can be easier for use (no need for mounting and unmounting the image), but it needs extra space since it should be extracted before use.

Pre-install

After downloading the tgz file, extract it and enter the directory that the files exist in:

[root@localhost]# mkdir /tmp/MLNX_OFED
[root@localhost]# cd /tmp/MLNX_OFED
[root@localhost]# tar xzf MLNX_OFED_LINUX-<ver>-<OS label><CPU arch>.tgz
[root@localhost]# cd MLNX_OFED_LINUX-<ver>-<OS label><CPU arch>

Post-install

Delete the generated directory that holds the extracted files:

[root@localhost]# cd
[root@localhost]# rm -fr /tmp/MLNX_OFED

Installation

Now, everything is ready and installation of MLNX-OFED can be started.

The installation script will:

  • Verify that installed kernel is supported.
  • Uninstall all installed RDMA packages, which are part of the Linux distribution or part of any RDMA package.
  • Install the MLNX-OFED binary RPMs.
  • Detect the Mellanox RDMA adapters that exist in the machine and update their Firmware, if needed.
  • Fix local configuration files, if needed.

The same installation procedure is used in all Linux distributions.

[root@localhost]# ./mlnxofedinstall
This program will install the MLNX_OFED_LINUX package on your machine.
Note that all other Mellanox, OEM, OFED, or Distribution IB packages will be removed.
Do you want to continue?[y/N]:

This "yes/no" question should be answered with "y". Only pressing the "enter" key will use the default (N) and the installation will be terminated.

/bin/rpm --nosignature -e --allmatches --nodeps --noscripts ib-bonding libibverbs libipathverbs libibcm ibsim librdmacm libsdp opensm compat-dapl dapl mstflint srptools rds-tools ibutils infiniband-diags libibverbs libipathverbs libibcm ibsim librdmacm libsdp opensm compat-dapl dapl mstflint srptools rds-tools infiniband-diags opensm ibvexdmtools ibutils libamso-rdmav2 ofed-kmp-pae-1.5.4.1_3.0.76_0.11-0.11.5 ofed-1.5.4.1-0.11.5 ofed-kmp-default-1.5.4.1_3.0.76_0.11-0.11.5 ofed-doc-1.5.4.1-0.11.5 ofed-kmp-trace-1.5.4.1_3.0.76_0.11-0.11.5 libmthca-rdmav2 libcxgb3-rdmav2 libmlx4-rdmav2 libibcommon1 ofed ofed-doc libibumad3 libibmad5
Starting MLNX_OFED_LINUX-2.0-3.0.0 installation ...
Installing mlnx-ofa_kernel RPM
Preparing... ##################################################
mlnx-ofa_kernel ##################################################
Installing mlnx-ofa_kernel-kmp-default RPM
Preparing... ##################################################
mlnx-ofa_kernel-kmp-default ##################################################
Installing mlnx-ofa_kernel-kmp-pae RPM
Preparing... ##################################################
mlnx-ofa_kernel-kmp-pae ##################################################
Installing mlnx-ofa_kernel-kmp-trace RPM
Preparing... ##################################################
mlnx-ofa_kernel-kmp-trace ##################################################
Installing mlnx-ofa_kernel-kmp-xen RPM
Preparing... ##################################################
mlnx-ofa_kernel-kmp-xen ##################################################
Installing mlnx-ofa_kernel-devel RPM
Preparing... ##################################################
mlnx-ofa_kernel-devel ##################################################
Installing kernel-mft-mlnx-kmp-default RPM
Preparing... ##################################################
kernel-mft-mlnx-kmp-default ##################################################
Installing kernel-mft-mlnx-kmp-pae RPM
Preparing... ##################################################
kernel-mft-mlnx-kmp-pae ##################################################
Installing kernel-mft-mlnx-kmp-trace RPM
Preparing... ##################################################
kernel-mft-mlnx-kmp-trace ##################################################
Installing kernel-mft-mlnx-kmp-xen RPM
Preparing... ##################################################
kernel-mft-mlnx-kmp-xen ##################################################
Installing user level RPMs:
Preparing... ##################################################
ofed-scripts ##################################################
Preparing... ##################################################
libibverbs ##################################################
Preparing... ##################################################
libibverbs-devel ##################################################
Preparing... ##################################################
libibverbs-devel-static ##################################################
Preparing... ##################################################
libibverbs-utils ##################################################
Preparing... ##################################################
libmlx4 ##################################################
Preparing... ##################################################
libmlx4-devel ##################################################
Preparing... ##################################################
libmlx5 ##################################################
Preparing... ##################################################
libmlx5-devel ##################################################
Preparing... ##################################################
libcxgb3 ##################################################
Preparing... ##################################################
libcxgb3-devel ##################################################
Preparing... ##################################################
libcxgb4 ##################################################
Preparing... ##################################################
libcxgb4-devel ##################################################
Preparing... ##################################################
libnes ##################################################
Preparing... ##################################################
libnes-devel-static ##################################################
Preparing... ##################################################
libibcm ##################################################
Preparing... ##################################################
libibcm-devel ##################################################
Preparing... ##################################################
libibumad ##################################################
Preparing... ##################################################
libibumad-devel ##################################################
Preparing... ##################################################
libibumad-static ##################################################
Preparing... ##################################################
libibmad ##################################################
Preparing... ##################################################
libibmad-devel ##################################################
Preparing... ##################################################
libibmad-static ##################################################
Preparing... ##################################################
ibsim ##################################################
Preparing... ##################################################
ibacm ##################################################
Preparing... ##################################################
librdmacm ##################################################
Preparing... ##################################################
librdmacm-utils ##################################################
Preparing... ##################################################
librdmacm-devel ##################################################
Preparing... ##################################################
opensm-libs ##################################################
Preparing... ##################################################
opensm ##################################################
opensmd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
Preparing... ##################################################
opensm-devel ##################################################
Preparing... ##################################################
opensm-static ##################################################
Preparing... ##################################################
dapl ##################################################
Preparing... ##################################################
dapl-devel ##################################################
Preparing... ##################################################
dapl-devel-static ##################################################
Preparing... ##################################################
dapl-utils ##################################################
Preparing... ##################################################
perftest ##################################################
Preparing... ##################################################
mstflint ##################################################
Preparing... ##################################################
mft ##################################################
Preparing... ##################################################
srptools ##################################################
Preparing... ##################################################
rds-tools ##################################################
Preparing... ##################################################
rds-devel ##################################################
Preparing... ##################################################
ibutils2 ##################################################
Preparing... ##################################################
ibutils ##################################################
Preparing... ##################################################
cc_mgr ##################################################
Preparing... ##################################################
dump_pr ##################################################
Preparing... ##################################################
ar_mgr ##################################################
Preparing... ##################################################
ibdump ##################################################
Preparing... ##################################################
infiniband-diags ##################################################
Preparing... ##################################################
infiniband-diags-compat ##################################################
Preparing... ##################################################
qperf ##################################################
Preparing... ##################################################
mlnxofed-docs ##################################################
Installation finished successfully.
Configuring /etc/security/limits.conf.

Different versions of MLNX-OFED or different Linux distributions may have different RPM names and versions.

Reboot is usually required to complete the installation.

Notes

Failure in uninstalling automatically an RPM

If there is a problem to uninstall any RPM, the installer will print an error message and the user will need to remove this RPM manually.

Adding new kernel support

If the installed kernel doesn't match any binary RPM which is provided within MLNX_OFED, one can build the MLNX_OFED package for this kernel manually. The script mlnx_add_kernel_support.sh is the tool for doing so.

This operation requires the kernel sources be available in the machine and assumes that adding this support can be done by compilation of the sources and doesn't need any changes/adjustments in the code.

The following command line will generate a tgz file for the installed kernel:

[root@localhost]# ./mlnx_add_kernel_support.sh -t /tmp/mlnx_ofed_build --make-tgz --mlnx_ofed /tmp/MLNX_OFED
Note: This program will create MLNX_OFED_LINUX TGZ for sles11sp3 under /tmp directory.
All Mellanox, OEM, OFED, or Distribution IB packages will be removed.
Do you want to continue?[y/N]:y
See log file /tmp/mlnx_ofed_iso.30869.log
Building OFED RPMs. Please wait...
Removing OFED RPMs...
Created /tmp/MLNX_OFED_LINUX-2.0-3.0.0-sles11sp3-i686.tgz

Similar command can be used for generating the ISO image for this kernel (using the parameter --make-iso).

The generated package can now be installed using the instructions that are written in the "Installation" section above in local machine or in any other machine with the same Linux distribution and kernel.

Uninstalling RDMA packages

This package comes with an uninstall script that will clean everything. The following command line will uninstall the MLNX_OFED packages:

[root@localhost]# /usr/sbin/ofed_uninstall.sh
This program will uninstall all MLNX_OFED_LINUX-2.0-3.0.0 packages on your machine.
Do you want to continue?[y/N]:

This "yes/no" question should be answered with "y". Only pressing the "enter" key will use the default (N) and the uninstallation will be terminated.

rpm -e --allmatches --nodeps kernel-mft-mlnx-kmp-pae-3.1.0_3.0.76_0.11-8.sles11sp3 libmlx4-1.0.4mlnx2-OFED.2.0.2.6.7.8.g5ddd0ea libcxgb4-rdmav2-1.1.1-0.7.32 libnes-1.1.3mlnx1-OFED.2.0.2.6.3.1.gae2c5c5 libibmad-devel-1.3.9.MLNX20130522.1e79ec6-0.1 opensm-4.0.5.MLNX20130808.c2b40b1-0.1 mstflint-3.0-0.1.g50289c4 dump_pr-1.0-0.8.g7b25a83 mlnx-ofa_kernel-kmp-pae-2.0_3.0.76_0.11-OFED.2.0.2.6.9.6.g3a2d7bf.sles11sp3 libmlx5-devel-1.0.0-OFED.2.0.3.1.0.3.g00bb19b libnes-rdmav2-1.1.3-0.11.1 libibcm-devel-1.0.5mlnx1-OFED.2.0.0.0.9.20130210.1800.gc8011c5 ibacm-1.0.8mlnx1-OFED.2.0.2.6.2.22.g8706262 dapl-2.0.38-1 rds-tools-2.0.6mlnx-1 infiniband-diags-1.6.1.MLNX20130822.dfac5dd-0.1 mlnx-ofa_kernel-kmp-default-2.0_3.0.76_0.11-OFED.2.0.2.6.9.6.g3a2d7bf.sles11sp3 mlnx-ofa_kernel-devel-2.0-OFED.2.0.2.6.9.6.g3a2d7bf.sles11sp3 kernel-mft-mlnx-kmp-xen-3.1.0_3.0.76_0.11-8.sles11sp3 libibverbs-devel-static-1.1.6mlnx2-OFED.2.0.2.6.8.10.g4c4c3f1 libmlx5-1.0.0-OFED.2.0.3.1.0.3.g00bb19b libcxgb4-1.3.0mlnx1-OFED.2.0.2.6.2.28.g3287317 libibcm-1.0.5mlnx1-OFED.2.0.0.0.9.20130210.1800.gc8011c5 libibumad-static-1.3.8.MLNX20130522.da65ddf-0.1 ibsim-0.5mlnx-0.5.g8bd64ec librdmacm-devel-1.0.17mlnx2-OFED.2.0.2.6.2.22.g13cb0e8 opensm-static-4.0.5.MLNX20130808.c2b40b1-0.1 dapl-utils-2.0.38-1 srptools-0.0.4mlnx3-OFED.2.0.2.6.7.11.ge863cb7 ibutils-1.5.7.1-0.5.g16c2d50 ibdump-2.0.0-5 mlnxofed-docs-2.0-3.0.0 mlnx-ofa_kernel-2.0-OFED.2.0.2.6.9.6.g3a2d7bf.sles11sp3 mlnx-ofa_kernel-kmp-xen-2.0_3.0.76_0.11-OFED.2.0.2.6.9.6.g3a2d7bf.sles11sp3 kernel-mft-mlnx-kmp-trace-3.1.0_3.0.76_0.11-8.sles11sp3 libibverbs-devel-1.1.6mlnx2-OFED.2.0.2.6.8.10.g4c4c3f1 libmlx4-devel-1.0.4mlnx2-OFED.2.0.2.6.7.8.g5ddd0ea libcxgb3-devel-1.3.1mlnx1-OFED.2.0.2.6.2.28.g58e86c2 libnes-devel-static-1.1.3mlnx1-OFED.2.0.2.6.3.1.gae2c5c5 libibumad-devel-1.3.8.MLNX20130522.da65ddf-0.1 libibmad-static-1.3.9.MLNX20130522.1e79ec6-0.1 librdmacm-utils-1.0.17mlnx2-OFED.2.0.2.6.2.22.g13cb0e8 opensm-devel-4.0.5.MLNX20130808.c2b40b1-0.1 dapl-devel-static-2.0.38-1 ibutils2-2.1.1-0.41.g7a6902e ar_mgr-1.0-0.11.g22fff4a qperf-0.4.9-1 mlnx-ofa_kernel-kmp-trace-2.0_3.0.76_0.11-OFED.2.0.2.6.9.6.g3a2d7bf.sles11sp3 libibverbs-1.1.6mlnx2-OFED.2.0.2.6.8.10.g4c4c3f1 libcxgb3-1.3.1mlnx1-OFED.2.0.2.6.2.28.g58e86c2 libibumad-1.3.8.MLNX20130522.da65ddf-0.1 librdmacm-1.0.17mlnx2-OFED.2.0.2.6.2.22.g13cb0e8 dapl-devel-2.0.38-1 infiniband-diags-compat-1.6.1.MLNX20130822.dfac5dd-0.1 kernel-mft-mlnx-kmp-default-3.1.0_3.0.76_0.11-8.sles11sp3 libibverbs-utils-1.1.6mlnx2-OFED.2.0.2.6.8.10.g4c4c3f1 libcxgb4-devel-1.3.0mlnx1-OFED.2.0.2.6.2.28.g3287317 libibmad-1.3.9.MLNX20130522.1e79ec6-0.1 opensm-libs-4.0.5.MLNX20130808.c2b40b1-0.1 perftest-2.0-0.58.gcf63bcd cc_mgr-1.0-0.10.g39d12f5 kernel-mft-mlnx-kmp-pae-3.1.0_3.0.76_0.11-8.sles11sp3 ofed-scripts-2.0-OFED.2.0.3.0.0 rds-tools-2.0.6mlnx-1 infiniband-diags-1.6.1.MLNX20130822.dfac5dd-0.1 kernel-mft-mlnx-kmp-xen-3.1.0_3.0.76_0.11-8.sles11sp3 kernel-mft-mlnx-kmp-trace-3.1.0_3.0.76_0.11-8.sles11sp3 mft-3.1.0-8 rds-devel-2.0.6mlnx-1 infiniband-diags-compat-1.6.1.MLNX20130822.dfac5dd-0.1 kernel-mft-mlnx-kmp-default-3.1.0_3.0.76_0.11-8.sles11sp3
Uninstall finished successfully

Starting the RDMA services

Load the RDMA drivers using the following command line:

[root@localhost]# /etc/init.d/openibd start

If one is using the InfiniBand transport and he doesn't have a managed switch in the subnet, he has to start the Subnet Manager (SM). Doing this in one of the machines in the subnet is enough, this can be done with the following command line:

[root@localhost]# /etc/init.d/opensmd start

By default, when installing the RDMA packages, the RDMA service will be loaded automatically when the operating system is loaded. If this doesn't happen, one should fix manually.

Stopping the RDMA services

If the SM is running, then it must be stopped before unloading the drivers. Stop the SM using the following command line:

[root@localhost]# /etc/init.d/opensmd stop

Unload the RDMA drivers using the following command line:

[root@localhost]# /etc/init.d/openibd stop

RDMA configuration file(s)

1. The openibd service loads the configuration file: /etc/infiniband/openib.conf. This file controls which modules will be loaded during the service startup and some attributes about the RDMA modules. The following parameters are supported:

Parameter name Description Supported values
ONBOOT Start RDMA modules/drivers upon machine boot yes/no
NODE_DESC Node description prefix command Command string. For example: $(hostname -s)
NODE_DESC_TIME_BEFORE_UPDATE Time, in seconds, to wait before updating node description Non-negative integer numbers
RUN_SYSCTL Run sysctl performance tuning on network attributes script yes/no
RUN_AFFINITY_TUNER Run affinity tuner script yes/no
RDMA_CM_LOAD Load RDMA_CM module yes/no
RDMA_UCM_LOAD Load RDMA_UCM module yes/no
UCM_LOAD Load UCM module yes/no
RENICE_IB_MAD Increase ib_mad thread priority yes/no
IPOIB_LOAD Load IPoIB module yes/no
SET_IPOIB_CHANNELS Set rx_channels/tx_channels to 1 to disable IPoIB RSS yes/no
SET_IPOIB_CM Enable IPoIB Connected Mode yes/no
E_IPOIB_LOAD Load E_IPOIB module yes/no
RDS_LOAD Load RDS module yes/no
SRP_LOAD Load SRP initiator module yes/no
SRP_DAEMON_ENABLE Enable SRP daemon yes/no
SRPHA_ENABLE Enable SRP High Availability daemon yes/no
SRP_TARGET_LOAD Load SRP Target module yes/no
MLX5_LOAD Load MLX5 driver yes/no
MLX4_LOAD Load MLX4 driver yes/no
MLX4_EN_LOAD Load MLX4_EN driver yes/no
MLX4_VNIC_LOAD Load MLX4_VNIC driver yes/no
MTHCA_LOAD Load MTHCA driver yes/no
CXGB3_LOAD Load CXGB3 driver yes/no
NES_LOAD Load iw_nes driver yes/no

2. RDMA needs to work with pinned memory, i.e. memory which cannot be swapped out by the kernel. By default, every process that is running as a non-root user is allowed to pin a low amount of memory (64KB). In order to work properly as a non-root user, it is highly recommended to increase the size of memory which can be locked. Edit the file /etc/security/limits.conf and add the following lines, if they weren't added by the installation script:

* soft memlock unlimited
* hard memlock unlimited

This will allow process that is running as any user to pin unlimited amount of memory. Changing this line will become effective for new login sessions.

After login again, executing the following command line will print how much memory (in KB) can be locked:

ulimit -l

(the expected output is: "unlimited").

  1. If one wishes to allow better control on this configuration: e.g. less memory to be pinned, or allow only specific user(s) to pin more memory - please refer to the Linux distribution manual.

More information

Detailed information on how to work with MLNX-OFED can be found in the User Manual of MLNX-OFED, in the following URL:

http://www.mellanox.com > Products > Software> InfiniBand
Drivers -> Linux SW/Drivers

Share Our Posts

Share this post through social bookmarks.

  • Delicious
  • Digg
  • Newsvine
  • RSS
  • StumbleUpon
  • Technorati

Comments

Tell us what do you think.

  1. David R. says: April 4, 2017

    Hi Dotan,

    I'm just curious what the difference is (if there is any) between the Mellanox OFED distributions and the packages that ship within the Linux kernel. Is one better or newer than the other? Or is it just a matter of how one wants to install/run? When working with the newest Linux kernels, do you recommend one over the other? Any advice would be greatly appreciated!

    Thanks,
    David

    • Dotan Barak says: April 14, 2017

      Hi.

      Let's start with a word of ethics: I'm an employee of Mellanox Technologies.

      The difference is that the MLNX-OFED contain features/commits that doesn't exist in the upstream kernel;
      * Changes that weren't submitted (yet)
      * Changes that can't be submitted (for many reasons)

      It depends on the version, but in most cases the MLNX-OFED will contain the features that exists in the upstream kernel code.

      Bottom line, if you would ask me what to use (even) in new kernels: MLNX-OFED.

      Thanks
      Dotan

      • David R. says: April 14, 2017

        Ah, I see. That makes sense. Thank you very much for the advice!

Add a Comment

This comment will be moderated; answer may be provided within 14 days.

Time limit is exhausted. Please reload CAPTCHA.