- 1 RDMA Programming Interfaces libraries for InfiniBand, RoCE and iWARP
- 2 Performance
- 3 HW Userspace drivers libraries and tools
- 4 Subnet Management (InfiniBand only)
- 5 Diagnostics (InfiniBand only)
- 6 Communication Management
- 7 Userlevel tools for ULPs
- 8 Misc
- 9 Summary
Here is a detailed list of the packages that handles RDMA in most popular Linux distributions, separated into categories. There names may be slightly different in different distributions.
Those packages may contain executables, libraries and headers.
Most libraries are separated into several packages:
- Package <library name> - The library files (shared library)
- <library name>-utils - Utilities that come with the library
- It is highly recommended to install this package to any library that is installed. This may come handy when there is a need to debug a problem.
- <library name>-devel - The headers of the library
- One should install it only if he plan to develop programs that uses this library
- <library name>-static - Static library
- One should install it only if he plans to link program that he develops as static
RDMA Programming Interfaces libraries for InfiniBand, RoCE and iWARP
Userspace library which implements the (hardware agnostic) verbs abstraction for using RDMA in software. Most RDMA programs are developed over this library, so it is quite mandatory to install it.
|libibverbs||libibverbs is a library that allows userspace processes to use RDMA "verbs" as described in the InfiniBand Architecture Specification and the RDMA Protocol Verbs Specification. This includes direct hardware access from userspace to InfiniBand/iWARP adapters (kernel bypass) for fast path operations.|
|libibverbs-utils||Useful libibverbs example programs such as ibv_devinfo, which displays information about RDMA devices.|
|libibverbs-devel||Header files for the libibverbs library.|
This is another library which implements common API for RDMA protocols (it implements the DAT API). However, not a big number of applications/libraries are using it (AFAIK, it is used mostly by specific MPI implementation). One should install it if he needs it.
|dapl||libdat and libdapl provide a userspace implementation of the DAT 2.0 API and is built to natively support InfiniBand/iWARP network technology.|
|dapl-utils||Useful test suites to validate the dapl library API's and operation.|
|compat-dapl||The DAT programming API provides a means of utilizing high performance network technologies, such as InfiniBand and iWARP, without needing to write your program to use those technologies directly. This package contains the libraries that implement version 1.2 of the DAT API. The current (and recommended version for any new code) is 2.0. These 1.2 libraries are provided solely for backward compatibility.|
|dapl-devel||Header files for libdat and libdapl library.|
Benchmark tools for RDMA (InfiniBand, RoCE and iWARP) subnets.
|perftest||Perftest is a collection of simple test programs designed to utilize RDMA communications and provide performance numbers over those RDMA connections. It does not work on normal TCP/IP networks, only on RDMA networks.|
|qperf||Measure socket and RDMA performance.|
HW Userspace drivers libraries and tools
Those are the hardware-specific userspace libraries (and tools) for various RDMA devices. One should install the libraries for the hardware that he has in his computers.
|libmlx5||libmlx5 provides a device-specific userspace driver for Mellanox Connect-IB HCAs for use with the libibverbs library.|
|libmlx4||libmlx4 provides a device-specific userspace driver for Mellanox ConnectX HCAs for use with the libibverbs library.|
|libmthca||libmthca provides a device-specific userspace driver for Mellanox HCAs (MT23108 InfiniHost and MT25208 InfiniHost III Ex) for use with the libibverbs library.|
|mstflint||This package contains a burning tool for Mellanox manufactured HCA cards. It also provides access to the relevant source code.|
|libcxgb4||Userspace hardware driver for use with the libibverbs InfiniBand/iWARP verbs library. This driver enables Chelsio T4 based iWARP capable Ethernet devices.|
|libcxgb3||Userspace hardware driver for use with the libibverbs InfiniBand/iWARP verbs library. This driver enables Chelsio iWARP capable ethernet devices.|
|libocrdma||libocrdma provides a device-specific userspace driver for Emulex One Command RoCE Adapters for use with the libibverbs library.|
|libehca||libehca provides a device-specific userspace driver for IBM HCAs for use with the libibverbs library.|
Intel (formally: NetEffect/QLogic)
|libnes||Userspace hardware driver for use with the libibverbs InfiniBand/iWARP verbs library. This driver enables NetEffect iWARP capable ethernet devices.|
|libipathverbs||libipathverbs provides a device-specific userspace driver for QLogic HCAs for use with the libibverbs library.|
|infinipath-psm||The PSM Messaging API, or PSM API, is QLogic's low-level user-level communications interface for the Truescale family of products. PSM users are enabled with mechanisms necessary to implement higher level communications interfaces in parallel environments.|
This is a software emulation of the RoCE standard, which can emulate RoCE over any other NIC.
|librxe||librxe provides a device-specific userspace driver RDMA over Converging Enhanced Ethernet for use with the libibverbs library.|
Subnet Management (InfiniBand only)
Utilities to configure the subnet, you may find them handy if you don't have a Subnet Manager in your switch (i.e. managed switch).
|opensm||OpenSM is the OpenIB project's Subnet Manager for Infiniband networks. The subnet manager is run as a system daemon on one of the machines in the infiniband fabric to manage the fabric's routing state. This package also contains various tools for diagnosing and testing Infiniband networks that can be used from any machine and do not need to be run on a machine running the opensm daemon.|
|ibsim||ibsim provides simulation of infiniband fabric for using with OFA OpenSM, diagnostic and management tools.|
|libibmad||libibmad provides low layer IB functions for use by the IB diagnostic and management programs. These include MAD, SA, SMP, and other basic IB functions.|
|libibumad||libibumad provides the user MAD library functions which sit on top of the user MAD modules in the kernel. These are used by the IB diagnostic and management tools, including OpenSM.|
|libibmad-devel||Development files for the libibmad library.|
|libibumad-devel||Development files for the libibumad library.|
|opensm-libs||Shared libraries for Infiniband user space access.|
|libibcommon||libibcommon provides common utility functions for the OFA diagnostic and management tools.|
Diagnostics (InfiniBand only)
Diagnostics tools for InfiniBand subnet. Using them will help you detect problems in the subnet.
|infiniband-diags||This package provides IB diagnostic programs and scripts needed to diagnose an IB subnet.|
|ibutils||ibutils provides IB network and path diagnostics.|
|ibutils-libs||Shared libraries used by the Mellanox Infiniband diagnostic utilities|
Libraries for establishing and tearing down connections between programs.
Communication Management for InfiniBand
Communications management library over InfiniBand addresses.
|libibcm||libibcm provides a userspace library that handles the majority of the low level work required to open an RDMA connection between two machines.|
|libibcm-devel||Development files for the libibcm library.|
Communication Management for RDMA (InfiniBand, RoCE and iWARP)
Communications management library for RDMA (InfiniBand, RoCE and iWARP) with a socket semantics.
|librdmacm||librdmacm provides a userspace RDMA Communication Management API.|
|librdmacm-utils||Example test programs for the librdmacm library.|
|librdmacm-devel||Development files for the librdmacm library.|
|ibacm||The ib_acm daemon helps reduce the load of managing path record lookups on large InfiniBand fabrics by providing a user space implementation of what is functionally similar to an ARP cache. The use of ib_acm, when properly configured, can reduce the SA packet load of a large IB cluster from O(n^2) to O(n). The ib_acm daemon is started and normally runs in the background, user applications need not know about this daemon as long as their app uses librdmacm to handle connection bring up/tear down. The librdmacm library knows how to talk directly to the ib_acm daemon to retrieve data.|
Userlevel tools for ULPs
User level tools for ULPs (Upper Layer Protocols). One should install them if he uses the ULP that they belong to.
|srptools||In conjunction with the kernel ib_srp driver, srptools allows you to discover and use SCSI devices via the SCSI RDMA Protocol over InfiniBand.|
|rds-tools||Various tools for support of the RDS (Reliable Datagram Socket) API. RDS is specific to InfiniBand and iWARP networks and does not work on non-RDMA.|
|rdma||User space initialization scripts for the kernel InfiniBand/iWARP drivers.|
In this post I tried to cover all the relevant packages for RDMA. If I missed a package that you think that should be added here, please update me.
Tell us what do you think.