Skip to content

ibv_query_device()

5.00 avg. rating (97% score) - 1 vote
int ibv_query_device(struct ibv_context *context,
                     struct ibv_device_attr *device_attr);

Description

ibv_query_device() returns the attributes of an RDMA device that is associated with a context.

Here is the full description of struct ibv_device_attr:

Name Description
fw_ver A NULL terminated string which describes the Firmware version of the RDMA device
node_guid GUID (in network byte order) associated with the RDMA device. This is the same GUID that was returned by ibv_get_device_guid()
sys_image_guid GUID (in network byte order) associated with this RDMA device and other devices which are part of a single system. For example: multiple Switch chips in the same Core switch
max_mr_size Size (in bytes) of the largest contiguous memory block that can be registered by this device
page_size_cap Memory page sizes supported by this device
vendor_id IEEE device's Vendor
vendor_part_id Devic's Part ID, as supplied by the Vendor
hw_ver Devic's Hardware version, as supplied by the Vendor
max_qp Maximum number of QPs, of UD/UC/RC transport types, supported by this device
max_qp_wr Maximum number of outstanding work requests on any Send or Receive Queue supported by this device
device_cap_flags Device's supported capabilities as bitwise ORed of the following enumerated values:

  • IBV_DEVICE_RESIZE_MAX_WR - Indication that the device supports modifying the maximum number of outstanding Work Requests of a QP
  • IBV_DEVICE_BAD_PKEY_CNTR - Indication that the device supports bad P_Key counting for each port
  • IBV_DEVICE_BAD_QKEY_CNTR - Indication that the device supports Q_Key violations counting for each port
  • IBV_DEVICE_RAW_MULTI - Indication that the device supports raw packet multicast
  • IBV_DEVICE_AUTO_PATH_MIG - Indication that the device supports automatic path migration
  • IBV_DEVICE_CHANGE_PHY_PORT - Indication that the device supports changing the primary port number of a QP when transitioning from SQD to SQD state
  • IBV_DEVICE_UD_AV_PORT_ENFORCE - Indication that the device supports AH port number enforcement
  • IBV_DEVICE_CURR_QP_STATE_MOD - Indication that the device supports the Current QP state modifier when calling ibv_modify_qp()
  • IBV_DEVICE_SHUTDOWN_PORT - Indication that the device supports shutdown port
  • IBV_DEVICE_INIT_TYPE - Indication that the device supports setting InitType and InitTypeReply
  • IBV_DEVICE_PORT_ACTIVE_EVENT - Indication that the device supports the IBV_EVENT_PORT_ACTIVE event generation
  • IBV_DEVICE_SYS_IMAGE_GUID - Indication that the device supports System Image GUID
  • IBV_DEVICE_RC_RNR_NAK_GEN - Indication that the device supports RNR-NAK generation for RC QPs
  • IBV_DEVICE_SRQ_RESIZE - Indication that the device supports modifying the maximum number of outstanding Work Requests in an SRQ
  • IBV_DEVICE_N_NOTIFY_CQ - Indication that the device supports Requesting Completion notification when N completions were added (and not only one) to a CQ
max_sge Maximum number of scatter/gather entries per Send or Receive Work Request, in a QP other than RD, supported by this device
max_sge_rd Maximum number of scatter/gather entries per Send or Receive Work Request, in an RD QP, supported by this device. If RD isn’t supported by this device, this value is zero
max_cq Maximum number of CQs supported by this device
max_cqe Maximum number of entries in each CQ supported by this device
max_mr Maximum number of MRs supported by this device
max_pd Maximum number of PDs supported by this device
max_qp_rd_atom Maximum number of RDMA Reads & atomic operations that can be outstanding per QP with this device as the target, if such are supported
max_ee_rd_atom Maximum number of RDMA Reads & atomic operations that can be outstanding per EEC with this device as the target, if such are supported
max_res_rd_atom Total maximum number of resources used for RDMA Reads & atomic operations by this device as the target, if such are supported
max_qp_init_rd_atom Maximum depth per QP for initiation of RDMA Read & atomic operations by this device, if such are supported
max_ee_init_rd_atom Maximum depth per EEC for initiation of RDMA Read & atomic operations by this device, if such are supported
atomic_cap Ability of the device to support atomic operations. The support level can be one of the following enumerated values:

  • IBV_ATOMIC_NONE - Atomic operations aren't supported at all
  • IBV_ATOMIC_HCA - Atomicity is guaranteed between QPs on this device only
  • IBV_ATOMIC_GLOB - Atomicity is guaranteed between this device and any other component, such as CPUs, IO devices and other RDMA devices
max_ee Maximum number of EE contexts supported by this device. If RD isn’t supported by this device, this value is zero
max_rdd Maximum number of RDDs supported by this device. If RD isn’t supported by this device, this value is zero
max_mw Maximum number of MWs supported by this device. If MWs aren't supported by this device, this value is zero
max_raw_ipv6_qp Maximum number of Raw IPv6 Datagram QPs supported by this device. If Raw IPv6 Datagram QPs aren't supported by this device, this value is zero
max_raw_ethy_qp Maximum number of Raw Ethertype Datagram QPs supported by this device. If Raw Ethertype Datagram QPs aren't supported by this device, this value is zero
max_mcast_grp Maximum number of multicast groups supported by this device. If unreliable multicast isn't supported by this device, this value is zero
max_mcast_qp_attach Maximum number of QPs per multicast group supported by this device. If unreliable multicast isn't supported by this device, this value is zero
max_total_mcast_qp_attach Maximum number of total QPs which can be attached to a multicast groups for this device. If unreliable multicast isn't supported by this device, this value is zero
max_ah Maximum number of AHs supported by this device
max_fmr Maximum number of FMRs supported by this device. If FMRs aren't supported by this device, this value is zero
max_map_per_fmr Maximum number of maps per FMR supported by this device. If FMRs aren't supported by this device, this value is zero
max_srq Maximum number of SRQs supported by this device. If SRQs aren't supported by this device, this value is zero
max_srq_wr Maximum number of outstanding work requests in an SRQ supported by this device
max_srq_sge Maximum number of scatter entries per Receive Work Request, in an SRQ, supported by this device
max_pkeys Maximum number of partitions supported by this device
local_ca_ack_delay Local CA ACK Delay. This value specifies the maximum expected time interval between the local device receiving a message and transmitting the associated ACK or NAK.
phys_port_cnt Number of physical ports on this device

The RDMA device attributes, returned by ibv_query_device(), are constant and won't be changed by the device or by the SM, thus a program can call to this verb and save those attributes for later use.

The maximum values returned by ibv_query_device() are the upper limits of supported resources of the device. However, it may not be possible to use these maximum values since the actual number of any resource that can be created may be limited by the machine configuration, the amount of host memory, user permissions, and the amount of resources already in use by other users/processes.

Parameters

Name Direction Description
context in RDMA device context, that was returned from ibv_open_device()
device_attr out RDMA device attributes

Return Values

Value Description
0 On success
errno On failure
ENOMEM Out of memory

Examples

Query a device attributes:

struct ibv_device_attr device_attr;
int rc;
 
rc = ibv_query_device(ctx, &device_attr);
if (rc) {
	fprintf(stderr, "Error, failed to query the device '%s' attributes\n",
		ibv_get_device_name(device_list[i]));
	return -1;
}

show_devices_attr.c

FAQs

ibv_query_device() indicated that I can create X elements from a resource, but I was able to create only Y (where Y is less than X), is this is o.k.?

Yes, this may happen since the values reported by ibv_query_device() are the upper limits supported by the device. If any other processes/modules created elements from this resource too, the available number this resource was decreased.

Can I know how many elements of a resource are available for me to create?

No, you can't. This value depends on a lot of factors.

Can I know how many elements of a resource were created by other processes/modules?

No, you can't. Currently, the RDMA stack doesn't support this feature.

Share Our Posts

Share this post through social bookmarks.

  • Delicious
  • Digg
  • Newsvine
  • RSS
  • StumbleUpon
  • Technorati

Comments

Tell us what do you think.

  1. Jay says: April 9, 2013

    Hi Dotan,

    I have a general question about adapters supporting RDMA over Ethernet. Does "Infiniband Spec" require vendors to support end to end flow control (based on
    credits) or is it optional? OR is it the
    verbs application developers duty to
    figure out if receiver is ready and has
    recv wqe posted before sending the data?

    • Dotan Barak says: May 3, 2013

      Hi Jay.

      Sorry it took some time to answer.

      Short answer:
      HCAs are required to implements end to end credits. It is a good practice for
      software to have flow control as well.

      Long answer:
      RC responders of HCAs shall support end to end credit generation (C9-158.2.1).
      However, TCAs are not required to (it is not a must).

      Some implementations weren't fully optimized for this case of running out of credits
      and therefore many applications use software flow control on top of this.

      The class of applications that use SW flow control is to ensure specific control message can always be sent to avoid cases of deadlock or forward progress, and hence needed to guarantee special room in remote machine.

      Last point, is when SRQ is being used, also flow control is not reflected. And hence even in HCA, if the app is using SRQ, then credits will not be reflected on the wire. (in such a case it is even hard to do SW flow control).

      I hope that this answer helps you
      Dotan

Add a Comment

Fill in the form and submit.

Time limit is exhausted. Please reload CAPTCHA.