Skip to content

ibv_create_ah()

5.00 avg. rating (97% score) - 1 vote
struct ibv_ah *ibv_create_ah(struct ibv_pd *pd, struct ibv_ah_attr *attr);

Description

ibv_create_ah() creates an Address Handle (AH) associated with a Protection Domain.

This AH will later be used when a Send Request (SR) will be posted to a Unreliable Datagram QP.

The struct ibv_ah_attr describes the requested attributes of the newly created AH.

struct ibv_ah_attr {
	struct ibv_global_route	grh;
	uint16_t		dlid;
	uint8_t			sl;
	uint8_t			src_path_bits;
	uint8_t			static_rate;
	uint8_t			is_global;
	uint8_t			port_num;
};

Here is the full description of struct ibv_ah_attr:

grh Attributes of the Global Routing Headers (GRH), as described in the table below. This is useful when sending packets to another subnet or to a multicast group
dlid If the destination is on the same subnet, the LID of the port to which the subnet delivers the packets to. If the destination is on another subnet, the LID of the Router
sl 4 bits. The Service Level to be used
src_path_bits The used Source Path Bits. This is useful when LMC is used in the port, i.e. each port covers a range of LIDs. The packets are being sent with the port's base LID, bitwised ORed with the value of the source path bits. The value 0 indicates the port's base LID is used
static_rate A value which limits the rate of packets that being sent to the subnet. This can be useful if the rate of the packet origin is higher than the rate of the destination
is_global If this value contains any value other than zero, then GRH information exists in this AH, thus the field grh if valid
port_num The local physical port that the packets will be sent from

struct ibv_global_route describes the values to be used in the GRH of the packets that will be sent when using this AH.

struct ibv_global_route {
	union ibv_gid		dgid;
	uint32_t		flow_label;
	uint8_t			sgid_index;
	uint8_t			hop_limit;
	uint8_t			traffic_class;
};

Here is the full description of struct ibv_global_route:

dgid The GID that is used to identify the destination port of the packets
flow_label 20 bits. If this value is set to a non-zero value, it gives a hint for switches and routers with multiple outbound paths that these sequences of packets must be delivered in order, those staying on the same path, so that they won't be reordered.
sgid_index An index in the port's GID table that will be used to identify the originator of the packet
hop_limit The number of hops (i.e. the number of routers) that the packet is permitted to take before being discarded. This ensures that a packet will not loop indefinitely between routers if a routing loop occur. Each router decrement by one this value at the packet and when this value reaches 0, this packet is discarded. Setting the value to 0 or 1 will ensure that the packet won't leave the local subnet.
traffic_class Using this value, the originator of the packets specifies the required delivery priority for handling them by the routers

Parameters

Name Direction Description
pd in Protection Domain that was returned from ibv_alloc_pd()
attr in Requested attributes for the Address Handle

Return Values

Value Description
AH pointer to the newly allocated Address Handle
NULL On failure, errno indicates the failure reason:

EINVAL Invalid pd or invalid value provided in attr
ENOMEM Not enough resources to complete this operation

Examples

Create an Address Handle and destroy it (this example assumed that the variables dlid, sl and port are declared and initialized with valid values):

struct ibv_pd *pd;
struct ibv_ah *ah;
struct ibv_ah_attr ah_attr;
 
memset(&ah_attr, 0, sizeof(ah_attr));
 
ah_attr.is_global     = 0;
ah_attr.dlid          = dlid;
ah_attr.sl            = sl;
ah_attr.src_path_bits = 0;
ah_attr.port_num      = port;
 
ah = ibv_create_ah(pd, &ah_attr);
if (!ah) {
	fprintf(stderr, "Error, ibv_create_ah() failed\n");
	return -1;
}
 
if (ibv_destroy_ah(ah)) {
	fprintf(stderr, "Error, ibv_destroy_ah() failed\n");
	return -1;
}

FAQs

Why is an AH good for anyway?

AH is used when calling ibv_post_send() to an UD QP. This object describes the path from the requester side to the responder side.

Can I use one AH with different QPs?

Yes. This can be done as long as all of those QPs and the AH are associated with the same PD.

How can I get the needed information for the AH when calling ibv_create_ah()?

There are several ways to obtain this information:

  • Perform path query to the Subnet Administrator (SA)
  • Out of band connection to the remote node, for example: using socket
  • Using well-known values, for example: this can be done in a static subnet (which all of the addresses are predefined) or using multicast groups

Share Our Posts

Share this post through social bookmarks.

  • Delicious
  • Digg
  • Newsvine
  • RSS
  • StumbleUpon
  • Technorati

Comments

Tell us what do you think.

  1. Omar Cardona says: February 6, 2013

    Can you describe how to leverage VLANs over RoCE? Assuming the VLAN ID is preset in the DLID somehow and the priority is explicitly provided via the SL on AH create. Any caveats with PFC?

    • Dotan Barak says: February 8, 2013

      Hi Omar.

      First, I plan to cover all of the RDMA verbs since I don't think that there is another point of knowledge at this level (detailed description and code samples).

      In the future, I plan to cover RoCE as well and explain how to port an application over IB to an application over RoCE.

  2. ocardona says: February 6, 2013

    Hi this is a great resource. Can you describe how this maps to VLAN IDs and priorities in RoCE? Assuming the VLAN ID was previously obtained somehow into the DLID and the SL is selected at AH create time. How does one discover/ensure the verb consumer selected VLAN and priorities are valid?

    • Dotan Barak says: February 8, 2013

      Hi Omar.

      Thanks for the complements.
      :)

      As I wrote in my previous reply: first, I plan to cover all of the RDMA verbs since I don't think that there is another point of knowledge at this level (detailed description and code samples).

      In the future, I plan to cover RoCE as well and explain how to port an application over IB to an application over RoCE.

  3. Alexey says: June 4, 2015

    Could you please clarify how to fill ah_attr.sl and may be some other fields in case of 3D Torus cluster topology.
    ah_attr.sl = sl
    Can sl value be used from ibv_query_port()?

    • Dotan Barak says: June 5, 2015

      Hi.

      The ibv_query_port() will tell you the number of supported VLs by the port and the SL that you need to use to get to the SM/SA.

      Did you try to send a path query to the SA (Subnet Administrator)?

      I think that you would like to read the following article:
      http://www.disca.upv.es/jflich/papers/icpp03.pdf

      Thanks
      Dotan

      • Alexey says: June 5, 2015

        Dotan,
        Thank you very much for response and the reference.
        No, I did not try to send a path query to the SA. Currently I do not have access to 3D Torus clusters to make experiments. I implemented the module that uses Infiniband UD transport where I just set ah_attr.sl = 0 and customer asked the support for 3D Torus cluster. I investigated mvapich2.1 sources and did not find any special changes according sl for UD. For RC they have a special function like get_path_record().

Add a Comment

Fill in the form and submit.

Time limit is exhausted. Please reload CAPTCHA.