Skip to content

ibv_modify_qp()

5.00 avg. rating (97% score) - 1 vote
int ibv_modify_qp(struct ibv_qp *qp, struct ibv_qp_attr *attr,
                  int attr_mask);

Description

ibv_modify_qp() modifies the attributes of a Queue Pair.

The changed attributes describe the send and receive attributes of the QP. In UC and RC QPs this means connecting the QP with a remote QP.

In Infiniband, one should perform path query to the Subnet Administrator (SA), in order to decide which attributes the QP should be configured with or as the best solution, use the Communication Manager (CM) or the Generic RDMA CM agent (CMA) to connect the QPs. However, there are some applications which prefer to connect the QPs on their own and decide which QP attributes to use by exchanging the data over sockets.

In RoCE, one should configure the GRH in the QP attributes, for connected QPs, or the GRH in the Address Handle for UD QP.

In iWARP, one should connect the QPs only using the Generic RDMA CM agent (CMA).

struct ibv_qp_attr describes the attributes of the Queue Pair.

struct ibv_qp_attr {
	enum ibv_qp_state	qp_state;
	enum ibv_qp_state	cur_qp_state;
	enum ibv_mtu		path_mtu;
	enum ibv_mig_state	path_mig_state;
	uint32_t		qkey;
	uint32_t		rq_psn;
	uint32_t		sq_psn;
	uint32_t		dest_qp_num;
	int			qp_access_flags;
	struct ibv_qp_cap	cap;
	struct ibv_ah_attr	ah_attr;
	struct ibv_ah_attr	alt_ah_attr;
	uint16_t		pkey_index;
	uint16_t		alt_pkey_index;
	uint8_t			en_sqd_async_notify;
	uint8_t			sq_draining;
	uint8_t			max_rd_atomic;
	uint8_t			max_dest_rd_atomic;
	uint8_t			min_rnr_timer;
	uint8_t			port_num;
	uint8_t			timeout;
	uint8_t			retry_cnt;
	uint8_t			rnr_retry;
	uint8_t			alt_port_num;
	uint8_t			alt_timeout;
};

Here is the full description of struct ibv_qp_attr:

qp_state The next QP state. It can be one of the following enumerated values:

  • IBV_QPS_RESET - Reset state
  • IBV_QPS_INIT - Initialized state
  • IBV_QPS_RTR - Ready To Receive state
  • IBV_QPS_RTS - Ready To Send state
  • IBV_QPS_SQD - Send Queue Drain state
  • IBV_QPS_SQE - Send Queue Error state
  • IBV_QPS_ERR - Error state
cur_qp_state Assume that this is the current QP state. This is useful if it is known to the application that the QP state is different from the assumed state by the low-level driver. It can be one of the enumerated values as qp_state
path_mtu The path MTU (Maximum Transfer Unit) i.e. the maximum payload size of a packet that can be transferred in the path. It can be one of the following enumerated values:

  • IBV_MTU_256 - 256 bytes
  • IBV_MTU_512 - 512 bytes
  • IBV_MTU_1024 - 1024 bytes
  • IBV_MTU_2048 - 2048 bytes
  • IBV_MTU_4096 - 4096 bytes

For UC and RC QPs, when needed, the RDMA device will automatically fragment the messages to packet of this size.

path_mig_state The state of the QP's path migration state machine if supported by the device (IBV_DEVICE_AUTO_PATH_MIG is set in dev_cap.device_cap_flags). It can be one of the following enumerated values:

  • IBV_MIG_MIGRATED - The state machine of path migration is Migrated, i.e. initial state of migration was done
  • IBV_MIG_REARM - The state machine of path migration is Rearm, i.e. attempt to try to coordinate the remote RC QP to move both local and remote QPs to Armed state
  • IBV_MIG_ARMED - The state machine of path migration is Armed, i.e. both local and remote QPs are ready to perform a path migration
qkey The Q_Key that incoming messages are check against and possibly used as the outgoing Q_Key (if the MSB of the q_key in the Send Request is set). Relevant only for UD QPs
rq_psn A 24 bits value of the Packet Sequence Number of the received packets for RC and UC QPs
sq_psn A 24 bits value of the Packet Sequence Number of the sent packets for any QP
dest_qp_num A 24 bits value of the remote QP number of RC and UC QPs; when sending data, packets will be sent to this QP number and when receiving data, packets will be accepted only from this QP number
qp_access_flags Allowed access flags of the remote operations for incoming packets of RC and UC QPs. It is either 0 or the bitwise OR of one or more of the following flags:

  • IBV_ACCESS_REMOTE_WRITE - Allow incoming RDMA Writes on this QP
  • IBV_ACCESS_REMOTE_READ - Allow incoming RDMA Reads on this QP
  • IBV_ACCESS_REMOTE_ATOMIC - Allow incoming Atomic operations on this QP
cap New attributes to the number of Wore Requests in the Queue Pair, as described in the table below, if resize QP is supported by the device (IBV_DEVICE_RESIZE_MAX_WR is set in dev_cap.device_cap_flags). Upon a successful Queue Pair resize, this structure will hold the actual Queue Pair attributes
ah_attr Address vector of the primary path which describes the path information to the remote QP as described in the table below
alt_ah_attr Address vector of the alternate path which describes the path information to the remote QP as described in the table below. Can be used only if supported by the device (IBV_DEVICE_AUTO_PATH_MIG is set in dev_cap.device_cap_flags)
pkey_index Primary P_Key index. The value of the entry in the P_Key table that outgoing packets from this QP will be sent with and incoming packets to this QP will be verified within the Primary path
alt_pkey_index Alternate P_Key index. The value of the entry in the P_Key table that outgoing packets from this QP will be sent with and incoming packets to this QP will be verified within the Alternate path
en_sqd_async_notify If non-zero, generate the affiliated asynchronous event IBV_EVENT_SQ_DRAINED when the QP state becomes SQD.drained, i.e. the Send Queue is drained
sq_draining If set, indication that Send Queue draining is in progress. Not relevant for ibv_modify_qp()
max_rd_atomic The number of RDMA Reads & atomic operations outstanding at any time that can be handled by this QP as an initiator. Relevant only for RC QPs
max_dest_rd_atomic The number of RDMA Reads & atomic operations outstanding at any time that can be handled by this QP as a destination. Relevant only for RC QPs
min_rnr_timer Minimum RNR NAK Timer Field Value. When an incoming message to this QP should consume a Work Request from the Receive Queue, but not Work Request is outstanding on that Queue, the QP will send an RNR NAK packet to the initiator. It does not affect RNR NAKs sent for other reasons. The value can be one of the following numeric values since those values aren’t enumerated:

  • 0 - 655.36 milliseconds delay
  • 1 - 0.01 milliseconds delay
  • 2 - 0.02 milliseconds delay
  • 3 - 0.03 milliseconds delay
  • 4 - 0.04 milliseconds delay
  • 5 - 0.06 milliseconds delay
  • 6 - 0.08 milliseconds delay
  • 7 - 0.12 milliseconds delay
  • 8 - 0.16 milliseconds delay
  • 9 - 0.24 milliseconds delay
  • 10 - 0.32 milliseconds delay
  • 11 - 0.48 milliseconds delay
  • 12 - 0.64 milliseconds delay
  • 13 - 0.96 milliseconds delay
  • 14 - 1.28 milliseconds delay
  • 15 - 1.92 milliseconds delay
  • 16 - 2.56 milliseconds delay
  • 17 - 3.84 milliseconds delay
  • 18 - 5.12 milliseconds delay
  • 19 - 7.68 milliseconds delay
  • 20 - 10.24 milliseconds delay
  • 21 - 15.36 milliseconds delay
  • 22 - 20.48 milliseconds delay
  • 23 - 30.72 milliseconds delay
  • 24 - 40.96 milliseconds delay
  • 25 - 61.44 milliseconds delay
  • 26 - 81.92 milliseconds delay
  • 27 - 122.88 milliseconds delay
  • 28 - 163.84 milliseconds delay
  • 29 - 245.76 milliseconds delay
  • 30 - 327.68 milliseconds delay
  • 31 - 491.52 milliseconds delay

Relevant only for RC QPs

port_num Primary physical port number associated with this QP
timeout The minimum timeout that a QP waits for ACK/NACK from remote QP before retransmitting the packet. The value zero is special value which means wait an infinite time for the ACK/NACK (useful for debugging). For any other value of timeout, the time calculation is: 4.096*2^{timeout} usec. For your convenient, here is the summary of each value and its timeout:

  • 0 - infinite
  • 1 - 8.192 usec (0.000008 sec)
  • 2 - 16.384 usec (0.000016 sec)
  • 3 - 32.768 usec (0.000032 sec)
  • 4 - 65.536 usec (0.000065 sec)
  • 5 - 131.072 usec (0.000131 sec)
  • 6 - 262.144 usec (0.000262 sec)
  • 7 - 524.288 usec (0.000524 sec)
  • 8 - 1048.576 usec (0.00104 sec)
  • 9 - 2097.152 usec (0.00209 sec)
  • 10 - 4194.304 usec (0.00419 sec)
  • 11 - 8388.608 usec (0.00838 sec)
  • 12 - 16777.22 usec (0.01677 sec)
  • 13 - 33554.43 usec (0.0335 sec)
  • 14 - 67108.86 usec (0.0671 sec)
  • 15 - 134217.7 usec (0.134 sec)
  • 16 - 268435.5 usec (0.268 sec)
  • 17 - 536870.9 usec (0.536 sec)
  • 18 - 1073742 usec (1.07 sec)
  • 19 - 2147484 usec (2.14 sec)
  • 20 - 4294967 usec (4.29 sec)
  • 21 - 8589935 usec (8.58 sec)
  • 22 - 17179869 usec (17.1 sec)
  • 23 - 34359738 usec (34.3 sec)
  • 24 - 68719477 usec (68.7 sec)
  • 25 - 137000000 usec (137 sec)
  • 26 - 275000000 usec (275 sec)
  • 27 - 550000000 usec (550 sec)
  • 28 - 1100000000 usec (1100 sec)
  • 29 - 2200000000 usec (2200 sec)
  • 30 - 4400000000 usec (4400 sec)
  • 31 - 8800000000 usec (8800 sec)

Relevant only to RC QPs

retry_cnt A 3 bits value of the total number of times that the QP will try to resend the packets before reporting an error because the remote side doesn't answer in the primary path
rnr_retry A 3 bits value of the total number of times that the QP will try to resend the packets when an RNR NACK was sent by the remote QP before reporting an error. The value 7 is special and specify to retry infinite times in case of RNR
alt_port_num Alternate physical port number associated with this QP
alt_timeout The total number of times that the QP will try to resend the packets before reporting an error because the remote side doesn't answer in the alternate path

Few caveats about some of the specific values in the QP attributes:

  • During development, it is a good practice to set attr.retry_cnt and attr.rnr_retry to 0. If there are races in the code, it is better to detect them during the development stage
  • The value 0 in attr.timeout means waiting infinite time for the ACK or NACK. This means that if any packet in a message is being lost and no ACK or NACK is being sent, no retry will ever occur and the QP will just stop sending data
  • The value 7 in attr.rnr_retry means to retry infinite number of times to send the message when RNR Nack is being sent by remote side

struct ibv_qp_cap describes the size of the Queue Pair (for both Send and Receive Queues).

struct ibv_qp_cap {
	uint32_t		max_send_wr;
	uint32_t		max_recv_wr;
	uint32_t		max_send_sge;
	uint32_t		max_recv_sge;
	uint32_t		max_inline_data;
};

Here is the full description of struct ibv_qp_cap:

max_send_wr The maximum number of outstanding Work Requests that can be posted to the Send Queue in that Queue Pair. Value can be [1..dev_cap.max_qp_wr]
max_recv_wr The maximum number of outstanding Work Requests that can be posted to the Receive Queue in that Queue Pair. Value can be [1..dev_cap.max_qp_wr]. This value is ignored if the Queue Pair is associated with an SRQ
max_send_sge The maximum number of scatter/gather elements in any Work Request that can be posted to the Send Queue in that Queue Pair. Value can be [1..dev_cap.max_sge]
max_recv_sge The maximum number of scatter/gather elements in any Work Request that can be posted to the Receive Queue in that Queue Pair. Value can be [1..dev_cap.max_sge]. This value is ignored if the Queue Pair is associated with an SRQ
max_inline_data The maximum message size (in bytes) that can be posted inline to the Send Queue. 0, if no inline message is requested

struct ibv_ah_attr describes the Address Vector of the QP.

struct ibv_ah_attr {
	struct ibv_global_route	grh;
	uint16_t		dlid;
	uint8_t			sl;
	uint8_t			src_path_bits;
	uint8_t			static_rate;
	uint8_t			is_global;
	uint8_t			port_num;
};

Here is the full description of struct ibv_ah_attr:

grh Attributes of the Global Routing Headers (GRH), as described in the table below. This is useful when sending packets to another subnet
dlid If the destination is in same subnet, the LID of the port to which the subnet delivers the packets to. If the destination is in another subnet, the LID of the Router
sl 4 bits. The Service Level to be used
src_path_bits The used Source Path Bits. This is useful when LMC is used in the port, i.e. each port covers a range of LIDs. The packets are being sent with the port's base LID, bitwised ORed with the value of the source path bits. The value 0 indicates the port's base LID is used
static_rate A value which limits the rate of packets that being sent to the subnet. This can be useful if the rate of the packet origin is higher than the rate of the destination
is_global If this value contains any value other than zero, then GRH information exists in this AH, thus the field grh if valid
port_num The local physical port that the packets will be sent from

struct ibv_global_route describes the values to be used in the GRH of the packets that will be sent when using this AH.

struct ibv_global_route {
	union ibv_gid		dgid;
	uint32_t		flow_label;
	uint8_t			sgid_index;
	uint8_t			hop_limit;
	uint8_t			traffic_class;
};

Here is the full description of struct ibv_global_route:

dgid The GID that is used to identify the destination port of the packets
flow_label 20 bits. If this value is set to a non-zero value, it gives a hint for switches and routers with multiple outbound paths that these sequence of packets must be delivered in order, those staying on the same path, so that they won't be reordered.
sgid_index An index in the port's GID table that will be used to identify the originator of the packet
hop_limit The number of hops (i.e. the number of routers) that the packet is permitted to take before being discarded. This ensures that a packet will not loop indefinitely between routers if a routing loop occur. Each router decrement by one this value at the packet and when this value reaches 0, this packet is discarded. Setting the value to 0 or 1 will ensure that the packet won't leave the local subnet.
traffic_class Using this value, the originator of the packets specifies the required delivery priority for handling them by the routers

attr_mask specifies the QP attributes to be modified. It is either 0 or the bitwise OR of one or more of the following flags:

IBV_QP_STATE Use the value of attr->qp_state
IBV_QP_CUR_STATE Use the value of attr->cur_qp_state
IBV_QP_EN_SQD_ASYNC_NOTIFY Use the value of attr->en_sqd_async_notify
IBV_QP_ACCESS_FLAGS Use the value of attr->qp_access_flags
IBV_QP_PKEY_INDEX Use the value of attr->pkey_index
IBV_QP_PORT Use the value of attr->port_num
IBV_QP_QKEY Use the value of attr->qkey
IBV_QP_AV Use the value of attr->ah_attr
IBV_QP_PATH_MTU Use the value of attr->path_mtu
IBV_QP_TIMEOUT Use the value of attr->timeout
IBV_QP_RETRY_CNT Use the value of attr->retry_cnt
IBV_QP_RNR_RETRY Use the value of attr->rnr_retry
IBV_QP_RQ_PSN Use the value of attr->rq_psn
IBV_QP_MAX_QP_RD_ATOMIC Use the value of attr->max_rd_atomic
IBV_QP_ALT_PATH Use the value of attr->alt_ah_attr, attr->alt_pkey_index, attr->alt_port_num, attr->alt_timeout
IBV_QP_MIN_RNR_TIMER Use the value of attr->min_rnr_timer
IBV_QP_SQ_PSN Use the value of attr->sq_psn
IBV_QP_MAX_DEST_RD_ATOMIC Use the value of attr->max_dest_rd_atomic
IBV_QP_PATH_MIG_STATE Use the value of attr->path_mig_state
IBV_QP_CAP Use the value of attr->cap
IBV_QP_DEST_QPN Use the value of attr->dest_qp_num

The following table specify the full supported transition for QP with service type IBV_QPT_UD and the attributes it accepts:

Transition Required Attributes Optional Attributes
*->RESET IBV_QP_STATE
*->ERR IBV_QP_STATE
RESET->INIT IBV_QP_STATE, IBV_QP_PKEY_INDEX, IBV_QP_PORT, IBV_QP_QKEY
INIT->INIT IBV_QP_PKEY_INDEX, IBV_QP_PORT, IBV_QP_QKEY
INIT->RTR IBV_QP_STATE IBV_QP_PKEY_INDEX, IBV_QP_QKEY
RTR->RTS IBV_QP_STATE, IBV_QP_SQ_PSN IBV_QP_CUR_STATE, IBV_QP_QKEY
RTS->RTS IBV_QP_CUR_STATE, IBV_QP_QKEY
RTS->SQD IBV_QP_STATE IBV_QP_EN_SQD_ASYNC_NOTIFY
SQD->RTS IBV_QP_STATE IBV_QP_CUR_STATE, IBV_QP_QKEY
SQD->SQD IBV_QP_PKEY_INDEX, IBV_QP_QKEY
SQE->RTS IBV_QP_STATE IBV_QP_CUR_STATE, IBV_QP_QKEY

The following table specify the full supported transition for QP with service type IBV_QPT_UC and the attributes it accepts:

Transition Required Attributes Optional Attributes
*->RESET IBV_QP_STATE
*->ERR IBV_QP_STATE
RESET->INIT IBV_QP_STATE, IBV_QP_PKEY_INDEX, IBV_QP_PORT, IBV_QP_ACCESS_FLAGS
INIT->INIT IBV_QP_PKEY_INDEX, IBV_QP_PORT, IBV_QP_ACCESS_FLAGS
INIT->RTR IBV_QP_STATE, IBV_QP_AV, IBV_QP_PATH_MTU, IBV_QP_DEST_QPN, IBV_QP_RQ_PSN IBV_QP_PKEY_INDEX, IBV_QP_ACCESS_FLAGS,
IBV_QP_ALT_PATH
RTR->RTS IBV_QP_STATE, IBV_QP_SQ_PSN IBV_QP_CUR_STATE,
IBV_QP_ACCESS_FLAGS,
IBV_QP_ALT_PATH, IBV_QP_PATH_MIG_STATE
RTS->RTS IBV_QP_CUR_STATE, IBV_QP_ACCESS_FLAGS, IBV_QP_ALT_PATH, IBV_QP_PATH_MIG_STATE
RTS->SQD IBV_QP_STATE IBV_QP_EN_SQD_ASYNC_NOTIFY
SQD->RTS IBV_QP_STATE IBV_QP_CUR_STATE, IBV_QP_ACCESS_FLAGS, IBV_QP_ALT_PATH, IBV_QP_PATH_MIG_STATE
SQD->SQD IBV_QP_PKEY_INDEX, IBV_QP_ACCESS_FLAGS,
IBV_QP_AV,
IBV_QP_ALT_PATH, IBV_QP_PATH_MIG_STATE
SQE->RTS IBV_QP_STATE IBV_QP_CUR_STATE, IBV_QP_ACCESS_FLAGS

The following table specify the full supported transition for QP with service type IBV_QPT_RC and the attributes it accepts:

Transition Required Attributes Optional Attributes
*->RESET IBV_QP_STATE
*->ERR IBV_QP_STATE
RESET->INIT IBV_QP_STATE, IBV_QP_PKEY_INDEX, IBV_QP_PORT, IBV_QP_ACCESS_FLAGS
INIT->INIT IBV_QP_PKEY_INDEX, IBV_QP_PORT, IBV_QP_ACCESS_FLAGS
INIT->RTR IBV_QP_STATE, IBV_QP_AV, IBV_QP_PATH_MTU, IBV_QP_DEST_QPN, IBV_QP_RQ_PSN, IBV_QP_MAX_DEST_RD_ATOMIC, IBV_QP_MIN_RNR_TIMER IBV_QP_PKEY_INDEX,
IBV_QP_ACCESS_FLAGS,
IBV_QP_ALT_PATH
RTR->RTS IBV_QP_STATE, IBV_QP_SQ_PSN, IBV_QP_TIMEOUT, IBV_QP_RETRY_CNT, IBV_QP_RNR_RETRY, IBV_QP_MAX_QP_RD_ATOMIC IBV_QP_CUR_STATE,
IBV_QP_ACCESS_FLAGS,
IBV_QP_MIN_RNR_TIMER, IBV_QP_ALT_PATH, IBV_QP_PATH_MIG_STATE
RTS->RTS IBV_QP_CUR_STATE, IBV_QP_ACCESS_FLAGS,
IBV_QP_MIN_RNR_TIMER,
IBV_QP_ALT_PATH, IBV_QP_PATH_MIG_STATE
RTS->SQD IBV_QP_STATE IBV_QP_EN_SQD_ASYNC_NOTIFY
SQD->RTS IBV_QP_STATE IBV_QP_CUR_STATE,
IBV_QP_ACCESS_FLAGS,
IBV_QP_MIN_RNR_TIMER, IBV_QP_ALT_PATH, IBV_QP_PATH_MIG_STATE
SQD->SQD IBV_QP_PKEY_INDEX,
IBV_QP_PORT,
IBV_QP_ACCESS_FLAGS,
IBV_QP_AV,
IBV_QP_MAX_QP_RD_ATOMIC,
IBV_QP_MIN_RNR_TIMER,
IBV_QP_ALT_PATH,
IBV_QP_TIMEOUT,
IBV_QP_RETRY_CNT,
IBV_QP_RNR_RETRY, IBV_QP_MAX_DEST_RD_ATOMIC,
IBV_QP_PATH_MIG_STATE

Parameters

Name Direction Description
qp in QP that was returned from ibv_create_qp()
attr in/out Requested attributes to be modified in the QP. If the QP is being resized, it will hold the actual attributes of the QP
attr_mask in Mask of the QP attributes to be modified

Return Values

Value Description
0 On success
errno On failure and no change will be done to the QP
EINVAL Invalid value provided in attr or in attr_mask
ENOMEM Not enough resources to complete this operation

Examples

1) Modify a UD QP from the RESET state to the RTS state:

/* this example assumed that the variables my_port, my_psn are declared and initialized with valid values */
struct ibv_qp *qp;
struct ibv_qp_attr attr;
 
memset(&attr, 0, sizeof(attr));
 
attr.qp_state        = IBV_QPS_INIT;
attr.pkey_index      = 0;
attr.port_num        = my_port;
attr.qkey            = 0x22222222;
 
if (ibv_modify_qp(qp, &attr,
		  IBV_QP_STATE      |
		  IBV_QP_PKEY_INDEX |
		  IBV_QP_PORT       |
		  IBV_QP_QKEY)) {
	fprintf(stderr, "Failed to modify QP to INIT\n");
	return -1;
}
 
memset(&attr, 0, sizeof(attr));
 
attr.qp_state		= IBV_QPS_RTR;
 
if (ibv_modify_qp(qp, &attr, IBV_QP_STATE)) {
	fprintf(stderr, "Failed to modify QP to RTR\n");
	return -1;
}
 
memset(&attr, 0, sizeof(attr));
 
attr.qp_state	    = IBV_QPS_RTS;
attr.sq_psn	    = my_psn;
 
if (ibv_modify_qp(qp, &attr,
		  IBV_QP_STATE |
		  IBV_QP_SQ_PSN)) {
	fprintf(stderr, "Failed to modify QP to RTS\n");
	return 1;
}

2) Modify a UC QP from the RESET state to the RTS state:

/* this example assumed that the variables my_port, my_psn, my_mtu, my_sl, remote_qpn, remote_psn, remote_lid are declared and initialized with valid values */
struct ibv_qp *qp;
struct ibv_qp_attr attr;
 
memset(&attr, 0, sizeof(attr));
 
attr.qp_state        = IBV_QPS_INIT;
attr.pkey_index      = 0;
attr.port_num        = my_port;
attr.qp_access_flags = 0;
 
if (ibv_modify_qp(qp, &attr,
		  IBV_QP_STATE        |
		  IBV_QP_PKEY_INDEX   |
		  IBV_QP_PORT         |
		  IBV_QP_ACCESS_FLAGS)) {
	fprintf(stderr, "Failed to modify QP to INIT\n");
	return -1;
}
 
memset(&attr, 0, sizeof(attr));
 
attr.qp_state		= IBV_QPS_RTR;
attr.path_mtu		= my_mtu;
attr.dest_qp_num	= remote_qpn;
attr.rq_psn		= remote_psn;
attr.ah_attr.is_global	   = 0;
attr.ah_attr.dlid	   = remote_lid;
attr.ah_attr.sl		   = my_sl;
attr.ah_attr.src_path_bits = 0;
attr.ah_attr.port_num	   = my_port;
 
if (ibv_modify_qp(qp, &attr,
		  IBV_QP_STATE    |
		  IBV_QP_AV       |
		  IBV_QP_PATH_MTU |
		  IBV_QP_DEST_QPN |
		  IBV_QP_RQ_PSN)) {
	fprintf(stderr, "Failed to modify QP to RTR\n");
	return -1;
}
 
memset(&attr, 0, sizeof(attr));
 
attr.qp_state	    = IBV_QPS_RTS;
attr.sq_psn	    = my_psn;
 
if (ibv_modify_qp(qp, &attr,
		  IBV_QP_STATE  |
		  IBV_QP_SQ_PSN)) {
	fprintf(stderr, "Failed to modify QP to RTS\n");
	return 1;
}

3) Modify an RC QP from the RESET state to the RTS state:

/* this example assumed that the variables my_port, my_psn, my_mtu, my_sl, remote_qpn, remote_psn, remote_lid are declared and initialized with valid values */
struct ibv_qp *qp;
struct ibv_qp_attr attr;
 
memset(&attr, 0, sizeof(attr));
 
attr.qp_state        = IBV_QPS_INIT;
attr.pkey_index      = 0;
attr.port_num        = my_port;
attr.qp_access_flags = 0;
 
if (ibv_modify_qp(qp, &attr,
		  IBV_QP_STATE      |
		  IBV_QP_PKEY_INDEX |
		  IBV_QP_PORT       |
		  IBV_QP_ACCESS_FLAGS)) {
	fprintf(stderr, "Failed to modify QP to INIT\n");
	return -1;
}
 
memset(&attr, 0, sizeof(attr));
 
attr.qp_state		= IBV_QPS_RTR;
attr.path_mtu		= my_mtu;
attr.dest_qp_num	= remote_qpn;
attr.rq_psn		= remote_psn;
attr.max_dest_rd_atomic	= 1;
attr.min_rnr_timer	= 12;
attr.ah_attr.is_global	   = 0;
attr.ah_attr.dlid	   = remote_lid;
attr.ah_attr.sl		   = my_sl;
attr.ah_attr.src_path_bits = 0;
attr.ah_attr.port_num	   = my_port;
 
if (ibv_modify_qp(qp, &attr,
		  IBV_QP_STATE              |
		  IBV_QP_AV                 |
		  IBV_QP_PATH_MTU           |
		  IBV_QP_DEST_QPN           |
		  IBV_QP_RQ_PSN             |
		  IBV_QP_MAX_DEST_RD_ATOMIC |
		  IBV_QP_MIN_RNR_TIMER)) {
	fprintf(stderr, "Failed to modify QP to RTR\n");
	return -1;
}
 
memset(&attr, 0, sizeof(attr));
 
attr.qp_state	    = IBV_QPS_RTS;
attr.sq_psn	    = my_psn;
attr.timeout	    = 14;
attr.retry_cnt	    = 7;
attr.rnr_retry	    = 7; /* infinite */
attr.max_rd_atomic  = 1;
 
if (ibv_modify_qp(qp, &attr,
		  IBV_QP_STATE              |
		  IBV_QP_TIMEOUT            |
		  IBV_QP_RETRY_CNT          |
		  IBV_QP_RNR_RETRY          |
		  IBV_QP_SQ_PSN             |
		  IBV_QP_MAX_QP_RD_ATOMIC)) {
	fprintf(stderr, "Failed to modify QP to RTS\n");
	return 1;
}

FAQs

Why do I need to call ibv_modify_qp()?

You need to call ibv_modify_qp() in order to modify the QP into a state where it can receive, and if needed, send data

Can I move a QP into RTR state and leave it in that state?

Yes, you can. If the QP will be only a receiver

How do I know which attributes to configure the QP with?

In Infiniband, you should perform path query to the SA or use CM or CMA, in iWARP, you should use the CMA

ibv_modify_qp() failed, what are the attributes of that QP now?

Exactly the same as before calling ibv_modify_qp(), no change was made to the QP. Although maybe the failure was caused because the QP moved by the RDMA device to the SQE or ERROR state because of an error

ibv_modify_qp() failed in the transition INIT->RTR for connected QPs over RoCE, what is the reason for this?

When using RoCE, GRH must be configured.
In connected QPs: as part of the QP attributes
In UD QP: as part of the Address Handle attributes

Share Our Posts

Share this post through social bookmarks.

  • Delicious
  • Digg
  • Newsvine
  • RSS
  • StumbleUpon
  • Technorati

Comments

Tell us what do you think.

  1. Igor R. says: February 11, 2014

    I'd appreciate if you could shed some light on the following subject.
    As per "Infinibeand Network Architecture"(* see quotations below), during RDMA Read request, the responder's RQ logic is allowed to re-read from memory when re-trying. Can we totally disable this functionality by setting retry_cnt=rnr_cnt=0?
    In other words, can one be sure in such a case, that a single RDMA Read request will never cause double read of any remote location? (This can be crucial if it's a device memory, and an access might have side effects.)

    Thanks!

    * Infiniband Network Architecture, p.34:
    <<
    This service is referred to as "reliable" for the following reasons:
    -- The destination QP's RQ Logic verifies the PSN in each request packet to ensure that all of the message's request packets are received in order, that none are missing, and, if any duplicate request packets are received, that they are only processed once. There is one exception; upon receipt of a duplicate memory read, the data is read from memory again.

    >>

    And in details (p.394):
    <<
    Effects of Retry on Requester and Responder The following list defines the sequence of events:

    6. RQ Logic receives duplicate copy of request. When the second copy (the retried one) of the request packet eventually arrives at the RQ Logic, it has the same PSN as the original request packet. Its PSN, therefore, falls into the Ack’ d PSN region (also referred to as the Duplicate region), thereby identifying it as a duplicate request packet. The actions taken by the RQ Logic are determined by the request type:

    — If it’ s a duplicate RDMA Read:
    – The RQ Logic re-executes the read from its local memory.
    – It sends back the requested read data in a series of one or more RDMA Read response packets.
    – The first response packet uses the PSN of the original request packet, and each subsequent response packet increments the PSN by one.
    – The RQ Logic does not change its ePSN (even if an error is detected while generating the response to the duplicate request).
    – After issuing the response to the duplicate request, the RQ Logic resumes waiting for request packet with the ePSN.

    • Igor R. says: February 12, 2014

      Well, it seems that the retry counter is a "red herring" here, as the real problem is that remote HCA doesn't buffer the read data. So, in case of failure, even if the automatic retransmission won't occur, a manual one will be required - so that the data will be read twice anyway. Thus, solely disabling automatic retries is not sufficient, and more complicated approach is needed.

      • Dotan Barak says: February 12, 2014

        I must admit that I don't really understand what you are trying to perform...

        Yes, if the retry_cnt will be 0 retransmission won't happen,
        and the QP will get into the error state in the requester side.

        Is this is the expected behavior that you need?
        If you must read the data from one side without any read from the device,
        maybe RDMA Read isn't the right approach
        (maybe cache the data in local memory and perform RDMA Write is better..)

        Thanks
        Dotan

      • Igor R. says: February 12, 2014

        Thanks for the input.
        In my case the responding side (where the device is) must remain as passive as possible. I'm just trying to consider various possibilities and to understand their consequences...

  2. Alan says: August 29, 2014

    Hi Dotan,

    Do we have to memset the ibv_qp_attr structure before using it in the ibv_modify_qp() function? I am thinking there are flags in the function API to indicate which fields are being used why we still need to memset it?

    Thanks.

    • Dotan Barak says: August 29, 2014

      Hi Alan.

      In general - you are correct (and in most cases it will work without any problem).

      However:
      * Most ibv_* structu don't have bitmask to specify the valid attributes
      * The struct ibv_qp_attr *may* be extended (although the developers try hard to keep backward binary compatibility), and in that case calling memset() on the struct will keep the behavior as it was in the past

      I believe, as a developer, in defensive programming and this is a good practice to handle API which is evolving.

      I hope that I answered your question.

      Thanks
      Dotan

      • Yanfang Le says: October 12, 2015

        Hi Dotan,

        I am trying to flush the work requests in qp, so I first turn the qp state to error state, and then convert to the RTS state. During this process, I don't know where can I get the ah_attr information. Could you tell me how can I get the ah_attr information? Thanks.

        I am trying to copy the information from the old qp (before turn the state to error), however, it doesn't work.

      • Dotan Barak says: October 17, 2015

        Hi.

        When the QP is in ERROR state,
        you cannot trust the attributes which were queried from it, expect for the QP state.

        If you must keep the AH attributes, maybe you can:
        1) Save a local copy of the attributes as a function of the QP number
        2) Use the qp_context pointer to hold a buffer which holds the AH attributes.

        Thanks
        Dotan

  3. Haomai says: January 21, 2015

    Hi, if I make a connected qp to error state, is it possible that peer side can detect this.

    For example, if we use tcp socket to communicate, if we close one side's socket, other side will get "0" when read. So peer side can know peer socket already closed. Is it possible that rdma can achieve this?

    Thanks!

    • Dotan Barak says: January 23, 2015

      Hi Haomai.

      In RDMA there aren't any keep-alive messages that send automatically,
      if needed you need to perform this in your application.

      If you close one side, only if you are using rdmacm, the remote side will be informed on this;
      but not in "standard" data verbs.

      If the other side (which didn't move to Error) won't send any message - it won't know that its remote
      QP moved to Error.

      Thanks
      Dotan

  4. Adrian says: March 23, 2015

    Hi Dotan,

    I've just started with verbs and when I try to modify a qp from INIT to RTR state, a 101 error is returned. The 101 error means #define ENETUNREACH 101 /* Network is unreachable */. The values are exchanged via TCP sockets (lid=4 qp_num=116 psn=9105394 from client to server and lid=3 qp_num=116 psn=14618275 from server to client). I don't know what is happening and there is difficult to obtain information.

    Thanks

    • Dotan Barak says: March 24, 2015

      Hi Adrian.

      Are you using explicit ibv_modify_qp(() or do you call librdmacm function?
      Are you using IB or RoCE?

      Thanks
      Dotan

      • Adrian says: March 24, 2015

        I'm using IB and explicit ibv_modify_qp()

      • Dotan Barak says: March 24, 2015

        It is weird; since this return value isn't really returned ...
        Which QP transport type are you using?
        Can you please write explicit the values that you set + the used mask in the ibv_modify_qp()?

        Thanks
        Dotan

  5. Adrian says: March 24, 2015

    Hi Dotan,
    the code is

    attr.qp_state = IBV_QPS_RTR;
    attr.path_mtu = mtu;
    attr.dest_qp_num = rmsg.qp_num[0];
    attr.rq_psn = rmsg.psn[0];
    attr.ah_attr.dlid = rmsg.lid;
    attr.max_dest_rd_atomic = 1;
    attr.min_rnr_timer = 12;
    attr.ah_attr.is_global = 0;
    attr.ah_attr.sl = 0;
    attr.ah_attr.src_path_bits = 0;

    if (ibv_modify_qp(qp, &attr, IBV_QP_STATE | IBV_QP_AV | IBV_QP_PATH_MTU |
    IBV_QP_DEST_QPN | IBV_QP_RQ_PSN | IBV_QP_MIN_RNR_TIMER |
    IBV_QP_MAX_DEST_RD_ATOMIC)){
    cerr << "Failed to modify QP 1 to RTR." << endl;
    abort();
    }

    lid, qp_num and psn are sent via TCP sockets.

    Thanks

    • Dotan Barak says: March 26, 2015

      Hi.

      I assume that you are using RC QP.
      What about attr.ah_attr.port_num?

      Thanks
      Dotan

      • Adrian says: March 31, 2015

        Yes, I use RC QP and the port number is 1. Finally, we reinstall the OFED drivers and it works.

        Thank you for your time.

      • Dotan Barak says: March 31, 2015

        Sure.

        Maybe there were some inconsistencies between the various libraries..

        I'm glad at the end everything worked for you
        :)
        Dotan

    • Sangwook says: April 14, 2015

      Hi Adrian & Dotan. Same error happens in my code too.
      Explicit ibv_modify_qp call to change QP state to RTR fails and returns 101. (My code is based on ibv_rc_pingpong program, an example from libibverb library. And ibv_rc_pingpong also fails in my environment.)
      You said the problem is solved after reinstalling OFED drivers.. Can you share your system environment and OFED version?
      (I installed OFED ver 2.4-1.0.4 on Intel Xeon E5-2670 + MLNX ConnectX-3 machine running Ubuntu 14.04 with kernel 3.13.0-48.)

    • Sangwook says: April 14, 2015

      Hi, I wrote a comment to ask for Adrian's system environment. But I figured out the cause of my problem- I had to add "gid-index" option in my program because I'm using RoCE. My program worked fine with that option. Anyway, thanks!

      • Dotan Barak says: April 16, 2015

        I'm glad that everything is fine.

        I'll add this note on RoCE, so people won't fail on it again...

        Thanks
        Dotan

  6. Jesus Camacho says: May 5, 2015

    Hi Dotan!

    I am running the ib_write_lat microbenchmark with OpenSM.

    I am disabling a link while running them and then OpenSM reconfigures all the network and the benchmark continues working without problems.

    If have reduce the number of retries in the QPs to 1 or 2, then ib_wirte_lat crash in some cases now.

    However, if I change the timeout of the QPs, I have the same behaviour.

    I expected to have less retries if I increase the QP timeout. I understand that every retry is triggered after the QP timeout expires.

    Am i right?

    If not, how is it affecting the QP timeout to the number of retries?

    Thanks for your help!

    Best regards,
    Jesus Camacho

    • Dotan Barak says: May 5, 2015

      Hi Jesus.

      Yes, you are right: every retry is triggered after a QP timeout expires.

      There is an internal clock in the RDMA device, and at least after the timeout expires
      (+ a delta which depends on this clock resolution), there is a retransmission.

      If (in your experience) the link will be stable until the last retransmission,
      ib_write_lat will continue working..

      Thanks
      Dotan
      If until the
      What is "important" is the

      • Jesus Camacho says: May 6, 2015

        Hi Dotan,

        I am replacing a LFT instead of switching on the link back.
        It takes less than 2 ms from the time I send the new block to the LFT and I receive its ACK later. Then the new path should be ready in this short period of time.

        In one of my experiments I am using only one retry and timeout 10 (4194.304 usec).
        Although I spend only 4 ms until the next retransmission, I have to spend up to 0.5 seconds for a successful completion when polling the completion queue (ibv_poll_cq using RC).
        Do you know why this takes all this time?

        Thank you!
        Jesus

      • Dotan Barak says: May 7, 2015

        Hi Jesus.

        What does LFT mean? (I don't really familiar with this term)
        Are you performing Automatic Path Migartion?
        What are the values of the timeout/retry_count?

        Thanks
        Dotan

  7. Jesus Camacho says: May 8, 2015

    Hi Dotan,

    instead of using Automatic Path Migration, I am replacing the forwarding tables (trough OpenSM) in some of the switches to avoid the broken link (I just say the switches to take another port to avoid the link that is down). This takes about 2 milliseconds.

    I am using this values in the ib_write_lat microbenchmark:
    timeout 10
    retry_cnt 1

    This seems to work (I am using a small network). Then I understand that the retransmission is OK in less than timeout+timeout.

    I am measuring the time to send a packet from one node to another and this takes usually several milliseconds.

    However, it takes longer after the fault. And most of the time is spent polling the completion queue -ibv_poll_cq- (up to 0.5 seconds).

    Do you know what could be the reason for this?

    Thanks for your time :-)
    Jesus

    • Dotan Barak says: May 8, 2015

      You are welcome
      :)

      I don't have a solid answer here (from knowledge), but I have an assumption:
      RDMA is very fine tune to handle good flow (since this happen most of the time).

      In case of error flow starts (i.e. retransmission, violation, etc.), maybe this flow isn't fully tuned.

      Another option is that this happening because of other issues
      (for example: is the NUMA node is local or remote? do you use CPU affinity to the running process? etc.)

      I would suggest to use a sniffing tool and try to measure this (when the message is send, when an ack is returned, and when there is a Work Completion available in the CQ) or send an email to the support team of the HW vendor about this.

      BTW, making the SM change the subnet may cause the subnet to start recovery flows
      (since *maybe* client reregister events are created in the subnet's nodes).

      Thanks
      Dotan

  8. Anonymous says: October 1, 2015

    Hi Dotan,

    When I tried to modify retry_cnt to 7 using ibv_modify_qp,
    I got "Invalid argument". Do you have any suggestion ?
    I listed my code here:

    struct ibv_qp_attr attr;
    int qret;
    attr.retry_cnt = (uint8_t)7;
    attr.timeout = (uint8_t)14;
    attr.rnr_retry = (uint8_t)7;
    qret = ibv_modify_qp(id->qp, &attr, IBV_QP_RETRY_CNT|IBV_QP_TIMEOU T|IBV_QP_RNR_RETRY);

    Thanks

    • Dotan Barak says: October 1, 2015

      Hi.

      From which QP state to which QP state did you try to do this?

      I suspect that you need more flags in the QP transition that you made.

      Thanks
      Dotan

  9. Junhyun says: August 12, 2016

    Hi Dotan, I recently built an event-driven RDMA send/recv IO engine for multiplexing multiple QP events and so far I worked out most kinks, but it suffers when more than one connection is introduced.
    What frustrates me is that the connection works perfectly until I get the WC for my first WR! (QP transition and post_send doesn't return any errors, even with the second connection)
    The second connection keeps throwing me status 12 (RETRY_EXC_ERROR) WCs while the first connection keeps on exchanging data smoothly. When I retry the post_send, I get status 5, vendor_err 249. I took it to mean that the QP has already transitioned to an error state.

    Environment:
    I use dual-port Mellanox CX-3 EN 10GBe on RoCE. Port 1 acts both as the port for my shell connection and RoCE.
    I have made it so that all QPs from the same device share device context and pd, but they use disjoint MRs.
    Here's the code I use for creating and transitioning QPs.
    Every created QPs follow this exact path.
    I wish I could find more explanation as to why this keeps failing.
    Please tell me if I'm doing anything wrong.
    conn.port is the local port I chose for use, and remote_info members are obtained from the other host over TCP connection. (byte-reordered)
    Again, thanks for maintaining such a helpful site.

    struct ibv_qp_init_attr qp_init_attr;
    memset(&qp_init_attr, 0, sizeof(qp_init_attr));
    qp_init_attr.qp_type = IBV_QPT_RC;
    qp_init_attr.sq_sig_all = 1;
    qp_init_attr.send_cq = conn.cq;
    qp_init_attr.recv_cq = conn.cq;
    qp_init_attr.cap.max_send_wr = DEFAULT_CQ_SIZE / 2;
    qp_init_attr.cap.max_recv_wr = DEFAULT_CQ_SIZE / 2;
    qp_init_attr.cap.max_send_sge = std::min((int)MAX_SGE, _per_device_attr[device].max_sge);
    qp_init_attr.cap.max_recv_sge = std::min((int)MAX_SGE, _per_device_attr[device].max_sge);
    CHECK_PTR(conn.qp = ibv_create_qp(conn.pd, &qp_init_attr), "Failed to create QP for device " << device << ", port " << port << ", error=" << strerror(errno) << ", pd=" < INIT
    struct ibv_qp_attr to_init;
    int flags;
    memset(&to_init, 0, sizeof(to_init));
    to_init.qp_state = IBV_QPS_INIT;
    to_init.port_num = conn.port_num;
    to_init.pkey_index = 0;
    to_init.qp_access_flags = IBV_ACCESS_LOCAL_WRITE | IBV_ACCESS_REMOTE_READ | IBV_ACCESS_REMOTE_WRITE;
    flags = IBV_QP_STATE | IBV_QP_PKEY_INDEX | IBV_QP_PORT | IBV_QP_ACCESS_FLAGS;
    CHECK_EQ0(ibv_modify_qp(conn.qp, &to_init, flags), "QP transition failed (RESET -> INIT): " <RTR
    struct ibv_qp_attr to_rtr;
    int flags;
    memset(&to_rtr, 0, sizeof(to_rtr));
    to_rtr.qp_state = IBV_QPS_RTR;
    to_rtr.path_mtu = DEFAULT_QP_MTU;
    to_rtr.dest_qp_num = remote_info.qp_num;
    to_rtr.rq_psn = 0;
    to_rtr.max_dest_rd_atomic = 0;
    to_rtr.min_rnr_timer = 12;
    to_rtr.ah_attr.dlid = remote_info.lid;
    to_rtr.ah_attr.sl = 0;
    to_rtr.ah_attr.src_path_bits = 0;
    to_rtr.ah_attr.port_num = conn.port_num;
    // Begin RoCE specific conf (currently universal)
    to_rtr.ah_attr.is_global = 1;
    memcpy(&to_rtr.ah_attr.grh.dgid, remote_info.gid, sizeof(remote_info.gid));
    to_rtr.ah_attr.grh.flow_label = 0;
    to_rtr.ah_attr.grh.hop_limit = 1;
    to_rtr.ah_attr.grh.sgid_index = 0;
    to_rtr.ah_attr.grh.traffic_class = 0;
    flags = IBV_QP_STATE | IBV_QP_AV | IBV_QP_PATH_MTU | IBV_QP_DEST_QPN |
    IBV_QP_RQ_PSN | IBV_QP_MAX_DEST_RD_ATOMIC | IBV_QP_MIN_RNR_TIMER;
    CHECK_EQ0(ibv_modify_qp(conn.qp, &to_rtr, flags), "QP transition failed (INIT -> RTR): " < RTS
    struct ibv_qp_attr to_rts;
    int flags;
    memset(&to_rts, 0, sizeof(to_rts));
    to_rts.qp_state = IBV_QPS_RTS;
    to_rts.timeout = 14;
    to_rts.retry_cnt = 7;
    to_rts.rnr_retry = 7;
    to_rts.sq_psn = 0;
    to_rts.max_rd_atomic = 0;
    flags = IBV_QP_STATE | IBV_QP_TIMEOUT | IBV_QP_RETRY_CNT |
    IBV_QP_RNR_RETRY | IBV_QP_SQ_PSN | IBV_QP_MAX_QP_RD_ATOMIC;
    CHECK_EQ0(ibv_modify_qp(conn.qp, &to_rts, flags), "QP transition failed (RTR -> RTS)" << getQPDescription(local_info));

    • Dotan Barak says: August 18, 2016

      Hi.

      Please make the following:

      1) verify that the DEFAULT_QP_MTU is less than the interface MTU
      (i.e. if the network I/F MTU is 1500, this value should be 1024).
      2) All remote attributes are correct
      3) Make sure that there is a synchronization between both sides
      (to prevent a case where the sender send a message and the receiver isn't in RTR state,
      or in Error state).

      Thanks
      Dotan

Add a Comment

Fill in the form and submit.

Time limit is exhausted. Please reload CAPTCHA.