Skip to content

ibv_create_srq()

5.00 avg. rating (97% score) - 1 vote
struct ibv_srq *ibv_create_srq(struct ibv_pd *pd, struct
                               ibv_srq_init_attr *srq_init_attr);

Description

ibv_create_srq() creates a Shared Receive Queue (SRQ) associated with a Protection Domain.

This SRQ will later be used when calling ibv_create_qp() to indicate that the Receive Queue of that Queue Pair is shared with other Queue Pairs.

If the value of dev_cap.max_srq is zero, this means that the RDMA device doesn't support Shared Receive Queues.

The user can define the minimum attributes to the SRQ: number of Work Requests and number of scatter/gather entries per Work Request. The actual attributes can be equal or higher than those values.

The struct ibv_srq_init_attr describes the requested attributes of the newly created SRQ.

struct ibv_srq_init_attr {
        void                   *srq_context;
        struct ibv_srq_attr     attr;
};

Here is the full description of struct ibv_srq_init_attr:

srq_context (optional) User defined value which will be available in srq->srq_context
attr Attributes of the Shared Receive Queue, as described in the table below

struct ibv_srq_attr describes the attributes of the Shared Receive Queue.

struct ibv_srq_attr {
        uint32_t                max_wr;
        uint32_t                max_sge;
        uint32_t                srq_limit;
};

Here is the full description of struct ibv_srq_attr:

max_wr The maximum number of outstanding Work Requests that can be posted to this Shared Receive Queue. Value can be [1..dev_cap.max_srq_wr]
max_sge The maximum number of scatter/gather elements in any Work Request that can be posted to this Shared Receive Queue. Value can be [1..dev_cap.max_srq_sge]
srq_limit The value that the SRQ will be armed with. This value is relevant only for iWARP and being ignored in Infiniband

Parameters

Name Direction Description
pd in Protection Domain that was returned from ibv_alloc_pd()
srq_init_attr in/out Requested attributes for the Shared Receive Queue. After the SRQ creation, it will hold the actual attributes of the SRQ

Return Values

Value Description
SRQ A pointer to the newly allocated Shared Receive Queue
NULL On failure, errno indicates the failure reason:

EINVAL Invalid pd or invalid value provided in max_wr or in max_sge
ENOMEM Not enough resources to complete this operation
ENOSYS Shared Receive Queue isn't supported by this RDMA device

Examples

Create an SRQ and destroy it:

struct ibv_pd *pd;
struct ibv_srq *srq;
struct ibv_srq_init_attr srq_init_attr;
 
memset(&srq_init_attr, 0, sizeof(srq_init_attr));
 
srq_init_attr.attr.max_wr  = 1;
srq_init_attr.attr.max_sge = 2;
 
srq = ibv_create_srq(pd, &srq_init_attr);
if (!srq) {
	fprintf(stderr, "Error, ibv_create_srq() failed\n");
	return -1;
}
 
if (ibv_destroy_srq(srq)) {
	fprintf(stderr, "Error, ibv_destroy_srq() failed\n");
	return -1;
}

FAQs

Why is an SRQ good for anyway?

SRQ, as its name states: is a Shared Receive Queue. Using SRQs decreases the total number of posted Receive Requests compared to using Queue Pairs, each with its own Receive Queue.

Can I use one SRQ with different QPs?

Yes. you can.

Can I use one SRQ with QPs with different transport types?

Yes. you can.

How do I use srq_limit?

If an SRQ was armed with a value, when the number of RRs in that SRQ drop below that value, the affiliated asynchronous event IBV_EVENT_SRQ_LIMIT_REACHED will be generated.

Which attributes should be used for an SRQ?

The number of Work Requests should be enough to hold Receive Requests for all of the incoming Messages in all of the QPs that are associated with this SRQ.
The number of scatter/gather elements should be the maximum number of scatter/gather elements in any Work Request that will be posted to this SRQ.

Which value can I use as the srq_context?

Since the srq_context is a void *, you can put any value that you wish.

Share Our Posts

Share this post through social bookmarks.

  • Delicious
  • Digg
  • Newsvine
  • RSS
  • StumbleUpon
  • Technorati

Comments

Tell us what do you think.

  1. Aunn Raza says: September 10, 2014

    Can a SRQ be associated with with more than one QP belonging to different connections (different NIC)?

    • Dotan Barak says: September 10, 2014

      One SRQ can be associated with multiple QPs, as long as the (both SRQ and QPs) are from the same RDMA device.
      Resources from different RDMA devices cannot be associated/used together.

      Thanks
      Dotan

  2. Rajesh says: November 14, 2015

    Hi Dotan,

    Can I associate SRQ with IB_QPT_GSI type of qp ?

    Thanks,
    Rajesh

    • Dotan Barak says: November 16, 2015

      Hi Rajesh.

      This is a hard question:
      According to the IB spec, SRQ should be supported for UD/UC/RC QPs.
      However, GSI QP behaves like a UD QP, so it *may* work.

      However, since GSI QP isn't a regular data QP,
      even if SRQ with GSI QP is supported - I would suggest not to use it

      Combine GSI QP and data QP isn't a good idea...

      Thanks
      Dotan

  3. Vu says: December 2, 2015

    Hi Dotan,

    Does the max_wr, and max_sge affect performance / memory in anycase ? Should I set it to the max the device can support or should I set it as low as possible to fit my work ?

    • Dotan Barak says: December 3, 2015

      Hi Vu.

      This is a good question, I'm answering here as a computer engineer and not from internal knowledge:
      IMHO using the lower possible value will be best because there are caches involved (for examples: for address translation):
      the lower the buffer is, the maximum cache hits that you will get;
      and maybe some more caches in the RDMA device
      +
      higher values means that more memory will be pinned (which has its own implication on system performance).

      Thanks
      Dotan

  4. kai says: July 21, 2016

    hi,
    do i have to use ibv_req_notify_cq() on sharedreceivequeues?

    • Dotan Barak says: July 29, 2016

      Hi.

      ibv_req_notify_cq() is relevant if one wished to work with CQ events,
      to decrease the CPU utilization by polling the CQ.

      You may use it with SRQ (or not).

      Thanks
      Dotan

  5. JD Sun says: June 7, 2017

    Hi Dotan,
    Can SRQ be associated with IBV_QPT_RAW_PACKET type of QP (Mellanox stack)? Thanks.

    • Dotan Barak says: July 2, 2017

      Hi.

      Sorry, but I prefer not to answer specific stack questions;
      this code is different in Mellanox stack compared to upstream kernel.

      Thanks
      Dotan

Add a Comment

Fill in the form and submit.

Time limit is exhausted. Please reload CAPTCHA.