Skip to content

Working with Unsignaled completions

Contents

4.56 avg. rating (91% score) - 9 votes

By default, all Work Requests generate Work Completions when their processing is finished. However, Send Request may or may not generate Work Completions when their processing is finished - this is fully controllable by the application and this is called Unsignaled Completions. In this post, I will cover when and how to use it.

What is Unsignaled Completion?

Unsignaled Completion is a mechanism that allows the application to post Send Requests that when their processing ends, Work Completions won't be generated in the Completion Queue, which is associated with the Send Queue as long as the Work Completion processing ends without an error. If a Send Request ended with an error, it will generate a Work Completion even if it was posted to use Unsignaled Completion.

Why use Unsignaled Completion?

Handling Work Requests consumes machine's resources:

  • Generation of the Work Completion by the RDMA device
  • Polling for the Completion Queue for Work Completions
  • Reading the Work Completion and check it status

Handling less Work Completions will help to reduce the CPU usage of the application, thus to increase the effectiveness of the application and to improve the latency of handled messages.

When to use Unsignaled Completion?

Unlike Work Completions that are generated from incoming messages that consume Receive Requests in asynchronous way, Work Requests that generated from Send Requests that their processing ended is quite expected by the application. Furthermore, Work Completions of finished Receive Requests may contain important information (opcode, message size, origin, etc.) but Work Completions of finished Send Requests doesn't contain important information (expect for information that is encoded in the wr_id, which is fully controlled by the application.

There are two groups of operations that can be used in Send Requests:

  1. Operations that request the remote side to send data (RDMA Read and Atomic operations)
  2. Operations that send data to the remote side (Send and RDMA Write operations with or without immediate data)

Unsignaled Completions can be used for operations from the first group if all the following conditions are met:

  • The application can request the remote side to read several buffers before it needs to use their content
  • All incoming data is written to different memory buffers
  • The Memory Regions that contain the incoming data don't need to be destroyed/reused

Unsignaled Completions can be used for operations from the second group if all the following conditions are met:

  • The application can send several messages before it needs acknowledge that there processing ended
  • The Memory Regions that contain the data to be sent don't need to be destroyed/reused
  • For UD QPs, Address Handles to the remote side don't need to be destroyed/reused

How to use Unsignaled Completion?

The support of Unsignaled Completion is configured per Queue Pair. When creating the Queue Pair, it should be created to support Unsignaled Completions in the Send Queue (the attribute qp_init_attr.sq_sig_all should be set to zero).

For every posted Send Request, if the flag IBV_SEND_SIGNALED is set in wr.send_flags then a Work Completion will be generated when this Send Request processing is ended. If this flags is cleared, then a Work Completion won't be generated (as long as it completed without an error).

Gotchas and Pitfalls

All posted Send Requested, Signaled and Unsignaled, are considered outstanding until a Work Completion that they, or Send Requests that were posted after them, was polled from the Completion Queue associated with the Send Queue. This means if one works with a Queue Pair that was configured to work with Unsignaled Completions, he must make sure that occasionally (before the Send Queue is full with outstanding Send Requests) a Send Request that generate Work Completion will be posted.

Not following this rule may lead to a case that the Send Queue is full with Send Requests that won't generate Work Completion:

  • The Send Queue is full, so no new Send Requests can be posted to it
  • The Send Queue can't be emptied, since no Work Completion can be generated anymore (the reason is that no Work Completion, that can generate Work Completion that polling it will empty the Send Queue, can be posted)
  • The status of all posted Send Request is considered unknown

Examples

1. Prepare a QP init attributes with an explicit setting of Unsignaled Completions:

struct ibv_qp_init_attr init_attr = {
	.send_cq = cq,
	.recv_cq = cq,
	.cap     = {
		.max_send_wr = 10,
		.max_recv_wr = 10,
		.max_send_sge = 1,
		.max_recv_sge = 1
	},
	.qp_type    = IBV_QPT_RC,
	.sq_sig_all = 0
};

2. Prepare a QP init attributes with an implicit setting of Unsignaled Completions:

struct ibv_qp_init_attr init_attr = {
	.send_cq = cq,
	.recv_cq = cq,
	.cap     = {
		.max_send_wr = 10,
		.max_recv_wr = 10,
		.max_send_sge = 1,
		.max_recv_sge = 1
	},
	.qp_type    = IBV_QPT_RC
};

3. Prepare a QP init attributes with an implicit setting of Unsignaled Completions:

struct ibv_qp_init_attr init_attr;
 
memset(&init_attr, 0, sizeof(init_attr));
 
init_attr.send_cq = cq;
init_attr.recv_cq = cq;
init_attr.cap.max_send_wr = 10;
init_attr.cap.max_recv_wr = 10;
init_attr.cap.max_send_sge = 1;
init_attr.cap.max_recv_sge = 1;
init_attr.qp_type = IBV_QPT_RC;

4. Prepare Send Request that ends with a Work Completion in a Queue Pair which configured to support Unsignaled Completions:

struct ibv_send_wr wr = {
	.num_sge    = 0,
	.opcode     = IBV_WR_SEND,
	.send_flags = IBV_SEND_SIGNALED
};

5. Prepare Send Request that won't end with a Work Completion in a Queue Pair which configured to support Unsignaled Completions:

struct ibv_send_wr wr = {
	.num_sge    = 0,
	.opcode     = IBV_WR_SEND,
	.send_flags = 0
};

FAQs

I accidentally filled the Send Queue with Send Requests that use Unsignaled Completion and I can't post anymore Send Requests to the Queue Pair. How can I recover from this state?

You cannot recover from this state. You need to destroy this Queue Pair and create a new one.

Share Our Posts

Share this post through social bookmarks.

  • Delicious
  • Digg
  • Newsvine
  • RSS
  • StumbleUpon
  • Technorati

Comments

Tell us what do you think.

  1. Abigor says: November 10, 2014

    Typo, you have written twice:
    "Unsignaled Completions can be used for operations from the first group if all the conditions are met:"
    instead of a first and then second group.

    Anyaway, great post, and blog, thank you!

    • Dotan Barak says: November 10, 2014

      Fixed. Thanks for noticing me on this!

      Dotan

  2. neuralcn says: December 16, 2014

    In "Gotchas and Pitfalls",what the mean of "Send Requests that were posted after them"?
    If that mean:when put several Send Requests with Unsignaled completions,then need put one Send Request and wait signal?

    • Dotan Barak says: December 17, 2014

      "Send Requests that were posted after them" means that if you post an Unsignaled Send Request, the first Send Request which is signaled
      (thus posted after them).

      Is it more clear now?

      Thanks
      Dotan

      • neuralcn says: December 18, 2014

        Thanks for your response,but I'm still confuse.Does it mean once the Signaled Send Request is finished,the previous Unsignaled Send Requests will also be finished?

      • Dotan Barak says: December 18, 2014

        Yes.

        Otherwise, you won't be able to know if they ended or not..

        Dotan

  3. Erfan says: February 17, 2015

    Hello Dotan,

    You wrote: "All posted Send Requested, Signaled and Unsignaled, are considered outstanding until a Work Completion that they, or Send Requests that were posted after them, was polled from the Completion Queue associated with the Send Queue". Now, let's say both send_cq and recv_cq are assigned to the same CQ when the QP was being created. I have this code snippet in my program:
    while (true) {
    RECEIVE (.....);
    unsignaled_SEND (...);
    poll_completion (....); // completion for RECEIVE
    }

    Basically, the completion event is only polled for RECEIVE. However, this code will create completion overflow. I don't understand why.

    • Erfan says: February 17, 2015

      Sorry, forgot to mention two things in my last comment:
      1- poll_completion(...) checks for the success of the wc_status. It's always successful.
      2- this machine that this code is run on, only RECEIVEs when it's SEND reaches the destination. Basically it's a client-server scheme. Before firing the SEND to the remote machine, I post the RECEIVE job request on the QP to be ready for the incoming message from the other side once it receives my SEND and responds to it.

      • Dotan Barak says: February 17, 2015

        Hi Efran.

        What do you mean by "completion overflow". What are the symptoms that you experience?

        Thanks
        Dotan

  4. Suveer says: May 29, 2015

    Hello Dotan, first of thanks a lot for all your posts. They help understand IB verbs better than any document.

    In your post you mention "All posted Send Requested, Signaled and Unsignaled, are considered outstanding until .....". Does this mean even unsignaled work requests on the send queue, a certain resource gets allocated that gets freed only upon querying a work completion of a subsequent signaled work request ?

    • Dotan Barak says: May 29, 2015

      Thanks for the feedback.

      The answer is short: Yes.
      Without a Completion, the SW doesn't know what is the status of the outstanding Work Request(s).

      Thanks
      Dotan

  5. Lingyu Zhu says: April 14, 2016

    Hi Dotan. I'm stuck with ENOMEN caused by unsignaled SQs for a day. Thanks to your blog! It really save lives!
    By the way, how do you figure out the way to solve this problem? Did you read the verbs library code?

    • Dotan Barak says: April 19, 2016

      Thanks for the feedback
      :)

      Can you describe what is the problem?
      (if you need my help, I just need more info)

      From the verbs library code, I can learn the return value;
      however, it doesn't describe the flow - the flow itself is described in the IB spec.

      Thanks
      Dotan

  6. Steeven says: September 9, 2016

    Just curious why not auto-send when the work queue reach to a limit?

    • Dotan Barak says: September 16, 2016

      Hi.

      Who do you expect to make the bookkeeping the driver or the HW itself?
      I believe that the idea was that the developer knows better how to utilize the technology...
      (and maybe it is just a minor hole in the architecture ...)

      Thanks
      Dotan

Add a Comment

This comment will be moderated; answer may be provided within 14 days.

Time limit is exhausted. Please reload CAPTCHA.