Skip to content

ibv_req_notify_cq()

5.00 avg. rating (98% score) - 3 votes
int ibv_req_notify_cq(struct ibv_cq *cq, int solicited_only);

Description

ibv_req_notify_cq() requests a Completion Notification on a Completion Queue (CQ).

ibv_req_notify_cq() requests a notification when the next Work Completion of a requested type is added to the CQ. Any Work Completions that existed in the CQ before calling ibv_req_notify_cq() will not result in creating a Completion Notification. The Completion Notification will be read using ibv_get_cq_event().

There are two types of Completion Events that can be requested:

  • Solicited Completion Event - Occurs when an incoming Send or RDMA Write with Immediate Data message with the Solicited Event indicator set (i.e. the remote side posted a Send Request with IBV_SEND_SOLICITED set in send_flags) causes a generation of a successful Receive Work Completion or any unsuccessful (either Send of Receive) Work Completion to be added to the CQ.
  • Unsolicited Completion Event - occurs when any Work Completion is added to the CQ, whether it is a Send or Receive Work Completion and whether is it successful or unsuccessful Work Completion.

If a Request Completion Notification is pending, i.e. ibv_req_notify_cq() was called and no Completion Notification occurred, subsequent calls to ibv_req_notify_cq() with the same CQ requesting the same Completion Event type will have no effect; only one Completion Notification will be generated. Calling ibv_req_notify_cq() will have an effect only after the Completion Notification will occur.

A Request Completion Notification for the next completion event takes precedence over a Request Completion Notification for a solicited event completion for the same CQ.

If multiple calls to ibv_req_notify_cq() have been made for the same CQ and at least one of the requests set the type to the next completion, a Completion Notification will be generated when the next Completion is added to that CQ (i.e. not for the next Solicited Completion).

Once a Completion Notification occurred, if one wishes to get more Completions Notification, he has to call ibv_req_notify_cq() again.

Parameters

Name Direction Description
cq in CQ that was returned from ibv_create_cq()
solicited_only in Type of the requested Completion Notification.

0 The next Completion whether it is Solicited or Unsolicited
otherwise the next Completion is Solicited only or for any unsuccessful Work Completion

Return Values

Value Description
0 On success
errno On failure.
EINVAL Invalid CQ handle

Examples

1) Request a notification for the next Completion:

struct ibv_cq *cq;
 
if (ibv_req_notify_cq(cq, 0)) {
	fprintf(stderr, "Error, ibv_req_notify_cq() failed when requested a "
		"notification for the next completion\n");
	return -1;
}

2) Request a notification for the next Solicited Completion:

struct ibv_cq *cq;
 
if (ibv_req_notify_cq(cq, 1)) {
	fprintf(stderr, "Error, ibv_req_notify_cq() failed when requested a "
		"notification for the next solicited completion\n");
	return -1;
}

FAQs

I can read Work Completions with ibv_poll_cq(), why would I want to use ibv_req_notify_cq()?

Handling Work Completions with events will decrease the CPU consumption of your application; your process will sleep until a new Work Completion will be added to the CQ.

Can I call ibv_req_notify_cq() with every CQ?

Yes. Although keep in mind that requesting for a notification on Solicited Completions has a meaning only for CQ that is associated with a Receive Queue.

I called ibv_req_notify_cq() when there is a Work Completion in the CQ (of the type that I've asked) will I get a Completion Notification?

No, you won't. A Completion Notification will be generated for the *next* Completion of the type that you've asked (Solicited or next) that will be added to the CQ *after* you called ibv_req_notify_cq().

I called ibv_req_notify_cq() several times before I got a Completion Notification, what is the effect of this?

If all calls to ibv_req_notify_cq() were done when requesting the same Completion type, you will get only one Completion Notification, according to the quested type. If at least one of them requested a Completion Notification for the next Completion, you will get a notification for the next Completion.

Can I ask to get a notification after N Completions will be added to the CQ

No, this isn't supported.

Can I ask to get a notification for every Completion that will add to the CQ automatically (instead of calling ibv_req_notify_cq() again each time)?

No, this isn't supported. After you got a notification, you need to call ibv_req_notify_cq() again if you want to get another notification.

Share Our Posts

Share this post through social bookmarks.

  • Delicious
  • Digg
  • Newsvine
  • RSS
  • StumbleUpon
  • Technorati

Comments

Tell us what do you think.

  1. Palmer Bachas says: April 17, 2013

    Just wish to say your article is as amazing. The clarity in your post is just excellent and i could assume you're an expert on this subject. Fine with your permission let me to grab your feed to keep up to date with forthcoming post. Thanks a million and please continue the rewarding work. Palmer Bachas

    • Dotan Barak says: April 18, 2013

      Hi.

      I hope that this isn't a spam...
      But if this is true, thank you

      :)
      Dotan

  2. Kaushal says: May 25, 2015

    Hi Dotan,

    I have a few newbie doubts. Can you please explain me the difference between ibv_create_comp_channel() and ibv_req_notify_cq()?
    Also, I understand that ibv_get_cq_event() will be required to read the notification event but which of the functions cause the thread to sleep, is it ibv_get_cq_event() or ibv_req_notify_cq()?

    Thank you very much,
    Regards,
    Kaushal

    • Dotan Barak says: May 26, 2015

      Sure.

      ibv_create_comp_channel() creates the object (Completion Event Channel) that will be used to pass the completion events from kernel to user.

      ibv_get_cq_event() reads a completion Event from a Completion Event Channel and returns the attribute of the Completion Queue that got this event.

      ibv_get_cq_event() is a blocking call by default (however, this behavior can be changed).
      ibv_req_notify_cq() only mark that Completion Event is required for a specific Completion Queue.

      I hope that I answered...
      Dotan

  3. Zhang Yue says: July 29, 2016

    hi Dotan

    I am using SEND/RECV, the problem I face is:
    I post 10 send quickly. but at the receive side,
    only 2 recv CQE will be notified and polled out by the code:( the code is armed to a epoll())

    if (ibv_get_cq_event(rc->comp_channel,&evt_cq, &cq_context))
    {
    printf("ibv get cq event fail, errno:%d\n", errno);
    perror(NULL);
    return 1;
    }
    ibv_ack_cq_events(evt_cq, 1);
    if (ibv_req_notify_cq(evt_cq, 0))
    return 3;
    (void)rcDirectPollCq(rc);

    but after that,if I directly poll CQ ( without ibv_get_cq_event() ), I can poll the rest 8 CQE out.

    How can I make all of them trigger the notify?
    Hope some suggest from you.

    Thanks

    Zhang Yue

    • Dotan Barak says: July 29, 2016

      Hi Zhang Yue.

      I can't see your code,
      but I suspect that you have a race and you wait for new events to happen AFTER all the CQEs were arrived.

      If you can share the code with me, maybe I'll be able to help you a little bit more.

      Thanks
      Dotan

      • Zhang Yue says: August 1, 2016

        Hi Dotan

        Thanks for your reply.
        I fixed it.
        The reason is my polling CQ code did not poll all the recv CQE out, it return early.
        I modiry it and make it poll all the CQE out, and test is OK.

        Thanks any way.

        Zhang Yue.

Add a Comment

Fill in the form and submit.

Time limit is exhausted. Please reload CAPTCHA.