Skip to content

ibv_create_comp_channel()

Contents

4.71 avg. rating (94% score) - 7 votes
struct ibv_comp_channel *ibv_create_comp_channel(struct ibv_context *context);

Description

ibv_create_comp_channel() creates a Completion event channel for an RDMA device context.

This Completion event channel is an abstraction introduced by libibverbs that does not exist in the InfiniBand Architecture verbs specification or RDMA Protocol Verbs Specification. A Completion event channel is essentially file descriptor that is used to deliver Work Completion notifications to a userspace process. When a Work Completion event is generated for a Completion Queue (CQ), the event is delivered via the completion event channel attached to that CQ. This may be useful to steer completion events to different threads by using multiple Completion event channels or to give different priority to different CQs.

One or more Completion Queues can be associated with the same Completion event channel.

Parameters

Name Direction Description
context in RDMA device context that was returned from ibv_open_device()

Return Values

Value Description
Completion event channel A pointer to the newly allocated Completion event channel
NULL On failure, errno indicates the failure reason:

EMFILE Too many files are opened by this process
ENOMEM Not enough resources to complete this operation

Examples

Create a Completion event channel and destroy it:

struct ibv_comp_channel *event_channel;
 
event_channel = ibv_create_comp_channel(context);
if (!event_channel) {
	fprintf(stderr, "Error, ibv_create_comp_channel() failed\n");
	return -1;
}
 
if (ibv_destroy_comp_channel(event_channel)) {
	fprintf(stderr, "Error, ibv_destroy_comp_channel() failed\n");
	return -1;
}

FAQs

Why is a Completion event channel good for anyway?

Completion event channel is an object that helps handling Work Completions in a userspace process using event mode rather than polling mode.

Why would I want to read Work Completions in event mode?

Reading Work Completions in event mode is useful if your process doesn't need low latency or if you need to decrease the CPU usage; your process will go to sleep and will be awakened by the kernel when a new Work Completion will be available in a CQ, which is associated with that Completion event channel.

Can more than one CQ be associated with the same Completion event channel?

Yes. More than one CQ can be associated with the same Completion event channel. One can use several Completion event channels in order to "aggregate" different CQs. This may be useful to steer completion events to different threads by using multiple Completion event channels or to give different priority to different CQs.

How can I work with multiple Completion event channels in order to give different priority to different CQs?

You can associate all of the CQs that have the same priority with the same Completion event channel, and handle Work Completion events according to the priority of the CQs group.

Share Our Posts

Share this post through social bookmarks.

  • Delicious
  • Digg
  • Newsvine
  • RSS
  • StumbleUpon
  • Technorati

Comments

Tell us what do you think.

  1. Jingcha says: September 18, 2013

    Can you please explain what you mean by "You can associate all of the CQs that have the same priority with the same Completion event channel, and handle Work Completion events according to the priority of the CQs group."

    Thanks,

    • Dotan Barak says: September 19, 2013

      Hi.

      I'll try to explain what I meant here:
      Let's assume that you have several CQs: CQx1..CQxN and CQy (CQy has higher priority),
      and your application needs to consume as less as CPU as possible
      (i.e. work with events and not with polling when reading Work Completions).

      Instead of using one Completion Event channel, you can have two of them,
      one for the CQx QPs and one for the CQy, each of them will be handled in a different thread.

      This way, as soon as CQy has an event, it will be handled
      (i.e. it gets higher priority than the other CQs).

      If your RDMA device supports multiple Completion vectors,
      CQy may get a different Completion vector than the other,
      and by doing this, get a better performance.

      Instead of using one CQy, you can have CQy1..CQyM but the idea is the same.

      I hope that I was clear with this explanation.

      Thanks
      Dotan

  2. Omar says: April 5, 2014

    Dear Dotan

    I want to associate different completion channel to each CQ i create. I have different threads waiting for notification from the CQ assigned to them. I think if i associate the same completion channel to all my CQ's and messages arrive at all of them simultaneously, i get a big performance hit. Should this happen? Can you share some code snippet where you are creating a new completion channel for each CQ. The code i use is
    "cb->channel = ibv_create_comp_channel(cm_id->verbs);
    cb->cq[SEND_CQ_INDEX] = ibv_create_cq(cm_id->verbs, cqe, cb, cb->channel, 0);
    cb->cq[RECV_CQ_INDEX] = ibv_create_cq(cm_id->verbs, cqe, cb, cb->channel, 0);"

    Since the context is the same, for each queue I suppose even if i call ibv_create_comp_channel() for each cq, they are shared. Also i am using the same channel for send/receive CQ. Will this cause a performance drop (due to some internal Locking mechanism).
    I hope i have made my point clear. To summarize: I want all my CQ's to receive notification from the completion channel independent of each other. I have a large number of CQ's.

    Warm regards

    Omar Khan

    • Dotan Barak says: April 6, 2014

      Hi Omar.

      I must admit that I never did it and never checked the performance difference.
      But you need to create Completion channels and create every group of CQs with different
      Completion channels.

      IMHO, using multiple completion vectors may provide even better performance improvement than using the same completion vector in all the CQs.

      Thanks
      Dotan

  3. Omar Khan says: April 6, 2014

    Dear dotan

    How do you know that your device supports multiple completion vectors? and what does it mean multiple completion vectors? if my device does not support multiple completion vectors, how can i set up a communication channel between multiple processes and have them communicate independently. can i have multiple threads using independent completion channels to notify completions from different processes?

    Warm regards
    Omar

    • Dotan Barak says: April 6, 2014

      Hi Omar.

      struct ibv_context contains an attribute called: num_comp_vectors.
      This value specify the number of completion vectors which the RDMA device supports.

      I'm not consider my self an expert in this, but I can try to explain *my* rational why using multiple completion vectors improves the performance:
      Using several completion vectors means that the interrupts will be spread across several vectors (which means that several cores in your system will handle them) and not only one vector.
      I'm sure that you can find information in the internet about this..

      If your device supports only one completion vector, I would try to set the affinity of the processes, that each one will use different core.

      I hope that this answer helped you.

      Thanks
      Dotan

  4. Tingyu says: July 7, 2015

    Hi Dotan,

    Can I use the same event channel for both accepting connection and creating connection? In this way, a program can connect to itself, and the server-side event such as RDMA_CM_EVENT_CONNECT_REQUEST and client-side event such as RDMA_CM_EVENT_ADDR_RESOLVED can be polled in the same loop. It is possible?

    Thanks,
    Tingyu

    • Dotan Barak says: July 13, 2015

      Hi Tingyu.

      The event channel is a mechanism for handle the events;
      it doesn't have any notion of the side it serves.

      For my understanding, the answer for you question is "yes".
      It should be possible to use the same event channel for both accepting and initiating connections.

      Thanks
      Dotan

  5. Lingyan says: January 25, 2017

    Hi Dotan,
    Could multiple threads share one completion event channel associated with multiple CQs?
    Thanks,
    Lingyan

    • Dotan Barak says: February 13, 2017

      Yes.

      Multiple threads can share one completion event channel associated with multiple CQs,
      it will be hard to predict though which thread will get the event.

      Thanks
      Dotan

  6. HuaiEn says: August 6, 2019

    Hi Dotan,
    I saw your reply above
    https://www.rdmamojo.com/2012/10/19/ibv_create_comp_channel/?replytocom=701#reply
    in your reply, you said "But you need to create Completion channels and create every group of CQs with different Completion channels."
    What does every group of CQs mean?
    As my realize, a completion channel is better to match to each CQ, or it is hard to know which CQ will receive event.
    Do I miss something?
    thanks

    BR,
    HuaiEn

    • Dotan Barak says: August 16, 2019

      Hi.

      "Every group of CQs" means that if for example you create 100 CQs and you want to use 2 completion channels,
      you can attach the first 50 CQs to the first completion channel and the last 50 CQs to the second completion channel.

      When you get a CQ event, you get the CQ handle - so it is easy to know which CQ received the event.
      However, depend on your flow, maybe you won't be able to predict ahead which CQ will get the event.

      Is it more clear now?

      Thanks
      Dotan

  7. HuaiEn says: August 18, 2019

    Hi Dotan,
    Thanks for clearly reply.
    There is another question.
    When do we need multiple cq?
    Thanks a lot

    • Dotan Barak says: August 21, 2019

      Hi.

      There may be several reasons for using multiple CQs, here are some reasons that I just thought about:
      * Using *many* QPs and the number of generated completions may cause a CQ overrun to one CQ
      * If you want to provide different QoS (application-wise) to different QPs
      * If you want to provide different functionality to different QPs
      * Separating Send and Receive Queues
      * more?

      It is up to the application to decide this, it isn't a technology decision ..

      Thanks
      Dotan

Add a Comment

This comment will be moderated; answer may be provided within 14 days.

Time limit is exhausted. Please reload CAPTCHA.