Skip to content

ibv_create_comp_channel()

4.33 avg. rating (88% score) - 3 votes
struct ibv_comp_channel *ibv_create_comp_channel(struct ibv_context *context);

Description

ibv_create_comp_channel() creates a Completion event channel for an RDMA device context.

This Completion event channel is an abstraction introduced by libibverbs that does not exist in the InfiniBand Architecture verbs specification or RDMA Protocol Verbs Specification. A Completion event channel is essentially file descriptor that is used to deliver Work Completion notifications to a userspace process. When a Work Completion event is generated for a Completion Queue (CQ), the event is delivered via the completion event channel attached to that CQ. This may be useful to steer completion events to different threads by using multiple Completion event channels or to give different priority to different CQs.

One or more Completion Queues can be associated with the same Completion event channel.

Parameters

Name Direction Description
context in RDMA device context that was returned from ibv_open_device()

Return Values

Value Description
Completion event channel A pointer to the newly allocated Completion event channel
NULL On failure, errno indicates the failure reason:

EMFILE Too many files are opened by this process
ENOMEM Not enough resources to complete this operation

Examples

Create a Completion event channel and destroy it:

struct ibv_comp_channel *event_channel;
 
event_channel = ibv_create_comp_channel(context);
if (!event_channel) {
	fprintf(stderr, "Error, ibv_create_comp_channel() failed\n");
	return -1;
}
 
if (ibv_destroy_comp_channel(event_channel)) {
	fprintf(stderr, "Error, ibv_destroy_comp_channel() failed\n");
	return -1;
}

FAQs

Why is a Completion event channel good for anyway?

Completion event channel is an object that helps handling Work Completions in a userspace process using event mode rather than polling mode.

Why would I want to read Work Completions in event mode?

Reading Work Completions in event mode is useful if your process doesn't need low latency or if you need to decrease the CPU usage; your process will go to sleep and will be awakened by the kernel when a new Work Completion will be available in a CQ, which is associated with that Completion event channel.

Can more than one CQ be associated with the same Completion event channel?

Yes. More than one CQ can be associated with the same Completion event channel. One can use several Completion event channels in order to "aggregate" different CQs. This may be useful to steer completion events to different threads by using multiple Completion event channels or to give different priority to different CQs.

How can I work with multiple Completion event channels in order to give different priority to different CQs?

You can associate all of the CQs that have the same priority with the same Completion event channel, and handle Work Completion events according to the priority of the CQs group.

Share Our Posts

Share this post through social bookmarks.

  • Delicious
  • Digg
  • Newsvine
  • RSS
  • StumbleUpon
  • Technorati

Comments

Tell us what do you think.

  1. Jingcha says: September 18, 2013

    Can you please explain what you mean by "You can associate all of the CQs that have the same priority with the same Completion event channel, and handle Work Completion events according to the priority of the CQs group."

    Thanks,

    • Dotan Barak says: September 19, 2013

      Hi.

      I'll try to explain what I meant here:
      Let's assume that you have several CQs: CQx1..CQxN and CQy (CQy has higher priority),
      and your application needs to consume as less as CPU as possible
      (i.e. work with events and not with polling when reading Work Completions).

      Instead of using one Completion Event channel, you can have two of them,
      one for the CQx QPs and one for the CQy, each of them will be handled in a different thread.

      This way, as soon as CQy has an event, it will be handled
      (i.e. it gets higher priority than the other CQs).

      If your RDMA device supports multiple Completion vectors,
      CQy may get a different Completion vector than the other,
      and by doing this, get a better performance.

      Instead of using one CQy, you can have CQy1..CQyM but the idea is the same.

      I hope that I was clear with this explanation.

      Thanks
      Dotan

  2. Omar says: April 5, 2014

    Dear Dotan

    I want to associate different completion channel to each CQ i create. I have different threads waiting for notification from the CQ assigned to them. I think if i associate the same completion channel to all my CQ's and messages arrive at all of them simultaneously, i get a big performance hit. Should this happen? Can you share some code snippet where you are creating a new completion channel for each CQ. The code i use is
    "cb->channel = ibv_create_comp_channel(cm_id->verbs);
    cb->cq[SEND_CQ_INDEX] = ibv_create_cq(cm_id->verbs, cqe, cb, cb->channel, 0);
    cb->cq[RECV_CQ_INDEX] = ibv_create_cq(cm_id->verbs, cqe, cb, cb->channel, 0);"

    Since the context is the same, for each queue I suppose even if i call ibv_create_comp_channel() for each cq, they are shared. Also i am using the same channel for send/receive CQ. Will this cause a performance drop (due to some internal Locking mechanism).
    I hope i have made my point clear. To summarize: I want all my CQ's to receive notification from the completion channel independent of each other. I have a large number of CQ's.

    Warm regards

    Omar Khan

    • Dotan Barak says: April 6, 2014

      Hi Omar.

      I must admit that I never did it and never checked the performance different.
      But you need to create Completion channels and create every group of CQs with different
      Completion channels.

      IMHO, using multiple completion vectors may provide even better performance improvement than using the same completion vector in all the CQs.

      Thanks
      Dotan

  3. Omar Khan says: April 6, 2014

    Dear dotan

    How do you know that your device supports multiple completion vectors? and what does it mean multiple completion vectors? if my device does not support multiple completion vectors, how can i set up a communication channel between multiple processes and have them communicate independently. can i have multiple threads using independent completion channels to notify completions from different processes?

    Warm regards
    Omar

    • Dotan Barak says: April 6, 2014

      Hi Omar.

      struct ibv_context contains an attribute called: num_comp_vectors.
      This value specify the number of completion vectors which the RDMA device supports.

      I'm not consider my self an expert in this, but I can try to explain *my* rational why using multiple completion vectors improves the performance:
      Using several completion vectors means that the interrupts will be spread across several vectors (which means that several cores in your system will handle them) and not only one vector.
      I'm sure that you can find information in the internet about this..

      If your device supports only one completion vector, I would try to set the affinity of the processes, that each one will use different core.

      I hope that this answer helped you.

      Thanks
      Dotan

  4. Tingyu says: July 7, 2015

    Hi Dotan,

    Can I use the same event channel for both accepting connection and creating connection? In this way, a program can connect to itself, and the server-side event such as RDMA_CM_EVENT_CONNECT_REQUEST and client-side event such as RDMA_CM_EVENT_ADDR_RESOLVED can be polled in the same loop. It is possible?

    Thanks,
    Tingyu

    • Dotan Barak says: July 13, 2015

      Hi Tingyu.

      The event channel is a mechanism for handle the events;
      it doesn't have any notion of the side it serves.

      For my understanding, the answer for you question is "yes".
      It should be possible to use the same event channel for both accepting and initiating connections.

      Thanks
      Dotan

  5. Lingyan says: January 25, 2017

    Hi Dotan,
    Could multiple threads share one completion event channel associated with multiple CQs?
    Thanks,
    Lingyan

    • Dotan Barak says: February 13, 2017

      Yes.

      Multiple threads can share one completion event channel associated with multiple CQs,
      it will be hard to predict though which thread will get the event.

      Thanks
      Dotan

Add a Comment

Fill in the form and submit.

Time limit is exhausted. Please reload CAPTCHA.