Skip to content

QP state machine

Contents

4.43 avg. rating (89% score) - 14 votes

Since a QP is one of the important objects in of RDMA, I will handle it in the next few posts.

QP State transitions

In a QP lifetime, the possible states can be:

  • Reset
  • Initialize (Init)
  • Ready To Receive (RTR)
  • Ready To Send (RTS)
  • Send Queue Drained (SQD)
  • Send Queue Error (SQE)
  • Error

A QP can be transitioned from one state to another state by two possible ways:

  1. Explicit call to ibv_modify_qp()
  2. Automatic transition by the device in case of a processing error

A QP is being created in the Reset state.

Any QP can be transitioned to Reset or Error state by calling ibv_modify_qp() from any state.

Reset state

Description

A QP is being created in the Reset state. In this state, all the needed resources of the QP are already allocated.

In order to reuse a QP, it can be transitioned to Reset state from any state by calling to ibv_modify_qp(). If prior to this state transition, there were any Work Requests or completions in the send or receive queues of that QP, they will be cleared from the queues.

Work Requests

Work Requests shouldn't be posted to either send or receive queues. Trying to do so will result in an immediate error.

Work Requests in both send and receive queues won't be processed.

Packets

Incoming packets which are targeted to this QP will be silently dropped.

No packets will be sent from this QP.

Init state

Description

In order to move to this state, one must call ibv_modify_qp() from Reset state.

Work Requests

Work Requests may be posted only to the receive queues. Trying to post Work Requests to the send queue will result in an immediate error.

Work Requests in both send and receive queues won't be processed.

Packets

Incoming packets which are targeted to this QP will be silently dropped.

No packets will be sent from this QP.

RTR state

Description

In order to move to this state, one must call ibv_modify_qp() from the Init state.

In RTR state, the QP may be used only as a responder.

Work Requests

Work Requests can be posted only to the receive queues. Trying to post Work Requests to the send queue will result in an immediate error.

Only Work Requests in the receive queue will be processed.

Packets

Incoming packets to this QP will be served. The QP will send packets from this QP as a responder (i.e: data, acks and nacks).

RTS state

Description

In order to move to this state, one must call ibv_modify_qp() from  RTR, or SQD states.

In RTS state, the QP may be used as a requester or as a responder.

Work Requests

Work Requests can be posted to both send and receive queues.

Work Requests in both send and receive queues will be processed.

Packets

Incoming packets to this QP will be served. The QP will send packets from this QP as a responder (i.e: data, acks and nacks).

The QP may initiate packets as a requester.

SQD state

In order to move to this state, one must call ibv_modify_qp() from  the RTS state.

This state affects only the send queue:

Work Requests that their processing have started will be handled until fully completed. However, new Work Requested won't be processed.

This state is separated to two internal states:

Draining - there are still Work Requests in the send queue that are being handled

Drained - there aren't any Work Requests in the send queue that are being processed

One can the state of that QP to RTS or SQD only if the internal state of the QP is Drained.

Work Requests

Work Requests can be posted to both send and receive queues.

Work Requests in the receive queue will be processed.

Work Requests in the send queue that their processing have started will be completed, new Work Requests won't be processed.

Packet

Incoming packets to this QP will be served. The QP will send packets from this QP as a responder (i.e: data, acks and nacks).

The QP may initiate packets as a requester.

SQE state

Description

Transition to this state will happen automatically by the device for all QP types, except for RC QPs, that were in the RTS state and processed a Work Request in the Send Queue that ended with completion error. The status of this Work Request is undefined (it is unknown if the Work Request has been partially or fully handled) and the content of the local or the remote buffers (depends on the used opcode) is undefined.

All subsequent Work Request in the send queue will be flushed with error.

One may call ibv_modify_qp() to move back to the RTS state to recover the Send Queue.

Work Requests

Work Requests can be posted to both send and receive queues.

Only Work Requests in the receive queue will be processed.

Work Requests to the send queue will be flushed with error.

Packets

Incoming packets to this QP will be served.

The QP will send packets from this QP as a responder (i.e: data, acks and nacks).

Error state

Description

This is the last state in the state machine of a QP.

A QP may be transitioned to this state automatically by the device, if a Work Request in the send queue was completed with error for an RC QP or if a Work Request in the receive queue was completed with error for any QP type, or by explicitly calling ibv_modify_qp() from any state.

If the QP state was transitioned automatically, the status of this Work Request is undefined (it is unknown if the Work Request has been partially or fully handled) and the content of the local or the remote buffers (depends on the used opcode) is undefined.

Work Requests

Work Requests can be posted in both send and receive queues.

If the QP was transitioned to this state automatically, the first Work Request that completed with error will indicate the reason for the error. All subsequent Work Requests in that queue and all Work Request in the other queue and new Work Requests posted to both of the queues will be flushed with error.

If the QP was transitioned to this state using ibv_modify_qp() all outstanding Work Request in both send and receive requests will be flushed with error.

Packet

Incoming packets which are targeted to this QP will be silently dropped.

No packets will be sent from this QP.

Summary

In this post, we covered the various states of a QP and the behavior of that QP.

The following table summarizes the behavior of a QP depends on its state:

Reset Init RTR RTS SQD SQE Error
Post receive request Disallowed Allowed Allowed Allowed Allowed Allowed Allowed
Post send request Disallowed Disallowed Disallowed Allowed Allowed Allowed Allowed
Receive Request processing Not processed Not processed Processed Processed Processed Processed Flushed with error
Send request processing Not processed Not processed Not processed Processed New WRs aren't processed Flushed with error Flushed with error
Incoming packets Silently dropped Silently dropped Handled Handled Handled Handled Silently dropped
Outgoing packets None None None Initiated Initiated None None

In the next post I will describe how to use the various QP states.

Share Our Posts

Share this post through social bookmarks.

  • Delicious
  • Digg
  • Newsvine
  • RSS
  • StumbleUpon
  • Technorati

Comments

Tell us what do you think.

  1. Elena says: July 8, 2015

    Hi Dotan,
    First of all i want to tell you that your blog is just amazing.
    second i have a question :)
    In hillard-iwarp-verbs-1.0 spec there are the following QP states:
    idle, RTS, closing, terminate and error.

    how this state machine correlates with IB QP state machine
    and how this is reflected by modify_qp verbs ?

    • Dotan Barak says: July 8, 2015

      Hi Elena.

      Thanks for the feedback
      :)

      Well, I'll try to answer from my understanding of this document
      (maybe I'm wrong here; I'm not an iWARP specialist).

      It is hard to compare InfiniBand and iWARP since we don't compare apples to apples,
      But let me try to answer:

      In iWARP, in the idle state SRs and RRs can be posted without processing,
      but in InfiniBand only RRs can be posted without processing in INIT state.
      But a QP start in the idle state.

      So idle = Both ~Reset and Init state

      There isn't any RTR state in iWARP.

      RTS state is the same for both protocols.

      iWARPs terminate state is ~ SQD state in InfiniBand.

      Both Error and Closing state are similar to the InfiniBand's Error state.

      Since the used verbs are InfiniBand verbs,
      I would have suggest you to follow the InfiniBand QP's state machine.

      Thanks
      Dotan

  2. Elena says: July 9, 2015

    Dotan thanks you for prompt response.
    I am new in IB and starting to work with iWarp, and wanted to see (in code preferably) how OFED generic library handles differences between IB and iWarp

  3. Elena says: July 9, 2015

    Hi Dotan, thanks for the prompt response.

    However i want to clarify my question,
    if i am using iWarp device and what to implement by myself connection establishment: at some time i am going to call ibv_modify_qp() (i failed to find rdma_modify_qp verb), so the input parametersis alligned with IB QP states, that differs from iWarp QP states -- how can i overcome this problem ? (let's suppose i am not going to use librdmacm)

    • Dotan Barak says: July 13, 2015

      Hi Elena.

      I don't have any experience using iWARP
      (I always programmed over InfiniBand and RoCE) and I don't have any iWARP adapters available to "play" with.

      I don't know if you can connect both sides without librdmacm
      (all the examples of iWARP are written over this library).

      Sorry. If you have another answer, I'll be happy to write about it...

      Thanks
      Dotan

  4. Rui Carlos Gonçalves says: August 6, 2015

    Hi Dotan,

    according to this post, when the QP transitions to the Error state automatically, one work requests will provide the cause of the problem, and the others will be flushed. But is it possible to only get work completions with status IBV_WC_WR_FLUSH_ERR? (That is, no work completions with other error status.) If so, do you have any idea of what could cause the failure of the QP?

    Thanks,
    Rui

    • Dotan Barak says: August 9, 2015

      Hi Rui.

      Yes, this is possible.
      Try to check if there is an asynchronous event or a Work Completion with an error status in the remote side of the QP.

      Thanks
      Dotan

      • Rui Carlos Gonçalves says: August 11, 2015

        Thank you very much for your help.

        In the remote side everything is normal when these errors happen.

        I checked for async events, and there is indeed an RDMA_CM_EVENT_DISCONNECTED event when I get the flush errors, even though there was no explicit call to disconnect. Could this be caused by an error/failure in the lower level libraries (in my case, Soft-iWARP), or network?
        (I really can't find any problem at the application level, and the errors I get are giving me little clues about the possible causes...)

        Thanks again for your help!
        Rui

      • Dotan Barak says: August 14, 2015

        Hi.

        What did you try to perform?
        which opcode?
        Did you get the error on the first message?

        Thanks
        Dotan

      • Rui Carlos Gonçalves says: September 4, 2015

        Hi again,

        Sorry for not replying before, but it seems that a reply I posted before was lost.

        Anyways, I have 2 processes, each one with 2 threads performing IBV_WC_RDMA_WRITE operations, and a thread performing IBV_WC_SEND and IBV_WC_RECV.
        Typically, I'm able to successfully complete thousands of requests before the failure occurs (actually, the application often finishes without errors).

        Meanwhile, I had the opportunity to test the application with InfiniBand hardware, and apparently those kinds of errors do not occur anymore. So, I guess this is probably an issue with Soft-iWARP.

        Thanks!
        Rui

Add a Comment

This comment will be moderated; answer may be provided within 14 days.

Time limit is exhausted. Please reload CAPTCHA.