What is Fence?
In the Send Queue there are operations that only send data from a local host to a remote one and there are operations that read data from a remote host and store it locally.
Sometimes, there is a need to read data from a remote host and send all/part of it back. Obviously, this should be done in two separated Send Requests. Some operations can begin processing before their previous ones were ended and this may lead to unexpected memory content to be sent. Furthermore, reading from a remote address and then writing to it (either directly using RDMA Write and Atomic or indirectly by Send) may lead to unexpected behavior.
There is a trivial solution for the above scenario: post one Send Request for the first operation, wait until it is completed (i.e. poll its Work Completion) and then post the second Send Request.
This solution will work, but it will consume more CPU time and will provide bad latency than posting those two Send Requests at once.
Luckily, RDMA provides us a mechanism to enforce an order of Send Request processing: Fence. When there is a Fence indicator on a Send Request, its processing won't begin until all prior RDMA Read and Atomic operations on the same Send Queue have completed. This is relevant only for reliable Queue Pair (since only them supports those operations).
Fence guarantees the processing order and Completion notification of Send Requests that were posted to the same Send Queue. The order between multiple Send Queues is undefined.
How to use Fence?
Fence can be used by setting the IBV_SEND_FENCE bit in the send_flags field of the Send Request and calling ibv_post_send().
When to use Fence?
As a rule of thumb, Fence should be used when RDMA Read or Atomic operation is followed a Send, RDMA Write, or Atomic operations and the first one can be outstanding.
Here is the description of what may happen when using/not-using Fence and how it affects the Send Requests processing order:
- RDMA Read which followed by Sends, RDMA Writes, or Atomics - not setting Fence may lead to a case where the RDMA Read response will contain data that was modified by the second operation.
Using Fence will make sure that the data that will be Read by the first operation will be the original (and expected) data.
- RDMA Read or Atomic operations, which followed by Sends, RDMA Writes, or Atomics - not setting Fence may lead to a case that if the first operation complete in error on the initiator side (because its ACK fail to return, local protection error when writing the data or any other reason) and the second operation could still be observed by the target, and it may even cause data to be written in the target's memory. Using Fence will prevent the second operation to be observed by the target if the first one fails.
Tell us what do you think.