Skip to content

ibv_fork_init()

5.00 avg. rating (97% score) - 1 vote
int ibv_fork_init(void);

Prerequisite

ibv_fork_init() should be called before calling any other function in libibverbs.

Description

ibv_fork_init() initializes libibverbs's data structures to handle the fork() function calls correctly and avoid data corruption, whether fork() is called explicitly or implicitly (such as in system(), popen(), etc.).

The fork() support isn't full and it assumes that the parent process will do the RDMA operations. If one will try to do RDMA operations from any child process, he may experience various problems.

It is not necessary to use this function if all parent process threads are always blocked until all child processes end or change address spaces via an exec() operation.

If fork() will be used without calling ibv_fork_init(), one may experience data corruption, segmentation fault, missing Work Request completions or any other phenomena.

ibv_fork_init() works on Linux kernels supporting the MADV_DONTFORK
flag for madvise() (2.6.17 and higher).

Calling ibv_fork_init() will reduce performance due to an extra system call for every memory registration, and the extra memory allocated to track memory regions. The precise performance impact depends on the workload and usually will not be significant.

Setting one of the environment variables RDMAV_FORK_SAFE or IBV_FORK_SAFE to any value (libibverbs only check if one of the environment variables was defined and completely ignores the value)  has the same effect as calling ibv_fork_init().

Setting the environment variable RDMAV_HUGEPAGES_SAFE tells the library to check the underlying page size used by the kernel for memory regions. This is required if an application uses huge pages either directly or indirectly via a library such as libhugetlbfs.

There isn't any function call that has the same effect as setting the environment variable RDMAV_HUGEPAGES_SAFE.

Setting RDMAV_HUGEPAGES_SAFE adds further overhead to all memory registrations.

Parameters

None

Return Values

Value Description
0 On success
errno On failure
EINVAL Function was called too late
ENOMEM Insufficient memory to complete the operation
ENOSYS No kernel support for RDMA

Examples

Setting environment variable for fork() protection (in bash), all the following lines have the same effect:

# export RDMAV_FORK_SAFE
# export RDMAV_FORK_SAFE=0
# export RDMAV_FORK_SAFE=1
# export RDMAV_FORK_SAFE="no"
# export RDMAV_FORK_SAFE="yes"
# export IBV_FORK_SAFE=0
# export IBV_FORK_SAFE=1
# export IBV_FORK_SAFE="no"

Setting environment variable for fork() protection when huge pages are used (in bash):

# export RDMAV_FORK_SAFE
# export RDMAV_HUGEPAGES_SAFE

Call function for fork() protection (in c source code):

int rc;
 
rc = ibv_fork_init()
if (rc)
        exit(rc);

FAQs

What will happen if I won't enable fork() support and I will still use fork()?

Bad things may happen, such as (but not limited to):

- Data corruption

- Missing Work Completions

- Segmentation fault

Does is matter if I call ibv_fork_init() or set the environment variable RDMAV_FORK_SAFE?

No, they care completely equivalent.

Does is matter which value I assign to the environment variable RDMAV_FORK_SAFE?

No, any value should do.

I'm using huge pages and I only enabled RDMAV_FORK_SAFE, will it fork for me?

No, any value should set the environment variable RDMAV_HUGEPAGES_SAFE too.

I don't know if I need fork() support, can I enable fork() support, just to be on the safe side?

Yes, you can. Just be aware that extra memory will be consumed when enabling fork support

I don't call explicitly to fork(), I only call to other system calls that may call fork(), do I still need to enable fork() support?

Yes, you need to enable fork() support, whether fork() is called directly or if any other system calls call fork() implicitly.

I only provide a library to my customer, and he calls fork(), does fork() support is still required?

Yes, fork() support needs to be enabled if the process calls fork(), it doesn't matter who calls it. If the user will sometimes use fork(), he can set the environment variable RDMAV_FORK_SAFE.

I called ibv_fork_init() and it failed, what does it mean?

It means that your kernel doesn't support madvise(), you should consult your support or update your kernel.

Can I inherit the RDMA resources after I call fork()?

The libibverbs has several file descriptor and after calling fork(), they still be available in the child process. However, it is highly recommended not to try to use them since it may cause bad things.

Share Our Posts

Share this post through social bookmarks.

  • Delicious
  • Digg
  • Newsvine
  • RSS
  • StumbleUpon
  • Technorati

Comments

Tell us what do you think.

There are no comments on this entry.

Add a Comment

Fill in the form and submit.

Time limit is exhausted. Please reload CAPTCHA.