Sharing cuDNN context between processes

AI & Data Science Deep Learning (Training & Inference) cuDNN
1

I’m trying to verify whether it’s possible to share a cuDNN context between multiple processes. Code I’m using to try to do this looks something like this:

foo ()
{
    ObjectWithCUDNN n("someplugin.so");

    int pid = fork ();
    if (pid == 0) // child process
    {
        n.Init ();  // Calls cudnnCreate, lots of other stuff
        exit (0);   // Child process exits cleanly if Init didn't crash
    }
    else  // parent process
    {
        int wstat;
        wait (&wstat);   // Wait for child process to exit
        if (wstat != 0)
            n.SetIsSafeToUse (false);
    }

    ...
}

I get CUDNN_STATUS_NOT_INITIALIZED failures in the child process trying to call cudnnCreate(), and I have a feeling this is because what I’m doing isn’t allowed. The reason for this is to avoid having to handle CPU exceptions that a plug-in, which may use cuDNN, may cause.

2

Hi @jbaumgart ,
Can you please share the API logs with us which may help to debug further.

Thanks!

3

This issue was due to my misunderstanding of shared resources after calling fork(). It seems that cuDNN constructs are not available across process boundaries, as opaque system memory allocations done by cuDNN cannot be done using shared memory segments.