Limitation in number of concurrent CUDA contexts on AGX Orin

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
1

We seem to hit a limitation on AGX Orin in the number of concurrent CUDA contexts that we can create. The following simple program runs N copies of itself, each of which calls cudaFree(0) to create a cuda context, and prints the result:

#include <iostream>
#include <thread>
#include <unistd.h>

int main(int argc, char ** argv)
{
    if (argc != 2)
    {
        std::cout << "Not enough arguments" << std::endl;
        return 1;
    }

    int n    = atoi(argv[1]);
    int i    = 1;
    int myid = 0;

    while (i < n)
    {
        std::cout << "Forking child " << i << std::endl;
        int res = fork();

        if (res == 0)
        {
            // child process
            myid = i;

            break;
        }
        else
        {
            // parent
            i++;
        }
    }

    std::this_thread::sleep_for(std::chrono::milliseconds(myid * 100));

    std::cout << "Instance " << myid << " about to call cudaFree" << std::endl;
    auto err = cudaFree(0);

    if (err != cudaSuccess)
    {
        std::cout << "Instance " << myid << ": cudaFree failed with " << cudaGetErrorString(err) << std::endl;
        return 1;
    }

    while (true)
    {
        std::this_thread::sleep_for(std::chrono::seconds(1));
    }

    return 0;
}

On AGX Orin, we can only run 32 instances - the 33rd one fails with “CUDA-capable device(s) is/are busy or unavailable”. At this point, we also see in dmesg:

[ 9846.508948] nvgpu: 17000000.gpu nvgpu_channel_open_new:1286 [ERR] out of hw chids
[ 9846.508968] nvgpu: 17000000.gpu __gk20a_channel_open:531 [ERR] failed to get f

In our system solution, we rely on being able run many instances of a video application, each of which has a cuda context. On TX2 we were able to comfortably run 48 instances of our program, but on Orin, the above error severly limits us. For comparison, we can run the above test with at least 80 instances without problem on TX2.

Is it possible to increase the number of allowed CUDA contexts?

4

Hi,

Try to reduce the CUDA_DEVICE_MAX_CONNECTIONS.
For example, we can launch 32 concurrent contexts after reducing the value to 1:

$ export CUDA_DEVICE_MAX_CONNECTIONS=1

The environment variable controls the maximum number of work queues from the different CUDA streams.
With the same hardware channel amount, reducing the value allows you to use more streams.

Please find the detailed explanation in our document below:

Thanks.

6

Thanks for your response. Can you just confirm that it is impossible to increase the number of channels on Orin?

In the meantime we have also tested this on an Orin Nano, and also there we can only start 32 programs that use CUDA.

8

Hi,

The maximal concurrent CUDA processes/contexts on Orin is 32.
But you can create more CUDA streams via multi-threads.

Thanks.