Optix 7.5-8.0 fails inside docker but works on host

Gaming and Visualization Technologies Visualization OptiX
, ,
1

Hi all,

I am trying to run the ./optixHello demo within a docker container and am getting the following error.

[ 2][   DEVICECTX]: Specified 0 as the CUDA context, but the current CUDA context could not be queried. (CUDA error code: 34)
Caught exception: OPTIX_ERROR_CUDA_ERROR: Optix call 'optixDeviceContextCreate( cuCtx, &options, &context )' failed: /home/robot/Optix-8/SDK/optixHello/optixHello.cpp:126)

Some background info:
Host:
Ubuntu 22.04
nvidia-driver-545
cuda 12.3

Container:
Ubuntu 20.04
nvidia-driver-545
cuda 12.3

nvidia-smi and nvcc --version works within the container, and I can even run some of the cuda-samples. I have tried other applications within the container like gazebo and they utilized the gpu.

I can build and run optix samples for versions 7.5-8.0 on the host.

Here is the strace output for ./optixHello in the container

openat(AT_FDCWD, "/home/robot/Optix-8/SDK/build/lib/libnvoptix.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/robot/otter_ws/devel/lib/libnvoptix.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/opt/ros/noetic/lib/libnvoptix.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/local/cuda/lib64/stubs/libnvoptix.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/local/cuda/lib64/libnvoptix.so.1", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/lib/x86_64-linux-gnu/libnvoptix.so.1", O_RDONLY|O_CLOEXEC) = 32
read(32, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260\22\7\0\0\0\0\0"..., 832) = 832
fstat(32, {st_mode=S_IFREG|0644, st_size=60689696, ...}) = 0
mmap(NULL, 62997440, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 32, 0) = 0x7f0172200000
mprotect(0x7f0175867000, 2097152, PROT_NONE) = 0
mmap(0x7f0175a67000, 3645440, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 32, 0x3667000) = 0x7f0175a67000
mmap(0x7f0175de1000, 209856, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f0175de1000
close(32)                               = 0
mprotect(0x7f0175a67000, 3194880, PROT_READ) = 0
futex(0x7f0175e130dc, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x7f0175e130e8, FUTEX_WAKE_PRIVATE, 2147483647) = 0
futex(0x7f0175df99fc, FUTEX_WAKE_PRIVATE, 2147483647) = 0
sched_get_priority_max(SCHED_RR)        = 99
sched_get_priority_min(SCHED_RR)        = 1
openat(AT_FDCWD, "/home/robot/Optix-8/SDK/build/lib/libcuda.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/home/robot/otter_ws/devel/lib/libcuda.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/opt/ros/noetic/lib/libcuda.so", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/usr/local/cuda/lib64/stubs/libcuda.so", O_RDONLY|O_CLOEXEC) = 32
read(32, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\204\0\0\0\0\0\0"..., 832) = 832
fstat(32, {st_mode=S_IFREG|0644, st_size=66272, ...}) = 0
mmap(NULL, 2162688, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 32, 0) = 0x7f01a8250000
mprotect(0x7f01a8260000, 2093056, PROT_NONE) = 0
mmap(0x7f01a845f000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 32, 0xf000) = 0x7f01a845f000
close(32)                               = 0
mprotect(0x7f01a845f000, 4096, PROT_READ) = 0
write(2, "[", 1[)                        = 1
write(2, " 2", 2 2)                       = 2
write(2, "][", 2][)                       = 2
write(2, " ", 1 )                        = 1
write(2, " ", 1 )                        = 1
write(2, " ", 1 )                        = 1
write(2, "DEVICECTX", 9DEVICECTX)                = 9
write(2, "]: ", 3]: )                      = 3
write(2, "Specified 0 as the CUDA context,"..., 105Specified 0 as the CUDA context, but the current CUDA context could not be queried. (CUDA error code: 34)) = 105
write(2, "\n", 1
)                       = 1
futex(0x7f01b75cb1e0, FUTEX_WAKE_PRIVATE, 2147483647) = 0
write(2, "Caught exception: ", 18Caught exception: )      = 18
write(2, "OPTIX_ERROR_CUDA_ERROR: Optix ca"..., 154OPTIX_ERROR_CUDA_ERROR: Optix call 'optixDeviceContextCreate( cuCtx, &options, &context )' failed: /home/robot/Optix-8/SDK/optixHello/optixHello.cpp:126)
) = 154
write(2, "\n", 1
)                       = 1
munmap(0x7f01a8250000, 2162688)         = 0
exit_group(1)                           = ?

I noticed from

and

That I need to manually mount the libnvoptix.so and libcuda.so files in the container but this does not seem to be my issue.

Any help with be appreciated, thanks!

2

Hi @bl3rimabdullai, welcome!

Which CUDA samples are running successfully in your container? Do the successfully running CUDA samples also load /usr/local/cuda/lib64/stubs/libcuda.so? Are there some CUDA samples that do not run?

Are there any relevant container images and/or packages to mention? How would one repro this problem from scratch?

The error message from optixHello appears to be failure of the CUDA context to initialize, which I assume means the CUDA runtime has not been fully or properly installed in the container, and that this may not (yet) be an OptiX problem.

I don’t have that much experience with docker and so I don’t want to speculate, but it’s probably worth mining for some current information outside of the OptiX forum. I worry those OptiX 6 threads are too old, and might not be relevant to getting CUDA working in docker. You could try one of the CUDA forums, perhaps the CUDA Setup and Installation channel, or search online for CUDA + docker setups.

Once any & all CUDA samples run in your container, then if OptiX still won’t launch, we can revisit and investigate the OptiX issues.

–
David.