Hi,
I run into a problem with using the compute sanitizer tool with an MPI code which, in the simplest case, just contains an MPI_Init() call. If I remove the mpi call, the tool works just fine. I used the following toolchain to build the executable:
nvcc: release 11.4, V11.4.120
nvcc flags: -G -Xcompiler -rdynamic
g++ (GCC) 11.2.0
OpenMPI v.4
I launched the program using srun:
srun -n 1 --cpu-bind=core compute-sanitizer
The error reported by the tool is as follows. Any feedback is much appreciated.
Thanks,
Soheil
========= COMPUTE-SANITIZER
========= Program hit invalid device context (error 201) on CUDA API call to cuCtxGetDevice.
========= Saved host backtrace up to driver entry point at error
========= Host Frame: [0x2caa7b]
========= in /usr/lib64/libcuda.so.1
========= Host Frame:uct_cuda_base_query_devices [0x51e0]
========= in /…/packages.norpm/x86_64/openucx/ucx-1.10.1_cuda/lib/ucx/libuct_cuda.so.0
========= Host Frame:uct_md_query_tl_resources [0x10f7e]
========= in /…/packages.norpm/x86_64/openucx/ucx-1.10.1_cuda/lib/libuct.so.0
========= Host Frame: [0x1461c]
========= in /…/packages.norpm/x86_64/openucx/ucx-1.10.1_cuda/lib/libucp.so.0
========= Host Frame:ucp_init_version [0x15684]
========= in /…/packages.norpm/x86_64/openucx/ucx-1.10.1_cuda/lib/libucp.so.0
========= Host Frame:mca_pml_ucx_open [0x657c]
========= in /…/openmpi_gpu/cuda_11.4-11.4.2-gcc_11-11.2.0/4.0.6/lib/openmpi/mca_pml_ucx.so
========= Host Frame:mca_base_framework_components_open [0x55524]
========= in /…/openmpi_gpu/cuda_11.4-11.4.2-gcc_11-11.2.0/4.0.6/lib/libopen-pal.so.40
========= Host Frame: [0xa6207]
========= in /…/openmpi_gpu/cuda_11.4-11.4.2-gcc_11-11.2.0/4.0.6/lib/libmpi.so.40
========= Host Frame:mca_base_framework_open [0x5ee04]
========= in /…/openmpi_gpu/cuda_11.4-11.4.2-gcc_11-11.2.0/4.0.6/lib/libopen-pal.so.40
========= Host Frame:ompi_mpi_init [0xaf99f]
========= in /…/openmpi_gpu/cuda_11.4-11.4.2-gcc_11-11.2.0/4.0.6/lib/libmpi.so.40
========= Host Frame:MPI_Init [0x6d4b9]
========= in /…/openmpi_gpu/cuda_11.4-11.4.2-gcc_11-11.2.0/4.0.6/lib/libmpi.so.40
========= Host Frame: [0xed4]
========= in /path/to/my/executable/test
========= Host Frame:__libc_start_main [0x243ea]
========= in /lib64/libc.so.6
========= Host Frame: [0xb4a]
========= in /path/to/my/executable/test