[Bug] multiple GPUs and the GPU device order problem in ISAACSIM 2023.1.0 version

[zhengzj]

OS: Ubuntu 20.04
GPU: RTX2080Ti x 2 (two GPUs)
Driver Version: 525.125.06 CUDA Version: 12.0

Example code:
ISAACSIM ROS Navigation Example

Bug:
[Error] [omni.isaac.core_nodes] CUDA error 801: cudaErrorNotSupported - operation not supported)
The bug is related to copying data from a wrong GPU. This problem was solved in the 2022.2.1 version, but it comes up again in the 2023.1.0 version.

Below are the GPU information from the log file.

Adding the below codes before initializing a simulation app solves the problem in standalone projects.

import os
os.environ["CUDA_VISIBLE_DEVICES"] = "1,0" // change to the sequence stated on the "active" column of your driver information in the log file
simulation_app = SimulationApp({"headless": False})

[rthaker]

Hi @zhengzj - Yes, whenever there are two GPUs on Linux device, you should use the script to avoid the GPU device order problem.

I confirmed with the dev team, this is not a bug.

[zhengzj]

Thanks for the clarification.