gymapi.Gym.create_camera_sensor() cause errors on multi-gpu server

Autonomous Machines Robotics - Isaac Isaac Gym
,
1

Hi
I tried to add cameras to my legged robot (which includes the function “create_camera_sensor()”). And it works perfectly when running on my single RTX 3090 Desktop, and it also works, according to my colleagues, on a server with multiple A100 GPUs.

However, when I tried to run it on a sever with multiple A5000 gpus, the following error occurs

` **[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on rgbImage buffer with error 101**

**[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on depthImage buffer with error 101**

**[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on segmentationImage buffer with error 101**

**[Error] [carb.gym.plugin] cudaExternamMemoryGetMappedBuffer failed on optical flow buffer with error 101**`

It occurs each time when I call “create_camera_sensor()”, and later when I call “get_camera_image_gpu_tensor” to retrieve the image it returns nothing.

I already tried the solutions on this forum, including this and this, but none of them work.

I also notice that Issac seems to have poor support (at least on my server) for CUDA_VISIBLE_DEVICES. When I set CUDA_VISIBLE_DEVICES and also --sim_device, --graphics_device, some of them make sense but would cause direct segmentation error (for example, set CUDA_VISIBLE_DEVICES=0 and sim_device=cuda:0), I am not sure if this is another bug.

I will really appreciate any help or suggestions!

3 Likes