# Segmentation fault when using different GPUs

**URL:** <https://forums.developer.nvidia.com/t/segmentation-fault-when-using-different-gpus/224136>\
**Category:** Isaac Gym\
**Created:** [August 16, 2022, 11:52pm UTC](https://forums.developer.nvidia.com/t/segmentation-fault-when-using-different-gpus/224136 "2022-08-16T23:52:00Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![xidong.feng.20](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@xidong.feng.20](https://forums.developer.nvidia.com/u/xidong.feng.20)\
**Post date:** [August 16, 2022, 11:52pm UTC](https://forums.developer.nvidia.com/t/segmentation-fault-when-using-different-gpus/224136/1 "2022-08-16T23:52:00Z")

</div>

I met a weird bug when running Issacgym’s joint\_monkey examples. For example, when I specify cuda:0 as the GPU to run the example, everything works well:

 ![image](https://global.discourse-cdn.com/nvidia/original/3X/f/b/fb63af68032668a2798e237b77994bd81eb0b08d.png)

However, when I use different GPU like cuda:1 (by seting CUDA\_VISIBLE\_DEVICES=1), the error happens:

 ![image](https://global.discourse-cdn.com/nvidia/original/3X/d/f/df4ac19d492240043a3cbb062bcf40bac543430f.png)

@gstate

---

<div class="post-metadata">

**Author:** ![gstate](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/gstate/32/48225_2.png) [@gstate](https://forums.developer.nvidia.com/u/gstate)\
**Post date:** [August 18, 2022, 5:10am UTC](https://forums.developer.nvidia.com/t/segmentation-fault-when-using-different-gpus/224136/2 "2022-08-18T05:10:55Z")

</div>

Hi @xidong.feng.20

Looks like we might have a problem to track down here with CUDA\_VISIBLE\_DEVICES. In the meantime, it is better anyways to explicitly specify your sim and/or graphics devices on the command line using:  
`--sim_device=cuda:n --graphics_device=k`

Note that the `sim_device` parameter uses CUDA style device syntax, while the `graphics_device` parameter uses the vulkan device ID, which may not always be the same as the CUDA device ID style.

Take care,  
-Gav

---

<div class="post-metadata">

**Author:** ![zeyanjie](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@zeyanjie](https://forums.developer.nvidia.com/u/zeyanjie)\
**Post date:** [March 14, 2023, 12:29pm UTC](https://forums.developer.nvidia.com/t/segmentation-fault-when-using-different-gpus/224136/3 "2023-03-14T12:29:37Z")

</div>

this helps!  
really hard to find solution btw :)

---

<div class="post-metadata">

**Author:** ![sashwat.mahalingam](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@sashwat.mahalingam](https://forums.developer.nvidia.com/u/sashwat.mahalingam)\
**Post date:** [May 25, 2023, 10:56am UTC](https://forums.developer.nvidia.com/t/segmentation-fault-when-using-different-gpus/224136/4 "2023-05-25T10:56:55Z")

</div>

How does the Vulkan ID relate to the GPU ID assigned by CUDA? I’m having a hard time trying to instantiate new renderers on devices that are not CUDA:0 and would appreciate any pointers to relevant documentation. Thanks!

---

<div class="post-metadata">

**Author:** ![oleh.rybkin](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@oleh.rybkin](https://forums.developer.nvidia.com/u/oleh.rybkin)\
**Post date:** [October 11, 2023, 10:48pm UTC](https://forums.developer.nvidia.com/t/segmentation-fault-when-using-different-gpus/224136/5 "2023-10-11T22:48:39Z")

</div>

This still doesn’t work for me. Most of the GPU memory is being used on GPU 1 when I specify sim\_device=cuda:1 rl\_device=cuda:1 graphics\_device\_id=1, however, some memory is still required on GPU 0. This crashes when GPU 0 is fully utilized, e.g. if tensorflow is running on that GPU. This makes it impossible to run isaac gym on machines shared across multiple users (since someone might be using tensorflow).

---

<div class="post-metadata">

**Author:** ![xiaochen.it](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@xiaochen.it](https://forums.developer.nvidia.com/u/xiaochen.it)\
**Post date:** [May 12, 2024, 8:18am UTC](https://forums.developer.nvidia.com/t/segmentation-fault-when-using-different-gpus/224136/6 "2024-05-12T08:18:07Z")

</div>

Agree. I also found that some memory is taken up on GPU 0, though I specify sim\_device=cuda:1 rl\_device=cuda:1 graphics\_device\_id=1.
