Error due to buffer overflow despite having enough GPU memory when setting large `numEnvs`

Robotics & Edge Computing Robotics - Isaac Isaac Gym
1

Hi,

I run into the following error when running Factory examples with a large numEnvs (e.g. 16384), despite having enough GPU memory in theory. The GPU I am using is A100 80G.
Would appreciate any solution / advice. Thank you!

/buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (2185) : internal error : PxgDynamicsMemoryConfig::collisionStackSize buffer overflow detected, please increase its size to at least 606286072 in the scene desc!


/buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (2185) : internal error : PxgDynamicsMemoryConfig::collisionStackSize buffer overflow detected, please increase its size to at least 606564784 in the scene desc!


/buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (2185) : internal error : PxgDynamicsMemoryConfig::collisionStackSize buffer overflow detected, please increase its size to at least 603825712 in the scene desc!


/buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (2185) : internal error : PxgDynamicsMemoryConfig::collisionStackSize buffer overflow detected, please increase its size to at least 596400504 in the scene desc!


/buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (2185) : internal error : PxgDynamicsMemoryConfig::collisionStackSize buffer overflow detected, please increase its size to at least 590987824 in the scene desc!


/buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (2185) : internal error : PxgDynamicsMemoryConfig::collisionStackSize buffer overflow detected, please increase its size to at least 581611688 in the scene desc!


/buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (2185) : internal error : PxgDynamicsMemoryConfig::collisionStackSize buffer overflow detected, please increase its size to at least 573989864 in the scene desc!


/buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (2185) : internal error : PxgDynamicsMemoryConfig::collisionStackSize buffer overflow detected, please increase its size to at least 567340536 in the scene desc!


/buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (2185) : internal error : PxgDynamicsMemoryConfig::collisionStackSize buffer overflow detected, please increase its size to at least 555458064 in the scene desc!


/buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (2185) : internal error : PxgDynamicsMemoryConfig::collisionStackSize buffer overflow detected, please increase its size to at least 546750176 in the scene desc!



[0] 0:bash  1:bash  2:python- 3:python* 4:bash  5:bash  6:bash  7:bash     "em13" 00:16 26-Feb-23
      sudo apt install xorg libxcb-randr0-dev libxrender-dev libxkbcommon-dev \
         libxkbcommon-x11-0 libavcodec-dev libavformat-dev libswscale-dev \


/buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (2098) : internal error : memcpy failed fail! 3 716


/buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (2076) : internal error : memcpy failed fail! 3 716


/buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (2098) : internal error : memcpy failed fail! 3 716


/buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h (54) : internal error : SynchronizeStreams cuEventRecord failed


/buildAgent/work/99bede84aa0a52c2/source/gpucommon/include/PxgCudaUtils.h (60) : internal error : SynchronizeStreams cuStreamWaitEvent failed


/buildAgent/work/99bede84aa0a52c2/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp (2408) : internal error : memcpy failed fail!
  716

Segmentation fault (core dumped)
2 Likes