Catching Flex Error in code?

Autonomous Machines Robotics - Isaac Isaac Gym
1

I am using Isaac Gym with Flex backend to simulate a deformable object interacting with various rigid bodies. We are experiencing a difficult to reproduce error which causes the simulator to error out:

[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMalloc(&lib->mSortTemp, sortTempSize) (in ../../../src/cuda/sort.cu:63)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(mNumChildren, 0, sizeof(int)*maxNodes) (in ../../../src/cuda/bvh.cu:739)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaFree(lib->mSortTemp) (in ../../../src/cuda/sort.cu:44)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMalloc(&lib->mSortTemp, sortTempSize) (in ../../../src/cuda/sort.cu:63)
Failed to launch CUDA graph
Failed to launch CUDA graph
Failed to launch CUDA graph
Failed to launch CUDA graph

Is there a way to catch these Flex errors in code so that we can react in some reasonable way? Currently, I can’t find a way to realize this is happening which makes it incredibly difficult to debug.

Thanks!

1 Like