Errors When Running Many Environments in Parallel

Hi NVIDIA and robotics friends,

I got the following errors when running many robots in parallel:

Failed to launch CUDA graph
Failed to launch CUDA graph
Failed to launch CUDA graph
Failed to launch CUDA graph
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(s->mParticleLower, &lower, sizeof(lower), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/flex.cu:5350)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(s->mParticleUpper, &upper, sizeof(upper), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/flex.cu:5351)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyToSymbolAsync(gBounds, s->mParticleLower, sizeof(Vec3), 0, cudaMemcpyDeviceToDevice) (in …/…/…/src/cuda/flex.cu:5361)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyToSymbolAsync(gBounds, s->mParticleUpper, sizeof(Vec3), sizeof(Vec3), cudaMemcpyDeviceToDevice) (in …/…/…/src/cuda/flex.cu:5362)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(s->mLib->mCellStarts, 0, sizeof(int)kGridDimkGridDimkGridDim) (in …/…/…/src/cuda/flex.cu:5389)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(s->mLib->mCellEnds, 0, sizeof(int)kGridDimkGridDim
kGridDim) (in …/…/…/src/cuda/flex.cu:5390)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(s->mStaticContactCounts, 0, n*sizeof(int)) (in …/…/…/src/cuda/flex.cu:5465)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(mTotalLower, &lower, sizeof(lower), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/bvh.cu:714)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(mTotalUpper, &upper, sizeof(upper), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/bvh.cu:715)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaFree(lib->mSortTemp) (in …/…/…/src/cuda/sort.cu:44)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMalloc(&lib->mSortTemp, sortTempSize) (in …/…/…/src/cuda/sort.cu:63)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(mNumChildren, 0, sizeof(int)maxNodes) (in …/…/…/src/cuda/bvh.cu:739)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaFree(lib->mSortTemp) (in …/…/…/src/cuda/sort.cu:44)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMalloc(&lib->mSortTemp, sortTempSize) (in …/…/…/src/cuda/sort.cu:63)
Failed to launch CUDA graph
Failed to launch CUDA graph
Failed to launch CUDA graph
Failed to launch CUDA graph
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(tempNormals, 0, sizeof(Vec4)s->mMaxParticles) (in …/…/…/src/cuda/flex.cu:6346)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(mTotalLower, &lower, sizeof(lower), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/bvh.cu:714)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(mTotalUpper, &upper, sizeof(upper), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/bvh.cu:715)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaFree(lib->mSortTemp) (in …/…/…/src/cuda/sort.cu:44)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMalloc(&lib->mSortTemp, sortTempSize) (in …/…/…/src/cuda/sort.cu:63)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(mNumChildren, 0, sizeof(int)maxNodes) (in …/…/…/src/cuda/bvh.cu:739)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpy(s->mDynamicTriangleFeatures, &triangleFeatureMasks[0], numTris * sizeof(int), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/flex.cu:8264)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(s->mParticleLower, &lower, sizeof(lower), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/flex.cu:5350)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(s->mParticleUpper, &upper, sizeof(upper), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/flex.cu:5351)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyToSymbolAsync(gBounds, s->mParticleLower, sizeof(Vec3), 0, cudaMemcpyDeviceToDevice) (in …/…/…/src/cuda/flex.cu:5361)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyToSymbolAsync(gBounds, s->mParticleUpper, sizeof(Vec3), sizeof(Vec3), cudaMemcpyDeviceToDevice) (in …/…/…/src/cuda/flex.cu:5362)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(s->mLib->mCellStarts, 0, sizeof(int)kGridDimkGridDim
kGridDim) (in …/…/…/src/cuda/flex.cu:5389)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(s->mLib->mCellEnds, 0, sizeof(int)kGridDimkGridDim
kGridDim) (in …/…/…/src/cuda/flex.cu:5390)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(s->mStaticContactCounts, 0, n
sizeof(int)) (in …/…/…/src/cuda/flex.cu:5465)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(mTotalLower, &lower, sizeof(lower), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/bvh.cu:714)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(mTotalUpper, &upper, sizeof(upper), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/bvh.cu:715)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaFree(lib->mSortTemp) (in …/…/…/src/cuda/sort.cu:44)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMalloc(&lib->mSortTemp, sortTempSize) (in …/…/…/src/cuda/sort.cu:63)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(mNumChildren, 0, sizeof(int)*maxNodes) (in …/…/…/src/cuda/bvh.cu:739)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaFree(lib->mSortTemp) (in …/…/…/src/cuda/sort.cu:44)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMalloc(&lib->mSortTemp, sortTempSize) (in …/…/…/src/cuda/sort.cu:63)

I only ran 40 robots when getting these errors. But in each environment, I have 1 soft object and 1 camera sensor. When I reduced the number of robots or eliminated the cameras, the simulation ran fine. I am just wondering if these errors are due to the limitation of my GPU or something else?

Do you have any advice on how to run many simulations in parallel without facing this issue?

Thank you,
Bradley

Hi @yuan.truyenbao,

Yes, it’s likely running out of memory. What size rendering are you doing, and how much VRAM does your GPU have? And how big are your soft body objects?

We have not done extensive work training RL networks that include vision as of yet. We are focused on Omniverse for that kind of work, given it’s better rendering capabilities.

Take care,
-Gav

Hi @gstate ,

I got a similar issue when running a self designed soft robot with multiple soft body sections. The problem is, it only reports me cuda 13 error and “Failed to launch CUDA graph” on my new pc (GPU: RTX2080 super, system: ubuntu 16.04.7LTS), with nvidia driver 460.91 and cuda 10.1. But the same scene runs bug free on my previous workstation, with the configuration like: (ubuntu 16.04.6, GPU: GTX 1080 Ti, driver: 418.56 and cuda 10.1). Do you know what might cause this issue in my case? Thank you very much!

Hi @xliu9

Do you get any CUDA errors with any of the examples, and in particular running with PhysX instead of Flex?

Note that CUDA 10.1 is fairly old. You might also want to try with CUDA 11.1 on your system. You could also try it inside a docker container.

Take care,
-Gav