Errors When Running Many Environments in Parallel

Hi NVIDIA and robotics friends,

I got the following errors when running many robots in parallel:

Failed to launch CUDA graph
Failed to launch CUDA graph
Failed to launch CUDA graph
Failed to launch CUDA graph
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(s->mParticleLower, &lower, sizeof(lower), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/flex.cu:5350)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(s->mParticleUpper, &upper, sizeof(upper), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/flex.cu:5351)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyToSymbolAsync(gBounds, s->mParticleLower, sizeof(Vec3), 0, cudaMemcpyDeviceToDevice) (in …/…/…/src/cuda/flex.cu:5361)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyToSymbolAsync(gBounds, s->mParticleUpper, sizeof(Vec3), sizeof(Vec3), cudaMemcpyDeviceToDevice) (in …/…/…/src/cuda/flex.cu:5362)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(s->mLib->mCellStarts, 0, sizeof(int)kGridDimkGridDimkGridDim) (in …/…/…/src/cuda/flex.cu:5389)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(s->mLib->mCellEnds, 0, sizeof(int)kGridDimkGridDim
kGridDim) (in …/…/…/src/cuda/flex.cu:5390)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(s->mStaticContactCounts, 0, n*sizeof(int)) (in …/…/…/src/cuda/flex.cu:5465)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(mTotalLower, &lower, sizeof(lower), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/bvh.cu:714)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(mTotalUpper, &upper, sizeof(upper), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/bvh.cu:715)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaFree(lib->mSortTemp) (in …/…/…/src/cuda/sort.cu:44)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMalloc(&lib->mSortTemp, sortTempSize) (in …/…/…/src/cuda/sort.cu:63)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(mNumChildren, 0, sizeof(int)maxNodes) (in …/…/…/src/cuda/bvh.cu:739)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaFree(lib->mSortTemp) (in …/…/…/src/cuda/sort.cu:44)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMalloc(&lib->mSortTemp, sortTempSize) (in …/…/…/src/cuda/sort.cu:63)
Failed to launch CUDA graph
Failed to launch CUDA graph
Failed to launch CUDA graph
Failed to launch CUDA graph
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(tempNormals, 0, sizeof(Vec4)s->mMaxParticles) (in …/…/…/src/cuda/flex.cu:6346)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(mTotalLower, &lower, sizeof(lower), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/bvh.cu:714)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(mTotalUpper, &upper, sizeof(upper), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/bvh.cu:715)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaFree(lib->mSortTemp) (in …/…/…/src/cuda/sort.cu:44)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMalloc(&lib->mSortTemp, sortTempSize) (in …/…/…/src/cuda/sort.cu:63)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(mNumChildren, 0, sizeof(int)maxNodes) (in …/…/…/src/cuda/bvh.cu:739)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpy(s->mDynamicTriangleFeatures, &triangleFeatureMasks[0], numTris * sizeof(int), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/flex.cu:8264)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(s->mParticleLower, &lower, sizeof(lower), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/flex.cu:5350)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(s->mParticleUpper, &upper, sizeof(upper), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/flex.cu:5351)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyToSymbolAsync(gBounds, s->mParticleLower, sizeof(Vec3), 0, cudaMemcpyDeviceToDevice) (in …/…/…/src/cuda/flex.cu:5361)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyToSymbolAsync(gBounds, s->mParticleUpper, sizeof(Vec3), sizeof(Vec3), cudaMemcpyDeviceToDevice) (in …/…/…/src/cuda/flex.cu:5362)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(s->mLib->mCellStarts, 0, sizeof(int)kGridDimkGridDim
kGridDim) (in …/…/…/src/cuda/flex.cu:5389)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(s->mLib->mCellEnds, 0, sizeof(int)kGridDimkGridDim
kGridDim) (in …/…/…/src/cuda/flex.cu:5390)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(s->mStaticContactCounts, 0, n
sizeof(int)) (in …/…/…/src/cuda/flex.cu:5465)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(mTotalLower, &lower, sizeof(lower), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/bvh.cu:714)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(mTotalUpper, &upper, sizeof(upper), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/bvh.cu:715)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaFree(lib->mSortTemp) (in …/…/…/src/cuda/sort.cu:44)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMalloc(&lib->mSortTemp, sortTempSize) (in …/…/…/src/cuda/sort.cu:63)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(mNumChildren, 0, sizeof(int)*maxNodes) (in …/…/…/src/cuda/bvh.cu:739)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaFree(lib->mSortTemp) (in …/…/…/src/cuda/sort.cu:44)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMalloc(&lib->mSortTemp, sortTempSize) (in …/…/…/src/cuda/sort.cu:63)

I only ran 40 robots when getting these errors. But in each environment, I have 1 soft object and 1 camera sensor. When I reduced the number of robots or eliminated the cameras, the simulation ran fine. I am just wondering if these errors are due to the limitation of my GPU or something else?

Do you have any advice on how to run many simulations in parallel without facing this issue?

Thank you,
Bradley

Hi @yuan.truyenbao,

Yes, it’s likely running out of memory. What size rendering are you doing, and how much VRAM does your GPU have? And how big are your soft body objects?

We have not done extensive work training RL networks that include vision as of yet. We are focused on Omniverse for that kind of work, given it’s better rendering capabilities.

Take care,
-Gav