Hi NVIDIA and robotics friends,
I got the following errors when running many robots in parallel:
Failed to launch CUDA graph
Failed to launch CUDA graph
Failed to launch CUDA graph
Failed to launch CUDA graph
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(s->mParticleLower, &lower, sizeof(lower), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/flex.cu:5350)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(s->mParticleUpper, &upper, sizeof(upper), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/flex.cu:5351)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyToSymbolAsync(gBounds, s->mParticleLower, sizeof(Vec3), 0, cudaMemcpyDeviceToDevice) (in …/…/…/src/cuda/flex.cu:5361)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyToSymbolAsync(gBounds, s->mParticleUpper, sizeof(Vec3), sizeof(Vec3), cudaMemcpyDeviceToDevice) (in …/…/…/src/cuda/flex.cu:5362)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(s->mLib->mCellStarts, 0, sizeof(int)kGridDimkGridDimkGridDim) (in …/…/…/src/cuda/flex.cu:5389)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(s->mLib->mCellEnds, 0, sizeof(int)kGridDimkGridDimkGridDim) (in …/…/…/src/cuda/flex.cu:5390)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(s->mStaticContactCounts, 0, n*sizeof(int)) (in …/…/…/src/cuda/flex.cu:5465)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(mTotalLower, &lower, sizeof(lower), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/bvh.cu:714)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(mTotalUpper, &upper, sizeof(upper), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/bvh.cu:715)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaFree(lib->mSortTemp) (in …/…/…/src/cuda/sort.cu:44)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMalloc(&lib->mSortTemp, sortTempSize) (in …/…/…/src/cuda/sort.cu:63)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(mNumChildren, 0, sizeof(int)maxNodes) (in …/…/…/src/cuda/bvh.cu:739)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaFree(lib->mSortTemp) (in …/…/…/src/cuda/sort.cu:44)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMalloc(&lib->mSortTemp, sortTempSize) (in …/…/…/src/cuda/sort.cu:63)
Failed to launch CUDA graph
Failed to launch CUDA graph
Failed to launch CUDA graph
Failed to launch CUDA graph
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(tempNormals, 0, sizeof(Vec4)s->mMaxParticles) (in …/…/…/src/cuda/flex.cu:6346)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind) (in …/…/…/src/cuda/flex.cu:7272)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventSynchronize(lib->mMemoryFence) (in …/…/…/src/cuda/flex.cu:6686)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(mTotalLower, &lower, sizeof(lower), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/bvh.cu:714)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(mTotalUpper, &upper, sizeof(upper), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/bvh.cu:715)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaFree(lib->mSortTemp) (in …/…/…/src/cuda/sort.cu:44)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMalloc(&lib->mSortTemp, sortTempSize) (in …/…/…/src/cuda/sort.cu:63)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(mNumChildren, 0, sizeof(int)maxNodes) (in …/…/…/src/cuda/bvh.cu:739)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpy(s->mDynamicTriangleFeatures, &triangleFeatureMasks[0], numTris * sizeof(int), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/flex.cu:8264)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(ByteOffset(dstPtr, dstByteOffset), ByteOffset(srcPtr, srcByteOffset), byteSize, kind, 0) (in …/…/…/src/cuda/flex.cu:7311)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaEventRecord(lib->mMemoryFence, 0) (in …/…/…/src/cuda/flex.cu:6673)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(s->mParticleLower, &lower, sizeof(lower), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/flex.cu:5350)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(s->mParticleUpper, &upper, sizeof(upper), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/flex.cu:5351)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyToSymbolAsync(gBounds, s->mParticleLower, sizeof(Vec3), 0, cudaMemcpyDeviceToDevice) (in …/…/…/src/cuda/flex.cu:5361)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyToSymbolAsync(gBounds, s->mParticleUpper, sizeof(Vec3), sizeof(Vec3), cudaMemcpyDeviceToDevice) (in …/…/…/src/cuda/flex.cu:5362)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(s->mLib->mCellStarts, 0, sizeof(int)kGridDimkGridDimkGridDim) (in …/…/…/src/cuda/flex.cu:5389)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(s->mLib->mCellEnds, 0, sizeof(int)kGridDimkGridDimkGridDim) (in …/…/…/src/cuda/flex.cu:5390)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(s->mStaticContactCounts, 0, nsizeof(int)) (in …/…/…/src/cuda/flex.cu:5465)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(mTotalLower, &lower, sizeof(lower), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/bvh.cu:714)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemcpyAsync(mTotalUpper, &upper, sizeof(upper), cudaMemcpyHostToDevice) (in …/…/…/src/cuda/bvh.cu:715)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaFree(lib->mSortTemp) (in …/…/…/src/cuda/sort.cu:44)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMalloc(&lib->mSortTemp, sortTempSize) (in …/…/…/src/cuda/sort.cu:63)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMemsetAsync(mNumChildren, 0, sizeof(int)*maxNodes) (in …/…/…/src/cuda/bvh.cu:739)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaFree(lib->mSortTemp) (in …/…/…/src/cuda/sort.cu:44)
[Error] [carb.gym.plugin] Flex error: Cuda error: 77 in cudaMalloc(&lib->mSortTemp, sortTempSize) (in …/…/…/src/cuda/sort.cu:63)
I only ran 40 robots when getting these errors. But in each environment, I have 1 soft object and 1 camera sensor. When I reduced the number of robots or eliminated the cameras, the simulation ran fine. I am just wondering if these errors are due to the limitation of my GPU or something else?
Do you have any advice on how to run many simulations in parallel without facing this issue?
Thank you,
Bradley