Isaac sim ROS2 lidar crashes

aspen.eyers · November 8, 2023, 2:55am

Hi, I’m just trying to get up to speed with ROS2 lidar sensors in Isaac sim, and Isaac sim crashes reliably with a simple scene following the Isaac sim ROS2 lidar example: 7.2.5. RTX Lidar Sensors — Omniverse IsaacSim latest documentation

Running the builtin ROS2 in Isaac sim (I’d prefer to be using my built-from-source ROS2 humble - but that’s another thread).

I also have issues where none of the sensors apart from the RTX Rotating lidar will work at all.

OS: Ubuntu 20.04
Isaac sim build: 2023.1.0-hotfix.1
ROS2: Humble

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------|   
NVIDIA GeForce RTX 4080 80W 12282MiB

Steps to reproduce:

source the isaac sim internal version of ros2

ROS_DOMAIN_ID=11
export RMW_IMPLEMENTATION=rmw_fastrtps_cpp
export PYTHONPATH=$PYTHONPATH:$HOME/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.ros2_bridge/humble/rclpy/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.ros2_bridge/humble/rclpy/rclpy
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.ros2_bridge/humble/lib

open isaac sim with ROS2 bridge, and launch

./isaac-sim.selector.sh

Create > isaac > sensors > RTX lidar > Rotating
Window > visual scripting > action graph
new action graph
add on playback tick node
add isaac create render product
add ros2 rtx lidar helper
connect playback tick to render product
connect render product tick to ros2 lidar helper
connect render product and ros2 lidar helper render product path
modify the render product input: select target rotating ldiar /World/Rotating
modify ros2 lidar helper output
- type: point_cloud
- topic: cloud
Now I play around, I haven’t found a reproducible set of steps (in any order, I only do the following) and at some point, it’ll crash, usually after less than a minute. But If I don’t touch the sim and just leave it, it appears to be stable. I.e. If I try to do anything, I can pretty well guarantee it’ll crash in a minute or so
- open rviz2 and display the cloud
- play and stop the simulation over and over
- move the lidar
- add object
- save scene

This is the error:

2023-11-08 02:22:46 [380,154ms] [Warning] [omni.syntheticdata.plugin] SdRenderVarPtr missing valid input renderVar RtxSensorCpu
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:37:an illegal memory access was encountered (cudaMemcpyAsync( &bufferParam->async, &parameter->async, sizeof(LidarAsyncParameter), cudaMemcpyHostToHost, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:47:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, ticks.azimuths, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:50:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, ticks.states, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:53:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, ticks.timestamps, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:57:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.azimuths, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:59:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.elevations, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:61:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.distances, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:63:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.intensities, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:66:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.velocities, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:68:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.hitPointNormals, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:71:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.deltaTimes, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:73:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.emitterIds, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:75:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.beamIds, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:77:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.materialIds, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:79:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.objectIds, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:137:an illegal memory access was encountered (cudaMemsetAsync(returns.azimuths, 0, sizeof(float) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:138:an illegal memory access was encountered (cudaMemsetAsync(returns.elevations, 0, sizeof(float) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:139:an illegal memory access was encountered (cudaMemsetAsync(returns.distances, 0, sizeof(float) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:140:an illegal memory access was encountered (cudaMemsetAsync(returns.intensities, 0, sizeof(float) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:141:an illegal memory access was encountered (cudaMemsetAsync(returns.velocities, 0, sizeof(float) * numReturns * 3, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:142:an illegal memory access was encountered (cudaMemsetAsync(returns.hitPointNormals, 0, sizeof(float) * numReturns * 3, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:143:an illegal memory access was encountered (cudaMemsetAsync(returns.deltaTimes, 0, sizeof(uint32_t) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:144:an illegal memory access was encountered (cudaMemsetAsync(returns.emitterIds, 0, sizeof(uint32_t) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:145:an illegal memory access was encountered (cudaMemsetAsync(returns.beamIds, 0, sizeof(uint32_t) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:146:an illegal memory access was encountered (cudaMemsetAsync(returns.materialIds, 0, sizeof(uint32_t) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:147:an illegal memory access was encountered (cudaMemsetAsync(returns.objectIds, 0, sizeof(uint32_t) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [carb.cudainterop.plugin] CUDA error 700: cudaErrorIllegalAddress - an illegal memory access was encountered)
2023-11-08 02:22:47 [380,409ms] [Error] [carb.cudainterop.plugin] Failed to signal external semaphore in CUDA.
2023-11-08 02:22:47 [380,409ms] [Error] [gpu.foundation.plugin] Signal external semaphore failed in CUDA submission in command list "Render graph command list (Render queue 0, device 0, frame submission index 2)"
2023-11-08 02:22:47 [380,413ms] [Error] [omni.physx.plugin] PhysX error: PhysX Internal CUDA error. Simulation can not continue! Error code 700!

, FILE /buildAgent/work/74336105e89c4a74/source/physx/src/NpScene.cpp, LINE 2982
2023-11-08 02:22:47 [380,413ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,413ms] [Error] [omni.physx.plugin] PhysX error: prepareGpuNarrowphase GPU error! code 700 
, FILE /buildAgent/work/74336105e89c4a74/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp, LINE 9199
2023-11-08 02:22:47 [380,413ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,413ms] [Error] [omni.physx.plugin] PhysX error: GPU kernel 'bpSignalComplete' failed to launch with error 700!!
, FILE /buildAgent/work/74336105e89c4a74/source/gpubroadphase/src/PxgCudaBroadPhaseSap.cpp, LINE 472
2023-11-08 02:22:47 [380,413ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,513ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuEventRecord failed with error 700
, FILE /buildAgent/work/74336105e89c4a74/source/gpucommon/include/PxgCudaUtils.h, LINE 53
2023-11-08 02:22:47 [380,513ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,513ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuStreamWaitEvent failed with error 700
, FILE /buildAgent/work/74336105e89c4a74/source/gpucommon/include/PxgCudaUtils.h, LINE 59
2023-11-08 02:22:47 [380,513ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,513ms] [Error] [omni.physx.plugin] PhysX error: memcpy failed fail!
  700, FILE /buildAgent/work/74336105e89c4a74/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp, LINE 1444
2023-11-08 02:22:47 [380,513ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuEventRecord failed with error 700
, FILE /buildAgent/work/74336105e89c4a74/source/gpucommon/include/PxgCudaUtils.h, LINE 53
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuStreamWaitEvent failed with error 700
, FILE /buildAgent/work/74336105e89c4a74/source/gpucommon/include/PxgCudaUtils.h, LINE 59
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuEventRecord failed with error 700
, FILE /buildAgent/work/74336105e89c4a74/source/gpucommon/include/PxgCudaUtils.h, LINE 75
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuStreamWaitEvent failed with error 700
, FILE /buildAgent/work/74336105e89c4a74/source/gpucommon/include/PxgCudaUtils.h, LINE 81
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] PhysX error: PhysX Internal CUDA error. Simulation can not continue! Error code 700!
, FILE /buildAgent/work/74336105e89c4a74/source/physx/src/NpSceneFetchResults.cpp, LINE 182
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,521ms] [Error] [carb.cudainterop.plugin] CUDA error 700: cudaErrorIllegalAddress - an illegal memory access was encountered)
2023-11-08 02:22:47 [380,521ms] [Error] [carb.cudainterop.plugin] Failed to wait on external semaphore in CUDA.
2023-11-08 02:22:47 [380,521ms] [Error] [gpu.foundation.plugin] Wait for external semaphore failed in CUDA submission in command list "Render graph command list (Render queue 0, device 0, frame submission index 0)"

Simplychenable · November 8, 2023, 11:19am

@aspen.eyers i wonder if it’s related to the driver version. 525.85.05/525.85.12 seem to be the the recommended version per the doc: Technical Requirements — Omniverse IsaacSim latest documentation

rchadha · November 8, 2023, 9:54pm

Hi @aspen.eyers , do you get the same error for the other sensors as well? and also to confirm, the crash occurs only if you open up rviz or before that?

aspen.eyers · November 10, 2023, 1:17am

Hi, so I don’t actually need to have RVIZ opened at all.
I don’t even need to be echoing the topic to my terminal for the crash.

I haven’t noticed the same issue with a camera

Create > Camera
on playback tick > isaac create tender product > ros2 camera helper
open Rviz and display the image

I havn’t got any of the coreate > Isaac > sensors > RGBD sensors working successfully yet.

I don’t have access to any radar examples (that I am aware of) so I can’t test that.

gongbingbing · November 13, 2023, 11:24am

Just like you。 I use the ros2 camera helper to publish RGB images, a cuda error 801 occurs, and the image is black when viewing with rqt.
OS : Ubuntu 22.04
Isaac sim build : 2023.1.0
ROS2 : Humble

aspen.eyers · November 15, 2023, 12:41pm

Hey @gongbingbing, your error sounds a little different.
I don’t appear to be having any issues with publishing RGB images and my issue with the lidar is that there is some sort of Cuda illegal memory access error, as if a buffer is misconfigured.

Your error is a ‘Cuda 801’ error. That looks like something is not supported by your device:

cudaErrorNotSupported = 801
This error indicates the attempted operation is not supported on the current system or device.

https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html

Simplychenable · November 15, 2023, 1:53pm

@gongbingbing if you want to make a new forum thread and upload the Isaac Sim log to determine if it’s related to your driver setup. here’s the location of the log:

https://docs.omniverse.nvidia.com/isaacsim/latest/installation/install_faq.html#common-path-locations

gongbingbing · November 16, 2023, 2:37am

thank U for your reply。
I checked the running environment， the “cuda error 801” occurred when I use Nomachine for remote desktop login to a tower server. But when I switched to the XDP remote desktop, everything was normal for the images published by the ROS2 camera helper.

aspen.eyers · November 16, 2023, 2:41am

That’s good to hear @gongbingbing! If you have more issues you should start your new topic in another thread.

@rchadha for whatever it’s worth I have cuda toolkit 12.2.
I haven’t had any luck in diagnosing the problem. Can you suggest anything to try?

gongbingbing · November 16, 2023, 2:41am

Perhaps the problem I encountered is related to remote desktop, and I will use xrdp for remote login later, which is the correct way to run.
Thank you still for your guidance！