Isaac sim ROS2 lidar crashes

Hi, I’m just trying to get up to speed with ROS2 lidar sensors in Isaac sim, and Isaac sim crashes reliably with a simple scene following the Isaac sim ROS2 lidar example: 7.2.5. RTX Lidar Sensors — Omniverse IsaacSim latest documentation

Running the builtin ROS2 in Isaac sim (I’d prefer to be using my built-from-source ROS2 humble - but that’s another thread).

I also have issues where none of the sensors apart from the RTX Rotating lidar will work at all.

OS: Ubuntu 20.04
Isaac sim build: 2023.1.0-hotfix.1
ROS2: Humble

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------|   
NVIDIA GeForce RTX 4080 80W 12282MiB 

Steps to reproduce:

  • source the isaac sim internal version of ros2
ROS_DOMAIN_ID=11
export RMW_IMPLEMENTATION=rmw_fastrtps_cpp
export PYTHONPATH=$PYTHONPATH:$HOME/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.ros2_bridge/humble/rclpy/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.ros2_bridge/humble/rclpy/rclpy
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.ros2_bridge/humble/lib
  • open isaac sim with ROS2 bridge, and launch
./isaac-sim.selector.sh
  • Create > isaac > sensors > RTX lidar > Rotating
  • Window > visual scripting > action graph
  • new action graph
  • add on playback tick node
  • add isaac create render product
  • add ros2 rtx lidar helper
  • connect playback tick to render product
  • connect render product tick to ros2 lidar helper
  • connect render product and ros2 lidar helper render product path
  • modify the render product input: select target rotating ldiar /World/Rotating
  • modify ros2 lidar helper output
    • type: point_cloud
    • topic: cloud
  • Now I play around, I haven’t found a reproducible set of steps (in any order, I only do the following) and at some point, it’ll crash, usually after less than a minute. But If I don’t touch the sim and just leave it, it appears to be stable. I.e. If I try to do anything, I can pretty well guarantee it’ll crash in a minute or so
    • open rviz2 and display the cloud
    • play and stop the simulation over and over
    • move the lidar
    • add object
    • save scene

This is the error:

2023-11-08 02:22:46 [380,154ms] [Warning] [omni.syntheticdata.plugin] SdRenderVarPtr missing valid input renderVar RtxSensorCpu
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:37:an illegal memory access was encountered (cudaMemcpyAsync( &bufferParam->async, &parameter->async, sizeof(LidarAsyncParameter), cudaMemcpyHostToHost, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:47:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, ticks.azimuths, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:50:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, ticks.states, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:53:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, ticks.timestamps, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:57:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.azimuths, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:59:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.elevations, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:61:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.distances, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:63:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.intensities, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:66:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.velocities, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:68:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.hitPointNormals, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:71:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.deltaTimes, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:73:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.emitterIds, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:75:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.beamIds, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:77:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.materialIds, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:79:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.objectIds, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:137:an illegal memory access was encountered (cudaMemsetAsync(returns.azimuths, 0, sizeof(float) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:138:an illegal memory access was encountered (cudaMemsetAsync(returns.elevations, 0, sizeof(float) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:139:an illegal memory access was encountered (cudaMemsetAsync(returns.distances, 0, sizeof(float) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:140:an illegal memory access was encountered (cudaMemsetAsync(returns.intensities, 0, sizeof(float) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:141:an illegal memory access was encountered (cudaMemsetAsync(returns.velocities, 0, sizeof(float) * numReturns * 3, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:142:an illegal memory access was encountered (cudaMemsetAsync(returns.hitPointNormals, 0, sizeof(float) * numReturns * 3, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:143:an illegal memory access was encountered (cudaMemsetAsync(returns.deltaTimes, 0, sizeof(uint32_t) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:144:an illegal memory access was encountered (cudaMemsetAsync(returns.emitterIds, 0, sizeof(uint32_t) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:145:an illegal memory access was encountered (cudaMemsetAsync(returns.beamIds, 0, sizeof(uint32_t) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:146:an illegal memory access was encountered (cudaMemsetAsync(returns.materialIds, 0, sizeof(uint32_t) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:147:an illegal memory access was encountered (cudaMemsetAsync(returns.objectIds, 0, sizeof(uint32_t) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [carb.cudainterop.plugin] CUDA error 700: cudaErrorIllegalAddress - an illegal memory access was encountered)
2023-11-08 02:22:47 [380,409ms] [Error] [carb.cudainterop.plugin] Failed to signal external semaphore in CUDA.
2023-11-08 02:22:47 [380,409ms] [Error] [gpu.foundation.plugin] Signal external semaphore failed in CUDA submission in command list "Render graph command list (Render queue 0, device 0, frame submission index 2)"
2023-11-08 02:22:47 [380,413ms] [Error] [omni.physx.plugin] PhysX error: PhysX Internal CUDA error. Simulation can not continue! Error code 700!
, FILE /buildAgent/work/74336105e89c4a74/source/physx/src/NpScene.cpp, LINE 2982
2023-11-08 02:22:47 [380,413ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,413ms] [Error] [omni.physx.plugin] PhysX error: prepareGpuNarrowphase GPU error! code 700 
, FILE /buildAgent/work/74336105e89c4a74/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp, LINE 9199
2023-11-08 02:22:47 [380,413ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,413ms] [Error] [omni.physx.plugin] PhysX error: GPU kernel 'bpSignalComplete' failed to launch with error 700!!
, FILE /buildAgent/work/74336105e89c4a74/source/gpubroadphase/src/PxgCudaBroadPhaseSap.cpp, LINE 472
2023-11-08 02:22:47 [380,413ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,513ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuEventRecord failed with error 700
, FILE /buildAgent/work/74336105e89c4a74/source/gpucommon/include/PxgCudaUtils.h, LINE 53
2023-11-08 02:22:47 [380,513ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,513ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuStreamWaitEvent failed with error 700
, FILE /buildAgent/work/74336105e89c4a74/source/gpucommon/include/PxgCudaUtils.h, LINE 59
2023-11-08 02:22:47 [380,513ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,513ms] [Error] [omni.physx.plugin] PhysX error: memcpy failed fail!
  700, FILE /buildAgent/work/74336105e89c4a74/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp, LINE 1444
2023-11-08 02:22:47 [380,513ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuEventRecord failed with error 700
, FILE /buildAgent/work/74336105e89c4a74/source/gpucommon/include/PxgCudaUtils.h, LINE 53
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuStreamWaitEvent failed with error 700
, FILE /buildAgent/work/74336105e89c4a74/source/gpucommon/include/PxgCudaUtils.h, LINE 59
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuEventRecord failed with error 700
, FILE /buildAgent/work/74336105e89c4a74/source/gpucommon/include/PxgCudaUtils.h, LINE 75
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuStreamWaitEvent failed with error 700
, FILE /buildAgent/work/74336105e89c4a74/source/gpucommon/include/PxgCudaUtils.h, LINE 81
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] PhysX error: PhysX Internal CUDA error. Simulation can not continue! Error code 700!
, FILE /buildAgent/work/74336105e89c4a74/source/physx/src/NpSceneFetchResults.cpp, LINE 182
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,521ms] [Error] [carb.cudainterop.plugin] CUDA error 700: cudaErrorIllegalAddress - an illegal memory access was encountered)
2023-11-08 02:22:47 [380,521ms] [Error] [carb.cudainterop.plugin] Failed to wait on external semaphore in CUDA.
2023-11-08 02:22:47 [380,521ms] [Error] [gpu.foundation.plugin] Wait for external semaphore failed in CUDA submission in command list "Render graph command list (Render queue 0, device 0, frame submission index 0)"
1 Like

@aspen.eyers i wonder if it’s related to the driver version. 525.85.05/525.85.12 seem to be the the recommended version per the doc: Technical Requirements — Omniverse IsaacSim latest documentation

Hi @aspen.eyers , do you get the same error for the other sensors as well? and also to confirm, the crash occurs only if you open up rviz or before that?

Hi, so I don’t actually need to have RVIZ opened at all.
I don’t even need to be echoing the topic to my terminal for the crash.

I haven’t noticed the same issue with a camera

  • Create > Camera
  • on playback tick > isaac create tender product > ros2 camera helper
  • open Rviz and display the image

I havn’t got any of the coreate > Isaac > sensors > RGBD sensors working successfully yet.

I don’t have access to any radar examples (that I am aware of) so I can’t test that.

Just like you。 I use the ros2 camera helper to publish RGB images, a cuda error 801 occurs, and the image is black when viewing with rqt.
OS : Ubuntu 22.04
Isaac sim build : 2023.1.0
ROS2 : Humble

1 Like

Hey @gongbingbing, your error sounds a little different.
I don’t appear to be having any issues with publishing RGB images and my issue with the lidar is that there is some sort of Cuda illegal memory access error, as if a buffer is misconfigured.

Your error is a ‘Cuda 801’ error. That looks like something is not supported by your device:

cudaErrorNotSupported = 801
This error indicates the attempted operation is not supported on the current system or device. 

https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html

@gongbingbing if you want to make a new forum thread and upload the Isaac Sim log to determine if it’s related to your driver setup. here’s the location of the log:

https://docs.omniverse.nvidia.com/isaacsim/latest/installation/install_faq.html#common-path-locations

thank U for your reply。
I checked the running environment, the “cuda error 801” occurred when I use Nomachine for remote desktop login to a tower server. But when I switched to the XDP remote desktop, everything was normal for the images published by the ROS2 camera helper.

That’s good to hear @gongbingbing! If you have more issues you should start your new topic in another thread.

@rchadha for whatever it’s worth I have cuda toolkit 12.2.
I haven’t had any luck in diagnosing the problem. Can you suggest anything to try?

Perhaps the problem I encountered is related to remote desktop, and I will use xrdp for remote login later, which is the correct way to run.
Thank you still for your guidance!

1 Like