Isaac sim ROS2 lidar crashes

aspen.eyers · November 8, 2023, 2:55am

Hi, I’m just trying to get up to speed with ROS2 lidar sensors in Isaac sim, and Isaac sim crashes reliably with a simple scene following the Isaac sim ROS2 lidar example: 7.2.5. RTX Lidar Sensors — Omniverse IsaacSim latest documentation

Running the builtin ROS2 in Isaac sim (I’d prefer to be using my built-from-source ROS2 humble - but that’s another thread).

I also have issues where none of the sensors apart from the RTX Rotating lidar will work at all.

OS: Ubuntu 20.04
Isaac sim build: 2023.1.0-hotfix.1
ROS2: Humble

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.129.03             Driver Version: 535.129.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------|   
NVIDIA GeForce RTX 4080 80W 12282MiB

Steps to reproduce:

source the isaac sim internal version of ros2

ROS_DOMAIN_ID=11
export RMW_IMPLEMENTATION=rmw_fastrtps_cpp
export PYTHONPATH=$PYTHONPATH:$HOME/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.ros2_bridge/humble/rclpy/
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.ros2_bridge/humble/rclpy/rclpy
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.ros2_bridge/humble/lib

open isaac sim with ROS2 bridge, and launch

./isaac-sim.selector.sh

Create > isaac > sensors > RTX lidar > Rotating
Window > visual scripting > action graph
new action graph
add on playback tick node
add isaac create render product
add ros2 rtx lidar helper
connect playback tick to render product
connect render product tick to ros2 lidar helper
connect render product and ros2 lidar helper render product path
modify the render product input: select target rotating ldiar /World/Rotating
modify ros2 lidar helper output
- type: point_cloud
- topic: cloud
Now I play around, I haven’t found a reproducible set of steps (in any order, I only do the following) and at some point, it’ll crash, usually after less than a minute. But If I don’t touch the sim and just leave it, it appears to be stable. I.e. If I try to do anything, I can pretty well guarantee it’ll crash in a minute or so
- open rviz2 and display the cloud
- play and stop the simulation over and over
- move the lidar
- add object
- save scene

This is the error:

2023-11-08 02:22:46 [380,154ms] [Warning] [omni.syntheticdata.plugin] SdRenderVarPtr missing valid input renderVar RtxSensorCpu
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:37:an illegal memory access was encountered (cudaMemcpyAsync( &bufferParam->async, &parameter->async, sizeof(LidarAsyncParameter), cudaMemcpyHostToHost, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:47:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, ticks.azimuths, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:50:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, ticks.states, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:53:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, ticks.timestamps, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:57:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.azimuths, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:59:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.elevations, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:61:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.distances, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:63:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.intensities, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:66:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.velocities, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:68:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.hitPointNormals, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:71:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.deltaTimes, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:73:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.emitterIds, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:75:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.beamIds, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:77:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.materialIds, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:79:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.objectIds, sizeInBytes, kind, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:137:an illegal memory access was encountered (cudaMemsetAsync(returns.azimuths, 0, sizeof(float) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:138:an illegal memory access was encountered (cudaMemsetAsync(returns.elevations, 0, sizeof(float) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:139:an illegal memory access was encountered (cudaMemsetAsync(returns.distances, 0, sizeof(float) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:140:an illegal memory access was encountered (cudaMemsetAsync(returns.intensities, 0, sizeof(float) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:141:an illegal memory access was encountered (cudaMemsetAsync(returns.velocities, 0, sizeof(float) * numReturns * 3, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:142:an illegal memory access was encountered (cudaMemsetAsync(returns.hitPointNormals, 0, sizeof(float) * numReturns * 3, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:143:an illegal memory access was encountered (cudaMemsetAsync(returns.deltaTimes, 0, sizeof(uint32_t) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:144:an illegal memory access was encountered (cudaMemsetAsync(returns.emitterIds, 0, sizeof(uint32_t) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:145:an illegal memory access was encountered (cudaMemsetAsync(returns.beamIds, 0, sizeof(uint32_t) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:146:an illegal memory access was encountered (cudaMemsetAsync(returns.materialIds, 0, sizeof(uint32_t) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [omni.sensors.nv.lidar.lidar_core.plugin] CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:147:an illegal memory access was encountered (cudaMemsetAsync(returns.objectIds, 0, sizeof(uint32_t) * numReturns, stream))
2023-11-08 02:22:47 [380,409ms] [Error] [carb.cudainterop.plugin] CUDA error 700: cudaErrorIllegalAddress - an illegal memory access was encountered)
2023-11-08 02:22:47 [380,409ms] [Error] [carb.cudainterop.plugin] Failed to signal external semaphore in CUDA.
2023-11-08 02:22:47 [380,409ms] [Error] [gpu.foundation.plugin] Signal external semaphore failed in CUDA submission in command list "Render graph command list (Render queue 0, device 0, frame submission index 2)"
2023-11-08 02:22:47 [380,413ms] [Error] [omni.physx.plugin] PhysX error: PhysX Internal CUDA error. Simulation can not continue! Error code 700!

, FILE /buildAgent/work/74336105e89c4a74/source/physx/src/NpScene.cpp, LINE 2982
2023-11-08 02:22:47 [380,413ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,413ms] [Error] [omni.physx.plugin] PhysX error: prepareGpuNarrowphase GPU error! code 700 
, FILE /buildAgent/work/74336105e89c4a74/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp, LINE 9199
2023-11-08 02:22:47 [380,413ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,413ms] [Error] [omni.physx.plugin] PhysX error: GPU kernel 'bpSignalComplete' failed to launch with error 700!!
, FILE /buildAgent/work/74336105e89c4a74/source/gpubroadphase/src/PxgCudaBroadPhaseSap.cpp, LINE 472
2023-11-08 02:22:47 [380,413ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,513ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuEventRecord failed with error 700
, FILE /buildAgent/work/74336105e89c4a74/source/gpucommon/include/PxgCudaUtils.h, LINE 53
2023-11-08 02:22:47 [380,513ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,513ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuStreamWaitEvent failed with error 700
, FILE /buildAgent/work/74336105e89c4a74/source/gpucommon/include/PxgCudaUtils.h, LINE 59
2023-11-08 02:22:47 [380,513ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,513ms] [Error] [omni.physx.plugin] PhysX error: memcpy failed fail!
  700, FILE /buildAgent/work/74336105e89c4a74/source/gpunarrowphase/src/PxgNarrowphaseCore.cpp, LINE 1444
2023-11-08 02:22:47 [380,513ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuEventRecord failed with error 700
, FILE /buildAgent/work/74336105e89c4a74/source/gpucommon/include/PxgCudaUtils.h, LINE 53
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuStreamWaitEvent failed with error 700
, FILE /buildAgent/work/74336105e89c4a74/source/gpucommon/include/PxgCudaUtils.h, LINE 59
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuEventRecord failed with error 700
, FILE /buildAgent/work/74336105e89c4a74/source/gpucommon/include/PxgCudaUtils.h, LINE 75
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] PhysX error: SynchronizeStreams cuStreamWaitEvent failed with error 700
, FILE /buildAgent/work/74336105e89c4a74/source/gpucommon/include/PxgCudaUtils.h, LINE 81
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] PhysX error: PhysX Internal CUDA error. Simulation can not continue! Error code 700!
, FILE /buildAgent/work/74336105e89c4a74/source/physx/src/NpSceneFetchResults.cpp, LINE 182
2023-11-08 02:22:47 [380,514ms] [Error] [omni.physx.plugin] Cuda context manager error, simulation will be stopped and new cuda context manager will be created.
2023-11-08 02:22:47 [380,521ms] [Error] [carb.cudainterop.plugin] CUDA error 700: cudaErrorIllegalAddress - an illegal memory access was encountered)
2023-11-08 02:22:47 [380,521ms] [Error] [carb.cudainterop.plugin] Failed to wait on external semaphore in CUDA.
2023-11-08 02:22:47 [380,521ms] [Error] [gpu.foundation.plugin] Wait for external semaphore failed in CUDA submission in command list "Render graph command list (Render queue 0, device 0, frame submission index 0)"

Simplychenable · November 8, 2023, 11:19am

@aspen.eyers i wonder if it’s related to the driver version. 525.85.05/525.85.12 seem to be the the recommended version per the doc: Technical Requirements — Omniverse IsaacSim latest documentation

rchadha · November 8, 2023, 9:54pm

Hi @aspen.eyers , do you get the same error for the other sensors as well? and also to confirm, the crash occurs only if you open up rviz or before that?

aspen.eyers · November 10, 2023, 1:17am

Hi, so I don’t actually need to have RVIZ opened at all.
I don’t even need to be echoing the topic to my terminal for the crash.

I haven’t noticed the same issue with a camera

Create > Camera
on playback tick > isaac create tender product > ros2 camera helper
open Rviz and display the image

I havn’t got any of the coreate > Isaac > sensors > RGBD sensors working successfully yet.

I don’t have access to any radar examples (that I am aware of) so I can’t test that.

gongbingbing · November 13, 2023, 11:24am

Just like you。 I use the ros2 camera helper to publish RGB images, a cuda error 801 occurs, and the image is black when viewing with rqt.
OS : Ubuntu 22.04
Isaac sim build : 2023.1.0
ROS2 : Humble

aspen.eyers · November 15, 2023, 12:41pm

Hey @gongbingbing, your error sounds a little different.
I don’t appear to be having any issues with publishing RGB images and my issue with the lidar is that there is some sort of Cuda illegal memory access error, as if a buffer is misconfigured.

Your error is a ‘Cuda 801’ error. That looks like something is not supported by your device:

cudaErrorNotSupported = 801
This error indicates the attempted operation is not supported on the current system or device.

https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html

Simplychenable · November 15, 2023, 1:53pm

@gongbingbing if you want to make a new forum thread and upload the Isaac Sim log to determine if it’s related to your driver setup. here’s the location of the log:

https://docs.omniverse.nvidia.com/isaacsim/latest/installation/install_faq.html#common-path-locations

gongbingbing · November 16, 2023, 2:37am

thank U for your reply。
I checked the running environment， the “cuda error 801” occurred when I use Nomachine for remote desktop login to a tower server. But when I switched to the XDP remote desktop, everything was normal for the images published by the ROS2 camera helper.

aspen.eyers · November 16, 2023, 2:41am

That’s good to hear @gongbingbing! If you have more issues you should start your new topic in another thread.

@rchadha for whatever it’s worth I have cuda toolkit 12.2.
I haven’t had any luck in diagnosing the problem. Can you suggest anything to try?

gongbingbing · November 16, 2023, 2:41am

Perhaps the problem I encountered is related to remote desktop, and I will use xrdp for remote login later, which is the correct way to run.
Thank you still for your guidance！

josiah7 · December 20, 2023, 9:12pm

Were you able to find a solution? I’m getting the same error.

CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:37:an illegal memory access was encountered (cudaMemcpyAsync( &bufferParam->async, &parameter->async, sizeof(LidarAsyncParameter), cudaMemcpyHostToHost, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:47:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, ticks.azimuths, sizeInBytes, kind, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:50:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, ticks.states, sizeInBytes, kind, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:53:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, ticks.timestamps, sizeInBytes, kind, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:57:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.azimuths, sizeInBytes, kind, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:59:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.elevations, sizeInBytes, kind, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:61:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.distances, sizeInBytes, kind, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:63:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.intensities, sizeInBytes, kind, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:66:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.velocities, sizeInBytes, kind, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:68:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.hitPointNormals, sizeInBytes, kind, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:71:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.deltaTimes, sizeInBytes, kind, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:73:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.emitterIds, sizeInBytes, kind, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:75:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.beamIds, sizeInBytes, kind, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:77:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.materialIds, sizeInBytes, kind, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:79:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, returns.objectIds, sizeInBytes, kind, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:137:an illegal memory access was encountered (cudaMemsetAsync(returns.azimuths, 0, sizeof(float) * numReturns, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:138:an illegal memory access was encountered (cudaMemsetAsync(returns.elevations, 0, sizeof(float) * numReturns, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:139:an illegal memory access was encountered (cudaMemsetAsync(returns.distances, 0, sizeof(float) * numReturns, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:140:an illegal memory access was encountered (cudaMemsetAsync(returns.intensities, 0, sizeof(float) * numReturns, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:141:an illegal memory access was encountered (cudaMemsetAsync(returns.velocities, 0, sizeof(float) * numReturns * 3, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:142:an illegal memory access was encountered (cudaMemsetAsync(returns.hitPointNormals, 0, sizeof(float) * numReturns * 3, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:143:an illegal memory access was encountered (cudaMemsetAsync(returns.deltaTimes, 0, sizeof(uint32_t) * numReturns, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:144:an illegal memory access was encountered (cudaMemsetAsync(returns.emitterIds, 0, sizeof(uint32_t) * numReturns, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:145:an illegal memory access was encountered (cudaMemsetAsync(returns.beamIds, 0, sizeof(uint32_t) * numReturns, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:146:an illegal memory access was encountered (cudaMemsetAsync(returns.materialIds, 0, sizeof(uint32_t) * numReturns, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:147:an illegal memory access was encountered (cudaMemsetAsync(returns.objectIds, 0, sizeof(uint32_t) * numReturns, stream))
2023-12-20 21:07:28 [68,361ms] [Error] [carb.cudainterop.plugin] CUDA error 700: cudaErrorIllegalAddress - an illegal memory access was encountered)
2023-12-20 21:07:28 [68,361ms] [Error] [carb.cudainterop.plugin] Failed to signal external semaphore in CUDA.
2023-12-20 21:07:28 [68,361ms] [Error] [gpu.foundation.plugin] Signal external semaphore failed in CUDA submission in command list "Render graph command list (Render queue 0, device 0, frame submission index 2)"
2023-12-20 21:07:28 [68,371ms] [Error] [carb.cudainterop.plugin] CUDA error 700: cudaErrorIllegalAddress - an illegal memory access was encountered)
2023-12-20 21:07:28 [68,371ms] [Error] [carb.cudainterop.plugin] Failed to wait on external semaphore in CUDA.
2023-12-20 21:07:28 [68,371ms] [Error] [gpu.foundation.plugin] Wait for external semaphore failed in CUDA submission in command list "Render graph command list (Render queue 0, device 0, frame submission index 0)"
2023-12-20 21:07:28 [68,379ms] [Error] [carb.cudainterop.plugin] CUDA error 700: cudaErrorIllegalAddress - an illegal memory access was encountered)
2023-12-20 21:07:28 [68,379ms] [Error] [carb.cudainterop.plugin] Failed to wait on external semaphore in CUDA.
2023-12-20 21:07:28 [68,379ms] [Error] [gpu.foundation.plugin] Wait for external semaphore failed in CUDA submission in command list "Render graph command list (Render queue 0, device 0, frame submission index 1)"

aspen.eyers · January 7, 2024, 10:22pm

No, solution yet sadly - I’ve been trying to keep my eye on the forums for anything tho.

rchadha · January 9, 2024, 6:51pm

hi @aspen.eyers and @josiah7 , this error should be fixed in the 2023.1.1 (bug fix release for Isaac Sim) can you please try it there and let us know if you still see the issue?

josiah7 · January 9, 2024, 6:55pm

Hi @rchadha, this is using the latest build 2023.1.1 with Ubuntu 22.04.3 I get this error+crash fairly frequently and randomly.

mcarlson1 · January 11, 2024, 8:00pm

@aspen.eyers This error should have been fixed in IsaacSim 2023.1.1. The workaround in 2023.1.0 may be to turn off gpu physics, but that it not guaranteed to work. Are you still getting this same error in IsaacSim 2023.1.1?

@josiah7 I am attempting to reproduce your errors, but have had no success. The crash that I am able to reproduce is a known issue with docking/resizing windows (Known Issues — Omniverse IsaacSim latest documentation)

ros2_rtx_crash.usda.txt (9.5 KB)

If I use the attached usda file I can get the windows docking crash, but it is not the one that starts with :

CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:37:an illegal memory access was encountered (cudaMemcpyAsync( &bufferParam->async, &parameter->async, sizeof(LidarAsyncParameter), cudaMemcpyHostToHost, stream))
CUDA error 700 in ../../../source/include/internal/omni/sensors/lidar/LidarReturnHelper.h:47:an illegal memory access was encountered (cudaMemcpyAsync(buffer + offset, ticks.azimuths, sizeInBytes, kind, stream))`

If you can reproduce your crash with the file I have attached, or give me a detailed repro then maybe I can help more!

Good luck,
Mark

Topic		Replies	Views
After upgrading to Isaac SIM 4 version, RTX Lidar sensor is not working properly Isaac Sim cuda , ubuntu , python , sensor , isaacsim , isaac-sim-v4-0-0	13	277	May 16, 2025
CUDA error 801: cudaErrorNotSupported - operation not supported Isaac Sim camera , cuda , isaacsim , isaac-sim-v4-0-0	2	115	October 7, 2024
The App is Crashing when i try to simulate my Robot Isaac Sim cuda , physx , isaac-sim-v4-2-0	11	159	April 16, 2025
Isaac Sim (2021.1.1) Wont Launch (Omniverse 1.3.4) Isaac Sim aws , setup	23	4196	April 5, 2024
Help Needed: PhysX GPU Kernel Launch Errors！！ Isaac Sim cuda , kernel , pytorch , physx , isaac-sim-v4-2-0	2	183	December 20, 2024
Issue with failed connection between Isaac Sim and Ros2 docker container Isaac Sim ros , docker , containers	12	1100	May 19, 2025
Error occured during the installation of IsaaC SIM Isaac Sim setup	10	2943	April 5, 2024
Isaac Sim 2022.2.1 crashes after opening USD file Isaac Sim	19	2003	September 27, 2023
Isaac SIM 4.2.0 version, RTX Lidar Sensor has a significant bug that prevents it from functioning properly Isaac Sim camera , ubuntu , sensor , isaacsim , rtx , isaac-sim-v4-2-0	16	468	February 16, 2025
Error in Isaac Lab Document "Reinforcement Learning Wrappers": No Attribute 'debug_vis' Isaac Sim isaac-lab , isaac-sim-v4-2-0	1	92	November 6, 2024

Isaac sim ROS2 lidar crashes

Related topics