Isaac SIM 4.2.0 version, RTX Lidar Sensor has a significant bug that prevents it from functioning properly

[Julie_A]

Hello everyone, I’m sorry that I encountered some issues while using the new version (Isaac SIM 4).
I am unable to create Rtx Lidar sensor using Isaac SIM 4.0 and Isaac SIM 4.1 versions, and encounter errors and crashes during use.

After upgrading to Isaac SIM 4 version, RTX Lidar sensor is not working properly

I am currently using ISACC SIM to configure the LiDAR. I started using version 2023.1.1 and it works fine. I also know that after Isaac SIM 4.0, some changes were made to the configuration files. I also know that after Isaac SIM 4.0, some changes were made to the configuration files. However, when I used the built-in configuration file for testing, unfortunately there were some errors and I was unable to use it. The configuration file I am using is the case file “ov/pkg/isaac-sim-4.0.0/exts/om…

Then I upgraded to Isaac SIM 4.2.0, which did not crash, but encountered new issues. When I was creating the RTX sensor Lidar using the official case, it started flashing and didn’t work properly. Completely unusable.

Is there a solution that can use Rtx Lidar normally? When I was using version 2023.1.1, Rtx Lidar worked normally, but after upgrading, it cannot work properly.

I’m not sure if there are any other bugs related to image acquisition, etc

|---------------------------------------------------------------------------------------------|
| Driver Version: 535.183.01    | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|     |                                  |        |     |            | Bus-ID    |            |
|---------------------------------------------------------------------------------------------|
| 0   | NVIDIA GeForce RTX 3070 Ti       | Yes: 0 |     | 8438    MB | 10de      | 0          |
|     |                                  |        |     |            | 2482      | 5713ffd4.. |
|     |                                  |        |     |            | 1         |            |
|---------------------------------------------------------------------------------------------|
| 1   | NVIDIA GeForce RTX 3070 Ti       | Yes: 1 |     | 8438    MB | 10de      | 0          |
|     |                                  |        |     |            | 2482      | 092c2cd5.. |
|     |                                  |        |     |            | 5         |            |
|=============================================================================================|
| OS: 22.04.4 LTS (Jammy Jellyfish) ubuntu, Version: 22.04.4, Kernel: 6.8.0-45-generic
| XServer Vendor: The X.Org Foundation, XServer Version: 12101004 (1.21.1.4)
| Processor: 12th Gen Intel(R) Core(TM) i7-12700KF | Cores: 12 | Logical: 20
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 31933 | Free Memory: 22964
| Total Page/Swap (MB): 2047 | Free Page/Swap: 1945

Mon Oct 14 18:15:45 2024       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.183.01             Driver Version: 535.183.01   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce RTX 3070 Ti     Off | 00000000:01:00.0  On |                  N/A |
| 46%   62C    P2              77W / 290W |   2601MiB /  8192MiB |      4%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce RTX 3070 Ti     Off | 00000000:05:00.0 Off |                  N/A |
| 39%   60C    P2              70W / 290W |   2250MiB /  8192MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|    0   N/A  N/A      2458      G   /usr/lib/xorg/Xorg                          594MiB |
|    0   N/A  N/A      2641      G   /usr/bin/gnome-shell                         54MiB |
|    0   N/A  N/A      2879      G   ...in/bin/sunloginclient --cmd=autorun       12MiB |
|    0   N/A  N/A      3060      G   ...) Chrome/58.0.3029.81 Safari/537.36        3MiB |
|    0   N/A  N/A      3135      G   ...en=A01DE550C902E4CE5E3F78984C7777B8        9MiB |
|    0   N/A  N/A      3251      G   ...ures=SpareRendererForSitePerProcess       24MiB |
|    0   N/A  N/A      7319      G   ...seed-version=20241011-130141.903000       78MiB |
|    0   N/A  N/A      7685      G   ...irefox/5091/usr/lib/firefox/firefox       18MiB |
|    0   N/A  N/A     10731      G   /proc/self/exe                               48MiB |
|    0   N/A  N/A     58898    C+G   ...hare/ov/pkg/isaac-sim-4.0.0/kit/kit     1529MiB |
|    1   N/A  N/A      2458      G   /usr/lib/xorg/Xorg                            4MiB |
|    1   N/A  N/A     58898    C+G   ...hare/ov/pkg/isaac-sim-4.0.0/kit/kit     2066MiB |
+---------------------------------------------------------------------------------------+

[Sirens_uk]

Hey Julie,

I’m also experiencing a similar issue if you want to look at this thread

MotionBVH for lidar model not enabled?

little more info here I only have the issue if i run IS with the standard bat file. If i run the standalone mode the fault dosent repo, again this is just using the standard script from the latest docs for rtx lidar [image]

I’ve been able to tun in standalone with no issues but when i run in the standard mode I have a similar issue without the flashing do you have the MotionBVH warning in the info in the console?

Scott

[Julie_A]

@Sirens_uk
Thank you for your reply
I saw this topic article, so I immediately tried using ‘standalone’, unfortunately, the result was the same, the flicker still exists.
No matter what method I used to create the Rtx Sensor, it never showed any “MotionBVH warning” or any error messages.
I tried generating a point cloud and saving it, and it still can generate a point cloud, but the flickering is severe and cannot be used normally. I think this is a serious bug in 4.2.0.
The following GIF is the result of my ‘standalone’ run

[Sirens_uk]

I never had the flickering, I’m running win 10 on a RTX 3090…

I have seen other people have this issue. I did some digging and there is something about Vulcan thats something to do with my error (I think)

I would say it would be worth trying to swap out the standard driver for the studio driver. Not sure if it will help but it’s probably worth a shot at this point

[Julie_A]

@Sirens_uk

Sorry for the late reply, as I have been conducting tests these past few days. I tried to collect RGB data and got the same result, but it kept flashing. But the collected data is normal, only the ‘viewport’ keeps flashing.
I have two RTX3070Ti graphics cards, which I think are sufficient for use. I still can’t find any solution to the flicker.

[Hammad_M]

@Julie_A can you try starting isaac sim with the following command line argument

--/rendergraph/mgpu/backend=copyQueue

[Julie_A]

@Hammad_M Thanks for your reply, how should I add this code? Where?

In “standalone”?I’m sorry, I don’t understand how to add this code.

[Hammad_M]

./python.sh my_script.py --/rendergraph/mgpu/backend=copyQueue

As long as your script only parses known args, all unknown args will be handled by the underlying sim

alternatively you can edit

apps/omni.isaac.sim.python.kit

and at the very bottom add

[settings]
rendergraph.mgpu.backend = "copyQueue"

[Julie_A]

@Hammad_M

I tried various startup methods, including ‘standalone’. I’m sorry, but I still got bad results and it still flashes. Is there any other way to solve this problem? I have seen many people on the forum who have encountered similar problems as me.

[Hammad_M]

One more thing to try, can you edit your SimulationApp call and disable mgpu

SimulationApp({“multi_gpu”:False})

[Julie_A]

@Hammad_M
Thank you very much, this method is effective，

[robocist404]

Hi, I tried to do the same thing using the GUI Render Setting option by unticking the Multi-GPU option for iw.hub warehouse navigation example. But still I get the flickering screen on running the simulation. Is there any solution to this?

[Julie_A]

@robocist404 @Hammad_M

I only succeeded in “standalone” mode, it still doesn’t work in GUI model.

[Sirens_uk]

Looking thru the code there seems to be work on going in the backend. there is a new RtxLidar class which i think will be the “final” form, this is good as it seems to simplify the process of setting up Lidars and extracting lidar data to the delight of everyone involved in RTX lidar.

If you look at

pkg\isaac-sim-4.2.0\standalone_examples\api\omni.isaac.debug_draw\rtx_lidar.py

it uses

_, sensor = omni.kit.commands.execute(
    "IsaacSensorCreateRtxLidar",
    path="/sensor",
    parent=None,
    config="Example_Rotary",
    translation=(0, 0, 1.0),
    orientation=Gf.Quatd(1.0, 0.0, 0.0, 0.0),  # Gf.Quatd is w,i,j,k
)

Where as

pkg\isaac-sim-4.2.0\standalone_examples\api\omni.isaac.sensor\rotating_lidar_rtx.py

uses

from omni.isaac.sensor import LidarRtx

if you go down the rabbit hole here it basically leads back to the same code block.

I would look at

pkg\isaac-sim-4.2.0\exts\omni.isaac.extension_templates\template_source_files\scripting_workflow

then look at line 127 in the extension.py this is the template for making extensions.

I think you need to run the lidar get_data/update in this

    def _on_physics_step(self, step):
        # HERE
        self.ui_builder.on_physics_step(step)

I’m pretty sure this is because you aren’t in sync with the physx simulation, as I understand it the reason the standalone works is that it is all inside the _on_physics_step method.

Also if you look in

    def _data_acquisition_callback(self, event: carb.events.IEvent):
        self._current_frame["rendering_frame"] = (
            og.Controller()
            .node("/Render/PostProcess/SDGPipeline/PostProcessDispatcher")
            .get_attribute("outputs:referenceTimeNumerator")
            .get()
        )

and

# RTX sensors are cameras and must be assigned to their own render product
hydra_texture = rep.create.render_product(sensor.GetPath(), [1, 1], name="Isaac")

simulation_context = SimulationContext(physics_dt=1.0 / 60.0, rendering_dt=1.0 / 60.0, stage_units_in_meters=1.0)
simulation_app.update()

# Create Point cloud publisher pipeline in the post process graph
writer = rep.writers.get("RtxLidar" + "ROS2PublishPointCloud")
writer.initialize(topicName="point_cloud", frameId="sim_lidar")
writer.attach([hydra_texture])

# Create the debug draw pipeline in the post process graph
writer = rep.writers.get("RtxLidar" + "DebugDrawPointCloud")
writer.attach([hydra_texture])

The long and short is it all seems to only work properly if it’s sync’d with the sim app / physx step, I’m pretty sure this is what’s going on here.

I’m looking into doing this myself, but I have some other work to complete before I get to this stage.

Going back to the

pkg\isaac-sim-4.2.0\exts\omni.isaac.extension_templates\template_source_files\scripting_workflow

you would want to create the lidar inside on_startup then collect the data and do debug draw inside of the on_step block.

class Extension(omni.ext.IExt):
    def on_startup(self, ext_id: str):
        """Initialize extension and UI elements"""

        self.ext_id = ext_id
        self._usd_context = omni.usd.get_context()

        # Build Window
        self._window = ScrollingWindow(
            title=EXTENSION_TITLE, width=600, height=500, visible=False, dockPreference=ui.DockPreference.LEFT_BOTTOM
        )
        self._window.set_visibility_changed_fn(self._on_window)

        action_registry = omni.kit.actions.core.get_action_registry()
        action_registry.register_action(
            ext_id,
            f"CreateUIExtension:{EXTENSION_TITLE}",
            self._menu_callback,
            description=f"Add {EXTENSION_TITLE} Extension to UI toolbar",
        )
        self._menu_items = [
            MenuItemDescription(name=EXTENSION_TITLE, onclick_action=(ext_id, f"CreateUIExtension:{EXTENSION_TITLE}"))
        ]

        add_menu_items(self._menu_items, EXTENSION_TITLE)

        # Filled in with User Functions
        self.ui_builder = UIBuilder()

        # Events
        self._usd_context = omni.usd.get_context()
        self._physxIFace = _physx.acquire_physx_interface()
        self._physx_subscription = None
        self._stage_event_sub = None
        self._timeline = omni.timeline.get_timeline_interface()

hopefully this works for you but i can totally see why this would be the issue and it would make sense that it only works in sim app and not the normal launcher method as its not syncd…

If you get it all working give me a shout I might build an ext doing this if i get time this week!

Cheers

Scott

[chsu8]

Hi @Sirens_uk did you happen to continue to work on these problems? I too am having issues with the RTX lidar and radar sensors. I am using now isaac sim 4.5. This is my current setup:

Whether I am using the script editor and following rtx lidar sensor or the standalone scripts given by ./python.sh standalone_examples/api/isaacsim.util.debug_draw/rtx_lidar.py I continue to have the same error.

I believe this error occurs when building the render product in step 2.
# 2. Create and Attach a render product to the camera render_product = rep.create.render_product(sensor.GetPath(), [1, 1])

I have also tried to follow the solutions mentioned prior to your most recent exposition into the problem to no avail.

It does seem that the PhysX lidar sensor works for me but I’d like to use a specific RTX sensor that is provided, i.e. the Hesai XT32. Thank you!

[Sirens_uk]

Looks like a driver issue, I’d check your driver.

I think there is another bug in the system as I was playing with it tonight and the point cloud seems to hang when you move the sensor…

I need to look into this more…

[chsu8]

Hi, please see this other thread where I figured out the driver issue which had to do with updating my cuda toolkit to 11.8. forum link

I believe I saw this issue to, let me know what you find.