Isaac Sim RTX LiDAR Sensor interacting with transparent and reflective objects

Hello everyone,

in my current usecase i need to utilize Isaac Sim with an LiDAR Sensor that correctly interacts with objects that are transparent or highly reflective. Therefore using the PhysX based LiDAR Sensor won’t be any use for me, since it only considers the PhysX collision API.
For this reason im using the RTX LiDAR Sensor offered by Isaac Sim, in hopes of better results. I followed the instructions step by step and didn’t encounter any problems there (as far as i know). Im running all Isaac Sim instances on an Isaac Sim docker container, and i utilize the following shell script to setup livestream capabilities and remaining RTX Sensor Material configurations:

# Fix the livestream
sed -i '550s/.*/"omni.isaac.quadruped" = {order = 10}/' ./apps/omni.isaac.sim.python.kit

# Include sensor material map for RTX LiDAR
map='rtx.materialDb.rtSensorNameToIdMap="DefaultMaterial:0;AsphaltStandardMaterial:1;AsphaltWeatheredMaterial:2;VegetationGrassMaterial:3;WaterStandardMaterial:4;GlassStandardMaterial:5;FiberGlassStandardMaterial:6;MetalAlloyMaterial:7;MetalAluminumMaterial:8;MetalAluminumOxidizedMaterial:9;PlasticStandardMaterial:10;RetroMarkingsMaterial:11;RetroSignMaterial:12;RubberStandardMaterial:13;SoilClayMaterial:14;ConcreteRoughMaterial:15;ConcreteSmoothMaterial:16;OakTreeBarkMaterial:17;FabricStandardMaterial:18;PlexiGlassStandardMaterial:19;MetalSilverMaterial:20"'
sed -i '61i\'"$map"'' ./exts/omni.isaac.sensor/config/extension.toml

# Add custom helios config to paths
directory_path='"${app}/../Isaac-Sim-Playground/install/",'
sed -i "58i${directory_path}" ./exts/omni.isaac.sensor/config/extension.toml

I didn’t modify the csv file located at ./kit/rendering-data/RtxSensorMaterialMap.csv, since i tried to use standard Materials. These are already defined inside this csv file, for example OmniGlass is mapped to: omniglass,GlassStandardMaterial.
The results can be seen below:


Omniverse-Streaming-Client-2023-09-06-15-14-41

It appears that the rays are wrapping around the glass wall in the first screenshot, and in the following gif it can be seen that the rays don’t hit the diffuse red cube behind the glass wall. Futhermore if a cube is put inside a glass cube, the rays travel through the glass cube and hit the red cuboid inside, but not a cube outside of it.
Omniverse-Streaming-Client-2023-09-06-15-16-38

Changing material properties such as IoR, or renderer settings such as max bounces, doesn’t appear to change anything. My question here is if material properties or render settings have any influence at all on the rays from the RTX LiDAR sensor. Also, i couldn’t figure out how to correctly modell a mirroring object that correclty interacts with the LiDAR sensor, since in a real world scenario the LiDAR shouldn’t detect it as well since the rays won’t travel back to the sensor.
In another post of mine i encountered an unrelated problem to this but also in regards to the RTX LiDAR sensor.

I hope an answer can be found for this.
Many thanks in advance

It looks like you have it mostly working! That’s great, and if you want to include the usd file for the scene we may even be able to make a good example out of it!
As for only hitting 2 sides of the glass, or one side and then the red cube… what is your maxReturns parameter? In the Example_Rotary it is “maxReturns”: 2. If you want it to hit more surfaces then 2, you will need to increase that number. Let me know if that works for you!

No, the ior and other visual material properties have no affect on the sensor material. So it’s only the set ones at RTX Lidar Sensor — Omniverse IsaacSim latest documentation which work. Maybe MetalSilver will work for a mirror?

Good luck!

Mark

Hey Mark, thanks for the reply.
You were right, i used as a parameter for maxReturns 2 and changing it to 3 did the trick! Now the RTX LiDAR sensor detects the red cube behind the glass wall. Here is how it looks like now:
Omniverse-Streaming-Client-2023-09-15-16-42-01

I also tried your advice on using MetalSilver for my mirror material and modified the
RtxSensorMaterialMap.csv file accordingly with the line mirror,MetalsilverMaterial. However this didn’t seem to do anything and the red cuboid isn’t really detected from angels where the mirror should reflect the rays. Here is how it looked like:
Omniverse-Streaming-Client-2023-09-15-17-18-14

Another question that i asked myself is, if it is possible to exclude the responses i get from the RTX LiDAR sensor which hit the glass wall, since it should not be visible and should not be detected from the sensor. The same applies for a mirroring object.
Finally as you wished, here is the usd file for the scene i used :)
rtx_lidar_glass_example.usd (6.3 KB)
I also used a custom config file for my RTX LiDAR sensor since im trying to replicate it within Isaac Sim. However its not fully accurate and i used the Hesai_XT32_SD10.json config as a template. Hopefully it will help:
RS-Helios-32-5515.json (2.4 KB)

So i have been doing a bit more testing and researched the exact specifications of the LiDAR sensor i used such that i can create a more accurate LiDAR config file.
First, i had to put the scanRateBaseHz to 60Hz, otherwise i wouldn’t get a full circular pattern, like i do when setting the rotation_rate of a PhysX LiDAR to 0.0. In my point cloud data from the real scans it was always a full circular pattern.
The best pulse time that worked for me were 1736.25ns, which seemed a bit large compared to other LiDAR config files where it was set lower then 10ns (in my config file i documentated how i came to this number).
Lastly, i tested out the ray type. With the default idealized one, i get pretty good results, however i still couldn’t figure out how to extract different return types from the point cloud, as i only want the Last Return, such that the glass won’t be visible. I hope there is a way for extracting only the specific return types. Here is how it looks like:
Omniverse Streaming Client 2023-09-29 15-41-39

When i tried out the gaussian beam, to achieve more realism, i got very strange results where the point cloud of the object would move into the inverse direction. The following gif illustrates this:
Omniverse Streaming Client 2023-09-29 15-37-47

I still didn’t manage to replicate mirrors with the LiDAR sensor, so thats still an issue. The maxReturns parameters is still set to 3.
Here is also my updated config file (For some reason i couldn’t upload a .json file):
RS-Helios-32-5515.txt (2.8 KB)

If have yet to do my own test, but I have stumbled over what seems to be retroreflective sensor materials (11 & 12)
image
in the Sensor Materials Documentation. Maybe you can build your mirror with those.

I have not seen mirrors give what people expect. It’s not like a ray bounces off the mirror and hits a wall and then the lidar can read the wall… the refection would have to come back to the lidar return to be seen.

You may want to try increasing the avgPowerW instead of the pulse time. That way the laser timing will be correct, but the intensity that is returned may be greater.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.