Isaac Sim RTX LiDAR Sensor interacting with transparent and reflective objects

[pat_n22]

Hello everyone,

in my current usecase i need to utilize Isaac Sim with an LiDAR Sensor that correctly interacts with objects that are transparent or highly reflective. Therefore using the PhysX based LiDAR Sensor won’t be any use for me, since it only considers the PhysX collision API.
For this reason im using the RTX LiDAR Sensor offered by Isaac Sim, in hopes of better results. I followed the instructions step by step and didn’t encounter any problems there (as far as i know). Im running all Isaac Sim instances on an Isaac Sim docker container, and i utilize the following shell script to setup livestream capabilities and remaining RTX Sensor Material configurations:

# Fix the livestream
sed -i '550s/.*/"omni.isaac.quadruped" = {order = 10}/' ./apps/omni.isaac.sim.python.kit

# Include sensor material map for RTX LiDAR
map='rtx.materialDb.rtSensorNameToIdMap="DefaultMaterial:0;AsphaltStandardMaterial:1;AsphaltWeatheredMaterial:2;VegetationGrassMaterial:3;WaterStandardMaterial:4;GlassStandardMaterial:5;FiberGlassStandardMaterial:6;MetalAlloyMaterial:7;MetalAluminumMaterial:8;MetalAluminumOxidizedMaterial:9;PlasticStandardMaterial:10;RetroMarkingsMaterial:11;RetroSignMaterial:12;RubberStandardMaterial:13;SoilClayMaterial:14;ConcreteRoughMaterial:15;ConcreteSmoothMaterial:16;OakTreeBarkMaterial:17;FabricStandardMaterial:18;PlexiGlassStandardMaterial:19;MetalSilverMaterial:20"'
sed -i '61i\'"$map"'' ./exts/omni.isaac.sensor/config/extension.toml

# Add custom helios config to paths
directory_path='"${app}/../Isaac-Sim-Playground/install/",'
sed -i "58i${directory_path}" ./exts/omni.isaac.sensor/config/extension.toml

I didn’t modify the csv file located at ./kit/rendering-data/RtxSensorMaterialMap.csv, since i tried to use standard Materials. These are already defined inside this csv file, for example OmniGlass is mapped to: omniglass,GlassStandardMaterial.
The results can be seen below:

It appears that the rays are wrapping around the glass wall in the first screenshot, and in the following gif it can be seen that the rays don’t hit the diffuse red cube behind the glass wall. Futhermore if a cube is put inside a glass cube, the rays travel through the glass cube and hit the red cuboid inside, but not a cube outside of it.

Changing material properties such as IoR, or renderer settings such as max bounces, doesn’t appear to change anything. My question here is if material properties or render settings have any influence at all on the rays from the RTX LiDAR sensor. Also, i couldn’t figure out how to correctly modell a mirroring object that correclty interacts with the LiDAR sensor, since in a real world scenario the LiDAR shouldn’t detect it as well since the rays won’t travel back to the sensor.
In another post of mine i encountered an unrelated problem to this but also in regards to the RTX LiDAR sensor.

I hope an answer can be found for this.
Many thanks in advance

[mcarlson1]

It looks like you have it mostly working! That’s great, and if you want to include the usd file for the scene we may even be able to make a good example out of it!
As for only hitting 2 sides of the glass, or one side and then the red cube… what is your maxReturns parameter? In the Example_Rotary it is “maxReturns”: 2. If you want it to hit more surfaces then 2, you will need to increase that number. Let me know if that works for you!

No, the ior and other visual material properties have no affect on the sensor material. So it’s only the set ones at RTX Sensor Materials — Omniverse IsaacSim latest documentation which work. Maybe MetalSilver will work for a mirror?

Good luck!

Mark

[pat_n22]

Hey Mark, thanks for the reply.
You were right, i used as a parameter for maxReturns 2 and changing it to 3 did the trick! Now the RTX LiDAR sensor detects the red cube behind the glass wall. Here is how it looks like now:

I also tried your advice on using MetalSilver for my mirror material and modified the
RtxSensorMaterialMap.csv file accordingly with the line mirror,MetalsilverMaterial. However this didn’t seem to do anything and the red cuboid isn’t really detected from angels where the mirror should reflect the rays. Here is how it looked like:

Another question that i asked myself is, if it is possible to exclude the responses i get from the RTX LiDAR sensor which hit the glass wall, since it should not be visible and should not be detected from the sensor. The same applies for a mirroring object.
Finally as you wished, here is the usd file for the scene i used :)
rtx_lidar_glass_example.usd (6.3 KB)
I also used a custom config file for my RTX LiDAR sensor since im trying to replicate it within Isaac Sim. However its not fully accurate and i used the Hesai_XT32_SD10.json config as a template. Hopefully it will help:
RS-Helios-32-5515.json (2.4 KB)