Isaac Sim RTX LiDAR Sensor interacting with Non Visual Materials

Omniverse Isaac Sim
, , ,
1

Topic Description

I need to utilize Isaac Sim with an RTX LiDAR Sensor to correctly simulate the interaction with objects that are absorptive or highly reflective. My hope is that the point cloud I see will be either sparse or highly noisy for that specific object.

I am modelling my Lidar on the HESAI JT128 Lidar. This is my config:

def OmniLidar "JT_128_10hz" (

        apiSchemas = ["OmniSensorGenericLidarCoreAPI"]

    )

    {

        float omni:sensor:Core:avgPowerW = 0.002

        float omni:sensor:Core:azimuthErrorMean = 0

        float omni:sensor:Core:azimuthErrorStd = 0.03

        float omni:sensor:Core:elevationErrorMean = 0

        float omni:sensor:Core:elevationErrorStd = 0.03

        float[] omni:sensor:Core:emitterState:s001:azimuthDeg = [-1.08, -9.71, -1.09, -9.71, -1.10, -9.72, -1.12, -9.73, 2.69, 11.33,  2.68, 11.34,  2.68, 11.36,  2.68, 11.39, -1.19, -9.92, -1.21, -9.98, -1.22, -10.04, -1.24, -10.11, 2.50, 11.47,  2.51, 11.56,  2.52, 11.65,  2.54, 11.75, -1.33, -10.60, -1.35, -10.71, -1.37, -10.84, -1.39, -10.97, 2.64, 12.39,  2.67, 12.55,  2.70, 12.71,  2.73, 12.89, -1.51, -11.77, -1.54, -11.96, -1.57, -12.15, -1.60, -12.36, 2.94, 13.96,  2.98, 14.21,  3.04, 14.48,  3.09, 14.76, -1.77, -13.62, -1.80, -13.91, -1.85, -14.22, -1.89, -14.55, 3.44, 16.55,  3.53, 16.98,  3.62, 17.43,  3.72, 17.92, -2.14, -16.65, -2.20, -17.16, -2.26, -17.70, -2.33, -18.30, 4.37, 21.24,  4.54, 22.06,  4.73, 22.96,  4.93, 23.96, -2.77, -22.52, -2.88, -23.61, -3.01, -24.83, -3.15, -26.20, 6.43, 32.52,  6.87, 34.78,  7.39, 37.40,  8.01, 40.46, -4.25, -41.40, -4.60, -45.72, -5.04, -50.95, -5.61, -57.28, 26.63, 31.42, 38.15, 47.90, -6.32, -0.86, 18.19, 117.30]

        uint[] omni:sensor:Core:emitterState:s001:channelId = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128]

        float[] omni:sensor:Core:emitterState:s001:distanceCorrectionM = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

        float[] omni:sensor:Core:emitterState:s001:elevationDeg = [-4.43, -3.50, -2.55, -1.65, -0.75,  0.13,  0.99,  1.85, 2.64,  3.46,  4.29,  5.09,  5.91,  6.69,  7.49,  8.25, 9.08,  9.84, 10.59, 11.34, 12.08, 12.81, 13.54, 14.26, 15.11, 15.83, 16.53, 17.23, 17.93, 18.62, 19.31, 19.99, 20.71, 21.38, 22.07, 22.73, 23.42, 24.07, 24.76, 25.40, 26.06, 26.73, 27.39, 28.05, 28.71, 29.35, 30.02, 30.66, 31.37, 32.05, 32.68, 33.34, 33.99, 34.63, 35.29, 35.92, 36.56, 37.23, 37.86, 38.51, 39.15, 39.79, 40.45, 41.07, 41.78, 42.47, 43.08, 43.75, 44.37, 45.02, 45.67, 46.29, 46.93, 47.65, 48.23, 48.91, 49.53, 50.18, 50.82, 51.44, 52.17, 52.84, 53.47, 54.11, 54.78, 55.38, 56.09, 56.66, 57.36, 58.11, 58.68, 59.37, 60.00, 60.64, 61.33, 61.90, 62.71, 63.44, 64.06, 64.72, 65.41, 66.00, 66.78, 67.29, 68.10, 69.12, 69.49, 70.36, 70.89, 71.58, 72.32, 72.77, 73.88, 75.22, 75.36, 76.40, 76.87, 77.51, 78.41, 78.51, 79.62, 81.04, 82.40, 83.64, 85.36, 87.02, 88.64, 88.99]

        uint[] omni:sensor:Core:emitterState:s001:fireTimeNs = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

        float[] omni:sensor:Core:emitterState:s001:focalDistM = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

        float[] omni:sensor:Core:emitterState:s001:focalSlope = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

        float[] omni:sensor:Core:emitterState:s001:horOffsetM = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

        float[] omni:sensor:Core:emitterState:s001:reportRateDiv = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

        float[] omni:sensor:Core:emitterState:s001:vertOffsetM = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

        float omni:sensor:Core:farRangeM = 60

        float omni:sensor:Core:focusDistM = 0.3

        token omni:sensor:Core:intensityMappingType = "LINEAR"

        token omni:sensor:Core:intensityProcessing = "NORMALIZATION"

        uint omni:sensor:Core:maxReturns = 2

        float omni:sensor:Core:minReflectance = 0.1

        float omni:sensor:Core:minReflectionRangeM = 40

        float omni:sensor:Core:nearRangeM = 0.3

        uint omni:sensor:Core:numberOfChannels = 128

        uint omni:sensor:Core:numberOfEmitters = 128

        uint omni:sensor:Core:pulseTimeNs = 10

        float omni:sensor:Core:rangeAccuracyM = 0.03

        float omni:sensor:Core:rangeResolutionM = 0.004

        token omni:sensor:Core:rayType = "IDEALIZED"

        uint omni:sensor:Core:reportRateBaseHz = 36000

        token omni:sensor:Core:rotationDirection = "CW"

        uint omni:sensor:Core:scanRateBaseHz = 20

        token omni:sensor:Core:scanType = "ROTARY"

        bool omni:sensor:Core:skipDroppingInvalidPoints = 1

        float omni:sensor:Core:waveLengthNm = 905

        string omni:sensor:modelName = "JT_128_10hz"

        float omni:sensor:tickRate = 20

        token ui:displayGroup = "Material Graphs"

        token ui:displayName = "JT_128_10hz"

        int ui:order = 1024

        float3 xformOp:rotateXYZ = (0, 0, 0)

        float3 xformOp:scale = (1, 1, 1)

        double3 xformOp:translate = (0, 0, 0.04784)

        uniform token[] xformOpOrder = ["xformOp:translate", "xformOp:rotateXYZ", "xformOp:scale"]

    }

I followed the instructions step by step to set the nodes and the Non Visual Materials but the Point Cloud looks the same no matter what material I apply to the object. Including the flags mentioned.

-–/app/sensors/nv/lidar/enableRtxReflectanceInformation=True 

--/app/sensors/nv/lidar/enableAdditionalRtxReturnInformation=True 

--/app/sensors/nv/lidar/enableRtxSensorGeometry=True 

--/app/sensors/nv/lidar/enablePolarization=True 

--/app/sensors/nv/materials/resetMaterials=True 

--/app/sensors/nv/materials/enableMaterialInterLayerContribution=True

I am the running Isaac Sim 5.1.0 executable on ubuntu 22.04 with ROS Humble.

image
image1920×1034 289 KB

2

Thank you for posting this. Isaac Sim 5.1.0 is designed to support RTX LiDAR simulation that considers non-visual material properties—meaning that it should allow you to model point clouds with behaviors such as sparsity or noise when objects are configured to be absorptive or highly reflective. However, if your point cloud does not change regardless of which materials are applied—even after enabling all the correct RTX material interaction flags—you may be running into current known limitations or missing some configuration steps in the material setup pipeline.

Critical Steps and Checks

  • Non-Visual Material Assignment: Ensure that the materials are correctly assigned at the USD/Omniverse level—not just visually but as sensor materials, using the attributes required by the RTX sensors system. Look for sensor material types specified with USD attributes in accordance with the official Isaac Sim non-visual materials documentation for 5.1.0. Visual materials alone do not affect sensor readings.
  • Material ID Debug View: Use the Non-Visual Material ID debug mode in Isaac Sim (found in the Debug View options) to confirm that correct sensor material IDs are registered and visually mapped to your objects. Each material ID should show up as a distinct color in the debug view.
  • Supported Materials: Only the designated sensor material types influence RTX Lidar behavior. Supported types include “reflective”, “absorptive”, and “composite”, among others, which must be directly referenced in the USD or through the Python APIs. Do not rely on ordinary PBR materials for LiDAR interaction.
  • Required Extension Enablement: Make sure the omni.sensors.nv.materials extension is loaded and active. This extension is critical for enabling non-visual material handling, and missing it will result in unchanged point clouds.
  • API Attribute Application: When applying materials, use the correct API attributes as shown in the NVIDIA documentation—such as setting omni.sensors.nv.materials:reflectance, absorption, or composite parameters, not just surface color or visual roughness.
3

Hello!

We noticed that this topic hasn’t received any recent responses, so we are closing it for now to help keep the forum organized.

If you’re still experiencing this issue or have additional questions, please feel free to create a new topic with updated details. When doing so, we recommend mentioning or linking to this original topic in your new post—this helps provide context and makes it easier for others to assist you.

Thank you for being part of the NVIDIA Isaac Sim community.

Best regards,
The NVIDIA Isaac Sim Forum Team