In Isaac Sim 5.1, the data obtained from the RTX sensor is all empty

[Julie_A]

I’m very sorry, but I have some confusion while using the RTX sensor.
First, I created a very simple scene with only a cube in it.

Additionally, I ran the official example code, which includes both the cube and the RTX sensor. However, the data I’m getting is all empty, and I’m not sure why.

{'rendering_time': 62.80000327527523, 'rendering_frame': (3781, 30), 'IsaacExtractRTXSensorPointCloudNoAccumulator': {'azimuth': array([], dtype=float64), 'beamId': array([], dtype=float64), 'data': array([], dtype=float64), 'distance': array([], dtype=float64), 'elevation': array([], dtype=float64), 'emitterId': array([], dtype=float64), 'index': array([], dtype=float64), 'intensity': array([], dtype=float64), 'materialId': array([], dtype=float64), 'normal': array([], dtype=float64), 'objectId': array([], dtype=float64), 'timestamp': array([], dtype=float64), 'velocity': array([], dtype=float64), 'info': {'numChannels': 0, 'numEchos': 0, 'numReturnsPerScan': 0, 'renderProductPath': '/Render/OmniverseKit/HydraTextures/Replicator', 'ticksPerScan': 0, 'transform': array([ 1.,  0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  1.,  0., -0.,
       -0., -0.,  1.]), 'azimuth': array([], dtype=float64), 'beamId': array([], dtype=float64), 'distance': array([], dtype=float64), 'elevation': array([], dtype=float64), 'emitterId': array([], dtype=float64), 'index': array([], dtype=float64), 'intensity': array([], dtype=float64), 'materialId': array([], dtype=float64), 'normal': array([], dtype=float64), 'objectId': array([], dtype=float64), 'timestamp': array([], dtype=float64), 'velocity': array([], dtype=float64)}}}

from isaacsim import SimulationApp
kit = SimulationApp({"headless": False})
from isaacsim.core.utils.stage import add_reference_to_stage
import numpy as np
import omni
from isaacsim.sensors.rtx import LidarRtx
from isaacsim.core.api import World

add_reference_to_stage(usd_path="/home/a/OM/test/cube.usd",
                       prim_path="/World")

# Create the RTX Lidar with the specified attributes.
sensor = LidarRtx(
    prim_path="/World/lidar",
    translation=np.array([0.0, 0.0, 0.0]),
    orientation=np.array([1.0, 0.0, 0.0, 0.0]),
    config_file_name="Example_Rotary",
)

# Initialize the LidarRtx object, which creates a render product for the sensor.
sensor.initialize()

# Attach an annotator to the sensor.
sensor.attach_annotator("IsaacExtractRTXSensorPointCloudNoAccumulator")

# Play the timeline to initialize the OmniGraph associated with the annotator and render product,
# and begin collecting data.
timeline = omni.timeline.get_timeline_interface()
timeline.play()

# Collect data from the annotator on each simulation frame.
while kit.is_running():
    # Step the simulation
    kit.update()
    # Print data collected by each annotator attached to the sensor as a Python dict
    print(sensor.get_current_frame())

timeline.stop()
kit.close()

[phennings]

Thank you for posting this. If your RTX Lidar sensor is returning all empty data in a simple scene (e.g., just a cube), the most common causes are related to USD asset configuration, render product setup, timeline playback, or Lidar parameters not matching the geometry present in the scene. Here are practical steps to diagnose and fix the issue, sourced from official NVIDIA Isaac Sim documentation and verified engineering practice:

Step-by-step Troubleshooting

1. Check USD Asset Properties

• Make sure your cube USD asset has proper collision, is visible, and has a valid material for RTX sensors to detect it. Minimal geometry or missing collision may result in zero returns.
• In USD Composer or Isaac Sim GUI, select the cube and ensure collision is enabled and the object is “renderable” and visible in the scene.

2. Sensor Placement and Orientation

• Place the Lidar at a position and orientation where it can “see” the cube. For example, moving the sensor above or to the side (e.g., translation=) can help.
• If placed at the same position as the cube (all zeros), the cube and sensor might overlap, leading to no hits.

3. Sensor Configuration File Validation

• The config_file_name="Example_Rotary" points to a config file that must fit your scene scale and object placement.
• Ensure the configuration sets a valid range, azimuth, elevation, and enough beams to intersect the cube. If the range is too large or too small, or the cube is out of FOV, you’ll get empty data.

4. Timeline Playback

• Data from sensors is only filled when the simulation is playing. Your code does not show an explicit command to start the simulation.

timeline.play()

• Add this line after all initializations to ensure the simulation runs and RTX data is generated.

5. Synchronous Data Retrieval

• If you attempt to access sensor data before any frames have been rendered, arrays will be empty.
• Allow the simulation to run for several frames for data collection:

for _ in range(100):
    kit.update()

6. Correct Annotator Usage

• Ensure "IsaacExtractRTXSensorPointCloudNoAccumulator" is a valid annotator for your sensor and your Omniverse/Isaac Sim version.

Example Code Update (partial and still under review)

Here is an improved baseline code incorporating these suggestions:

from isaacsim import SimulationApp
kit = SimulationApp({"headless": False})
from isaacsim.core.utils.stage import add_reference_to_stage
import numpy as np
import omni
from isaacsim.sensors.rtx import LidarRtx

add_reference_to_stage(usd_path="/home/a/OM/test/cube.usd", prim_path="/World")

sensor = LidarRtx(
    prim_path="/World/lidar",
    translation=np.array([0.0, 0.0, 2.0]),              # Move above the cube
    orientation=np.array([1.0, 0.0, 0.0, 0.0]),
    config_file_name="Example_Rotary",
)
sensor.initialize()
sensor.attach_annotator("IsaacExtractRTXSensorPointCloudNoAccumulator")

timeline = omni.timeline.get_timeline_interface()
timeline.play()

for _ in range(100):
    kit.update()

data = sensor.get_annotator_data("IsaacExtractRTXSensorPointCloudNoAccumulator")
print(data)

Key changes:

• Sensor translation changed to to avoid overlap.
• Added explicit timeline.play().
• Waited 100 frames with kit.update() before fetching data.