Remove elements from the depth frame received from RealSense camera before rendering it

Isaac Sim Version

  • 4.0.0

Operating System

  • Ubuntu 22.04

GPU Information

  • Model: RTX 3080 + RTX 3090
  • Driver Version: 535.183.01

Goal

To edit the scene we get from the Intel RealSense 415 before it is rendered

Description

Currently I am getting the depth data of the whole scene, I need to remove some of the elements from the scene;
Like, to remove the objects common in both the simulated scene and the real scene. How can I modify the data before rendering it. I have tried frame by frame manipulation using SSIM but the depth result has to be converted into numpyarray format for calculation, but we can only render the depth frame format and the conversion back to depth frame from numpyarray is not natively possible yet (as I found when I searched).

What are some of the other tools or methods I should look into to get the expected result.