Getting a scene's normal map

[xDynames]

At present I have been using Isaac’s synthetic data helper to generate ground truth data. I would also like to capture the normal maps of objects in the scene. I got the impression this would be supported due to the preview available in the GUI. It looks like the API has been setup to do this but it is missing from the configuration dictionary as a valid option. So far I have dug a little deeper and tried to call it directly using:

normals = self._sd_helper.sensor_helper_lib.get_normals(self._viewport)

But this yields an error:

ValueError: cannot reshape array of size 0 into shape (0,0,newaxis)

Almost like the buffer was empty when the request was made. Am I missing something about how to use this? Is there a more standard way?

Any help would be much appreciated

[xDynames]

After some more digging I found I can modify the SyntheticDataHelperclass to allow get_groundtruth to accept "normals" as a valid key. To do this I modified SyntheticDataHelper’s constructor in .local/share/ov/pkg/isaac_sim-2021.1.1/exts/omni.isaac.synthetic_utils/omni/isaac/synthetic_utils/scripts/sytheticdata.py
by adding this pair to the dictionary self.sensor_helpers:

"normals": sensors.get_normals,

And this pair to the self.sensor_types dictionary:

"normals": self.sd.SensorType.Normal,

After doing this you will be able to add "normals" to your list of ground truth and SyntheticDataHelper will retrieve it for you along with the other requested types.

Instead of modifying source you can use this monkey patch for now:

    def _get_custom_synthetic_data_helper():
        sd_helper = SyntheticDataHelper()
        sd_helper.sensor_helpers["normals"] = omni.syntheticdata.sensors.get_normals
        sd_helper.sensor_types["normals"] = sd_helper.sd.SensorType.Normal
        return sd_helper