Superpixels freespace segmentation for autonomous data collection

Hey everyone,
I’m trying to get a freespace DNN to work in my environment (following this tutorial), and need autonomous real-world image freespace labeling because I can’t work with simulation.

I need help with the following problems I’m facing :

  • I can’t find documentation on isaac::superpixels::RgbdSuperpixelFreespace component, and can’t seem to obtain good freespace segmentation results. I’m especially wondering how to tweak height_angle and height_tolerance parameters, and what’s their role.

  • From what I understand, this component also needs pose between camera and robot, the default one being [0.270598, -0.653281, 0.653281, -0.270598, 0.0, 0.0, 0.775]. What does this correspond to exactly? How could I tweak this for usage with Kaya?

  • When I try to run the training script on a prerecorded log containing depth and color image from the Realsense camera, the sample buffer successfully fills up, but I get this error :
    Number of samples in sample buffer: 0
    Number of samples in sample buffer: 1
    2020-06-22 14:48:22.626 ERROR ./messages/image.hpp@74: Image element type does not match: actual=1, expected=2
    This doesn’t crash the program but I still would like to fix it.

Can somebody help me with that?

Also I noticed that in packages/freespace_dnn/apps/freespace_dnn_data_annotation.subgraph.json , value for “label_invalid” parameter was originally 0.2, but this parameter expects an int, which was also giving me an error.

Sill stuck on this, I really would need some help.
I’d like at least some documentation on RgbdSuperpixelFreespace component…

Hi there,

Sorry for the delay. Will make sure to get back to you soon!

1 Like

Hello atorabi,
Any update? :-)

Hello, thank you for the interest in freespace segmentation and apologies for the delay! Please find the answers to your queries below -

  1. The superpixels are transformed into the ground coordinate frame by assuming that the ground plane will be Z = 0 in the ground coordinate frame.
    The height_tolerance parameter is a threshold on the height (z axis) between the superpixel position and ground plane to decide if a superpixel is part of the ground plane. This helps us provide some flexibility in deciding which pixels are part of the ground plane.
    The height_angle parameter helps increase the height_tolerance based on the distance of the superpixel from the robot. The sine of this angle is added to the height_tolerance.

  2. The default pose was given as an example pose transformation between Carter and its camera. The first 4 values in the pose corresponds to the rotation transform between the camera and the robot (in quaternion form), while the last 3 correspond to the translation (X, Y and Z). Isaac SDK also supports a more straightforward form of specifying the pose in the following format-
    “PoseInitializer”: {
    “lhs_frame”: “camera”,
    “rhs_frame”: “robot”,
    “pose”: {
    “translation”: [1.0, 0.0, 0.0],
    “rotation”: {
    “yaw_degrees”: 90.0
    In this way, the translation values and rotation angles can be directly filled in for Kaya.

  3. Would you mind sharing if you’ve seen this affect any part of the application, for example, the viewers in Sight? This would help us narrow down the source!

thanks for your answer.

It doesn’t seem to affect any other part of the app (packages/freespace_dnn/apps:freespace_dnn_training), there is no other errors, the app creates network checkpoints regularly and ends when there is no more image in the replay file. In Sight, I can see the images used for training (color - depth - segmentation). I have no idea if the training is actually taking place, I didn’t test the resulting networks as I would like to have freespace segmentation working first for correct annotation.

EDIT : I forgot that the app was opening a tensorflow instance, I went and checked it : I can correctly see input_image, however ground_truth is completelly black! So apparently the problem is linked to this, and training is not taking place…

Also, as you can see I have not managed to make segmentation work, even after changing the pose. Note that I don’t have Kaya built at the moment, I’m using the Realsense on a small tripod, but once it will be used on Kaya, pose should be similar.

(freespace segmentation corresponds to the green pixels in segmentation_viewer window)

Do you think these results are because of wrong pose, or wrong freespace parameters (height_angle, height_tolerance, or maybe normal_threshold?)