CoarseToFineStereoDepth doesn't work with ImageLoaders

Hi,

I have tried to connected two ImageLoaders to a CoarseToFineStereoDepth instance. The left depth image output goes to a DepthCameraViewer.
Right and left image are 1279x1080 rgb png files. Both are rectified.
The tick_period of the ImageLoaders is 1Hz.

According to the statistics window neither CoarseToFineStereoDepth nor DepthCameraViewer work.
Also there is no depth channel available in the channel list.

Any suggestions?

Are there any restrictions for CoarseToFineStereoDepth, e.g. image size?

Hi Jemide

Have you looked at the following? What camera are you using? Can you give the details?

https://docs.nvidia.com/isaac/isaac/packages/stereo_depth/doc/c2f-stereo-depth.html

Also there is no depth channel available in the channel list.

https://docs.nvidia.com/isaac/isaac/doc/component_api.html#isaac-stereo-depth-coarsetofinestereodepth

Are there any restrictions for CoarseToFineStereoDepth, e.g. image size?

No known limitations. But, I’d like to learn about the camera you are using. Thanks!

Hello Shrinv,

thank you for your answer.

My goal is to try the stereo algorithm offline with a pair of test images.

As a starting point I used the sample ‘stereo_matching_depth’ and replaced the zed-camera with two ‘imageloader’ from the ‘dummy’ module.

Image data see above. The images are taken by two standard color cameras and have been rectified.
Reading and displaying the images works fine.

The configuration looks like this:

{
  "name": "stereo_matching_depth1",
  "modules": [
    "//packages/stereo_depth:coarse_to_fine",
    "dummies",
    "viewers"
  ],
  "graph": {
    "nodes": [
      {
        "name": "left_image",
        "components": [
          {
            "name": "isaac.alice.MessageLedger",
            "type": "isaac::alice::MessageLedger"
          },
          {
            "name": "isaac.dummies.ImageLoader",
            "type": "isaac::dummies::ImageLoader"
          }
        ]
      },
      {
        "name": "right_image",
        "components": [
          {
            "name": "isaac.alice.MessageLedger",
            "type": "isaac::alice::MessageLedger"
          },
          {
            "name": "isaac.dummies.ImageLoader",
            "type": "isaac::dummies::ImageLoader"
          }
        ]
      },
      {
        "name": "depth_visualization",
        "components": [
          {
            "name": "isaac.alice.MessageLedger",
            "type": "isaac::alice::MessageLedger"
          },
          {
            "name": "isaac.viewers.DepthCameraViewer",
            "type": "isaac::viewers::DepthCameraViewer"
          }
        ]
      },
      {
        "name": "depth_estimation",
        "components": [
          {
            "name": "isaac.alice.MessageLedger",
            "type": "isaac::alice::MessageLedger"
          },
          {
            "name": "isaac.stereo_depth.CoarseToFineStereoDepth",
            "type": "isaac::stereo_depth::CoarseToFineStereoDepth"
          }
        ]
      }
    ],
    "edges": [
      {
        "source": "left_image/isaac.dummies.ImageLoader/color",
        "target": "depth_estimation/isaac.stereo_depth.CoarseToFineStereoDepth/left_image"
      },
      {
        "source": "right_image/isaac.dummies.ImageLoader/color",
        "target": "depth_estimation/isaac.stereo_depth.CoarseToFineStereoDepth/right_image"
      },
      {
        "source": "depth_estimation/isaac.stereo_depth.CoarseToFineStereoDepth/left_depth_image",
        "target": "depth_visualization/isaac.viewers.DepthCameraViewer/depth_listener"
      }
    ]
  },
  "config": {
    "depth_estimation": {
      "isaac.stereo_depth.CoarseToFineStereoDepth": {
        "tick_period": "1Hz",
        "min_depth": 0.0,
        "max_depth": 1.0,
        "baseline" : 0.1
      }
    },
    "depth_visualization": {
      "isaac.viewers.DepthCameraViewer": {
	"target_fps": "1Hz",
        "tick_period": "1Hz",
        "min_visualization_depth": 0.0,
        "max_visualization_depth": 200.0,
	"colormap": [1.0, 1.0, 1.0]
      }
    },
    "left_image": {
      "isaac.dummies.ImageLoader": {
        "tick_period": "3s",
        "color_filename": "/home/jens/isaac/apps/samples/stereo_matching_depth1/board1_cam0_rectified_cropped.png",
	"focal_length": [1000,1000],
        "optical_center": [640, 540],
        "distortion_coefficients": [0.0, 0.0, 0.0, 0.0, 0.0]
      }
    },
    "right_image": {
      "isaac.dummies.ImageLoader": {
        "tick_period": "3s",
        "color_filename": "/home/jens/isaac/apps/samples/stereo_matching_depth1/board1_cam1_rectified_cropped.png",
	"focal_length": [1000,1000],
        "optical_center": [640, 540],
        "distortion_coefficients": [0.0, 0.0, 0.0, 0.0, 0.0]
      }
    },
    "websight": {
      "WebsightServer": {
        "tick_period": "1Hz",
        "port": 3000,
        "ui_config": {
          "windows": {
            "Depth": {
              "renderer": "2d",
              "channels": [
                { "name": "stereo_matching_depth1/depth_visualization/isaac.viewers.DepthCameraViewer/Depth" }
              ]
            },
            "Cameras": {
              "renderer": "2d",
              "channels": [
                { "name": "stereo_matching_depth1/color_visualization_left/isaac.viewers.ColorCameraViewer/Color" },
                {  "name": "stereo_matching_depth1/color_visualization_right/isaac.viewers.ColorCameraViewer/Color" }
              ]
            }
          }
        }
      }
    }
  }
}

The whole thing leads to the (mis)behavior described above.

Hi, jemide,

Unfortunately two image loaders won’t trigger CoarseToFineStereoDepth properly as it tries to tick on synchronized messages for left_image and right_image, i.e., same acqTime on them.

You may need to code your own ImageLoader for that.