Custom Python API Isaac module to serve images

I am trying to build a small Python application with the Isaac SDK which reads a pair of RGB-D images, sends it to the superpixels module, then retrieves the superpixels. I have followed the PyAlice buffer app test to load and send images and the configurations in the superpixels package to write my configuration.

The following is my configuration which works. Notice that this connectes Nvidia’s image_loader to the superpixel interface. What does not work is replacing the image_loader sources with my own ImageProvider on the relevant edges. When using the image_loader, the length of the output superpixels is correctly displayed (see python code below) and I can see the output in WebSight as well. When using my ImageProvider, no output is received, and the superpixels channels do not appear in WebSight.
As per the python code, I am the messages of the image_loader and ImageProvider seem to be the same (checking the proto and the buffer). I was also able to check that the input gets at least to rgbd_processing/DepthEdges as putting the wrong information into the proto (e.g uint16 instead of float32 for the depth image) produces an error.

Any help would be appreciated.

Configuration (.app.json file):

{
   "name":"segmentation",
   "modules":[
      "superpixels",
      "viewers",
      "message_generators",
      "sight"
   ],
   "graph":{
      "nodes":[
         {
            "name":"segmentation_node",
            "components":[
               {
                  "name": "ImageProvider",
                  "type": "isaac::alice::PyCodelet"
               },
               {
                  "name": "SuperpixelConsumer",
                  "type": "isaac::alice::PyCodelet"
               },
               {
                  "name": "message_ledger",
                  "type": "isaac::alice::MessageLedger"
               },
               {
                  "name":"CameraIntrinsicsGenerator",
                  "type":"isaac::message_generators::CameraIntrinsicsGenerator"
               },
               {
                  "name":"left_rgb_pose",
                  "type":"isaac::alice::PoseInitializer"
               },
               {
                  "name":"image_loader",
                  "type":"isaac::message_generators::ImageLoader"
               }
            ]
         },
         {
            "name": "camera_viewer",
            "components": [
               {
                  "name": "ledger",
                  "type": "isaac::alice::MessageLedger"
               },
               {
                  "name": "viewer",
                  "type": "isaac::viewers::ImageViewer"
               }
            ]
         },
         {
            "name":"superpixels",
            "subgraph":"packages/superpixels/apps/superpixels.subgraph.json"
         }
      ],
      "edges":[
         {
            "source":"segmentation_node/image_loader/color",
            "target":"superpixels.subgraph/interface/color"
         },
         {
            "source":"segmentation_node/image_loader/depth",
            "target":"superpixels.subgraph/interface/depth"
         },
         {
            "source":"segmentation_node/CameraIntrinsicsGenerator/intrinsics",
            "target":"superpixels.subgraph/interface/depth_intrinsics"
         },
         {
            "source":"segmentation_node/image_loader/color",
            "target":"segmentation_node/CameraIntrinsicsGenerator/image"
         },
         {
            "source":"superpixels.subgraph/interface/superpixels",
            "target":"segmentation_node/SuperpixelConsumer/superpixels"
         },
         {
            "source":"segmentation_node/image_loader/color",
            "target":"segmentation_node/SuperpixelConsumer/loader_rgb"
         },
         {
            "source":"segmentation_node/ImageProvider/color",
            "target":"segmentation_node/SuperpixelConsumer/provider_rgb"
         }
      ]
   },
   "config": {
      "segmentation_node":{
         "SuperpixelConsumer":{
            "tick_period": "1Hz"
         },
         "CameraIntrinsicsGenerator": {
            "focal_length": [570, 570],
            "optical_center": [240, 320]
         },
         "left_rgb_pose": {
            "lhs_frame": "robot",
            "rhs_frame": "camera",
            "pose": [0.270598, -0.653281, 0.653281, -0.270598, 0.0, 0.0, 0.775]
         },
         "ImageProvider":{
            "color_filename": "assets/rgb_test.png",
            "depth_filename": "assets/depth_test.png",
            "tick_period": "1Hz"
         },
         "image_loader": {
            "color_filename": "assets/rgb_test.png",
            "depth_filename": "assets/depth_test",
            "tick_period": "1Hz"
         }
      },
      "websight":{
         "WebsightServer":{
            "port":5000,
            "ui_config":{
               "windows":{
                  "color":{
                     "renderer":"2d",
                     "dims":{"width":640, "height":480},
                     "channels":[
                        { "name":"segmentation/superpixels.camera_viewer/viewer/image" },
                        { "name":"segmentation/superpixels.rgbd_edges/edges/edges" }
                     ]
                  },
                  "depth": {
                     "renderer": "2d",
                     "dims": { "width": 640, "height": 480 },
                     "channels": [
                        { "name": "segmentation/superpixels.rgbd_points/points/depth" },
                        { "name": "segmentation/superpixels.rgbd_points/points/points" },
                        { "name": "segmentation/superpixels.superpixels/superpixels/superpixel_indices" }
                     ]
                  } ,
                  "codelet_images": {
                     "renderer": "2d",
                     "dims": { "width": 640, "height": 480 },
                     "channels": [
                        { "name": "camera_viewer/viewer/image" }
                     ]
                  }
               }
            }
         }
      }
   }
}

My python file is below:

from isaac import *
import numpy as np
import PIL.Image
import cv2
import skimage


class SuperpixelConsumer(Codelet):
    def start(self):
        self.superpixels_rx = self.isaac_proto_rx("SuperpixelsProto", "superpixels")
        self.provider_rgb_rx = self.isaac_proto_rx("ImageProto", "provider_rgb")
        self.loader_rgb_rx = self.isaac_proto_rx("ImageProto", "loader_rgb")
        self.log_info("SuperpixelConsumer restart tick")
        self.tick_periodically(0.1)
        self.cnt = 0

    def tick(self):
        superpixel_msg = self.superpixels_rx.message
        if superpixel_msg is not None:
            numpy_superpixels = np.array(superpixel_msg.buffers[superpixel_msg.proto.indices.dataBufferIndex])
            print(len(numpy_superpixels))

        provider_rgb_msg = self.provider_rgb_rx.message
        loader_rgb_msg = self.loader_rgb_rx.message
        if provider_rgb_msg is not None and loader_rgb_msg is not None:
            shape_provider = (provider_rgb_msg.proto.rows, provider_rgb_msg.proto.cols, provider_rgb_msg.proto.channels)
            shape_loader = (loader_rgb_msg.proto.rows, loader_rgb_msg.proto.cols, loader_rgb_msg.proto.channels)

            buffer_provider = provider_rgb_msg.buffers[provider_rgb_msg.proto.dataBufferIndex]
            buffer_loader = loader_rgb_msg.buffers[loader_rgb_msg.proto.dataBufferIndex]

            type_provider = provider_rgb_msg.proto.elementType
            type_loader = loader_rgb_msg.proto.elementType

            assert shape_provider == shape_loader
            assert type_provider == type_loader
            assert buffer_provider == buffer_loader
            print("Both messages received")
        else:
            if provider_rgb_msg is None:
                print("Provider message not received")
            if loader_rgb_msg is None:
                print("Loader message not received")

        self.cnt += 1

    def stop(self):
        assert self.cnt > 0, "ticking count {}".format(self.cnt)


class ImageProvider(Codelet):
    def start(self):
        self.color_tx = self.isaac_proto_tx("ImageProto", "color")
        self.depth_tx = self.isaac_proto_tx("ImageProto", "depth")
        self.log_info("ImageProvider restart tick")
        self.tick_periodically(0.1)
        self.cnt = 0

        color_filename = self.config.color_filename
        self.color_image = np.array(PIL.Image.open(color_filename).convert(mode="RGB"))

        depth_filename = self.config.depth_filename
        self.depth_image = skimage.img_as_float32(cv2.imread(depth_filename, cv2.IMREAD_ANYDEPTH))
        self.depth_image = self.depth_image[..., np.newaxis]

    def tick(self):
        # get isaac parameters. Both ways are equivalent
        message_color = self.color_tx.init()
        message_color.buffers = [self.color_image]
        message_color.proto.dataBufferIndex = 0

        shape_color = self.color_image.shape
        message_color.proto.rows = shape_color[0]
        message_color.proto.cols = shape_color[1]
        message_color.proto.channels = shape_color[2]
        message_color.proto.elementType = 'uint8'
        self.color_tx.publish()

        message_depth = self.depth_tx.init()

        message_depth.buffers = [np.array(self.depth_image)]
        message_depth.proto.dataBufferIndex = 0

        shape = self.depth_image.shape
        message_depth.proto.rows = shape[0]
        message_depth.proto.cols = shape[1]
        message_depth.proto.channels = shape[2]
        message_depth.proto.elementType = 'float32'
        self.depth_tx.publish()

        self.cnt += 1

    def stop(self):
        assert self.cnt > 0, "ticking count {}".format(self.cnt)

def main():
    app = Application("apps/fullfusion/segmentation.app.json")
    app.nodes["segmentation_node"].add(ImageProvider, name="ImageProvider")
    app.nodes["segmentation_node"].add(SuperpixelConsumer, name="SuperpixelConsumer")
    app.run()


if __name__ == '__main__':
    main()

Your code seems fine to me at least. Have you tried wiring the image_loader’s output through your ImageProvider as a pass-through (maybe change a pixel) and then to CameraIntrinsicsGenerator and SuperpixelConsumer just to eliminate anything about the ImageProvider codelet configuration itself? ImageProvider does not have an edge to CameraIntrinsicsGenerator like image_loader, etc. but that may be innocuous. Could you attach the configuration that fails as well for reference?

Thanks for the answer. The config that fails is replacing the image_loader inputs with my ImageProvider. Please see below:

{
   "name":"segmentation",
   "modules":[
      "superpixels",
      "viewers",
      "message_generators",
      "sight"
   ],
   "graph":{
      "nodes":[
         {
            "name":"segmentation_node",
            "components":[
               {
                  "name": "ImageProvider",
                  "type": "isaac::alice::PyCodelet"
               },
               {
                  "name": "SuperpixelConsumer",
                  "type": "isaac::alice::PyCodelet"
               },
               {
                  "name": "message_ledger",
                  "type": "isaac::alice::MessageLedger"
               },
               {
                  "name":"CameraIntrinsicsGenerator",
                  "type":"isaac::message_generators::CameraIntrinsicsGenerator"
               },
               {
                  "name":"left_rgb_pose",
                  "type":"isaac::alice::PoseInitializer"
               },
               {
                  "name":"image_loader",
                  "type":"isaac::message_generators::ImageLoader"
               }
            ]
         },
         {
            "name": "camera_viewer",
            "components": [
               {
                  "name": "ledger",
                  "type": "isaac::alice::MessageLedger"
               },
               {
                  "name": "my_viewer",
                  "type": "isaac::viewers::ImageViewer"
               }
            ]
         },
         {
            "name":"superpixels",
            "subgraph":"packages/superpixels/apps/superpixels.subgraph.json"
         }
      ],
      "edges":[
         {
            "source":"segmentation_node/ImageProvider/color",
            "target":"superpixels.subgraph/interface/color"
         },
         {
            "source":"segmentation_node/ImageProvider/depth",
            "target":"superpixels.subgraph/interface/depth"
         },
         {
            "source":"segmentation_node/CameraIntrinsicsGenerator/intrinsics",
            "target":"superpixels.subgraph/interface/depth_intrinsics"
         },
         {
            "source":"segmentation_node/ImageProvider/color",
            "target":"segmentation_node/CameraIntrinsicsGenerator/image"
         },
         {
            "source":"superpixels.subgraph/interface/superpixels",
            "target":"segmentation_node/SuperpixelConsumer/superpixels"
         },
         {
            "source":"segmentation_node/image_loader/color",
            "target":"segmentation_node/SuperpixelConsumer/loader_rgb"
         },
         {
            "source":"segmentation_node/ImageProvider/color",
            "target":"segmentation_node/SuperpixelConsumer/provider_rgb"
         },
         {
            "source":"segmentation_node/ImageProvider/color",
            "target":"camera_viewer/my_viewer/image"
         }
      ]
   },
   "config": {
      "segmentation_node":{
         "SuperpixelConsumer":{
            "tick_period": "1Hz"
         },
         "CameraIntrinsicsGenerator": {
            "focal_length": [570, 570],
            "optical_center": [240, 320]
         },
         "left_rgb_pose": {
            "lhs_frame": "robot",
            "rhs_frame": "camera",
            "pose": [0.270598, -0.653281, 0.653281, -0.270598, 0.0, 0.0, 0.775]
         },
         "ImageProvider":{
            "color_filename": "assets/rgb_test.png",
            "depth_filename": "assets/depth_test.png",
            "tick_period": "1Hz"
         },
         "image_loader": {
            "color_filename": "assets/rgb_test.png",
            "depth_filename": "assets/depth_test",
            "tick_period": "1Hz"
         }
      },
      "websight":{
         "WebsightServer":{
            "port":5000,
            "ui_config":{
               "windows":{
                  "color":{
                     "renderer":"2d",
                     "dims":{"width":640, "height":480},
                     "channels":[
                        { "name":"segmentation/superpixels.camera_viewer/viewer/image" },
                        { "name":"segmentation/superpixels.rgbd_edges/edges/edges" }
                     ]
                  },
                  "depth": {
                     "renderer": "2d",
                     "dims": { "width": 640, "height": 480 },
                     "channels": [
                        { "name": "segmentation/superpixels.rgbd_points/points/depth" },
                        { "name": "segmentation/superpixels.rgbd_points/points/points" },
                        { "name": "segmentation/superpixels.superpixels/superpixels/superpixel_indices" }
                     ]
                  } ,
                  "codelet_images": {
                     "renderer": "2d",
                     "dims": { "width": 640, "height": 480 },
                     "channels": [
                        { "name": "segmentation/camera_viewer/my_viewer/image" }
                     ]
                  }
               }
            }
         }
      }
   }
}

Even if the only modification I make is replacing image_loader/color → CameraIntrinsicsGenerator/image with my ImageProvider/color → CameraIntrinsicsGenerator/image and leave everything else the same, it doesn’t work.

I’ve also tried wiring the image_loader output through the ImageProvider, and it still doesn’t work.

With the config that fails I continue to get an output from superpixels.rgbd_edges but nothing from rgbd_points, rgbd_normals, or superpixel_indices. Please see the screenshot of the Job Statistics Report below.

It seems like if I replace the camera intrinsics image edge with depth rather than color, the normals and points work, but the superpixels are still at 0.

I.e instead of

 {
    "source":"segmentation_node/ImageProvider/color",
    "target":"segmentation_node/CameraIntrinsicsGenerator/image"
 }

use

 {
    "source":"segmentation_node/ImageProvider/depth",
    "target":"segmentation_node/CameraIntrinsicsGenerator/image"
 }

So I’m wondering whether somehow there’s a problem with the RGB image but:

  1. segmentation_node/ImageProvider/color gets correctly displayed in WebSight
  2. I have another version of the same application, with the same config but detecting ORB keypoints as well, coming from the ImageProvider color. That has no problem getting computed and displayed.
  3. As mentioned previously, I’ve checked the protos and the buffer of ImageProvider vs image_loader (as can be seen in the python code from the original post). They appear to be identical for both the RGB and the Depth images

One issue might be that the color and depth image do not have the same acquisition time so they do not ever sync up (Superpixels codelet syncs on the color and depth channels). You may need to add this to the custom ImageProvider

message_color.acqtime = acqtime
message_depth.acqtime = acqtime

That was it! Thanks a lot.