Publish message in custom PyCodelet

I am developing a custom PyCodelet that receives a ColorCameraProto, extracts camera intrinsics, and publishes them as a TensorListProto. I have followed the PyCodelet tutorial and have successfully received the ColorCameraProto and initialized the TensorListProto, but I cannot figure out how to add my data to the TensorListProto. The SDK documentation says the Data field is formatted as raw bytes. How would I assign a NP array to this field?

The code I have tried is below:

class PinholeToTensor(Codelet):
    def start(self):
        # This part will be run once in the beginning of the program
        print("Running initialization")

        # Input and output messages for the Codelet
        self.rx = self.isaac_proto_rx("ColorCameraProto", "rgb_image")
        self.tx = self.isaac_proto_tx("TensorListProto", "tensor_list")
        self.tx_inner = self.isaac_proto_tx("TensorProto", "tensor")

        # Tick every time we receive an image
        self.tick_on_message(self.rx)

    def tick(self):
        print("Ticking")

        # Extract pinhole proto from rx
        pinhole = self.rx.get_proto().pinhole

        # Extract proto attributes
        focal_x = pinhole.focal.x
        focal_y = pinhole.focal.y
        center_x = pinhole.center.x
        center_y = pinhole.center.y

        # Create 1D camera matrix
        camera_mat = [int(focal_x), 0, int(center_x),
                      0, int(focal_y), int(center_y),
                      0, 0, 1]

        # Initialize TensorProto and TensorListProto for the camera matrix
        tx_message = self.tx.init_proto()
        tx_contents = self.tx_inner.init_proto()

        # Set element type
        tx_contents.elementType = 2

        # Set tensor dimensions
        size = tx_contents.init('sizes', 2) # = [1, 9]
        size[0] = 9
        size[1] = 1

        # Add data to TensorProto
        data = tx_contents.init('data',len(camera_mat))
        for i in range(len(camera_mat)):
            data[i] = camera_mat[i] ## ERROR HERE ##

        # Create TensorList
        tensor_list = tx_message.init('tensors', 1)

        # Add tensor to TensorListProto
        tensor_list[0] = tx_contents

        # Publish TensorListProto

        self.tx.publish()

When I run this, I get the following error:

TypeError: ‘bytes’ object does not support item assignment

Don’t know if this is of any help but would it work to use the numpy.getbuffer function?

When I was trying to read the values from the image proto and use them to create numpy arrays I had to use numpy.frombuffer function.

https://devtalk.nvidia.com/default/topic/1052351/sdk/developing-pycodelets-to-be-used-by-other-applications/

Perhaps the reverse will work for what you need?

np.getbuffer() seems to be deprecated, but I am able to use np.tobytes() for the same functionality. What I can’t figure out is how to pass back the buffer into an ImageProto I would like to publish. In your program @d20.hall, you used get_buffer_content() to retrieve the buffer from Isaac SDK, but there does not seem to be any method to set the buffer; something like “set_buffer_content()”. Do you have any idea how I could do this?

Relevant code:

depth_map = predict_depth(image)
depth_map = depth_map.tobytes() # Create buffer of depth map

# Initialize tx proto
depth_image_proto = self.isaac_proto_tx("ImageProto", "rgb_image")
depth_image = depth_image_proto.init_proto()
depth_image.dataBufferIndex = depth_map # This is the line I can't figure out