Sending ImageProto from python codelet

[ngilbert]

I am developing a machine learning application (in python) which requires me to send/receive messages of the ImageProto type. The ImageProto protobuffer has the field “dataBufferIndex”. After a bit of digging, I found out how to use this field to read a buffer, using self.rx_hook.get_buffer_content(index), which allows me to receive ImageProto messages. However, I was unable to find any way of setting the buffer contents for transmitting. I know I could probably get something to work using the TensorProto, but then I have no way of sending results to a ColorCameraViewer. Am I missing something?

[BlackToothGrin]

Hi, sorry to disappoint but I don’t have an answer; I’m looking for advice.

I am trying to do something similar; I am working with the ROS-Isaac bridge and am trying to pass image data out of Isaac and into Ros via the bridge.

Could you tell me a little bit about how you accessed the data buffer?

I have a node hooked into the left_camera_subscriber receiving ColorCameraProto messages which I am able to deconstruct, but (as i’m sure you have noticed) the messages don’t actually contain the image data.

If I can gain access to the image data I should be able to convert the data into a sensor_msgs::Image message type for ROS.

If you think I can help you in any way, I’d be happy to return the favour.

Thanks

[ngilbert]

I’ve been using https://docs.nvidia.com/isaac/isaac/doc/message_api.html as a reference for how the Isaaac messages are set up. The ColorCameraProto contains an ImageProto in the image field. This ImageProta has a field called dataBufferIndex. This tells you which buffer to look in for the image data. I’ve been reading this data into a numpy array. So to get the image data I do something like this:

image_data = np.frombuffer(rx_color_camera_hook.get_buffer_content(color_camera_proto.image.dataBufferIndex))

Hope that helps!

[yangl]

Hi, @ngilbert,

Sorry we don’t have the support in PyCodelet for publishing messages with buffers yet. Coding a C++ codelet might help before the feature is there.

[nitikesh.bhad]

How can we do this in c++ Codelet ? Can you provide a sample ?

[yangl]

@nitikesh.bhad
Check packages/dummies/ImageLoader.cpp for sample.

[nitikesh.bhad]

In my packages/dummies/ directory there is just BUILD file which loads the libdummies_module.so shared object.

[yangl]

@nitikesh.bhad
Maybe packages/realsense has what you need.

[excit0n]

Hi,

Here is a helpful breakdown of Protos in general (https://devtalk.nvidia.com/default/topic/1064465/sdk/reading-content-of-detections2proto-message/post/5390513/#reply[)

Hi excit0n,
from the page I linked:

“For every field bar defined in Foo, Foo::Reader has a method getBar(). For primitive types, get just returns the type, but for structs, lists, and blobs, it returns a Reader for the type.”

This means that for a Detection2Proto::Reader you have the methods

getPredictions(); //returns a List(PredictionProto)
getBoundingBoxes(); // returns a List(RectangleProto)

Accessing an element of the list with the operator gives you a Reader for the element in the list

auto detections_list = rx_samples().getProto().getPredictions(); //returns a List(PredictionProto)
auto detection = detections_list[0]; //returns a PredictionProto::Reader

Now you have a new Reader, the PredictionProto::Reader. Just like the previous one you have a get() method for each element in the Struct, i.e.

getLabel(); //returns a Text (i.e. a string)
getConfidence(); // returns a Float64

So basically to access the elements in a Proto, all you need to know are the elements contained in the Struct (https://docs.nvidia.com/isaac/isaac/doc/message_api.html#protobuffers).
Then the reader will provide you a get() method to access them (just be careful with the syntax and the upper/lower cases). You have to use those methods to “dig” into the nested structures until you get a primitive type.

You can find the definition of the Proto::Reader in "isaac-sdk-folder/engine/alice/hooks/message_hook.hpp, but TBH it’s not that easy to understand from there.

Good luck.
Cheers

[adhamija]

Hi, is there any update on if it’s possible to use PyCodelets for publishing messages with buffers now? I want to use my PyCodelet for getting an image from a camera and then applying my image processing algorithms. I then want to publish the image for viewing on WebSight. Thanks!

[yangl]

Hi, @adhamija,

With 2020.1 release it is possible to publish message with buffers now. Please check the unit tests in message_test.py that comes with the release.