Viewing Object Detections in Isaac Omniverse Viewport

Isaac Omniverse communicates using a TCP subscriber and publisher, but I’m not familiar with what kind of information can be published that the TCP subscriber will receive. Specifically, I have 2D bounding box detections that I can currently view in Isaac Sight via DetectionsViewer, but want to display those detections in the viewport of Isaac Omniverse. How can I do this, if at all possible? Thank you!

The TCP subscriber/publisher send a payload that both sides can interpret, such as a serialized Cap’n Proto message. You should be able to write an extension in Isaac Sim to receive messages from your SDK app. Once you have it receiving messages, you should be able to render whatever you need to. It should be possible but there may not be many examples yet to follow.

Sorry, its not possible to do this in the current release of Isaac Sim, in the next release there will be ways to visualize 3d polylines via proto messages. Additionally pyalice will be available inside of sim so you could in theory receive a message and then use one of the python debug rendering APIs to render these boxes.