Interacting with Skeleton2ListProto

Hello,
I’m trying to add depth data from an Intel Realsense camera to the 2D data obtained through 2D Skeleton Pose Estimation, but can’t seem to extract the data from Skeleton2ListProto into something I can interact with in my C++ custom Codelet.

What is the syntax for converting the Skeleton2ListProto into a list of structs in a C++ Codelet?

Thanks for the help.

Hi @felix.chippendale, please have a look at the edge detection sample located at:

<Isaac SDK Home>/apps/tutorials/opencv_edge_detection/EdgeDetector.cpp

Following piece of code shows how to extract focal length, center, rows, and, cols information from CameraColorProto -> PinholeProto -> focal length or center or rows or cols.

auto output = tx_output_image().initProto();
auto in_pinhole = input.getPinhole();
const Vector2d focal = FromProto(in_pinhole.getFocal());
const Vector2d center = FromProto(in_pinhole.getCenter());
auto out_pinhole = output.initPinhole();
ToProto(focal, out_pinhole.getFocal());
ToProto(center, out_pinhole.getCenter());
out_pinhole.setCols(in_pinhole.getCols());
out_pinhole.setRows(in_pinhole.getRows());

I hope this helps.

1 Like

Thank you for your help Swapnesh, it got me going in the right direction.

For anyone facing this problem in the future, below you can see my solution to it. Hope this can be of help.

// create cap'n Proto List of Skeleton2Proto
    auto skeleton_list = rx_skeletons().getProto().getSkeletons();
  
    for(size_t i = 0; i < skeleton_list.size(); i++) {
  
      // for each Skeleton2Proto, create cap'n Proto List of Joint2Proto
      auto joint_list = skeleton_list[i].getJoints();

      for(size_t ii = 0; ii < joint_list.size(); ii++) {
        
         // retrieve joint prediction confidence, this is a float
         float joint_conf = joint_list[ii].getLabel().getConfidence();

         // retrieve X position in image of joint (be warned, this is a float too)
         float joint_x = joint_list[ii].getPosition().getX();

         // retrieve Y position in image of joint (be warned, this is a float too)
         float joint_y = joint_list[ii].getPosition().getY();
       }
     }
1 Like

Thank you Felix do you think this is missing in tutorials/doc on how to get access to joints?

Hello atorabi,
I was able to piece this together from the example provided by Swapnesh, the Message API overview and additional information from the Cap’n Proto documentation, however it took me several hours of investigating and trial and error to obtain a working solution. I believe it would be useful to anyone who is attempting to use the 2d skeleton pose estimation package provided, as the documentation on this didn’t give me any idea of how to access the data itself.

This is naturally only my opinion, and I may have been alone in experiencing difficulties with this.

Kind regards,
Felix

1 Like

Hello,
Here is a minor update to get the label, I added the following lines :

//retrive joint label, this is a string
     std::string joint_label= joint_list[ii].getLabel().getLabel();
 std::printf("Label:  %s    |    ",joint_label.c_str());

Full code :

#include "poseRX.hpp"
#include <string>

namespace isaac {

void PoseRX::start() {
  // By using tickOnMessage instead of tickPeriodically we instruct the codelet to only tick when
  // a new message is received on the incoming data channel `trigger`.
  tickOnMessage(rx_skeletons());
}

void PoseRX::tick() {
  // This function will now only be executed whenever we receive a new message. This is guaranteed
  // by the Isaac Robot Engine.

  // Parse the message we received
  // create cap'n Proto List of Skeleton2Proto
    auto skeleton_list = rx_skeletons().getProto().getSkeletons();
  
    for(size_t i = 0; i < skeleton_list.size(); i++) {
  
      // for each Skeleton2Proto, create cap'n Proto List of Joint2Proto
      auto joint_list = skeleton_list[i].getJoints();

      for(size_t ii = 0; ii < joint_list.size(); ii++) {
	//retrive joint label, this is a string
         std::string joint_label= joint_list[ii].getLabel().getLabel();
	 std::printf("Label:  %s    |    ",joint_label.c_str());
         // retrieve joint prediction confidence, this is a float
         float joint_conf = joint_list[ii].getLabel().getConfidence();
	 std::printf("conf :  %f    |    ",joint_conf);
         // retrieve X position in image of joint (be warned, this is a float too)
         float joint_x = joint_list[ii].getPosition().getX();
	 std::printf("x :  %f    |    ",joint_x);
         // retrieve Y position in image of joint (be warned, this is a float too)
         float joint_y = joint_list[ii].getPosition().getY();
	 std::printf("y :  %f   \n",joint_y);
       }
     }
     
}

}  // namespace isaac