Running custom models with Keypoints / Landmarks

Accelerated Computing Intelligent Video Analytics DeepStream SDK
#1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)

GPU

• DeepStream Version

Any

• JetPack Version (valid for Jetson only)
• TensorRT Version

Any

• NVIDIA GPU Driver Version (valid for GPU only)

Any

• Issue Type( questions, new requirements, bugs)

Questions

• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)

N/A

• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

There are models from tensorflow (for example) that include keypoints (landmarks), such as from here: models/tf2_detection_zoo.md at master · tensorflow/models · GitHub

One example is: CenterNet Resnet50 V2 Keypoints 512x512

Would the following be possible:

  • Train a custom model with both bounding boxes and keypoints, e.g. CenterNet Resnet50 V2 Keypoints 512x512
  • Run the model with Deepstream
  • See both the bounding boxes and the keypoints displayed?
  • If the keypoints cannot be displayed due to OSD limitations, I should still be able to access these and e.g. send them over kafka?

I do see NVIDIA pose estimation examples, which I think is along the same lines as what I’m trying to achieve, but if I want custom landmarks for e.g. fish “nose” and fish tail then it seems I need to wait for fpenet-generic in a future release.

#3

1 if want to use nvidia framework to train model , please refer to TAO,Overview — TAO Toolkit 3.22.02 documentation
2 you can use deepstream to run model, but maybe you need to custom your post process if network-type is special. please refer to Gst-nvinfer — DeepStream 6.1 Release documentation.
3 deepstream supports kafka, please refer to Gst-nvmsgbroker — DeepStream 6.1 Release documentation, and demo is deepstream-test4 or deepstream-test5.

#5

There is no update from you for a period, assuming this is not an issue anymore.
Hence we are closing this topic. If need further support, please open a new one.
Thanks