BodyPose3DNet: training data, license and deployment for a commercial product

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Hello,

We are evaluating BodyPose3DNet (NGC: nvidia/tao/bodypose3dnet, deployable_accuracy_v1.0 and deployable_performance_v1.0) for a commercial sports-science app. The app measures vertical jumps from a single side-view smartphone video, and it is used by coaches with their athletes. Before we test the model, we need to confirm four things:

  1. Training data. Which datasets were used to train and evaluate BodyPose3DNet? Were they real images, synthetic data, or both? Did they include any third-party datasets (for example COCO, MPII, Human3.6M or similar)? If so, under which licenses? The model card does not describe the training data.

  2. License. The NGC page mentions both the NVIDIA Open Model License and the Creative Commons Attribution 4.0 License. Which one governs the model weights? Does either one allow commercial use in a cloud service?

  3. Deployment. The page says the models “can only be used with Train Adapt Optimize (TAO) Toolkit”. Can we run the ONNX version (deployable_performance_onnx_v1.0) with ONNX Runtime or TensorRT on a cloud GPU, outside DeepStream, in a commercial product?

  4. Keypoints and inputs.

    • Where exactly are “big_toe” and “heel” located anatomically? For the big toe: the tip of the toe, the nail, or the metatarsophalangeal joint? For the heel: the surface or the inner center?
    • Are the “pose2d” coordinates in the pixel frame of the 192 × 256 input crop or of the original image?
    • Is there documentation for the inputs k_inv, t_form_inv and the mean limb lengths?

Thank you very much for your help.

Galo Hernández
HEGO (hego.lat)

3

Hi,
1)The training data is internal data. It does not include any third-party datasets.
2)The models are covered by the NVIDIA Open Model License: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/ .
3) Please refer to https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-license/ . For example, that license states that models are commercially usable and grants a worldwide right to use, publicly perform, sell, and distribute the Model, subject to the whole agreement. It does not prohibit cloud deployment by location. Please read the full agreement, including its Trustworthy AI terms, notice requirements, and the separate licenses for runtimes such as TensorRT and CUDA.
4) The 34 joints are listed on the NGC model card. The same 34-keypoint skeleton is documented here:
Appendix B: 3D Body Pose Keypoint Format — NVIDIA Augmented Reality (AR) SDK User Guide

NVIDIA has not published a finer anatomical definition for big_toe / heel (for example tip of the toe vs MTP joint, or heel surface vs inner center). In the skeleton, both big_toe and heel are foot keypoints whose parent is the corresponding ankle.

Outputs:

  • pose2d: 34 x 3 [x, y, conf] in the 192 x 256 network-input crop.
  • pose2d_org_img: 34 x 3 [x, y, conf] in the original image.
  • pose25d: 34 x 4 [x, y, zRel, conf]. In the DeepStream sample, x,y are in the original image and zRel is relative depth from the pelvis.
  • pose3d: 3D keypoints in camera/world coordinates (millimeters in the DeepStream sample).

The extra inputs are documented on the NGC card. The DeepStream sample shows the intended meaning and the default values:

https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps/blob/master/deepstream-bodypose-3d/sources/nvdsinfer_custom_impl_BodyPose3DNet/nvdsinitinputlayers_BodyPose3DNet.cpp

  • k_inv is the inverse camera intrinsic matrix K^{-1}. It should match your camera. The sample comment is “k_inv would change for camera parameters”. The default value in the sample corresponds to fx = fy ≈ 800.79, with the principal point at the center of the 192 x 256 crop (cx = 96, cy = 128).
  • t_form_inv is the inverse of the 2D affine transform from original-image coordinates to the 192 x 256 crop. It is used to map crop-space 2D points back to the original image (`pose2d_org_img`). The sample initializes it to identity; if you run outside DeepStream, set it to the inverse of your actual crop/resize transform.