# Retraining BodyPoseNet

**URL:** <https://forums.developer.nvidia.com/t/retraining-bodyposenet/221777>\
**Category:** TAO Toolkit\
**Created:** [July 25, 2022, 8:44am UTC](https://forums.developer.nvidia.com/t/retraining-bodyposenet/221777 "2022-07-25T08:44:42Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![taoonvision](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@taoonvision](https://forums.developer.nvidia.com/u/taoonvision)\
**Post date:** [July 25, 2022, 8:44am UTC](https://forums.developer.nvidia.com/t/retraining-bodyposenet/221777/1 "2022-07-25T08:44:42Z")

</div>

• Hardware (NVIDIA GeForce RTX 2060 SUPER)  
• Network Type (BodyPoseNet)  
• TAO Toolkit Version (3.22.05)

Hello,  
I am currently trying to do a transfer learning of BodyPoseNet on a dataset with 21 keypoints instead of 18 (to get a hand pose detection model).  
I created my own dataset under COCO format and generated tfrecords files.

**tao bpnet train** command works fine with default files but not with my files, here is the full output:

> !tao bpnet train -e $SPECS\_DIR/bpnet\_train\_hand1\_coco.yaml   
> -r $USER\_EXPERIMENT\_DIR/models/exp\_m1\_unpruned   
> -k $KEY   
> –gpus $NUM\_GPUS  
> 2022-07-25 09:28:34,621 [INFO] root: Registry: [‘[nvcr.io](http://nvcr.io)’]  
> 2022-07-25 09:28:34,663 [INFO] tlt.components.instance\_handler.local\_instance: Running command in container: [nvcr.io/nvidia/tao/tao-toolkit-tf:v3.22.05-tf1.15.5-py3](http://nvcr.io/nvidia/tao/tao-toolkit-tf:v3.22.05-tf1.15.5-py3)  
> 2022-07-25 09:28:34,674 [WARNING] tlt.components.docker\_handler.docker\_handler:  
> Docker will run the commands as root. If you would like to retain your  
> local host permissions, please add the “user”:“UID:GID” in the  
> DockerOptions portion of the “/home/mm/.tao\_mounts.json” file. You can obtain your  
> users UID and GID by using the “id -u” and “id -g” commands on the  
> terminal.  
> 2022-07-25 07:28:36.871415: I tensorflow/stream\_executor/platform/default/dso\_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0  
> WARNING:tensorflow:Deprecation warnings have been disabled. Set TF\_ENABLE\_DEPRECATION\_WARNINGS=1 to re-enable them.  
> /usr/local/lib/python3.6/dist-packages/requests/ **init**.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn’t match a supported version!  
> RequestsDependencyWarning)  
> Using TensorFlow backend.  
> WARNING:tensorflow:From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/checkpoint\_saver\_hook.py:25: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead.
> 
> 2022-07-25 07:28:38,845 [WARNING] tensorflow: From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/checkpoint\_saver\_hook.py:25: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead.
> 
> WARNING:tensorflow:Deprecation warnings have been disabled. Set TF\_ENABLE\_DEPRECATION\_WARNINGS=1 to re-enable them.  
> /usr/local/lib/python3.6/dist-packages/requests/ **init**.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn’t match a supported version!  
> RequestsDependencyWarning)  
> Using TensorFlow backend.  
> WARNING:tensorflow:From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/checkpoint\_saver\_hook.py:25: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead.
> 
> 2022-07-25 07:28:41,239 [WARNING] tensorflow: From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/checkpoint\_saver\_hook.py:25: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead.
> 
> WARNING:tensorflow:From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/scripts/train.py:91: The name tf.logging.set\_verbosity is deprecated. Please use tf.compat.v1.logging.set\_verbosity instead.
> 
> WARNING 2022-07-25 07:28:41,239| tensorflow: From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/scripts/train.py:91: The name tf.logging.set\_verbosity is deprecated. Please use tf.compat.v1.logging.set\_verbosity instead.
> 
> WARNING:tensorflow:From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/scripts/train.py:91: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.
> 
> WARNING 2022-07-25 07:28:41,239| tensorflow: From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/scripts/train.py:91: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.
> 
> /usr/local/lib/python3.6/dist-packages/driveix/bpnet/scripts/train.py:110: YAMLLoadWarning: calling yaml.load() without Loader=… is deprecated, as the default Loader is unsafe. Please read [https://msg.pyyaml.org/load](https://msg.pyyaml.org/load) for full details.  
> /workspace/tao-experiments/bpnet/models/exp\_m1\_unpruned  
> WARNING:tensorflow:From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/dataloaders/bpnet\_dataloader.py:484: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.
> 
> WARNING 2022-07-25 07:28:41,705| tensorflow: From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/dataloaders/bpnet\_dataloader.py:484: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.
> 
> Traceback (most recent call last):  
> File “/root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/scripts/train.py”, line 146, in   
> File “/root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/scripts/train.py”, line 132, in main  
> File “/root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build\_wheel.runfiles/ai\_infra/moduluspy/modulus/modulusobject/modulusobject.py”, line 158, in deserialize\_maglev\_object  
> File “/root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build\_wheel.runfiles/ai\_infra/moduluspy/modulus/modulusobject/modulusobject.py”, line 145, in \_deserialize\_recursively  
> File “/root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build\_wheel.runfiles/ai\_infra/moduluspy/modulus/modulusobject/modulusobject.py”, line 167, in deserialize\_maglev\_object  
> File “/root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build\_wheel.runfiles/ai\_infra/moduluspy/modulus/modulusobject/modulusobject.py”, line 432, in wrapper  
> File “/root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/dataloaders/bpnet\_dataloader.py”, line 150, in **init**  
> File “/root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/dataloaders/processors/label\_processor.py”, line 57, in **init**  
> AssertionError  
> Traceback (most recent call last):  
> File “/usr/local/bin/bpnet”, line 8, in   
> sys.exit(main())  
> File “/root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/entrypoint/bpnet.py”, line 12, in main  
> File “/root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/common/entrypoint/entrypoint.py”, line 300, in launch\_job  
> AssertionError: Process run failed.  
> 2022-07-25 09:28:42,438 [INFO] tlt.components.docker\_handler.docker\_handler: Stopping container.

To me it seems like a wrong configuration in my dataset files, but I can’t see where it comes from.  
I also checked my tfrecords file and the value of bytes\_list is way bigger in the default tfrecord for coco than in mine, should I be worried about it?  
**Check of a part of my tfrecord file:**  
[val\_hand\_tfrecord.txt](https://forums.developer.nvidia.com/uploads/short-url/lxZNVIDBD5yImeEKvaNNm1GGjSG.txt) (23.9 KB)  
**Check of a part of default tfrecord file:**  
[val\_default\_tfrecord.txt](https://forums.developer.nvidia.com/uploads/short-url/dg6uk1qC6A24eae1AiLDeUmv2U8.txt) (31.1 KB)

Here are the other files I used:  
[bpnet\_train\_hand1\_coco.yaml](https://forums.developer.nvidia.com/uploads/short-url/es3DYAEpmTwJRHUBgYRmr9zsCJK.yaml) (2.9 KB)  
[coco\_hand\_spec.json](https://forums.developer.nvidia.com/uploads/short-url/oJRBwVgLt5GPw3XyGwq6Aj8vHAg.json) (2.7 KB)  
[act1\_coco\_annotations\_test.json](https://forums.developer.nvidia.com/uploads/short-url/uf2S5ZZ2F2iWjRqUG4d3JaGqHZ5.json) (1.4 MB)  
[act1\_coco\_annotations\_train.json](https://forums.developer.nvidia.com/uploads/short-url/b23wqCu5Wn4Rte2HbIi0HgVirBn.json) (1.4 MB)

Ask me if any other config file is needed, thanks in advance.

---

<div class="post-metadata">

**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\
**Post date:** [July 26, 2022, 2:10am UTC](https://forums.developer.nvidia.com/t/retraining-bodyposenet/221777/3 "2022-07-26T02:10:17Z")

</div>

Would you please modify the title? Seems that this topic is related to bpnet instead of Gesturenet.  
Thanks a lot!

---

<div class="post-metadata">

**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\
**Post date:** [July 27, 2022, 7:30am UTC](https://forums.developer.nvidia.com/t/retraining-bodyposenet/221777/4 "2022-07-27T07:30:10Z")

</div>

Refer to [BodyPoseNet TAO training error - #4 by Morganh](https://forums.developer.nvidia.com/t/bodyposenet-tao-training-error/211960/4)

- The `target_shape` depends on the input shape. This can be computed based on the model stride. In the default setting, the model has a stride of 8.

The assertion error is due to

> assert (image\_shape[0] // target\_shape[0]) == (image\_shape[1] // target\_shape[1])

---

<div class="post-metadata">

**Author:** ![taoonvision](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@taoonvision](https://forums.developer.nvidia.com/u/taoonvision)\
**Post date:** [August 1, 2022, 7:47am UTC](https://forums.developer.nvidia.com/t/retraining-bodyposenet/221777/5 "2022-08-01T07:47:57Z")

</div>

Thank you,

I changed my input\_shape to [256,256] so that I don’t have to change the target\_shape.

I had another question though. In the BodyPoseNet doc I read that:  
" Currently, BodyPoseNet only supports the given default skeleton configuration at `pose_config_path`. The inference pipelines do not support custom skeleton configuration at the moment."

Is the doc up-to-date or does that mean I can’t have a custom skeleton like the one below in the json of /model\_pose\_config?

**Custom json in /model\_pose\_config:**

> {  
> “pose\_config\_type”: “bpnet\_18\_joints”,  
> “categories”: [  
> {  
> “supercategory”: “person”,  
> “id”: 1,  
> “name”: “person”,  
> “num\_joints”: 21,  
> “keypoints”: [  
> “wrist”, “thumb\_cmc”, “thumb\_mcp”, “thumb\_ip”, “thumb\_tip”,  
> “index\_finger\_mcp”, “index\_finger\_pip”, “index\_finger\_dip”, “index\_finger\_tip”, “middle\_finger\_mcp”,  
> “middle\_finger\_pip”, “middle\_finger\_dip”, “middle\_finger\_tip”,“ring\_finger\_mcp”, “ring\_finger\_pip”,  
> “ring\_finger\_dip”, “ring\_finger\_tip”, “pinky\_mcp”,“pinky\_pip”,“pinky\_dip”,“pinky\_tip”  
> ],  
> “skeleton”: [  
> [0,1],[1,2],[2,3],[3,4],[0,5],[5,6],[6,7],[7,8],  
> [9,10],[10,11],[11,12],[13,14],[14,15],[15,16],[17,18],[18,19],[19,20],[0,17],[5,9], [9,13], [13,17]  
> ],  
> “skeleton\_edge\_names”: [  
> [“wrist”, “thumb\_cmc”], [“thumb\_cmc”, “thumb\_mcp”], [“thumb\_mcp”, “thumb\_ip”],  
> [“thumb\_ip”, “thumb\_tip”], [“wrist”, “index\_finger\_mcp”], [“index\_finger\_mcp”, “index\_finger\_pip”],  
> [“index\_finger\_pip”, “index\_finger\_dip”], [“index\_finger\_dip”, “index\_finger\_tip”], [“middle\_finger\_mcp”, “middle\_finger\_pip”],  
> [“middle\_finger\_pip”, “middle\_finger\_dip”], [“middle\_finger\_dip”, “middle\_finger\_tip”], [“ring\_finger\_mcp”, “ring\_finger\_pip”],  
> [“ring\_finger\_pip”,“ring\_finger\_dip”], [“ring\_finger\_dip”,“ring\_finger\_tip”], [“pinky\_mcp”,“pinky\_pip”],  
> [“pinky\_pip”,“pinky\_dip”], [“pinky\_dip”,“pinky\_tip”], [“wrist”, “pinky\_mcp”],  
> [“index\_finger\_mcp”, “middle\_finger\_mcp”], [“middle\_finger\_mcp”, “ring\_finger\_mcp”], [“ring\_finger\_mcp”, “pinky\_mcp”]  
> ]  
> }  
> ]  
> }

---

<div class="post-metadata">

**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\
**Post date:** [August 5, 2022, 3:39am UTC](https://forums.developer.nvidia.com/t/retraining-bodyposenet/221777/6 "2022-08-05T03:39:51Z")

</div>

Yes, it does not support custom skeleton configuration at the moment.

---

<div class="post-metadata">

**Author:** ![taoonvision](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@taoonvision](https://forums.developer.nvidia.com/u/taoonvision)\
**Post date:** [August 5, 2022, 6:08am UTC](https://forums.developer.nvidia.com/t/retraining-bodyposenet/221777/7 "2022-08-05T06:08:18Z")

</div>

Ok then doing a transfer learning of BodyPoseNet for detecting 21 hand joints isn’t possible.  
Thanks for the help and have a great weekend!

---

<div class="post-metadata">

**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\
**Post date:** [August 19, 2022, 6:08am UTC](https://forums.developer.nvidia.com/t/retraining-bodyposenet/221777/8 "2022-08-19T06:08:44Z")

</div>

Closed on August 19, 2022, 6:08am UTC

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.