Please provide the following information when requesting support.
• Network Type fpenet
• TLT Version 3.22.05
• Training spec file
experiment_spec_pig.yaml (2.3 KB)
I want to use 27 custom keypoints train fpenet
deadpig_weight_1660549603537_1.pts (249 Bytes)
I modified the dataset to be the same as afw.
After that, I modified function convert_dataset and convert data successful
if key_points == 80:
for num, keypoint in enumerate(image_keypoints):
annotations["P{}x".format(num+1)] = keypoint[0]
annotations["P{}y".format(num+1)] = keypoint[1]
# fill in dummy keypoints for keypoints 69 to 80
for num in range(69, 81, 1):
annotations["P{}x".format(num)] = image_keypoints[0][0]
annotations["P{}y".format(num)] = image_keypoints[0][1]
annotations["P{}occluded".format(num)] = True
elif key_points==10:
key_id = 1
for num, keypoint in enumerate(image_keypoints):
# change to 10-points dataset:
if (num+1) in [1, 9, 17, 20, 25, 39, 45, 34, 49, 55]:
annotations["P{}x".format(key_id)] = keypoint[0]
annotations["P{}y".format(key_id)] = keypoint[1]
key_id += 1
elif key_points == 27:
for num, keypoint in enumerate(image_keypoints):
if keypoint[0] == 0 or keypoint[1] == 0:
annotations["P{}x".format(num)] = image_keypoints[0][0]
annotations["P{}y".format(num)] = image_keypoints[0][1]
annotations["P{}occluded".format(num)] = True
else:
annotations["P{}x".format(num+1)] = keypoint[0]
annotations["P{}y".format(num+1)] = keypoint[1]
else:
raise ValueError("This script only generates 10 & 80 keypoints dataset.")
convert dataset command
python data_utils.py --afw_data_path /home/nxin/zyan/cv_samples_v1.4.0/fpenet/afw --output_json_path /home/nxin/zyan/cv_samples_v1.4.0/fpenet/data/afw/afw.json --afw_image_save_path /workspace/tao-experiments/fpenet/afw --num_key_points 27
train error
2022-10-24 18:02:41,861 [INFO] root: Registry: ['nvcr.io']
2022-10-24 18:02:41,904 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.22.05-tf1.15.5-py3
2022-10-24 18:02:41,982 [WARNING] tlt.components.docker_handler.docker_handler:
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the "user":"UID:GID" in the
DockerOptions portion of the "/home/nxin/.tao_mounts.json" file. You can obtain your
users UID and GID by using the "id -u" and "id -g" commands on the
terminal.
2022-10-24 10:02:42.603162: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
/usr/local/lib/python3.6/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn't match a supported version!
RequestsDependencyWarning)
Using TensorFlow backend.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/checkpoint_saver_hook.py:25: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead.
2022-10-24 10:02:44,431 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/checkpoint_saver_hook.py:25: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead.
WARNING 2022-10-24 10:02:44,960| tensorflow: Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
/usr/local/lib/python3.6/dist-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.5) or chardet (3.0.4) doesn't match a supported version!
RequestsDependencyWarning)
Using TensorFlow backend.
WARNING 2022-10-24 10:02:46,637| tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/checkpoint_saver_hook.py:25: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead.
WARNING 2022-10-24 10:02:46,647| tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/fpenet/scripts/train.py:95: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.
WARNING 2022-10-24 10:02:46,647| tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/fpenet/scripts/train.py:95: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.
/usr/local/lib/python3.6/dist-packages/driveix/fpenet/scripts/train.py:105: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
/workspace/tao-experiments/fpenet/models/exp1
WARNING 2022-10-24 10:02:46,984| tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/fpenet/dataloader/fpenet_dataloader.py:249: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.
WARNING 2022-10-24 10:02:46,988| tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
/workspace/tao-experiments/fpenet/data/tfrecords/afw/FpeTfRecords/data.tfrecords: 50
Total Samples: 50
WARNING 2022-10-24 10:02:47,030| tensorflow: From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.
WARNING 2022-10-24 10:02:47,061| tensorflow: Entity <bound method Processor.__call__ of <modulus.processors.parse_example_proto.ParseExampleProto object at 0x7f17e0a50b00>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of <bound method Processor.__call__ of <modulus.processors.parse_example_proto.ParseExampleProto object at 0x7f17e0a50b00>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
WARNING 2022-10-24 10:02:47,074| tensorflow: Entity <bound method FpeNetDataloader._load_and_decode of <driveix.fpenet.dataloader.fpenet_dataloader.FpeNetDataloader object at 0x7f17f404bb70>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of <bound method FpeNetDataloader._load_and_decode of <driveix.fpenet.dataloader.fpenet_dataloader.FpeNetDataloader object at 0x7f17f404bb70>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
WARNING 2022-10-24 10:02:48,456| tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/fpenet/dataloader/fpenet_dataloader.py:562: The name tf.matrix_inverse is deprecated. Please use tf.linalg.inv instead.
WARNING 2022-10-24 10:02:48,464| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:48,468| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:48,469| tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/common/augment/random_gamma.py:75: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.
WARNING 2022-10-24 10:02:48,475| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:48,636| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:48,640| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:48,647| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:48,810| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:48,813| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:48,820| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:48,989| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:48,992| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:48,999| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:49,165| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:49,169| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:49,176| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:49,340| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:49,344| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:49,351| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:49,514| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:49,518| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:49,525| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:49,744| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:49,748| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:49,755| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:49,918| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:49,921| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:49,928| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:50,090| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:50,094| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:50,101| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:50,263| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:50,267| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:50,274| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:50,436| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:50,440| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:50,446| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:50,609| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:50,612| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:50,619| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:50,780| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:50,784| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:50,791| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:50,955| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:50,959| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:50,966| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:51,128| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:51,131| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:51,138| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:51,302| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:51,305| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:51,312| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:51,475| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:51,479| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:51,485| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:51,648| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:51,652| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:51,659| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:51,888| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:51,891| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:51,898| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:52,060| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:52,063| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:52,070| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:52,235| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:52,239| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:52,246| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:52,409| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:52,412| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:52,419| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:52,582| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:52,585| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:52,592| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:52,754| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:52,757| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:52,764| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:52,927| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:52,930| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:52,937| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:53,099| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:53,103| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:53,110| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:53,272| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:53,276| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:53,283| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:53,445| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:53,449| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:53,456| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:53,621| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:53,625| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:53,632| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:53,795| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:53,798| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:53,805| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:53,967| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:53,970| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:53,977| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:02:54,014| tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/fpenet/visualization/fpenet_visualization.py:62: The name tf.summary.image is deprecated. Please use tf.compat.v1.summary.image instead.
INFO 2022-10-24 10:02:54,015| root: model type is: FpeNet_public
WARNING 2022-10-24 10:02:54,025| tensorflow: From /opt/nvidia/third_party/keras/tensorflow_backend.py:183: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.
WARNING 2022-10-24 10:02:54,279| tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
WARNING 2022-10-24 10:02:54,481| tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.
WARNING 2022-10-24 10:02:54,481| tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
WARNING 2022-10-24 10:02:54,481| tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.
WARNING 2022-10-24 10:02:55,211| tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.
/usr/local/lib/python3.6/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.
warnings.warn('No training configuration found in save file: '
INFO 2022-10-24 10:02:56,387| driveix.fpenet.models.fpenet_basemodel: Loading weights from pretrained model file. /workspace/tao-experiments/fpenet/pretrained_models/fpenet_vtrainable_v1.0/model.tlt
WARNING 2022-10-24 10:03:19,593| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,597| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,606| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,609| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,618| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,622| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,632| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,636| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,646| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,650| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,659| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,663| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,672| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,676| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,686| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,690| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,700| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,703| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,713| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,716| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,725| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,729| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,739| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,742| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,751| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,755| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,764| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,768| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,777| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,781| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,790| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,794| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,803| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,807| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,816| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,820| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,829| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,833| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,842| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,846| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,855| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,859| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,868| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,872| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,881| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,885| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,894| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,898| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,907| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,910| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,920| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,923| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,933| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,936| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,946| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,950| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,959| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,962| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,971| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,975| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,984| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,988| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:19,997| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
WARNING 2022-10-24 10:03:20,001| tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical.
/workspace/tao-experiments/fpenet/data/tfrecords/afw/FpeTfRecords/data.tfrecords: 50
Total Samples: 50
WARNING 2022-10-24 10:03:21,017| tensorflow: Entity <bound method FpeNetDataloader._load_and_decode of <driveix.fpenet.dataloader.fpenet_dataloader.FpeNetDataloader object at 0x7f17f404bb70>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of <bound method FpeNetDataloader._load_and_decode of <driveix.fpenet.dataloader.fpenet_dataloader.FpeNetDataloader object at 0x7f17f404bb70>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
INFO 2022-10-24 10:03:22,520| __main__: Build trainer finished. Starting training...
WARNING 2022-10-24 10:03:22,525| tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/fpenet/trainers/fpenet_trainer.py:239: The name tf.train.get_global_step is deprecated. Please use tf.compat.v1.train.get_global_step instead.
WARNING 2022-10-24 10:03:22,527| tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:61: The name tf.train.LoggingTensorHook is deprecated. Please use tf.estimator.LoggingTensorHook instead.
WARNING 2022-10-24 10:03:22,528| tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:62: The name tf.train.StopAtStepHook is deprecated. Please use tf.estimator.StopAtStepHook instead.
WARNING 2022-10-24 10:03:22,528| tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:75: The name tf.train.StepCounterHook is deprecated. Please use tf.estimator.StepCounterHook instead.
INFO 2022-10-24 10:03:22,528| tensorflow: Create CheckpointSaverHook.
WARNING 2022-10-24 10:03:22,528| tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:104: The name tf.train.SummarySaverHook is deprecated. Please use tf.estimator.SummarySaverHook instead.
INFO 2022-10-24 10:03:26,869| tensorflow: Graph was finalized.
INFO 2022-10-24 10:03:28,318| tensorflow: Running local_init_op.
INFO 2022-10-24 10:03:29,808| tensorflow: Done running local_init_op.
INFO 2022-10-24 10:03:40,529| tensorflow: Saving checkpoints for step-0.
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: {{function_node __inference_Dataset_map__map_func_set_random_wrapper_38375}} assertion failed: [target_width must be > 0.]
[[{{node crop_to_bounding_box/Assert_2/Assert}}]]
[[IteratorGetNext_1]]
[[RegexFullMatch_55/_1508]]
(1) Invalid argument: {{function_node __inference_Dataset_map__map_func_set_random_wrapper_38375}} assertion failed: [target_width must be > 0.]
[[{{node crop_to_bounding_box/Assert_2/Assert}}]]
[[IteratorGetNext_1]]
0 successful operations.
0 derived errors ignored.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/fpenet/scripts/train.py", line 141, in <module>
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/fpenet/scripts/train.py", line 137, in main
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/fpenet/trainers/fpenet_trainer.py", line 286, in train
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/fpenet/trainers/fpenet_trainer.py", line 332, in run_training_loop
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 754, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1360, in run
raise six.reraise(*original_exc_info)
File "/usr/local/lib/python3.6/dist-packages/six.py", line 696, in reraise
raise value
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1345, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1418, in run
run_metadata=run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/training/monitored_session.py", line 1176, in run
return self._sess.run(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: assertion failed: [target_width must be > 0.]
[[{{node crop_to_bounding_box/Assert_2/Assert}}]]
[[IteratorGetNext_1]]
[[RegexFullMatch_55/_1508]]
(1) Invalid argument: assertion failed: [target_width must be > 0.]
[[{{node crop_to_bounding_box/Assert_2/Assert}}]]
[[IteratorGetNext_1]]
0 successful operations.
0 derived errors ignored.
Traceback (most recent call last):
File "/usr/local/bin/fpenet", line 8, in <module>
sys.exit(main())
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/fpenet/entrypoint/fpenet.py", line 12, in main
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/common/entrypoint/entrypoint.py", line 300, in launch_job
AssertionError: Process run failed.
2022-10-24 18:04:03,294 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.
How should I train custom keypoints with fpenet?
