# tensorflow.python.framework.errors\_impl.InvalidArgumentError: 2 root error(s) found **URL:** \ **Category:** TAO Toolkit\ **Created:** [January 20, 2022, 4:04am UTC](https://forums.developer.nvidia.com/t/tensorflow-python-framework-errors-impl-invalidargumenterror-2-root-error-s-found/201073 "2022-01-20T04:04:06Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![oomg](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@oomg](https://forums.developer.nvidia.com/u/oomg)\ **Post date:** [January 20, 2022, 4:04am UTC](https://forums.developer.nvidia.com/t/tensorflow-python-framework-errors-impl-invalidargumenterror-2-root-error-s-found/201073/1 "2022-01-20T04:04:06Z")
Prepare your own dataset of [https://docs.nvidia.com/tao/tao-toolkit/text/data\_annotation\_format.html#bodyposenet-coco-format](https://docs.nvidia.com/tao/tao-toolkit/text/data_annotation_format.html#bodyposenet-coco-format) This is my dataset.Make according to notebook [keypoints\_train.json](https://forums.developer.nvidia.com/uploads/short-url/8V9s4YBTNpG17md9pYmmuIFP5mZ.json) (317.0 KB) I encountered a mistake during training(batch\_size=1): 2022-01-20 10:59:07,277 [INFO] root: Registry: [‘[nvcr.io](http://nvcr.io)’] 2022-01-20 10:59:07,471 [INFO] tlt.components.instance\_handler.local\_instance: Running command in container: [nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3](http://nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3) 2022-01-20 10:59:08,543 [WARNING] tlt.components.docker\_handler.docker\_handler: Docker will run the commands as root. If you would like to retain your local host permissions, please add the “user”:“UID:GID” in the DockerOptions portion of the “/home/chenhongzhao/.tao\_mounts.json” file. You can obtain your users UID and GID by using the “id -u” and “id -g” commands on the terminal. 2022-01-20 02:59:10.763322: I tensorflow/stream\_executor/platform/default/dso\_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 WARNING:tensorflow:Deprecation warnings have been disabled. Set TF\_ENABLE\_DEPRECATION\_WARNINGS=1 to re-enable them. Using TensorFlow backend. WARNING:tensorflow:From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/checkpoint\_saver\_hook.py:25: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead. 2022-01-20 02:59:15,408 [WARNING] tensorflow: From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/checkpoint\_saver\_hook.py:25: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead. WARNING:tensorflow:Deprecation warnings have been disabled. Set TF\_ENABLE\_DEPRECATION\_WARNINGS=1 to re-enable them. Using TensorFlow backend. WARNING:tensorflow:From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/checkpoint\_saver\_hook.py:25: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead. 2022-01-20 02:59:21,195 [WARNING] tensorflow: From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/checkpoint\_saver\_hook.py:25: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead. WARNING:tensorflow:From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/scripts/train.py:91: The name tf.logging.set\_verbosity is deprecated. Please use tf.compat.v1.logging.set\_verbosity instead. WARNING 2022-01-20 02:59:21,196| tensorflow: From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/scripts/train.py:91: The name tf.logging.set\_verbosity is deprecated. Please use tf.compat.v1.logging.set\_verbosity instead. WARNING:tensorflow:From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/scripts/train.py:91: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead. WARNING 2022-01-20 02:59:21,196| tensorflow: From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/scripts/train.py:91: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead. /usr/local/lib/python3.6/dist-packages/driveix/bpnet/scripts/train.py:110: YAMLLoadWarning: calling yaml.load() without Loader=… is deprecated, as the default Loader is unsafe. Please read [https://msg.pyyaml.org/load](https://msg.pyyaml.org/load) for full details. /workspace/tao-experiments/bpnet/models/exp\_m1\_unpruned WARNING:tensorflow:From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/dataloaders/bpnet\_dataloader.py:484: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead. WARNING 2022-01-20 02:59:22,746| tensorflow: From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/dataloaders/bpnet\_dataloader.py:484: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead. INFO 2022-01-20 02:59:22,757| **main** : done WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow\_backend.py:153: The name tf.get\_default\_graph is deprecated. Please use tf.compat.v1.get\_default\_graph instead. WARNING 2022-01-20 02:59:22,757| tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow\_backend.py:153: The name tf.get\_default\_graph is deprecated. Please use tf.compat.v1.get\_default\_graph instead. /workspace/tao-experiments/bpnet/data/train-fold-000-of-001: 178 Total Samples: 178 WARNING:tensorflow:From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/dataloaders/bpnet\_dataloader.py:319: The name tf.matrix\_inverse is deprecated. Please use tf.linalg.inv instead. WARNING 2022-01-20 02:59:22,848| tensorflow: From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/dataloaders/bpnet\_dataloader.py:319: The name tf.matrix\_inverse is deprecated. Please use tf.linalg.inv instead. WARNING:tensorflow:From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/dataloaders/bpnet\_dataloader.py:224: The name tf.image.resize\_images is deprecated. Please use tf.image.resize instead. WARNING 2022-01-20 02:59:22,878| tensorflow: From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/dataloaders/bpnet\_dataloader.py:224: The name tf.image.resize\_images is deprecated. Please use tf.image.resize instead. INFO 2022-01-20 02:59:22,893| driveix.bpnet.trainers.bpnet\_trainer: Building model graph from model defintion … WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow\_backend.py:4138: The name tf.random\_uniform is deprecated. Please use tf.random.uniform instead. WARNING 2022-01-20 02:59:22,895| tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow\_backend.py:4138: The name tf.random\_uniform is deprecated. Please use tf.random.uniform instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow\_backend.py:1834: The name tf.nn.fused\_batch\_norm is deprecated. Please use tf.compat.v1.nn.fused\_batch\_norm instead. WARNING 2022-01-20 02:59:22,918| tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow\_backend.py:1834: The name tf.nn.fused\_batch\_norm is deprecated. Please use tf.compat.v1.nn.fused\_batch\_norm instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow\_backend.py:4115: The name tf.random\_normal is deprecated. Please use tf.random.normal instead. WARNING 2022-01-20 02:59:23,250| tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow\_backend.py:4115: The name tf.random\_normal is deprecated. Please use tf.random.normal instead. INFO 2022-01-20 02:59:23,643| driveix.bpnet.trainers.bpnet\_trainer: Not first run and not finetuning experiment → Loading from latest checkpoint… WARNING:tensorflow:From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/losses/bpnet\_loss.py:120: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead. WARNING 2022-01-20 02:59:23,654| tensorflow: From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/losses/bpnet\_loss.py:120: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead. INFO 2022-01-20 02:59:26,849| **main** : training WARNING:tensorflow:From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/utils.py:59: The name tf.train.LoggingTensorHook is deprecated. Please use tf.estimator.LoggingTensorHook instead. WARNING 2022-01-20 02:59:26,853| tensorflow: From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/utils.py:59: The name tf.train.LoggingTensorHook is deprecated. Please use tf.estimator.LoggingTensorHook instead. WARNING:tensorflow:From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/utils.py:60: The name tf.train.StopAtStepHook is deprecated. Please use tf.estimator.StopAtStepHook instead. WARNING 2022-01-20 02:59:26,853| tensorflow: From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/utils.py:60: The name tf.train.StopAtStepHook is deprecated. Please use tf.estimator.StopAtStepHook instead. WARNING:tensorflow:From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/utils.py:73: The name tf.train.StepCounterHook is deprecated. Please use tf.estimator.StepCounterHook instead. WARNING 2022-01-20 02:59:26,854| tensorflow: From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/utils.py:73: The name tf.train.StepCounterHook is deprecated. Please use tf.estimator.StepCounterHook instead. INFO:tensorflow:Create CheckpointSaverHook. INFO 2022-01-20 02:59:26,854| tensorflow: Create CheckpointSaverHook. WARNING:tensorflow:From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/utils.py:99: The name tf.train.SummarySaverHook is deprecated. Please use tf.estimator.SummarySaverHook instead. WARNING 2022-01-20 02:59:26,854| tensorflow: From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/utils.py:99: The name tf.train.SummarySaverHook is deprecated. Please use tf.estimator.SummarySaverHook instead. WARNING:tensorflow:From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/trainers/bpnet\_trainer.py:300: The name tf.train.NanTensorHook is deprecated. Please use tf.estimator.NanTensorHook instead. WARNING 2022-01-20 02:59:26,854| tensorflow: From /root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/trainers/bpnet\_trainer.py:300: The name tf.train.NanTensorHook is deprecated. Please use tf.estimator.NanTensorHook instead. INFO:tensorflow:Graph was finalized. INFO 2022-01-20 02:59:36,833| tensorflow: Graph was finalized. INFO:tensorflow:Restoring parameters from /tmp/tmpmdw\_ct22/model.ckpt-115240 INFO 2022-01-20 02:59:37,243| tensorflow: Restoring parameters from /tmp/tmpmdw\_ct22/model.ckpt-115240 INFO:tensorflow:Running local\_init\_op. INFO 2022-01-20 02:59:38,394| tensorflow: Running local\_init\_op. INFO:tensorflow:Done running local\_init\_op. INFO 2022-01-20 02:59:38,531| tensorflow: Done running local\_init\_op. INFO:tensorflow:Saving checkpoints for step-115240. INFO 2022-01-20 03:00:21,579| tensorflow: Saving checkpoints for step-115240. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow\_backend.py:174: The name tf.get\_default\_session is deprecated. Please use tf.compat.v1.get\_default\_session instead. WARNING 2022-01-20 03:00:33,133| tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow\_backend.py:174: The name tf.get\_default\_session is deprecated. Please use tf.compat.v1.get\_default\_session instead. Traceback (most recent call last): File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/client/session.py”, line 1365, in \_do\_call return fn(\*args) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/client/session.py”, line 1350, in \_run\_fn target\_list, run\_metadata) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/client/session.py”, line 1443, in \_call\_tf\_sessionrun run\_metadata) tensorflow.python.framework.errors\_impl.InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: assertion failed: [32.3707886] [[{{node Optimizer/Assert/AssertGuard/Assert}}]] (1) Invalid argument: assertion failed: [32.3707886] [[{{node Optimizer/Assert/AssertGuard/Assert}}]] [[Model/block\_3d\_bn\_1/AssignMovingAvg\_1/\_2705]] 0 successful operations. 0 derived errors ignored. During handling of the above exception, another exception occurred: Traceback (most recent call last): File “/root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/scripts/train.py”, line 146, in File “/root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/scripts/train.py”, line 137, in main File “/root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/trainers/bpnet\_trainer.py”, line 316, in train File “/root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build\_wheel.runfiles/ai\_infra/moduluspy/modulus/blocks/trainers/trainer.py”, line 119, in run\_training\_loop File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/training/monitored\_session.py”, line 754, in run run\_metadata=run\_metadata) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/training/monitored\_session.py”, line 1360, in run raise six.reraise(\*original\_exc\_info) File “/usr/local/lib/python3.6/dist-packages/six.py”, line 696, in reraise raise value File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/training/monitored\_session.py”, line 1345, in run return self.\_sess.run(\*args, \*\*kwargs) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/training/monitored\_session.py”, line 1418, in run run\_metadata=run\_metadata) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/training/monitored\_session.py”, line 1176, in run return self.\_sess.run(\*args, \*\*kwargs) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/client/session.py”, line 956, in run run\_metadata\_ptr) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/client/session.py”, line 1180, in \_run feed\_dict\_tensor, options, run\_metadata) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/client/session.py”, line 1359, in \_do\_run run\_metadata) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/client/session.py”, line 1384, in \_do\_call raise type(e)(node\_def, op, message) tensorflow.python.framework.errors\_impl.InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: assertion failed: [32.3707886] [[node Optimizer/Assert/AssertGuard/Assert (defined at usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/framework/ops.py:1748)]] (1) Invalid argument: assertion failed: [32.3707886] [[node Optimizer/Assert/AssertGuard/Assert (defined at usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/framework/ops.py:1748)]] [[Model/block\_3d\_bn\_1/AssignMovingAvg\_1/\_2705]] 0 successful operations. 0 derived errors ignored. Original stack trace for ‘Optimizer/Assert/AssertGuard/Assert’: File “root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/scripts/train.py”, line 146, in File “root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/scripts/train.py”, line 134, in main File “root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/trainers/bpnet\_trainer.py”, line 255, in build File “root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/trainers/bpnet\_trainer.py”, line 245, in \_build\_distributed File “root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/optimizers/weighted\_momentum\_optimizer.py”, line 55, in build File “root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build\_wheel.runfiles/ai\_infra/moduluspy/modulus/blocks/learning\_rate\_schedules/softstart\_annealing\_schedule.py”, line 114, in get\_tensor File “root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/core/build\_wheel.runfiles/ai\_infra/moduluspy/modulus/hooks/utils.py”, line 40, in get\_softstart\_annealing\_learning\_rate File “usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/util/tf\_should\_use.py”, line 198, in wrapped return \_add\_should\_use\_warning(fn(\*args, \*\*kwargs)) File “usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/ops/control\_flow\_ops.py”, line 173, in Assert guarded\_assert = cond(condition, no\_op, true\_assert, name=“AssertGuard”) File “usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/util/deprecation.py”, line 513, in new\_func return func(\*args, \*\*kwargs) File “usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/ops/control\_flow\_ops.py”, line 1235, in cond orig\_res\_f, res\_f = context\_f.BuildCondBranch(false\_fn) File “usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/ops/control\_flow\_ops.py”, line 1061, in BuildCondBranch original\_result = fn() File “usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/ops/control\_flow\_ops.py”, line 171, in true\_assert condition, data, summarize, name=“Assert”) File “usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/ops/gen\_logging\_ops.py”, line 74, in \_assert name=name) File “usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/framework/op\_def\_library.py”, line 794, in \_apply\_op\_helper op\_def=op\_def) File “usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/util/deprecation.py”, line 513, in new\_func return func(\*args, \*\*kwargs) File “usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/framework/ops.py”, line 3357, in create\_op attrs, op\_def, compute\_device) File “usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/framework/ops.py”, line 3426, in \_create\_op\_internal op\_def=op\_def) File “usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/framework/ops.py”, line 1748, in **init** self.\_traceback = tf\_stack.extract\_stack() Traceback (most recent call last): File “/usr/local/bin/bpnet”, line 8, in sys.exit(main()) File “/root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/bpnet/entrypoint/bpnet.py”, line 12, in main File “/root/.cache/bazel/\_bazel\_root/ed34e6d125608f91724fda23656f1726/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build\_wheel.runfiles/ai\_infra/driveix/common/entrypoint/entrypoint.py”, line 300, in launch\_job AssertionError: Process run failed. 2022-01-20 11:00:47,788 [INFO] tlt.components.docker\_handler.docker\_handler: Stopping container. -------------------------------- if i set batch\_size=2. The error message will change: … tensorflow.python.framework.errors\_impl.InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: assertion failed: [64.7415771] [[{{node Optimizer/Assert/AssertGuard/Assert}}]] (1) Invalid argument: assertion failed: [64.7415771] [[{{node Optimizer/Assert/AssertGuard/Assert}}]] [[Optimizer/Exp/\_2677]] … -------------------------------- it’s too hard, --!, please help me. ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 20, 2022, 4:43am UTC](https://forums.developer.nvidia.com/t/tensorflow-python-framework-errors-impl-invalidargumenterror-2-root-error-s-found/201073/3 "2022-01-20T04:43:01Z")
If you have trained the default bpnet notebook successfully, could you please compare it to narrow down? ---
**Author:** ![oomg](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@oomg](https://forums.developer.nvidia.com/u/oomg)\ **Post date:** [January 20, 2022, 5:21am UTC](https://forums.developer.nvidia.com/t/tensorflow-python-framework-errors-impl-invalidargumenterror-2-root-error-s-found/201073/4 "2022-01-20T05:21:04Z")
I successfully trained the default BPnet notebook, but I used the dataset from coco2017. If I use a custom dataset, an error will be reported。 I compared their JSON files and found nothing。 At the same time, my JSON file is designed according to dpnet notebook。 The error must have happened to my dataset, but I don’t have a clue。 ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 21, 2022, 9:38am UTC](https://forums.developer.nvidia.com/t/tensorflow-python-framework-errors-impl-invalidargumenterror-2-root-error-s-found/201073/5 "2022-01-21T09:38:24Z")
Can you use a new result folder and retry? ---
**Author:** ![oomg](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@oomg](https://forums.developer.nvidia.com/u/oomg)\ **Post date:** [January 22, 2022, 7:01am UTC](https://forums.developer.nvidia.com/t/tensorflow-python-framework-errors-impl-invalidargumenterror-2-root-error-s-found/201073/6 "2022-01-22T07:01:40Z")
I want to retest with other labeling software. ---
**Author:** ![oomg](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@oomg](https://forums.developer.nvidia.com/u/oomg)\ **Post date:** [January 23, 2022, 7:56am UTC](https://forums.developer.nvidia.com/t/tensorflow-python-framework-errors-impl-invalidargumenterror-2-root-error-s-found/201073/7 "2022-01-23T07:56:46Z")
I tried to modify it many times, but I still failed. Can you teach me how to make this coco keypoint data set ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [January 23, 2022, 10:24am UTC](https://forums.developer.nvidia.com/t/tensorflow-python-framework-errors-impl-invalidargumenterror-2-root-error-s-found/201073/8 "2022-01-23T10:24:52Z")
Could you please refer to [Body Pose Estimation - NVIDIA Docs](https://docs.nvidia.com/tao/tao-toolkit/text/bodypose_estimation/bodyposenet.html#data-input-for-bodyposenet) ? > The BodyPoseNet apps in TAO Toolkit expect data in COCO format for training and evaluation. > > See the [Data Annotation Format](https://docs.nvidia.com/tao/tao-toolkit/text/data_annotation_format.html#id11) page for more information about the COCO data format. > > The [Sloth](https://github.com/cvhciKIT/sloth) and [Label Studio](https://labelstud.io/) tools may be used for labeling. ---
**Author:** ![oomg](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@oomg](https://forums.developer.nvidia.com/u/oomg)\ **Post date:** [January 23, 2022, 10:33am UTC](https://forums.developer.nvidia.com/t/tensorflow-python-framework-errors-impl-invalidargumenterror-2-root-error-s-found/201073/9 "2022-01-23T10:33:44Z")
I’ve read this document before, but I haven’t seen the recommended tools。 I’m too careless。 ---
**Author:** ![ajayverma81](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@ajayverma81](https://forums.developer.nvidia.com/u/ajayverma81)\ **Post date:** [January 28, 2022, 5:10pm UTC](https://forums.developer.nvidia.com/t/tensorflow-python-framework-errors-impl-invalidargumenterror-2-root-error-s-found/201073/10 "2022-01-28T17:10:39Z")
I’ve got the same error using the same version of tao and try to train on the tao toolkit’s cv\_samples\_v1.3.0 for mask rcnn. ---
**Author:** ![oomg](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@oomg](https://forums.developer.nvidia.com/u/oomg)\ **Post date:** [January 29, 2022, 1:29am UTC](https://forums.developer.nvidia.com/t/tensorflow-python-framework-errors-impl-invalidargumenterror-2-root-error-s-found/201073/11 "2022-01-29T01:29:51Z")
I use the labelme tool for annotation. Sloth and labelstudio haven’t tried yet. What labeling tool do you use. ---
**Author:** ![oomg](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@oomg](https://forums.developer.nvidia.com/u/oomg)\ **Post date:** [January 30, 2022, 9:46am UTC](https://forums.developer.nvidia.com/t/tensorflow-python-framework-errors-impl-invalidargumenterror-2-root-error-s-found/201073/12 "2022-01-30T09:46:46Z")
What labeling tools do you use ---
**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\ **Post date:** [February 13, 2022, 9:47am UTC](https://forums.developer.nvidia.com/t/tensorflow-python-framework-errors-impl-invalidargumenterror-2-root-error-s-found/201073/13 "2022-02-13T09:47:29Z")
Closed on February 13, 2022, 9:47am UTC This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.