# Error while training on tlt **URL:** \ **Category:** TAO Toolkit\ **Created:** [July 6, 2021, 10:06am UTC](https://forums.developer.nvidia.com/t/error-while-training-on-tlt/182884 "2021-07-06T10:06:57Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![aksinghsubscriptions](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@aksinghsubscriptions](https://forums.developer.nvidia.com/u/aksinghsubscriptions)\ **Post date:** [July 6, 2021, 10:06am UTC](https://forums.developer.nvidia.com/t/error-while-training-on-tlt/182884/1 "2021-07-06T10:06:58Z")
Please provide the following information when requesting support. i am trying to run the tlt training inside the tlt container and while doing so i am getting this error. WARNING:tensorflow:Deprecation warnings have been disabled. Set TF\_ENABLE\_DEPRECATION\_WARNINGS=1 to re-enable them. Using TensorFlow backend. WARNING:tensorflow:From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/cost\_function/cost\_auto\_weight\_hook.py:43: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/checkpoint\_saver\_hook.py:25: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/scripts/train.py:68: The name tf.logging.set\_verbosity is deprecated. Please use tf.compat.v1.logging.set\_verbosity instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/scripts/train.py:68: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/ **init**.py:117: The name tf.global\_variables is deprecated. Please use tf.compat.v1.global\_variables instead. 2021-07-06 09:00:40,409 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/ **init**.py:117: The name tf.global\_variables is deprecated. Please use tf.compat.v1.global\_variables instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/ **init**.py:143: The name tf.get\_default\_graph is deprecated. Please use tf.compat.v1.get\_default\_graph instead. 2021-07-06 09:00:40,409 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/ **init**.py:143: The name tf.get\_default\_graph is deprecated. Please use tf.compat.v1.get\_default\_graph instead. 2021-07-06 09:00:40,616 [INFO] **main** : Loading experiment spec at /workspace/tlt-experiments/specs/detectnet\_v2\_train\_resnet18\_kitti.txt. 2021-07-06 09:00:40,618 [INFO] iva.detectnet\_v2.spec\_handler.spec\_loader: Merging specification from /workspace/tlt-experiments/specs/detectnet\_v2\_train\_resnet18\_kitti.txt 2021-07-06 09:00:40,930 [INFO] **main** : Cannot iterate over exactly 87 samples with a batch size of 4; each epoch will therefore take one extra step. WARNING:tensorflow:From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/cost\_function/cost\_auto\_weight\_hook.py:107: The name tf.variable\_scope is deprecated. Please use tf.compat.v1.variable\_scope instead. 2021-07-06 09:00:40,932 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/cost\_function/cost\_auto\_weight\_hook.py:107: The name tf.variable\_scope is deprecated. Please use tf.compat.v1.variable\_scope instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/cost\_function/cost\_auto\_weight\_hook.py:110: The name tf.get\_variable is deprecated. Please use tf.compat.v1.get\_variable instead. 2021-07-06 09:00:40,933 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/cost\_function/cost\_auto\_weight\_hook.py:110: The name tf.get\_variable is deprecated. Please use tf.compat.v1.get\_variable instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/cost\_function/cost\_auto\_weight\_hook.py:113: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead. 2021-07-06 09:00:40,937 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/cost\_function/cost\_auto\_weight\_hook.py:113: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow\_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. 2021-07-06 09:00:41,046 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow\_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow\_backend.py:4138: The name tf.random\_uniform is deprecated. Please use tf.random.uniform instead. 2021-07-06 09:00:41,047 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow\_backend.py:4138: The name tf.random\_uniform is deprecated. Please use tf.random.uniform instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow\_backend.py:1834: The name tf.nn.fused\_batch\_norm is deprecated. Please use tf.compat.v1.nn.fused\_batch\_norm instead. 2021-07-06 09:00:41,079 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow\_backend.py:1834: The name tf.nn.fused\_batch\_norm is deprecated. Please use tf.compat.v1.nn.fused\_batch\_norm instead. WARNING:tensorflow:From /opt/nvidia/third\_party/keras/tensorflow\_backend.py:187: The name tf.nn.avg\_pool is deprecated. Please use tf.nn.avg\_pool2d instead. 2021-07-06 09:00:42,705 [WARNING] tensorflow: From /opt/nvidia/third\_party/keras/tensorflow\_backend.py:187: The name tf.nn.avg\_pool is deprecated. Please use tf.nn.avg\_pool2d instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow\_backend.py:174: The name tf.get\_default\_session is deprecated. Please use tf.compat.v1.get\_default\_session instead. 2021-07-06 09:00:43,062 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow\_backend.py:174: The name tf.get\_default\_session is deprecated. Please use tf.compat.v1.get\_default\_session instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow\_backend.py:199: The name tf.is\_variable\_initialized is deprecated. Please use tf.compat.v1.is\_variable\_initialized instead. 2021-07-06 09:00:43,062 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow\_backend.py:199: The name tf.is\_variable\_initialized is deprecated. Please use tf.compat.v1.is\_variable\_initialized instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow\_backend.py:206: The name tf.variables\_initializer is deprecated. Please use tf.compat.v1.variables\_initializer instead. 2021-07-06 09:00:43,466 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow\_backend.py:206: The name tf.variables\_initializer is deprecated. Please use tf.compat.v1.variables\_initializer instead. 2021-07-06 09:00:49,745 [INFO] iva.detectnet\_v2.objectives.bbox\_objective: Default L1 loss function will be used. * * * # Layer (type) Output Shape Param # Connected to input\_1 (InputLayer) (None, 3, 384, 1248) 0 * * * conv1 (Conv2D) (None, 64, 192, 624) 9472 input\_1[0][0] * * * bn\_conv1 (BatchNormalization) (None, 64, 192, 624) 256 conv1[0][0] * * * activation\_1 (Activation) (None, 64, 192, 624) 0 bn\_conv1[0][0] * * * block\_1a\_conv\_1 (Conv2D) (None, 64, 96, 312) 36928 activation\_1[0][0] * * * block\_1a\_bn\_1 (BatchNormalizati (None, 64, 96, 312) 256 block\_1a\_conv\_1[0][0] * * * block\_1a\_relu\_1 (Activation) (None, 64, 96, 312) 0 block\_1a\_bn\_1[0][0] * * * block\_1a\_conv\_2 (Conv2D) (None, 64, 96, 312) 36928 block\_1a\_relu\_1[0][0] * * * block\_1a\_conv\_shortcut (Conv2D) (None, 64, 96, 312) 4160 activation\_1[0][0] * * * block\_1a\_bn\_2 (BatchNormalizati (None, 64, 96, 312) 256 block\_1a\_conv\_2[0][0] * * * block\_1a\_bn\_shortcut (BatchNorm (None, 64, 96, 312) 256 block\_1a\_conv\_shortcut[0][0] * * * add\_1 (Add) (None, 64, 96, 312) 0 block\_1a\_bn\_2[0][0] block\_1a\_bn\_shortcut[0][0] * * * block\_1a\_relu (Activation) (None, 64, 96, 312) 0 add\_1[0][0] * * * block\_1b\_conv\_1 (Conv2D) (None, 64, 96, 312) 36928 block\_1a\_relu[0][0] * * * block\_1b\_bn\_1 (BatchNormalizati (None, 64, 96, 312) 256 block\_1b\_conv\_1[0][0] * * * block\_1b\_relu\_1 (Activation) (None, 64, 96, 312) 0 block\_1b\_bn\_1[0][0] * * * block\_1b\_conv\_2 (Conv2D) (None, 64, 96, 312) 36928 block\_1b\_relu\_1[0][0] * * * block\_1b\_bn\_2 (BatchNormalizati (None, 64, 96, 312) 256 block\_1b\_conv\_2[0][0] * * * add\_2 (Add) (None, 64, 96, 312) 0 block\_1b\_bn\_2[0][0] block\_1a\_relu[0][0] * * * block\_1b\_relu (Activation) (None, 64, 96, 312) 0 add\_2[0][0] * * * block\_2a\_conv\_1 (Conv2D) (None, 128, 48, 156) 73856 block\_1b\_relu[0][0] * * * block\_2a\_bn\_1 (BatchNormalizati (None, 128, 48, 156) 512 block\_2a\_conv\_1[0][0] * * * block\_2a\_relu\_1 (Activation) (None, 128, 48, 156) 0 block\_2a\_bn\_1[0][0] * * * block\_2a\_conv\_2 (Conv2D) (None, 128, 48, 156) 147584 block\_2a\_relu\_1[0][0] * * * block\_2a\_conv\_shortcut (Conv2D) (None, 128, 48, 156) 8320 block\_1b\_relu[0][0] * * * block\_2a\_bn\_2 (BatchNormalizati (None, 128, 48, 156) 512 block\_2a\_conv\_2[0][0] * * * block\_2a\_bn\_shortcut (BatchNorm (None, 128, 48, 156) 512 block\_2a\_conv\_shortcut[0][0] * * * add\_3 (Add) (None, 128, 48, 156) 0 block\_2a\_bn\_2[0][0] block\_2a\_bn\_shortcut[0][0] * * * block\_2a\_relu (Activation) (None, 128, 48, 156) 0 add\_3[0][0] * * * block\_2b\_conv\_1 (Conv2D) (None, 128, 48, 156) 147584 block\_2a\_relu[0][0] * * * block\_2b\_bn\_1 (BatchNormalizati (None, 128, 48, 156) 512 block\_2b\_conv\_1[0][0] * * * block\_2b\_relu\_1 (Activation) (None, 128, 48, 156) 0 block\_2b\_bn\_1[0][0] * * * block\_2b\_conv\_2 (Conv2D) (None, 128, 48, 156) 147584 block\_2b\_relu\_1[0][0] * * * block\_2b\_bn\_2 (BatchNormalizati (None, 128, 48, 156) 512 block\_2b\_conv\_2[0][0] * * * add\_4 (Add) (None, 128, 48, 156) 0 block\_2b\_bn\_2[0][0] block\_2a\_relu[0][0] * * * block\_2b\_relu (Activation) (None, 128, 48, 156) 0 add\_4[0][0] * * * block\_3a\_conv\_1 (Conv2D) (None, 256, 24, 78) 295168 block\_2b\_relu[0][0] * * * block\_3a\_bn\_1 (BatchNormalizati (None, 256, 24, 78) 1024 block\_3a\_conv\_1[0][0] 2021-07-06 09:00:49,806 [INFO] modulus.blocks.data\_loaders.multi\_source\_loader.data\_loader: Serial augmentation enabled = False 2021-07-06 09:00:49,806 [INFO] modulus.blocks.data\_loaders.multi\_source\_loader.data\_loader: Pseudo sharding enabled = False 2021-07-06 09:00:49,806 [INFO] modulus.blocks.data\_loaders.multi\_source\_loader.data\_loader: Max Image Dimensions (all sources): (0, 0) 2021-07-06 09:00:49,806 [INFO] modulus.blocks.data\_loaders.multi\_source\_loader.data\_loader: number of cpus: 4, io threads: 8, compute threads: 4, buffered batches: 4 2021-07-06 09:00:49,806 [INFO] modulus.blocks.data\_loaders.multi\_source\_loader.data\_loader: total dataset size 87, number of sources: 1, batch size per gpu: 4, steps: 22 WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/autograph/converters/directives.py:119: The name tf.set\_random\_seed is deprecated. Please use tf.compat.v1.set\_random\_seed instead. 2021-07-06 09:00:49,868 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/autograph/converters/directives.py:119: The name tf.set\_random\_seed is deprecated. Please use tf.compat.v1.set\_random\_seed instead. WARNING:tensorflow:Entity \\> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of \\>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do\_not\_convert. Original error: could not get source code 2021-07-06 09:00:49,933 [WARNING] tensorflow: Entity \\> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of \\>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do\_not\_convert. Original error: could not get source code 2021-07-06 09:00:49,967 [INFO] iva.detectnet\_v2.dataloader.default\_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates. 2021-07-06 09:00:50,348 [INFO] modulus.blocks.data\_loaders.multi\_source\_loader.data\_loader: shuffle: True - shard 0 of 1 2021-07-06 09:00:50,357 [INFO] modulus.blocks.data\_loaders.multi\_source\_loader.data\_loader: sampling 1 datasets with weights: 2021-07-06 09:00:50,357 [INFO] modulus.blocks.data\_loaders.multi\_source\_loader.data\_loader: source: 0 weight: 1.000000 WARNING:tensorflow:Entity \\> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of \\>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do\_not\_convert. Original error: could not get source code 2021-07-06 09:00:50,377 [WARNING] tensorflow: Entity \\> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of \\>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do\_not\_convert. Original error: could not get source code 2021-07-06 09:00:50,957 [INFO] **main** : Found 87 samples in training set WARNING:tensorflow:From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/rasterizers/bbox\_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead. 2021-07-06 09:00:51,126 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/rasterizers/bbox\_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/training/training\_proto\_utilities.py:89: The name tf.train.get\_or\_create\_global\_step is deprecated. Please use tf.compat.v1.train.get\_or\_create\_global\_step instead. 2021-07-06 09:00:51,330 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/training/training\_proto\_utilities.py:89: The name tf.train.get\_or\_create\_global\_step is deprecated. Please use tf.compat.v1.train.get\_or\_create\_global\_step instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/training/training\_proto\_utilities.py:36: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead. 2021-07-06 09:00:51,355 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/training/training\_proto\_utilities.py:36: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/cost\_function/cost\_functions.py:17: The name tf.log is deprecated. Please use tf.math.log instead. 2021-07-06 09:00:51,613 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/cost\_function/cost\_functions.py:17: The name tf.log is deprecated. Please use tf.math.log instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/cost\_function/cost\_auto\_weight\_hook.py:235: The name tf.assign\_add is deprecated. Please use tf.compat.v1.assign\_add instead. 2021-07-06 09:00:51,697 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/cost\_function/cost\_auto\_weight\_hook.py:235: The name tf.assign\_add is deprecated. Please use tf.compat.v1.assign\_add instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/model/detectnet\_model.py:587: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead. 2021-07-06 09:00:51,712 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/model/detectnet\_model.py:587: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead. 2021-07-06 09:00:54,274 [INFO] modulus.blocks.data\_loaders.multi\_source\_loader.data\_loader: Serial augmentation enabled = False 2021-07-06 09:00:54,274 [INFO] modulus.blocks.data\_loaders.multi\_source\_loader.data\_loader: Pseudo sharding enabled = False 2021-07-06 09:00:54,274 [INFO] modulus.blocks.data\_loaders.multi\_source\_loader.data\_loader: Max Image Dimensions (all sources): (0, 0) 2021-07-06 09:00:54,275 [INFO] modulus.blocks.data\_loaders.multi\_source\_loader.data\_loader: number of cpus: 4, io threads: 8, compute threads: 4, buffered batches: 4 2021-07-06 09:00:54,275 [INFO] modulus.blocks.data\_loaders.multi\_source\_loader.data\_loader: total dataset size 14, number of sources: 1, batch size per gpu: 4, steps: 4 WARNING:tensorflow:Entity \\> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of \\>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do\_not\_convert. Original error: could not get source code 2021-07-06 09:00:54,289 [WARNING] tensorflow: Entity \\> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of \\>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do\_not\_convert. Original error: could not get source code 2021-07-06 09:00:54,318 [INFO] iva.detectnet\_v2.dataloader.default\_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates. 2021-07-06 09:00:54,676 [INFO] modulus.blocks.data\_loaders.multi\_source\_loader.data\_loader: shuffle: False - shard 0 of 1 2021-07-06 09:00:54,683 [INFO] modulus.blocks.data\_loaders.multi\_source\_loader.data\_loader: sampling 1 datasets with weights: 2021-07-06 09:00:54,683 [INFO] modulus.blocks.data\_loaders.multi\_source\_loader.data\_loader: source: 0 weight: 1.000000 WARNING:tensorflow:Entity \\> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of \\>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do\_not\_convert. Original error: could not get source code 2021-07-06 09:00:54,704 [WARNING] tensorflow: Entity \\> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of \\>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do\_not\_convert. Original error: could not get source code 2021-07-06 09:00:55,093 [INFO] **main** : Found 14 samples in validation set WARNING:tensorflow:From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/validation\_hook.py:40: The name tf.summary.FileWriterCache is deprecated. Please use tf.compat.v1.summary.FileWriterCache instead. 2021-07-06 09:00:56,000 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/validation\_hook.py:40: The name tf.summary.FileWriterCache is deprecated. Please use tf.compat.v1.summary.FileWriterCache instead. 2021-07-06 09:00:57,502 [INFO] **main** : Checkpoint interval: 10 WARNING:tensorflow:From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/scripts/train.py:108: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead. 2021-07-06 09:00:57,502 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/scripts/train.py:108: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/common/graph/initializers.py:14: The name tf.local\_variables\_initializer is deprecated. Please use tf.compat.v1.local\_variables\_initializer instead. 2021-07-06 09:00:57,502 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/common/graph/initializers.py:14: The name tf.local\_variables\_initializer is deprecated. Please use tf.compat.v1.local\_variables\_initializer instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/common/graph/initializers.py:15: The name tf.tables\_initializer is deprecated. Please use tf.compat.v1.tables\_initializer instead. 2021-07-06 09:00:57,503 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/common/graph/initializers.py:15: The name tf.tables\_initializer is deprecated. Please use tf.compat.v1.tables\_initializer instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/common/graph/initializers.py:16: The name tf.get\_collection is deprecated. Please use tf.compat.v1.get\_collection instead. 2021-07-06 09:00:57,504 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/common/graph/initializers.py:16: The name tf.get\_collection is deprecated. Please use tf.compat.v1.get\_collection instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/utils.py:59: The name tf.train.LoggingTensorHook is deprecated. Please use tf.estimator.LoggingTensorHook instead. 2021-07-06 09:00:57,507 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/utils.py:59: The name tf.train.LoggingTensorHook is deprecated. Please use tf.estimator.LoggingTensorHook instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/utils.py:60: The name tf.train.StopAtStepHook is deprecated. Please use tf.estimator.StopAtStepHook instead. 2021-07-06 09:00:57,507 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/utils.py:60: The name tf.train.StopAtStepHook is deprecated. Please use tf.estimator.StopAtStepHook instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/utils.py:73: The name tf.train.StepCounterHook is deprecated. Please use tf.estimator.StepCounterHook instead. 2021-07-06 09:00:57,507 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/utils.py:73: The name tf.train.StepCounterHook is deprecated. Please use tf.estimator.StepCounterHook instead. INFO:tensorflow:Create CheckpointSaverHook. 2021-07-06 09:00:57,507 [INFO] tensorflow: Create CheckpointSaverHook. WARNING:tensorflow:From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/utils.py:99: The name tf.train.SummarySaverHook is deprecated. Please use tf.estimator.SummarySaverHook instead. 2021-07-06 09:00:57,507 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/tfhooks/utils.py:99: The name tf.train.SummarySaverHook is deprecated. Please use tf.estimator.SummarySaverHook instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead. 2021-07-06 09:00:57,509 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead. INFO:tensorflow:Graph was finalized. 2021-07-06 09:00:58,904 [INFO] tensorflow: Graph was finalized. INFO:tensorflow:Running local\_init\_op. 2021-07-06 09:01:00,721 [INFO] tensorflow: Running local\_init\_op. INFO:tensorflow:Done running local\_init\_op. 2021-07-06 09:01:01,682 [INFO] tensorflow: Done running local\_init\_op. INFO:tensorflow:Saving checkpoints for step-0. 2021-07-06 09:01:10,103 [INFO] tensorflow: Saving checkpoints for step-0. * * * block\_3a\_relu\_1 (Activation) (None, 256, 24, 78) 0 block\_3a\_bn\_1[0][0] * * * block\_3a\_conv\_2 (Conv2D) (None, 256, 24, 78) 590080 block\_3a\_relu\_1[0][0] * * * block\_3a\_conv\_shortcut (Conv2D) (None, 256, 24, 78) 33024 block\_2b\_relu[0][0] * * * block\_3a\_bn\_2 (BatchNormalizati (None, 256, 24, 78) 1024 block\_3a\_conv\_2[0][0] * * * block\_3a\_bn\_shortcut (BatchNorm (None, 256, 24, 78) 1024 block\_3a\_conv\_shortcut[0][0] * * * add\_5 (Add) (None, 256, 24, 78) 0 block\_3a\_bn\_2[0][0] block\_3a\_bn\_shortcut[0][0] * * * block\_3a\_relu (Activation) (None, 256, 24, 78) 0 add\_5[0][0] * * * block\_3b\_conv\_1 (Conv2D) (None, 256, 24, 78) 590080 block\_3a\_relu[0][0] * * * block\_3b\_bn\_1 (BatchNormalizati (None, 256, 24, 78) 1024 block\_3b\_conv\_1[0][0] * * * block\_3b\_relu\_1 (Activation) (None, 256, 24, 78) 0 block\_3b\_bn\_1[0][0] * * * block\_3b\_conv\_2 (Conv2D) (None, 256, 24, 78) 590080 block\_3b\_relu\_1[0][0] * * * block\_3b\_bn\_2 (BatchNormalizati (None, 256, 24, 78) 1024 block\_3b\_conv\_2[0][0] * * * add\_6 (Add) (None, 256, 24, 78) 0 block\_3b\_bn\_2[0][0] block\_3a\_relu[0][0] * * * block\_3b\_relu (Activation) (None, 256, 24, 78) 0 add\_6[0][0] * * * block\_4a\_conv\_1 (Conv2D) (None, 512, 24, 78) 1180160 block\_3b\_relu[0][0] * * * block\_4a\_bn\_1 (BatchNormalizati (None, 512, 24, 78) 2048 block\_4a\_conv\_1[0][0] * * * block\_4a\_relu\_1 (Activation) (None, 512, 24, 78) 0 block\_4a\_bn\_1[0][0] * * * block\_4a\_conv\_2 (Conv2D) (None, 512, 24, 78) 2359808 block\_4a\_relu\_1[0][0] * * * block\_4a\_conv\_shortcut (Conv2D) (None, 512, 24, 78) 131584 block\_3b\_relu[0][0] * * * block\_4a\_bn\_2 (BatchNormalizati (None, 512, 24, 78) 2048 block\_4a\_conv\_2[0][0] * * * block\_4a\_bn\_shortcut (BatchNorm (None, 512, 24, 78) 2048 block\_4a\_conv\_shortcut[0][0] * * * add\_7 (Add) (None, 512, 24, 78) 0 block\_4a\_bn\_2[0][0] block\_4a\_bn\_shortcut[0][0] * * * block\_4a\_relu (Activation) (None, 512, 24, 78) 0 add\_7[0][0] * * * block\_4b\_conv\_1 (Conv2D) (None, 512, 24, 78) 2359808 block\_4a\_relu[0][0] * * * block\_4b\_bn\_1 (BatchNormalizati (None, 512, 24, 78) 2048 block\_4b\_conv\_1[0][0] * * * block\_4b\_relu\_1 (Activation) (None, 512, 24, 78) 0 block\_4b\_bn\_1[0][0] * * * block\_4b\_conv\_2 (Conv2D) (None, 512, 24, 78) 2359808 block\_4b\_relu\_1[0][0] * * * block\_4b\_bn\_2 (BatchNormalizati (None, 512, 24, 78) 2048 block\_4b\_conv\_2[0][0] * * * add\_8 (Add) (None, 512, 24, 78) 0 block\_4b\_bn\_2[0][0] block\_4a\_relu[0][0] * * * block\_4b\_relu (Activation) (None, 512, 24, 78) 0 add\_8[0][0] * * * output\_bbox (Conv2D) (None, 12, 24, 78) 6156 block\_4b\_relu[0][0] * * * # output\_cov (Conv2D) (None, 3, 24, 78) 1539 block\_4b\_relu[0][0] Total params: 11,203,023 Trainable params: 11,193,295 Non-trainable params: 9,728 * * * Traceback (most recent call last): File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/client/session.py”, line 1365, in \_do\_call return fn(\*args) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/client/session.py”, line 1350, in \_run\_fn target\_list, run\_metadata) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/client/session.py”, line 1443, in \_call\_tf\_sessionrun run\_metadata) tensorflow.python.framework.errors\_impl.InvalidArgumentError: Conv2DCustomBackpropInputOp only supports NHWC. [[{{node gradients/resnet18\_nopool\_bn\_detectnet\_v2/output\_bbox/convolution\_grad/Conv2DBackpropInput}}]] During handling of the above exception, another exception occurred: Traceback (most recent call last): File “/opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/scripts/train.py”, line 843, in File “/opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/scripts/train.py”, line 832, in File “”, line 2, in main File “/opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/utilities/timer.py”, line 46, in wrapped\_fn File “/opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/scripts/train.py”, line 821, in main File “/opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/scripts/train.py”, line 702, in run\_experiment File “/opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/scripts/train.py”, line 638, in train\_gridbox File “/opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/scripts/train.py”, line 154, in run\_training\_loop File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/training/monitored\_session.py”, line 754, in run run\_metadata=run\_metadata) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/training/monitored\_session.py”, line 1360, in run raise six.reraise(\*original\_exc\_info) File “/usr/local/lib/python3.6/dist-packages/six.py”, line 696, in reraise raise value File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/training/monitored\_session.py”, line 1345, in run return self.\_sess.run(\*args, \*\*kwargs) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/training/monitored\_session.py”, line 1418, in run run\_metadata=run\_metadata) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/training/monitored\_session.py”, line 1176, in run return self.\_sess.run(\*args, \*\*kwargs) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/client/session.py”, line 956, in run run\_metadata\_ptr) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/client/session.py”, line 1180, in \_run feed\_dict\_tensor, options, run\_metadata) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/client/session.py”, line 1359, in \_do\_run run\_metadata) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/client/session.py”, line 1384, in \_do\_call raise type(e)(node\_def, op, message) tensorflow.python.framework.errors\_impl.InvalidArgumentError: Conv2DCustomBackpropInputOp only supports NHWC. [[node gradients/resnet18\_nopool\_bn\_detectnet\_v2/output\_bbox/convolution\_grad/Conv2DBackpropInput (defined at /usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/framework/ops.py:1748)]] Original stack trace for ‘gradients/resnet18\_nopool\_bn\_detectnet\_v2/output\_bbox/convolution\_grad/Conv2DBackpropInput’: File “/opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/scripts/train.py”, line 832, in File “”, line 2, in main File “/opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/utilities/timer.py”, line 46, in wrapped\_fn File “/opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/scripts/train.py”, line 821, in main File “/opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/scripts/train.py”, line 702, in run\_experiment File “/opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/scripts/train.py”, line 613, in train\_gridbox File “/opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/scripts/train.py”, line 468, in build\_training\_graph File “/opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/model/detectnet\_model.py”, line 598, in build\_training\_graph File “/opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/training/train\_op\_generator.py”, line 59, in get\_train\_op File “/opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/training/train\_op\_generator.py”, line 74, in \_get\_train\_op\_without\_cost\_scaling File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/training/optimizer.py”, line 419, in minimize grad\_loss=grad\_loss) File “/usr/local/lib/python3.6/dist-packages/horovod/tensorflow/ **init**.py”, line 253, in compute\_gradients gradients = self.\_optimizer.compute\_gradients(\*args, \*\*kwargs) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/training/optimizer.py”, line 537, in compute\_gradients colocate\_gradients\_with\_ops=colocate\_gradients\_with\_ops) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/ops/gradients\_impl.py”, line 158, in gradients unconnected\_gradients) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/ops/gradients\_util.py”, line 703, in \_GradientsHelper lambda: grad\_fn(op, \*out\_grads)) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/ops/gradients\_util.py”, line 362, in \_MaybeCompile return grad\_fn() # Exit early File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/ops/gradients\_util.py”, line 703, in lambda: grad\_fn(op, \*out\_grads)) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/ops/nn\_grad.py”, line 596, in \_Conv2DGrad data\_format=data\_format), File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/ops/gen\_nn\_ops.py”, line 1407, in conv2d\_backprop\_input name=name) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/framework/op\_def\_library.py”, line 794, in \_apply\_op\_helper op\_def=op\_def) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/util/deprecation.py”, line 513, in new\_func return func(\*args, \*\*kwargs) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/framework/ops.py”, line 3357, in create\_op attrs, op\_def, compute\_device) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/framework/ops.py”, line 3426, in \_create\_op\_internal op\_def=op\_def) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/framework/ops.py”, line 1748, in **init** self.\_traceback = tf\_stack.extract\_stack() …which was originally created as op ‘resnet18\_nopool\_bn\_detectnet\_v2/output\_bbox/convolution’, defined at: File “/opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/scripts/train.py”, line 832, in [elided 5 identical lines from previous traceback] File “/opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/scripts/train.py”, line 468, in build\_training\_graph File “/opt/tlt/.cache/dazel/\_dazel\_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai\_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build\_wheel.runfiles/ai\_infra/iva/detectnet\_v2/model/detectnet\_model.py”, line 572, in build\_training\_graph File “/usr/local/lib/python3.6/dist-packages/keras/engine/base\_layer.py”, line 457, in **call** output = self.call(inputs, \*\*kwargs) File “/usr/local/lib/python3.6/dist-packages/keras/engine/network.py”, line 564, in call output\_tensors, \_, \_ = self.run\_internal\_graph(inputs, masks) File “/usr/local/lib/python3.6/dist-packages/keras/engine/network.py”, line 721, in run\_internal\_graph layer.call(computed\_tensor, \*\*kwargs)) File “/usr/local/lib/python3.6/dist-packages/keras/layers/convolutional.py”, line 171, in call dilation\_rate=self.dilation\_rate) File “/opt/nvidia/third\_party/keras/tensorflow\_backend.py”, line 113, in conv2d data\_format=tf\_data\_format, File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/ops/nn\_ops.py”, line 921, in convolution name=name) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/ops/nn\_ops.py”, line 1032, in convolution\_internal name=name) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/ops/gen\_nn\_ops.py”, line 1071, in conv2d data\_format=data\_format, dilations=dilations, name=name) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/framework/op\_def\_library.py”, line 794, in \_apply\_op\_helper op\_def=op\_def) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/util/deprecation.py”, line 513, in new\_func return func(\*args, \*\*kwargs) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/framework/ops.py”, line 3357, in create\_op attrs, op\_def, compute\_device) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/framework/ops.py”, line 3426, in \_create\_op\_internal op\_def=op\_def) File “/usr/local/lib/python3.6/dist-packages/tensorflow\_core/python/framework/ops.py”, line 1748, in **init** self.\_traceback = tf\_stack.extract\_stack() Using TensorFlow backend. I am using the specification files for detectnet\_v2 example with 100 training images, and 30 test images ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [July 6, 2021, 1:42pm UTC](https://forums.developer.nvidia.com/t/error-while-training-on-tlt/182884/2 "2021-07-06T13:42:14Z")
Which gpu did you use ? ---
**Author:** ![aksinghsubscriptions](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@aksinghsubscriptions](https://forums.developer.nvidia.com/u/aksinghsubscriptions)\ **Post date:** [July 6, 2021, 5:57pm UTC](https://forums.developer.nvidia.com/t/error-while-training-on-tlt/182884/3 "2021-07-06T17:57:29Z")
tesla t4 ---
**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\ **Post date:** [July 7, 2021, 1:47am UTC](https://forums.developer.nvidia.com/t/error-while-training-on-tlt/182884/4 "2021-07-07T01:47:10Z")
**There is no update from you for a period, assuming this is not an issue any more. Hence we are closing this topic. If need further support, please open a new one. Thanks** Can you share the training spec? ---
**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\ **Post date:** [September 5, 2021, 1:47am UTC](https://forums.developer.nvidia.com/t/error-while-training-on-tlt/182884/5 "2021-09-05T01:47:19Z")
This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.