To run with multigpu, please change --gpus based on the number of available GPUs in your machine. 2022-04-07 16:23:16,171 [INFO] root: Registry: ['nvcr.io'] 2022-04-07 16:23:16,356 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3 2022-04-07 16:23:17,142 [WARNING] tlt.components.docker_handler.docker_handler: Docker will run the commands as root. If you would like to retain your local host permissions, please add the "user":"UID:GID" in the DockerOptions portion of the "/home/sysadmin/.tao_mounts.json" file. You can obtain your users UID and GID by using the "id -u" and "id -g" commands on the terminal. Using TensorFlow backend. Using TensorFlow backend. WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them. Using TensorFlow backend. WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py:40: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py:40: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead. 2022-04-07 09:23:28,422 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py:40: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead. 2022-04-07 09:23:28,422 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py:40: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py:43: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py:43: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead. 2022-04-07 09:23:28,423 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py:43: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead. 2022-04-07 09:23:28,423 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py:43: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. 2022-04-07 09:23:29,181 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/data_loader/generate_shape_tensors.py:8: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead. 2022-04-07 09:23:29,206 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/data_loader/generate_shape_tensors.py:8: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/data_loader/generate_shape_tensors.py:8: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead. 2022-04-07 09:23:29,207 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/data_loader/generate_shape_tensors.py:8: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/data_loader/generate_shape_tensors.py:9: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead. 2022-04-07 09:23:29,207 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/data_loader/generate_shape_tensors.py:9: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/data_loader/generate_shape_tensors.py:55: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead. 2022-04-07 09:23:29,212 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/data_loader/generate_shape_tensors.py:55: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. 2022-04-07 09:23:29,253 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. 2022-04-07 09:23:29,255 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. 2022-04-07 09:23:29,278 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. WARNING:tensorflow:From /opt/nvidia/third_party/keras/tensorflow_backend.py:183: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead. 2022-04-07 09:23:29,487 [WARNING] tensorflow: From /opt/nvidia/third_party/keras/tensorflow_backend.py:183: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. 2022-04-07 09:23:29,586 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/data_loader/generate_shape_tensors.py:8: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead. 2022-04-07 09:23:29,615 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/data_loader/generate_shape_tensors.py:8: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/data_loader/generate_shape_tensors.py:8: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead. 2022-04-07 09:23:29,615 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/data_loader/generate_shape_tensors.py:8: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/data_loader/generate_shape_tensors.py:9: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead. 2022-04-07 09:23:29,615 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/data_loader/generate_shape_tensors.py:9: The name tf.get_variable is deprecated. Please use tf.compat.v1.get_variable instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/data_loader/generate_shape_tensors.py:55: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead. 2022-04-07 09:23:29,620 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/data_loader/generate_shape_tensors.py:55: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. 2022-04-07 09:23:29,654 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. 2022-04-07 09:23:29,656 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. 2022-04-07 09:23:29,678 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. WARNING:tensorflow:From /opt/nvidia/third_party/keras/tensorflow_backend.py:183: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead. 2022-04-07 09:23:29,880 [WARNING] tensorflow: From /opt/nvidia/third_party/keras/tensorflow_backend.py:183: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:2018: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead. 2022-04-07 09:23:29,942 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:2018: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:2018: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead. 2022-04-07 09:23:30,326 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:2018: The name tf.image.resize_nearest_neighbor is deprecated. Please use tf.compat.v1.image.resize_nearest_neighbor instead. 2022-04-07 09:23:31,878 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False 2022-04-07 09:23:31,878 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False 2022-04-07 09:23:31,878 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0) 2022-04-07 09:23:31,878 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 64, io threads: 128, compute threads: 64, buffered batches: -1 2022-04-07 09:23:31,878 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 25335, number of sources: 1, batch size per gpu: 20, steps: 1267 WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. 2022-04-07 09:23:31,937 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-04-07 09:23:31,991 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-04-07 09:23:32,014 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates. 2022-04-07 09:23:32,125 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False 2022-04-07 09:23:32,126 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False 2022-04-07 09:23:32,126 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0) 2022-04-07 09:23:32,126 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 64, io threads: 128, compute threads: 64, buffered batches: -1 2022-04-07 09:23:32,126 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 25335, number of sources: 1, batch size per gpu: 20, steps: 1267 WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. 2022-04-07 09:23:32,174 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-04-07 09:23:32,225 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-04-07 09:23:32,249 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates. 2022-04-07 09:23:32,492 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: True - shard 0 of 1 2022-04-07 09:23:32,499 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights: 2022-04-07 09:23:32,499 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000 WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-04-07 09:23:32,517 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-04-07 09:23:32,720 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: True - shard 0 of 1 2022-04-07 09:23:32,727 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights: 2022-04-07 09:23:32,727 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000 WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-04-07 09:23:32,745 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code /opt/nvidia/third_party/keras/tensorflow_backend.py:356: UserWarning: Seed 43 from outer graph might be getting used by function Dataset_map__map_func_set_random_wrapper, if the random op has not been provided any seed. Explicitly set the seed in the function if this is not the intended behavior. self, _map_func_set_random_wrapper, num_parallel_calls=num_parallel_calls /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py:302: UserWarning: tf.data static optimizations are not compatible with tf.Variable. The following optimizations will be disabled: map_and_batch_fusion, noop_elimination, shuffle_and_repeat_fusion. To enable optimizations, use resource variables instead by calling `tf.enable_resource_variables()` at the start of the program. ", ".join(static_optimizations)) WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/dataio/tf_data_pipe.py:131: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead. 2022-04-07 09:23:35,157 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/dataio/tf_data_pipe.py:131: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead. /opt/nvidia/third_party/keras/tensorflow_backend.py:356: UserWarning: Seed 42 from outer graph might be getting used by function Dataset_map__map_func_set_random_wrapper, if the random op has not been provided any seed. Explicitly set the seed in the function if this is not the intended behavior. self, _map_func_set_random_wrapper, num_parallel_calls=num_parallel_calls /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/data/ops/dataset_ops.py:302: UserWarning: tf.data static optimizations are not compatible with tf.Variable. The following optimizations will be disabled: map_and_batch_fusion, noop_elimination, shuffle_and_repeat_fusion. To enable optimizations, use resource variables instead by calling `tf.enable_resource_variables()` at the start of the program. ", ".join(static_optimizations)) WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/dataio/tf_data_pipe.py:131: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead. 2022-04-07 09:23:35,674 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/dataio/tf_data_pipe.py:131: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. 2022-04-07 09:23:40,495 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. 2022-04-07 09:23:40,495 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. 2022-04-07 09:23:40,496 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. 2022-04-07 09:23:41,327 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. 2022-04-07 09:23:41,327 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. 2022-04-07 09:23:41,329 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. 2022-04-07 09:23:41,714 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. 2022-04-07 09:23:42,522 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead. 2022-04-07 09:23:43,288 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3295: The name tf.log is deprecated. Please use tf.math.log instead. 2022-04-07 09:23:43,293 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3295: The name tf.log is deprecated. Please use tf.math.log instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead. 2022-04-07 09:23:43,906 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead. 2022-04-07 09:23:44,012 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3295: The name tf.log is deprecated. Please use tf.math.log instead. 2022-04-07 09:23:44,017 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3295: The name tf.log is deprecated. Please use tf.math.log instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead. 2022-04-07 09:23:44,511 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead. 2022-04-07 09:24:38,880 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False 2022-04-07 09:24:38,880 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False 2022-04-07 09:24:38,881 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0) 2022-04-07 09:24:38,881 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 64, io threads: 128, compute threads: 64, buffered batches: -1 2022-04-07 09:24:38,881 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 6333, number of sources: 1, batch size per gpu: 8, steps: 792 WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-04-07 09:24:38,906 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-04-07 09:24:38,930 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates. 2022-04-07 09:24:39,081 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False 2022-04-07 09:24:39,081 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False 2022-04-07 09:24:39,082 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0) 2022-04-07 09:24:39,082 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 64, io threads: 128, compute threads: 64, buffered batches: -1 2022-04-07 09:24:39,082 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 6333, number of sources: 1, batch size per gpu: 8, steps: 792 WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-04-07 09:24:39,106 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-04-07 09:24:39,127 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates. 2022-04-07 09:24:39,247 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: False - shard 0 of 1 2022-04-07 09:24:39,253 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights: 2022-04-07 09:24:39,253 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000 WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-04-07 09:24:39,270 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-04-07 09:24:39,403 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: False - shard 0 of 1 2022-04-07 09:24:39,408 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights: 2022-04-07 09:24:39,408 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000 WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2022-04-07 09:24:39,423 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code /usr/local/lib/python3.6/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually. warnings.warn('No training configuration found in save file: ' WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/utils/tensor_utils.py:7: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead. 2022-04-07 09:24:51,961 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/utils/tensor_utils.py:7: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/utils/tensor_utils.py:8: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead. 2022-04-07 09:24:51,961 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/utils/tensor_utils.py:8: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/utils/tensor_utils.py:9: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead. 2022-04-07 09:24:51,962 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/utils/tensor_utils.py:9: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead. __________________________________________________________________________________________________ Layer (type) Output Shape Param # Connected to ================================================================================================== Input (InputLayer) (None, 3, None, None 0 __________________________________________________________________________________________________ conv_0 (Conv2D) (None, 32, None, Non 864 Input[0][0] __________________________________________________________________________________________________ conv_0_bn (BatchNormalization) (None, 32, None, Non 128 conv_0[0][0] __________________________________________________________________________________________________ conv_0_mish (LeakyReLU) (None, 32, None, Non 0 conv_0_bn[0][0] __________________________________________________________________________________________________ conv_1 (Conv2D) (None, 64, None, Non 18432 conv_0_mish[0][0] __________________________________________________________________________________________________ conv_1_bn (BatchNormalization) (None, 64, None, Non 256 conv_1[0][0] __________________________________________________________________________________________________ conv_1_mish (LeakyReLU) (None, 64, None, Non 0 conv_1_bn[0][0] __________________________________________________________________________________________________ conv_2_conv_0 (Conv2D) (None, 64, None, Non 36864 conv_1_mish[0][0] __________________________________________________________________________________________________ conv_2_conv_0_bn (BatchNormaliz (None, 64, None, Non 256 conv_2_conv_0[0][0] __________________________________________________________________________________________________ conv_2_conv_0_mish (LeakyReLU) (None, 64, None, Non 0 conv_2_conv_0_bn[0][0] __________________________________________________________________________________________________ conv_2_split_0 (Split) (None, 32, None, Non 0 conv_2_conv_0_mish[0][0] __________________________________________________________________________________________________ conv_2_conv_1 (Conv2D) (None, 32, None, Non 9216 conv_2_split_0[0][0] __________________________________________________________________________________________________ conv_2_conv_1_bn (BatchNormaliz (None, 32, None, Non 128 conv_2_conv_1[0][0] __________________________________________________________________________________________________ conv_2_conv_1_mish (LeakyReLU) (None, 32, None, Non 0 conv_2_conv_1_bn[0][0] __________________________________________________________________________________________________ conv_2_conv_2 (Conv2D) (None, 32, None, Non 9216 conv_2_conv_1_mish[0][0] __________________________________________________________________________________________________ conv_2_conv_2_bn (BatchNormaliz (None, 32, None, Non 128 conv_2_conv_2[0][0] __________________________________________________________________________________________________ conv_2_conv_2_mish (LeakyReLU) (None, 32, None, Non 0 conv_2_conv_2_bn[0][0] __________________________________________________________________________________________________ conv_2_concat_0 (Concatenate) (None, 64, None, Non 0 conv_2_conv_2_mish[0][0] conv_2_conv_1_mish[0][0] __________________________________________________________________________________________________ conv_2_conv_3 (Conv2D) (None, 64, None, Non 4096 conv_2_concat_0[0][0] __________________________________________________________________________________________________ conv_2_conv_3_bn (BatchNormaliz (None, 64, None, Non 256 conv_2_conv_3[0][0] __________________________________________________________________________________________________ conv_2_conv_3_mish (LeakyReLU) (None, 64, None, Non 0 conv_2_conv_3_bn[0][0] __________________________________________________________________________________________________ conv_2_concat_1 (Concatenate) (None, 128, None, No 0 conv_2_conv_0_mish[0][0] conv_2_conv_3_mish[0][0] __________________________________________________________________________________________________ conv_2_pool_0 (MaxPooling2D) (None, 128, None, No 0 conv_2_concat_1[0][0] __________________________________________________________________________________________________ conv_3_conv_0 (Conv2D) (None, 128, None, No 147456 conv_2_pool_0[0][0] __________________________________________________________________________________________________ conv_3_conv_0_bn (BatchNormaliz (None, 128, None, No 512 conv_3_conv_0[0][0] __________________________________________________________________________________________________ conv_3_conv_0_mish (LeakyReLU) (None, 128, None, No 0 conv_3_conv_0_bn[0][0] __________________________________________________________________________________________________ conv_3_split_0 (Split) (None, 64, None, Non 0 conv_3_conv_0_mish[0][0] __________________________________________________________________________________________________ conv_3_conv_1 (Conv2D) (None, 64, None, Non 36864 conv_3_split_0[0][0] __________________________________________________________________________________________________ conv_3_conv_1_bn (BatchNormaliz (None, 64, None, Non 256 conv_3_conv_1[0][0] __________________________________________________________________________________________________ conv_3_conv_1_mish (LeakyReLU) (None, 64, None, Non 0 conv_3_conv_1_bn[0][0] __________________________________________________________________________________________________ conv_3_conv_2 (Conv2D) (None, 64, None, Non 36864 conv_3_conv_1_mish[0][0] __________________________________________________________________________________________________ conv_3_conv_2_bn (BatchNormaliz (None, 64, None, Non 256 conv_3_conv_2[0][0] __________________________________________________________________________________________________ conv_3_conv_2_mish (LeakyReLU) (None, 64, None, Non 0 conv_3_conv_2_bn[0][0] __________________________________________________________________________________________________ conv_3_concat_0 (Concatenate) (None, 128, None, No 0 conv_3_conv_2_mish[0][0] conv_3_conv_1_mish[0][0] __________________________________________________________________________________________________ conv_3_conv_3 (Conv2D) (None, 128, None, No 16384 conv_3_concat_0[0][0] __________________________________________________________________________________________________ conv_3_conv_3_bn (BatchNormaliz (None, 128, None, No 512 conv_3_conv_3[0][0] __________________________________________________________________________________________________ conv_3_conv_3_mish (LeakyReLU) (None, 128, None, No 0 conv_3_conv_3_bn[0][0] __________________________________________________________________________________________________ conv_3_concat_1 (Concatenate) (None, 256, None, No 0 conv_3_conv_0_mish[0][0] conv_3_conv_3_mish[0][0] __________________________________________________________________________________________________ conv_3_pool_0 (MaxPooling2D) (None, 256, None, No 0 conv_3_concat_1[0][0] __________________________________________________________________________________________________ conv_4_conv_0 (Conv2D) (None, 256, None, No 589824 conv_3_pool_0[0][0] __________________________________________________________________________________________________ conv_4_conv_0_bn (BatchNormaliz (None, 256, None, No 1024 conv_4_conv_0[0][0] __________________________________________________________________________________________________ conv_4_conv_0_mish (LeakyReLU) (None, 256, None, No 0 conv_4_conv_0_bn[0][0] __________________________________________________________________________________________________ conv_4_split_0 (Split) (None, 128, None, No 0 conv_4_conv_0_mish[0][0] __________________________________________________________________________________________________ conv_4_conv_1 (Conv2D) (None, 128, None, No 147456 conv_4_split_0[0][0] __________________________________________________________________________________________________ conv_4_conv_1_bn (BatchNormaliz (None, 128, None, No 512 conv_4_conv_1[0][0] __________________________________________________________________________________________________ conv_4_conv_1_mish (LeakyReLU) (None, 128, None, No 0 conv_4_conv_1_bn[0][0] __________________________________________________________________________________________________ conv_4_conv_2 (Conv2D) (None, 128, None, No 147456 conv_4_conv_1_mish[0][0] __________________________________________________________________________________________________ conv_4_conv_2_bn (BatchNormaliz (None, 128, None, No 512 conv_4_conv_2[0][0] __________________________________________________________________________________________________ conv_4_conv_2_mish (LeakyReLU) (None, 128, None, No 0 conv_4_conv_2_bn[0][0] __________________________________________________________________________________________________ conv_4_concat_0 (Concatenate) (None, 256, None, No 0 conv_4_conv_2_mish[0][0] conv_4_conv_1_mish[0][0] __________________________________________________________________________________________________ conv_4_conv_3 (Conv2D) (None, 256, None, No 65536 conv_4_concat_0[0][0] __________________________________________________________________________________________________ conv_4_conv_3_bn (BatchNormaliz (None, 256, None, No 1024 conv_4_conv_3[0][0] __________________________________________________________________________________________________ conv_4_conv_3_mish (LeakyReLU) (None, 256, None, No 0 conv_4_conv_3_bn[0][0] __________________________________________________________________________________________________ conv_4_concat_1 (Concatenate) (None, 512, None, No 0 conv_4_conv_0_mish[0][0] conv_4_conv_3_mish[0][0] __________________________________________________________________________________________________ conv_4_pool_0 (MaxPooling2D) (None, 512, None, No 0 conv_4_concat_1[0][0] __________________________________________________________________________________________________ conv_5 (Conv2D) (None, 512, None, No 2359296 conv_4_pool_0[0][0] __________________________________________________________________________________________________ conv_5_bn (BatchNormalization) (None, 512, None, No 2048 conv_5[0][0] __________________________________________________________________________________________________ conv_5_mish (LeakyReLU) (None, 512, None, No 0 conv_5_bn[0][0] __________________________________________________________________________________________________ yolo_conv1_1 (Conv2D) (None, 256, None, No 131072 conv_5_mish[0][0] __________________________________________________________________________________________________ yolo_conv1_1_bn (BatchNormaliza (None, 256, None, No 1024 yolo_conv1_1[0][0] __________________________________________________________________________________________________ yolo_conv1_1_lrelu (LeakyReLU) (None, 256, None, No 0 yolo_conv1_1_bn[0][0] __________________________________________________________________________________________________ yolo_conv2 (Conv2D) (None, 128, None, No 32768 yolo_conv1_1_lrelu[0][0] __________________________________________________________________________________________________ yolo_conv2_bn (BatchNormalizati (None, 128, None, No 512 yolo_conv2[0][0] __________________________________________________________________________________________________ yolo_conv2_lrelu (LeakyReLU) (None, 128, None, No 0 yolo_conv2_bn[0][0] __________________________________________________________________________________________________ upsample0 (UpSampling2D) (None, 128, None, No 0 yolo_conv2_lrelu[0][0] __________________________________________________________________________________________________ concatenate_2 (Concatenate) (None, 384, None, No 0 upsample0[0][0] conv_4_conv_3_mish[0][0] __________________________________________________________________________________________________ yolo_conv1_6 (Conv2D) (None, 512, None, No 1179648 yolo_conv1_1_lrelu[0][0] __________________________________________________________________________________________________ yolo_conv3_6 (Conv2D) (None, 256, None, No 884736 concatenate_2[0][0] __________________________________________________________________________________________________ yolo_conv1_6_bn (BatchNormaliza (None, 512, None, No 2048 yolo_conv1_6[0][0] __________________________________________________________________________________________________ yolo_conv3_6_bn (BatchNormaliza (None, 256, None, No 1024 yolo_conv3_6[0][0] __________________________________________________________________________________________________ yolo_conv1_6_lrelu (LeakyReLU) (None, 512, None, No 0 yolo_conv1_6_bn[0][0] __________________________________________________________________________________________________ yolo_conv3_6_lrelu (LeakyReLU) (None, 256, None, No 0 yolo_conv3_6_bn[0][0] __________________________________________________________________________________________________ conv_big_object (Conv2D) (None, 18, None, Non 9234 yolo_conv1_6_lrelu[0][0] __________________________________________________________________________________________________ conv_mid_object (Conv2D) (None, 18, None, Non 4626 yolo_conv3_6_lrelu[0][0] __________________________________________________________________________________________________ bg_permute (Permute) (None, None, None, 1 0 conv_big_object[0][0] __________________________________________________________________________________________________ md_permute (Permute) (None, None, None, 1 0 conv_mid_object[0][0] __________________________________________________________________________________________________ bg_reshape (Reshape) (None, None, 6) 0 bg_permute[0][0] __________________________________________________________________________________________________ md_reshape (Reshape) (None, None, 6) 0 md_permute[0][0] __________________________________________________________________________________________________ bg_anchor (YOLOAnchorBox) (None, None, 6) 0 conv_big_object[0][0] __________________________________________________________________________________________________ bg_bbox_processor (BBoxPostProc (None, None, 6) 0 bg_reshape[0][0] __________________________________________________________________________________________________ md_anchor (YOLOAnchorBox) (None, None, 6) 0 conv_mid_object[0][0] __________________________________________________________________________________________________ md_bbox_processor (BBoxPostProc (None, None, 6) 0 md_reshape[0][0] __________________________________________________________________________________________________ encoded_bg (Concatenate) (None, None, 12) 0 bg_anchor[0][0] bg_bbox_processor[0][0] __________________________________________________________________________________________________ encoded_md (Concatenate) (None, None, 12) 0 md_anchor[0][0] md_bbox_processor[0][0] __________________________________________________________________________________________________ encoded_detections (Concatenate (None, None, 12) 0 encoded_bg[0][0] encoded_md[0][0] ================================================================================================== Total params: 5,880,324 Trainable params: 5,874,116 Non-trainable params: 6,208 __________________________________________________________________________________________________ /usr/local/lib/python3.6/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually. warnings.warn('No training configuration found in save file: ' WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/utils/tensor_utils.py:7: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead. 2022-04-07 09:24:52,408 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/utils/tensor_utils.py:7: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/utils/tensor_utils.py:8: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead. 2022-04-07 09:24:52,408 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/utils/tensor_utils.py:8: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/utils/tensor_utils.py:9: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead. 2022-04-07 09:24:52,409 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/utils/tensor_utils.py:9: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead. Epoch 1/80 39b4a0586816:165:428 [0] NCCL INFO Bootstrap : Using lo:127.0.0.1<0> 39b4a0586816:165:428 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation 39b4a0586816:165:428 [0] NCCL INFO NET/IB : No device found. 39b4a0586816:165:428 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0> [1]eth0:172.17.0.4<0> 39b4a0586816:165:428 [0] NCCL INFO Using network Socket NCCL version 2.9.9+cuda11.3 39b4a0586816:166:425 [1] NCCL INFO Bootstrap : Using lo:127.0.0.1<0> 39b4a0586816:166:425 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation 39b4a0586816:166:425 [1] NCCL INFO NET/IB : No device found. 39b4a0586816:166:425 [1] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0> [1]eth0:172.17.0.4<0> 39b4a0586816:166:425 [1] NCCL INFO Using network Socket 39b4a0586816:165:428 [0] NCCL INFO Channel 00/04 : 0 1 39b4a0586816:165:428 [0] NCCL INFO Channel 01/04 : 0 1 39b4a0586816:165:428 [0] NCCL INFO Channel 02/04 : 0 1 39b4a0586816:165:428 [0] NCCL INFO Channel 03/04 : 0 1 39b4a0586816:165:428 [0] NCCL INFO Trees [0] 1/-1/-1->0->-1 [1] 1/-1/-1->0->-1 [2] 1/-1/-1->0->-1 [3] 1/-1/-1->0->-1 39b4a0586816:165:428 [0] NCCL INFO Setting affinity for GPU 0 to ffff,0000ffff 39b4a0586816:166:425 [1] NCCL INFO Trees [0] -1/-1/-1->1->0 [1] -1/-1/-1->1->0 [2] -1/-1/-1->1->0 [3] -1/-1/-1->1->0 39b4a0586816:166:425 [1] NCCL INFO Setting affinity for GPU 1 to ffff,0000ffff 39b4a0586816:166:425 [1] NCCL INFO Channel 00 : 1[3e000] -> 0[1a000] via P2P/IPC 39b4a0586816:166:425 [1] NCCL INFO Channel 01 : 1[3e000] -> 0[1a000] via P2P/IPC 39b4a0586816:166:425 [1] NCCL INFO Channel 02 : 1[3e000] -> 0[1a000] via P2P/IPC 39b4a0586816:166:425 [1] NCCL INFO Channel 03 : 1[3e000] -> 0[1a000] via P2P/IPC 39b4a0586816:165:428 [0] NCCL INFO Channel 00 : 0[1a000] -> 1[3e000] via P2P/IPC 39b4a0586816:165:428 [0] NCCL INFO Channel 01 : 0[1a000] -> 1[3e000] via P2P/IPC 39b4a0586816:165:428 [0] NCCL INFO Channel 02 : 0[1a000] -> 1[3e000] via P2P/IPC 39b4a0586816:165:428 [0] NCCL INFO Channel 03 : 0[1a000] -> 1[3e000] via P2P/IPC 39b4a0586816:166:425 [1] NCCL INFO Connected all rings 39b4a0586816:166:425 [1] NCCL INFO Connected all trees 39b4a0586816:166:425 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 39b4a0586816:166:425 [1] NCCL INFO 4 coll channels, 4 p2p channels, 4 p2p channels per peer 39b4a0586816:166:425 [1] NCCL INFO comm 0x7fed64357cb0 rank 1 nranks 2 cudaDev 1 busId 3e000 - Init COMPLETE 39b4a0586816:165:428 [0] NCCL INFO Connected all rings 39b4a0586816:165:428 [0] NCCL INFO Connected all trees 39b4a0586816:165:428 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512 39b4a0586816:165:428 [0] NCCL INFO 4 coll channels, 4 p2p channels, 4 p2p channels per peer 39b4a0586816:165:428 [0] NCCL INFO comm 0x7f199835fc60 rank 0 nranks 2 cudaDev 0 busId 1a000 - Init COMPLETE 39b4a0586816:165:428 [0] NCCL INFO Launch mode Parallel 1/1584 [..............................] - ETA: 12:57:02 - loss: 5013.8589Traceback (most recent call last): File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py", line 110, in File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/utils.py", line 528, in return_func File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/utils.py", line 516, in return_func File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py", line 106, in main File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py", line 63, in run_experiment File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/models/yolov4_model.py", line 631, in train File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1039, in fit validation_steps=validation_steps) File "/usr/local/lib/python3.6/dist-packages/keras/engine/training_arrays.py", line 154, in fit_loop outs = f(ins) File "/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py", line 2715, in __call__ return self._call(inputs) File "/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py", line 2675, in _call fetched = self._callable_fn(*array_vals) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1472, in __call__ run_metadata_ptr) tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: {{function_node __inference_Dataset_map__map_func_set_random_wrapper_2821}} Expected image (JPEG, PNG, or GIF), got unknown format starting with 'BM6\020\016\000\000\000\000\0006\000\000\000(\000' [[{{node AssetLoader/DecodePng}}]] [[data_loader_out]] (1) Invalid argument: {{function_node __inference_Dataset_map__map_func_set_random_wrapper_2821}} Expected image (JPEG, PNG, or GIF), got unknown format starting with 'BM6\020\016\000\000\000\000\0006\000\000\000(\000' [[{{node AssetLoader/DecodePng}}]] [[data_loader_out]] [[SparseSplit/_3395]] 0 successful operations. 0 derived errors ignored. Traceback (most recent call last): File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py", line 110, in File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/utils.py", line 528, in return_func File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/utils.py", line 516, in return_func File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py", line 106, in main File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/scripts/train.py", line 63, in run_experiment File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v4/models/yolov4_model.py", line 631, in train File "/usr/local/lib/python3.6/dist-packages/keras/engine/training.py", line 1039, in fit validation_steps=validation_steps) File "/usr/local/lib/python3.6/dist-packages/keras/engine/training_arrays.py", line 154, in fit_loop outs = f(ins) File "/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py", line 2715, in __call__ return self._call(inputs) File "/usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py", line 2675, in _call fetched = self._callable_fn(*array_vals) File "/usr/local/lib/python3.6/dist-packages/tensorflow_core/python/client/session.py", line 1472, in __call__ run_metadata_ptr) tensorflow.python.framework.errors_impl.UnknownError: Horovod has been shut down. This was caused by an exception on one of the ranks or an attempt to allreduce, allgather or broadcast a tensor after one of the ranks finished execution. If the shutdown was caused by an exception, you should see the exception in the log before the first shutdown message. [[{{node training_1/Adam/DistributedAdam_Allreduce/cond_58/HorovodAllreduce_training_1_Adam_gradients_conv_big_object_1_BiasAdd_grad_BiasAddGrad_0}}]] -------------------------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code. Per user-direction, the job has been aborted. -------------------------------------------------------------------------- -------------------------------------------------------------------------- mpirun.real detected that one or more processes exited with non-zero status, thus causing the job to be terminated. The first process to do so was: Process name: [[47132,1],1] Exit code: 1 -------------------------------------------------------------------------- 2022-04-07 16:25:46,154 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.