2022-05-17 13:18:42,705 [INFO] root: Registry: ['nvcr.io'] 2022-05-17 13:18:42,769 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3 2022-05-17 13:18:42,875 [WARNING] tlt.components.docker_handler.docker_handler: Docker will run the commands as root. If you would like to retain your local host permissions, please add the "user":"UID:GID" in the DockerOptions portion of the "/home/ataka/.tao_mounts.json" file. You can obtain your users UID and GID by using the "id -u" and "id -g" commands on the terminal. Using TensorFlow backend. WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them. Using TensorFlow backend. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/evaluate.py:43: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead. WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/evaluate.py:43: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead. 2022-05-17 04:18:50,519 [DEBUG] __main__: Starting experiment. 2022-05-17 04:18:50,520 [INFO] __main__: Loading experiment spec at /workspace/tao-experiments/unet/specs/unet_train_resnet_unet_kitti_retrain.txt. 2022-05-17 04:18:50,520 [INFO] iva.unet.spec_handler.spec_loader: Merging specification from /workspace/tao-experiments/unet/specs/unet_train_resnet_unet_kitti_retrain.txt WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. 2022-05-17 04:18:50,527 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. 2022-05-17 04:18:50,541 [INFO] iva.unet.model.utilities: Label Id 0: Train Id 0 2022-05-17 04:18:50,541 [INFO] iva.unet.model.utilities: Label Id 1: Train Id 1 2022-05-17 04:18:50,541 [INFO] iva.unet.model.utilities: Label Id 2: Train Id 2 2022-05-17 04:18:50,541 [INFO] iva.unet.model.utilities: Label Id 3: Train Id 3 2022-05-17 04:18:50,541 [INFO] iva.unet.model.utilities: Label Id 4: Train Id 4 2022-05-17 04:18:50,541 [INFO] iva.unet.model.utilities: Label Id 5: Train Id 5 2022-05-17 04:18:50,541 [INFO] iva.unet.model.utilities: Label Id 6: Train Id 6 2022-05-17 04:18:50,541 [INFO] iva.unet.model.utilities: Label Id 7: Train Id 7 2022-05-17 04:18:50,542 [INFO] iva.unet.model.utilities: Label Id 8: Train Id 8 2022-05-17 04:18:50,542 [INFO] iva.unet.model.utilities: Label Id 9: Train Id 9 2022-05-17 04:18:50,542 [INFO] iva.unet.model.utilities: Label Id 10: Train Id 10 2022-05-17 04:18:50,542 [INFO] iva.unet.model.utilities: Label Id 11: Train Id 11 2022-05-17 04:18:50,542 [INFO] iva.unet.model.utilities: Label Id 12: Train Id 12 2022-05-17 04:18:50,542 [INFO] iva.unet.model.utilities: Label Id 13: Train Id 13 2022-05-17 04:18:50,542 [INFO] iva.unet.model.utilities: Label Id 14: Train Id 14 2022-05-17 04:18:50,542 [INFO] iva.unet.model.utilities: Label Id 15: Train Id 15 2022-05-17 04:18:50,542 [INFO] iva.unet.model.utilities: Label Id 16: Train Id 16 2022-05-17 04:18:50,542 [INFO] iva.unet.model.utilities: Label Id 17: Train Id 17 2022-05-17 04:18:50,542 [INFO] iva.unet.model.utilities: Label Id 18: Train Id 18 2022-05-17 04:18:50,542 [INFO] iva.unet.model.utilities: Label Id 19: Train Id 19 2022-05-17 04:18:50,542 [INFO] iva.unet.model.utilities: Label Id 20: Train Id 20 2022-05-17 04:18:50,542 [INFO] iva.unet.model.utilities: Label Id 21: Train Id 21 2022-05-17 04:18:50,542 [INFO] iva.unet.model.utilities: Label Id 22: Train Id 22 2022-05-17 04:18:50,542 [INFO] iva.unet.model.utilities: Label Id 23: Train Id 23 2022-05-17 04:18:50,542 [INFO] iva.unet.model.utilities: Label Id 24: Train Id 24 2022-05-17 04:18:50,542 [INFO] iva.unet.model.utilities: Label Id 25: Train Id 25 2022-05-17 04:18:50,542 [INFO] iva.unet.model.utilities: Label Id 26: Train Id 26 2022-05-17 04:18:50,542 [INFO] iva.unet.model.utilities: Label Id 27: Train Id 27 2022-05-17 04:18:50,542 [INFO] iva.unet.model.utilities: Label Id 28: Train Id 28 2022-05-17 04:18:50,542 [INFO] iva.unet.model.utilities: Label Id 29: Train Id 29 2022-05-17 04:18:50,542 [INFO] iva.unet.model.utilities: Label Id 30: Train Id 30 2022-05-17 04:18:50,543 [INFO] iva.unet.model.utilities: Label Id 31: Train Id 31 2022-05-17 04:18:50,543 [INFO] iva.unet.model.utilities: Label Id 32: Train Id 32 2022-05-17 04:18:50,543 [INFO] iva.unet.model.utilities: Label Id 33: Train Id 33 2022-05-17 04:18:50,551 [DEBUG] PIL.PngImagePlugin: STREAM b'IHDR' 16 13 2022-05-17 04:18:50,551 [DEBUG] PIL.PngImagePlugin: STREAM b'IDAT' 41 8192 Phase val: Total 40 files. 0%| | 0/14 [00:00 File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/evaluate.py", line 392, in main File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/evaluate.py", line 300, in run_experiment File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/evaluate.py", line 262, in evaluate_unet File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/evaluate.py", line 218, in run_evaluate_trt File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/evaluate.py", line 129, in print_compute_metrics File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/evaluate.py", line 74, in compute_metrics_masks ValueError: operands could not be broadcast together with shapes (34,34) (12,12) (34,34) 2022-05-17 13:19:03,670 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.