Using TensorFlow backend. WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them. Using TensorFlow backend. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/hooks/checkpoint_saver_hook.py:21: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/hooks/pretrained_restore_hook.py:23: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/hooks/pretrained_restore_hook.py:23: The name tf.logging.WARN is deprecated. Please use tf.compat.v1.logging.WARN instead. WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/train.py:405: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. Loading experiment spec at /new/media/hdd/datasets/mapillary/vistas/experiment_config.cfg. 2021-06-15 08:08:49,298 [INFO] __main__: Loading experiment spec at /new/media/hdd/datasets/mapillary/vistas/experiment_config.cfg. 2021-06-15 08:08:49,300 [INFO] iva.unet.spec_handler.spec_loader: Merging specification from /new/media/hdd/datasets/mapillary/vistas/experiment_config.cfg 2021-06-15 08:08:49,314 [INFO] iva.unet.model.utilities: Label Id 0: Train Id 0 2021-06-15 08:08:49,314 [INFO] iva.unet.model.utilities: Label Id 1: Train Id 1 2021-06-15 08:08:49,314 [INFO] iva.unet.model.utilities: Label Id 2: Train Id 2 2021-06-15 08:08:49,314 [INFO] iva.unet.model.utilities: Label Id 3: Train Id 3 2021-06-15 08:08:49,314 [INFO] iva.unet.model.utilities: Label Id 4: Train Id 4 2021-06-15 08:08:49,314 [INFO] iva.unet.model.utilities: Label Id 5: Train Id 5 2021-06-15 08:08:49,314 [INFO] iva.unet.model.utilities: Label Id 6: Train Id 6 2021-06-15 08:08:49,314 [INFO] iva.unet.model.utilities: Label Id 7: Train Id 7 2021-06-15 08:08:49,314 [INFO] iva.unet.model.utilities: Label Id 8: Train Id 8 2021-06-15 08:08:49,314 [INFO] iva.unet.model.utilities: Label Id 9: Train Id 9 2021-06-15 08:08:49,314 [INFO] iva.unet.model.utilities: Label Id 10: Train Id 10 2021-06-15 08:08:49,314 [INFO] iva.unet.model.utilities: Label Id 11: Train Id 11 2021-06-15 08:08:49,314 [INFO] iva.unet.model.utilities: Label Id 12: Train Id 12 2021-06-15 08:08:49,314 [INFO] iva.unet.model.utilities: Label Id 13: Train Id 13 2021-06-15 08:08:49,314 [INFO] iva.unet.model.utilities: Label Id 14: Train Id 14 2021-06-15 08:08:49,314 [INFO] iva.unet.model.utilities: Label Id 15: Train Id 15 2021-06-15 08:08:49,314 [INFO] iva.unet.model.utilities: Label Id 16: Train Id 16 2021-06-15 08:08:49,314 [INFO] iva.unet.model.utilities: Label Id 17: Train Id 17 2021-06-15 08:08:49,314 [INFO] iva.unet.model.utilities: Label Id 18: Train Id 18 2021-06-15 08:08:49,314 [INFO] iva.unet.model.utilities: Label Id 19: Train Id 19 2021-06-15 08:08:49,314 [INFO] iva.unet.model.utilities: Label Id 20: Train Id 20 2021-06-15 08:08:49,314 [INFO] iva.unet.model.utilities: Label Id 21: Train Id 21 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 22: Train Id 22 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 23: Train Id 23 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 24: Train Id 24 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 25: Train Id 25 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 26: Train Id 26 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 27: Train Id 27 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 28: Train Id 28 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 29: Train Id 29 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 30: Train Id 30 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 31: Train Id 31 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 32: Train Id 32 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 33: Train Id 33 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 34: Train Id 34 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 35: Train Id 35 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 36: Train Id 36 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 37: Train Id 37 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 38: Train Id 38 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 39: Train Id 39 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 40: Train Id 40 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 41: Train Id 41 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 42: Train Id 42 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 43: Train Id 43 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 44: Train Id 44 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 45: Train Id 45 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 46: Train Id 46 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 47: Train Id 47 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 48: Train Id 48 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 49: Train Id 49 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 50: Train Id 50 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 51: Train Id 51 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 52: Train Id 52 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 53: Train Id 53 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 54: Train Id 54 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 55: Train Id 55 2021-06-15 08:08:49,315 [INFO] iva.unet.model.utilities: Label Id 56: Train Id 56 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 57: Train Id 57 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 58: Train Id 58 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 59: Train Id 59 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 60: Train Id 60 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 61: Train Id 61 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 62: Train Id 62 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 63: Train Id 63 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 64: Train Id 64 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 65: Train Id 65 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 66: Train Id 66 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 67: Train Id 67 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 68: Train Id 68 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 69: Train Id 69 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 70: Train Id 70 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 71: Train Id 71 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 72: Train Id 72 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 73: Train Id 73 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 74: Train Id 74 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 75: Train Id 75 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 76: Train Id 76 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 77: Train Id 77 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 78: Train Id 78 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 79: Train Id 79 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 80: Train Id 80 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 81: Train Id 81 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 82: Train Id 82 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 83: Train Id 83 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 84: Train Id 84 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 85: Train Id 85 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 86: Train Id 86 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 87: Train Id 87 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 88: Train Id 88 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 89: Train Id 89 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 90: Train Id 90 2021-06-15 08:08:49,316 [INFO] iva.unet.model.utilities: Label Id 91: Train Id 91 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 92: Train Id 92 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 93: Train Id 93 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 94: Train Id 94 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 95: Train Id 95 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 96: Train Id 96 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 97: Train Id 97 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 98: Train Id 98 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 99: Train Id 99 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 100: Train Id 100 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 101: Train Id 101 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 102: Train Id 102 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 103: Train Id 103 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 104: Train Id 104 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 105: Train Id 105 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 106: Train Id 106 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 107: Train Id 107 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 108: Train Id 108 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 109: Train Id 109 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 110: Train Id 110 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 111: Train Id 111 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 112: Train Id 112 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 113: Train Id 113 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 114: Train Id 114 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 115: Train Id 115 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 116: Train Id 116 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 117: Train Id 117 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 118: Train Id 118 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 119: Train Id 119 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 120: Train Id 120 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 121: Train Id 121 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 122: Train Id 122 2021-06-15 08:08:49,317 [INFO] iva.unet.model.utilities: Label Id 123: Train Id 123 2021-06-15 08:08:49,319 [INFO] iva.unet.hooks.latest_checkpoint: Getting the latest checkpoint for restoring /new/media/hdd/datasets/mapillary/vistas/segmentation_training_2gpu/model.step-0.tlt INFO:tensorflow:Using config: {'_model_dir': '/new/media/hdd/datasets/mapillary/vistas/segmentation_training_2gpu', '_tf_random_seed': None, '_save_summary_steps': 5, '_save_checkpoints_steps': None, '_save_checkpoints_secs': None, '_session_config': intra_op_parallelism_threads: 1 inter_op_parallelism_threads: 38 gpu_options { allow_growth: true visible_device_list: "0" force_gpu_compatible: true } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': None, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} 2021-06-15 08:08:50,299 [INFO] tensorflow: Using config: {'_model_dir': '/new/media/hdd/datasets/mapillary/vistas/segmentation_training_2gpu', '_tf_random_seed': None, '_save_summary_steps': 5, '_save_checkpoints_steps': None, '_save_checkpoints_secs': None, '_session_config': intra_op_parallelism_threads: 1 inter_op_parallelism_threads: 38 gpu_options { allow_growth: true visible_device_list: "0" force_gpu_compatible: true } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': None, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} Phase train: Total 18000 files. 2021-06-15 08:08:50,387 [INFO] iva.unet.model.utilities: The total number of training samples 18000 and the batch size per GPU 7 2021-06-15 08:08:50,387 [INFO] iva.unet.model.utilities: Cannot iterate over exactly 18000 samples with a batch size of 7; each epoch will therefore take one extra step. 2021-06-15 08:08:50,387 [INFO] iva.unet.model.utilities: Steps per epoch taken: 2572 Running for 30 Epochs 2021-06-15 08:08:50,387 [INFO] __main__: Running for 30 Epochs INFO:tensorflow:Create CheckpointSaverHook. 2021-06-15 08:08:50,388 [INFO] tensorflow: Create CheckpointSaverHook. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. 2021-06-15 08:08:51,223 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-06-15 08:08:51,266 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity . at 0x7f4815e5fae8> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7f4815e5fae8>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-06-15 08:08:51,278 [WARNING] tensorflow: Entity . at 0x7f4815e5fae8> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7f4815e5fae8>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow: The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see: * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md * https://github.com/tensorflow/addons * https://github.com/tensorflow/io (for I/O related ops) If you depend on functionality not listed there, please file an issue. 2021-06-15 08:08:51,280 [WARNING] tensorflow: The TensorFlow contrib module will not be included in TensorFlow 2.0. For more information, please see: * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md * https://github.com/tensorflow/addons * https://github.com/tensorflow/io (for I/O related ops) If you depend on functionality not listed there, please file an issue. /opt/nvidia/third_party/keras/tensorflow_backend.py:356: UserWarning: Creating resources inside a function passed to Dataset.map() is not supported. Create each resource outside the function, and capture it inside the function to use it. self, _map_func_set_random_wrapper, num_parallel_calls=num_parallel_calls WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-06-15 08:08:51,333 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-06-15 08:08:51,339 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-06-15 08:08:51,344 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/data_loader.py:403: The name tf.image.resize_image_with_pad is deprecated. Please use tf.compat.v1.image.resize_image_with_pad instead. 2021-06-15 08:08:51,344 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/data_loader.py:403: The name tf.image.resize_image_with_pad is deprecated. Please use tf.compat.v1.image.resize_image_with_pad instead. WARNING:tensorflow:Entity . at 0x7f46c87e1e18> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7f46c87e1e18>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-06-15 08:08:51,420 [WARNING] tensorflow: Entity . at 0x7f46c87e1e18> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7f46c87e1e18>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity . at 0x7f46c87406a8> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7f46c87406a8>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-06-15 08:08:51,426 [WARNING] tensorflow: Entity . at 0x7f46c87406a8> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7f46c87406a8>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/data_loader.py:262: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. 2021-06-15 08:08:51,426 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/data_loader.py:262: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. WARNING:tensorflow:The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical. 2021-06-15 08:08:51,480 [WARNING] tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical. WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-06-15 08:08:51,490 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:Entity . at 0x7f46c75a2d08> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7f46c75a2d08>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-06-15 08:08:51,496 [WARNING] tensorflow: Entity . at 0x7f46c75a2d08> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of . at 0x7f46c75a2d08>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code INFO:tensorflow:Calling model_fn. 2021-06-15 08:08:51,512 [INFO] tensorflow: Calling model_fn. 2021-06-15 08:08:51,512 [INFO] iva.unet.utils.model_fn: {'exec_mode': 'train', 'model_dir': '/new/media/hdd/datasets/mapillary/vistas/segmentation_training_2gpu', 'log_dir': None, 'batch_size': 7, 'learning_rate': 0.0005000000237487257, 'crossvalidation_idx': None, 'max_steps': None, 'regularizer_type': 2, 'weight_decay': 3.000000026176508e-09, 'log_summary_steps': 10, 'warmup_steps': 0, 'augment': True, 'use_amp': False, 'use_trt': False, 'use_xla': False, 'loss': 'cross_entropy', 'epochs': 30, 'pretrained_weights_file': None, 'unet_model': , 'key': 'nvidia_tlt', 'experiment_spec': dataset_config { augment: true dataset: "custom" input_image_type: "color" train_images_path: "/new/media/hdd/datasets/mapillary/vistas/images/train" train_masks_path: "/new/media/hdd/datasets/mapillary/vistas/masks/train" val_images_path: "/new/media/hdd/datasets/mapillary/vistas/images/val" val_masks_path: "/new/media/hdd/datasets/mapillary/vistas/masks/val" test_images_path: "/new/media/hdd/datasets/mapillary/vistas/images/val" data_class_config { target_classes { name: "animal--bird" mapping_class: "animal--bird" } target_classes { name: "animal--ground-animal" label_id: 1 mapping_class: "animal--ground-animal" } target_classes { name: "construction--barrier--ambiguous" label_id: 2 mapping_class: "construction--barrier--ambiguous" } target_classes { name: "construction--barrier--concrete-block" label_id: 3 mapping_class: "construction--barrier--concrete-block" } target_classes { name: "construction--barrier--curb" label_id: 4 mapping_class: "construction--barrier--curb" } target_classes { name: "construction--barrier--fence" label_id: 5 mapping_class: "construction--barrier--fence" } target_classes { name: "construction--barrier--guard-rail" label_id: 6 mapping_class: "construction--barrier--guard-rail" } target_classes { name: "construction--barrier--other-barrier" label_id: 7 mapping_class: "construction--barrier--other-barrier" } target_classes { name: "construction--barrier--road-median" label_id: 8 mapping_class: "construction--barrier--road-median" } target_classes { name: "construction--barrier--road-side" label_id: 9 mapping_class: "construction--barrier--road-side" } target_classes { name: "construction--barrier--separator" label_id: 10 mapping_class: "construction--barrier--separator" } target_classes { name: "construction--barrier--temporary" label_id: 11 mapping_class: "construction--barrier--temporary" } target_classes { name: "construction--barrier--wall" label_id: 12 mapping_class: "construction--barrier--wall" } target_classes { name: "construction--flat--bike-lane" label_id: 13 mapping_class: "construction--flat--bike-lane" } target_classes { name: "construction--flat--crosswalk-plain" label_id: 14 mapping_class: "construction--flat--crosswalk-plain" } target_classes { name: "construction--flat--curb-cut" label_id: 15 mapping_class: "construction--flat--curb-cut" } target_classes { name: "construction--flat--driveway" label_id: 16 mapping_class: "construction--flat--driveway" } target_classes { name: "construction--flat--parking" label_id: 17 mapping_class: "construction--flat--parking" } target_classes { name: "construction--flat--parking-aisle" label_id: 18 mapping_class: "construction--flat--parking-aisle" } target_classes { name: "construction--flat--pedestrian-area" label_id: 19 mapping_class: "construction--flat--pedestrian-area" } target_classes { name: "construction--flat--rail-track" label_id: 20 mapping_class: "construction--flat--rail-track" } target_classes { name: "construction--flat--road" label_id: 21 mapping_class: "construction--flat--road" } target_classes { name: "construction--flat--road-shoulder" label_id: 22 mapping_class: "construction--flat--road-shoulder" } target_classes { name: "construction--flat--service-lane" label_id: 23 mapping_class: "construction--flat--service-lane" } target_classes { name: "construction--flat--sidewalk" label_id: 24 mapping_class: "construction--flat--sidewalk" } target_classes { name: "construction--flat--traffic-island" label_id: 25 mapping_class: "construction--flat--traffic-island" } target_classes { name: "construction--structure--bridge" label_id: 26 mapping_class: "construction--structure--bridge" } target_classes { name: "construction--structure--building" label_id: 27 mapping_class: "construction--structure--building" } target_classes { name: "construction--structure--garage" label_id: 28 mapping_class: "construction--structure--garage" } target_classes { name: "construction--structure--tunnel" label_id: 29 mapping_class: "construction--structure--tunnel" } target_classes { name: "human--person--individual" label_id: 30 mapping_class: "human--person--individual" } target_classes { name: "human--person--person-group" label_id: 31 mapping_class: "human--person--person-group" } target_classes { name: "human--rider--bicyclist" label_id: 32 mapping_class: "human--rider--bicyclist" } target_classes { name: "human--rider--motorcyclist" label_id: 33 mapping_class: "human--rider--motorcyclist" } target_classes { name: "human--rider--other-rider" label_id: 34 mapping_class: "human--rider--other-rider" } target_classes { name: "marking--continuous--dashed" label_id: 35 mapping_class: "marking--continuous--dashed" } target_classes { name: "marking--continuous--solid" label_id: 36 mapping_class: "marking--continuous--solid" } target_classes { name: "marking--continuous--zigzag" label_id: 37 mapping_class: "marking--continuous--zigzag" } target_classes { name: "marking--discrete--ambiguous" label_id: 38 mapping_class: "marking--discrete--ambiguous" } target_classes { name: "marking--discrete--arrow--left" label_id: 39 mapping_class: "marking--discrete--arrow--left" } target_classes { name: "marking--discrete--arrow--other" label_id: 40 mapping_class: "marking--discrete--arrow--other" } target_classes { name: "marking--discrete--arrow--right" label_id: 41 mapping_class: "marking--discrete--arrow--right" } target_classes { name: "marking--discrete--arrow--split-left-or-straight" label_id: 42 mapping_class: "marking--discrete--arrow--split-left-or-straight" } target_classes { name: "marking--discrete--arrow--split-right-or-straight" label_id: 43 mapping_class: "marking--discrete--arrow--split-right-or-straight" } target_classes { name: "marking--discrete--arrow--straight" label_id: 44 mapping_class: "marking--discrete--arrow--straight" } target_classes { name: "marking--discrete--crosswalk-zebra" label_id: 45 mapping_class: "marking--discrete--crosswalk-zebra" } target_classes { name: "marking--discrete--give-way-row" label_id: 46 mapping_class: "marking--discrete--give-way-row" } target_classes { name: "marking--discrete--give-way-single" label_id: 47 mapping_class: "marking--discrete--give-way-single" } target_classes { name: "marking--discrete--hatched--chevron" label_id: 48 mapping_class: "marking--discrete--hatched--chevron" } target_classes { name: "marking--discrete--hatched--diagonal" label_id: 49 mapping_class: "marking--discrete--hatched--diagonal" } target_classes { name: "marking--discrete--other-marking" label_id: 50 mapping_class: "marking--discrete--other-marking" } target_classes { name: "marking--discrete--stop-line" label_id: 51 mapping_class: "marking--discrete--stop-line" } target_classes { name: "marking--discrete--symbol--bicycle" label_id: 52 mapping_class: "marking--discrete--symbol--bicycle" } target_classes { name: "marking--discrete--symbol--other" label_id: 53 mapping_class: "marking--discrete--symbol--other" } target_classes { name: "marking--discrete--text" label_id: 54 mapping_class: "marking--discrete--text" } target_classes { name: "marking-only--continuous--dashed" label_id: 55 mapping_class: "marking-only--continuous--dashed" } target_classes { name: "marking-only--discrete--crosswalk-zebra" label_id: 56 mapping_class: "marking-only--discrete--crosswalk-zebra" } target_classes { name: "marking-only--discrete--other-marking" label_id: 57 mapping_class: "marking-only--discrete--other-marking" } target_classes { name: "marking-only--discrete--text" label_id: 58 mapping_class: "marking-only--discrete--text" } target_classes { name: "nature--mountain" label_id: 59 mapping_class: "nature--mountain" } target_classes { name: "nature--sand" label_id: 60 mapping_class: "nature--sand" } target_classes { name: "nature--sky" label_id: 61 mapping_class: "nature--sky" } target_classes { name: "nature--snow" label_id: 62 mapping_class: "nature--snow" } target_classes { name: "nature--terrain" label_id: 63 mapping_class: "nature--terrain" } target_classes { name: "nature--vegetation" label_id: 64 mapping_class: "nature--vegetation" } target_classes { name: "nature--water" label_id: 65 mapping_class: "nature--water" } target_classes { name: "object--banner" label_id: 66 mapping_class: "object--banner" } target_classes { name: "object--bench" label_id: 67 mapping_class: "object--bench" } target_classes { name: "object--bike-rack" label_id: 68 mapping_class: "object--bike-rack" } target_classes { name: "object--catch-basin" label_id: 69 mapping_class: "object--catch-basin" } target_classes { name: "object--cctv-camera" label_id: 70 mapping_class: "object--cctv-camera" } target_classes { name: "object--fire-hydrant" label_id: 71 mapping_class: "object--fire-hydrant" } target_classes { name: "object--junction-box" label_id: 72 mapping_class: "object--junction-box" } target_classes { name: "object--mailbox" label_id: 73 mapping_class: "object--mailbox" } target_classes { name: "object--manhole" label_id: 74 mapping_class: "object--manhole" } target_classes { name: "object--parking-meter" label_id: 75 mapping_class: "object--parking-meter" } target_classes { name: "object--phone-booth" label_id: 76 mapping_class: "object--phone-booth" } target_classes { name: "object--pothole" label_id: 77 mapping_class: "object--pothole" } target_classes { name: "object--sign--advertisement" label_id: 78 mapping_class: "object--sign--advertisement" } target_classes { name: "object--sign--ambiguous" label_id: 79 mapping_class: "object--sign--ambiguous" } target_classes { name: "object--sign--back" label_id: 80 mapping_class: "object--sign--back" } target_classes { name: "object--sign--information" label_id: 81 mapping_class: "object--sign--information" } target_classes { name: "object--sign--other" label_id: 82 mapping_class: "object--sign--other" } target_classes { name: "object--sign--store" label_id: 83 mapping_class: "object--sign--store" } target_classes { name: "object--street-light" label_id: 84 mapping_class: "object--street-light" } target_classes { name: "object--support--pole" label_id: 85 mapping_class: "object--support--pole" } target_classes { name: "object--support--pole-group" label_id: 86 mapping_class: "object--support--pole-group" } target_classes { name: "object--support--traffic-sign-frame" label_id: 87 mapping_class: "object--support--traffic-sign-frame" } target_classes { name: "object--support--utility-pole" label_id: 88 mapping_class: "object--support--utility-pole" } target_classes { name: "object--traffic-cone" label_id: 89 mapping_class: "object--traffic-cone" } target_classes { name: "object--traffic-light--general-single" label_id: 90 mapping_class: "object--traffic-light--general-single" } target_classes { name: "object--traffic-light--pedestrians" label_id: 91 mapping_class: "object--traffic-light--pedestrians" } target_classes { name: "object--traffic-light--general-upright" label_id: 92 mapping_class: "object--traffic-light--general-upright" } target_classes { name: "object--traffic-light--general-horizontal" label_id: 93 mapping_class: "object--traffic-light--general-horizontal" } target_classes { name: "object--traffic-light--cyclists" label_id: 94 mapping_class: "object--traffic-light--cyclists" } target_classes { name: "object--traffic-light--other" label_id: 95 mapping_class: "object--traffic-light--other" } target_classes { name: "object--traffic-sign--ambiguous" label_id: 96 mapping_class: "object--traffic-sign--ambiguous" } target_classes { name: "object--traffic-sign--back" label_id: 97 mapping_class: "object--traffic-sign--back" } target_classes { name: "object--traffic-sign--direction-back" label_id: 98 mapping_class: "object--traffic-sign--direction-back" } target_classes { name: "object--traffic-sign--direction-front" label_id: 99 mapping_class: "object--traffic-sign--direction-front" } target_classes { name: "object--traffic-sign--front" label_id: 100 mapping_class: "object--traffic-sign--front" } target_classes { name: "object--traffic-sign--information-parking" label_id: 101 mapping_class: "object--traffic-sign--information-parking" } target_classes { name: "object--traffic-sign--temporary-back" label_id: 102 mapping_class: "object--traffic-sign--temporary-back" } target_classes { name: "object--traffic-sign--temporary-front" label_id: 103 mapping_class: "object--traffic-sign--temporary-front" } target_classes { name: "object--trash-can" label_id: 104 mapping_class: "object--trash-can" } target_classes { name: "object--vehicle--bicycle" label_id: 105 mapping_class: "object--vehicle--bicycle" } target_classes { name: "object--vehicle--boat" label_id: 106 mapping_class: "object--vehicle--boat" } target_classes { name: "object--vehicle--bus" label_id: 107 mapping_class: "object--vehicle--bus" } target_classes { name: "object--vehicle--car" label_id: 108 mapping_class: "object--vehicle--car" } target_classes { name: "object--vehicle--caravan" label_id: 109 mapping_class: "object--vehicle--caravan" } target_classes { name: "object--vehicle--motorcycle" label_id: 110 mapping_class: "object--vehicle--motorcycle" } target_classes { name: "object--vehicle--on-rails" label_id: 111 mapping_class: "object--vehicle--on-rails" } target_classes { name: "object--vehicle--other-vehicle" label_id: 112 mapping_class: "object--vehicle--other-vehicle" } target_classes { name: "object--vehicle--trailer" label_id: 113 mapping_class: "object--vehicle--trailer" } target_classes { name: "object--vehicle--truck" label_id: 114 mapping_class: "object--vehicle--truck" } target_classes { name: "object--vehicle--vehicle-group" label_id: 115 mapping_class: "object--vehicle--vehicle-group" } target_classes { name: "object--vehicle--wheeled-slow" label_id: 116 mapping_class: "object--vehicle--wheeled-slow" } target_classes { name: "object--water-valve" label_id: 117 mapping_class: "object--water-valve" } target_classes { name: "void--car-mount" label_id: 118 mapping_class: "void--car-mount" } target_classes { name: "void--dynamic" label_id: 119 mapping_class: "void--dynamic" } target_classes { name: "void--ego-vehicle" label_id: 120 mapping_class: "void--ego-vehicle" } target_classes { name: "void--ground" label_id: 121 mapping_class: "void--ground" } target_classes { name: "void--static" label_id: 122 mapping_class: "void--static" } target_classes { name: "void--unlabeled" label_id: 123 mapping_class: "void--unlabeled" } } } model_config { num_layers: 10 use_batch_norm: true training_precision { backend_floatx: FLOAT32 } freeze_bn: true arch: "resnet" all_projections: true model_input_height: 640 model_input_width: 640 model_input_channels: 3 } training_config { batch_size: 7 regularizer { type: L2 weight: 3.000000026176508e-09 } optimizer { adam { epsilon: 9.99999993922529e-09 beta1: 0.8999999761581421 beta2: 0.9990000128746033 } } checkpoint_interval: 1 log_summary_steps: 10 learning_rate: 0.0005000000237487257 loss: "cross_entropy" epochs: 30 } , 'seed': 0, 'benchmark': False, 'temp_dir': '/tmp/tmpldnbc3i2', 'num_classes': 124, 'start_step': 0, 'checkpoint_interval': 1, 'model_json': None, 'load_graph': False, 'phase': None} WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. 2021-06-15 08:08:51,513 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. Traceback (most recent call last): File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/train.py", line 419, in File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/train.py", line 413, in main File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/train.py", line 314, in run_experiment File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/train.py", line 229, in train_unet File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/train.py", line 105, in run_training_loop File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 370, in train loss = self._train_model(input_fn, hooks, saving_listeners) File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1161, in _train_model return self._train_model_default(input_fn, hooks, saving_listeners) File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1191, in _train_model_default features, labels, ModeKeys.TRAIN, self.config) File "/usr/local/lib/python3.6/dist-packages/tensorflow_estimator/python/estimator/estimator.py", line 1149, in _call_model_fn model_fn_results = self._model_fn(features=features, **kwargs) File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/model_fn.py", line 121, in unet_fn File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/model/unet_model.py", line 104, in construct_model File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/model/resnet_unet.py", line 52, in construct_decoder_model File "/usr/local/lib/python3.6/dist-packages/keras/engine/network.py", line 358, in get_layer raise ValueError('No such layer: ' + name) ValueError: No such layer: block_1b_relu