Using TensorFlow backend. WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them. Using TensorFlow backend. WARNING:tensorflow:From /home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/hooks/checkpoint_saver_hook.py:21: The name tf.train.CheckpointSaverHook is deprecated. Please use tf.estimator.CheckpointSaverHook instead. WARNING:tensorflow:From /home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/hooks/pretrained_restore_hook.py:23: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead. WARNING:tensorflow:From /home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/hooks/pretrained_restore_hook.py:23: The name tf.logging.WARN is deprecated. Please use tf.compat.v1.logging.WARN instead. WARNING:tensorflow:From /home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/scripts/train.py:389: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead. Loading experiment spec at /new/media/hdd/datasets/mapillary/vistas/experiment_config.cfg. 2021-06-13 10:49:52,100 [INFO] __main__: Loading experiment spec at /new/media/hdd/datasets/mapillary/vistas/experiment_config.cfg. 2021-06-13 10:49:52,133 [INFO] iva.unet.spec_handler.spec_loader: Merging specification from /new/media/hdd/datasets/mapillary/vistas/experiment_config.cfg 2021-06-13 10:49:52,234 [INFO] root: Initializing the pre-trained weights from /new/media/hdd/datasets/mapillary/vistas/tlt_semantic_segmentation_vresnet10/resnet_10.hdf5 WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. 2021-06-13 10:49:52,300 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. 2021-06-13 10:49:52,332 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. 2021-06-13 10:49:52,362 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead. 2021-06-13 10:49:52,366 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:133: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead. WARNING:tensorflow:From /opt/nvidia/third_party/keras/tensorflow_backend.py:187: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead. 2021-06-13 10:49:52,664 [WARNING] tensorflow: From /opt/nvidia/third_party/keras/tensorflow_backend.py:187: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. 2021-06-13 10:49:52,956 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. 2021-06-13 10:49:52,956 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. 2021-06-13 10:49:53,034 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead. 2021-06-13 10:49:53,241 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead. 2021-06-13 10:49:53,258 [INFO] iva.unet.hooks.latest_checkpoint: Getting the latest checkpoint for restoring /new/media/hdd/datasets/mapillary/vistas/segmentation_training_2gpu/model.step-0.tlt INFO:tensorflow:Using config: {'_model_dir': '/new/media/hdd/datasets/mapillary/vistas/segmentation_training_2gpu', '_tf_random_seed': None, '_save_summary_steps': 5, '_save_checkpoints_steps': None, '_save_checkpoints_secs': None, '_session_config': intra_op_parallelism_threads: 1 inter_op_parallelism_threads: 38 gpu_options { allow_growth: true visible_device_list: "0" force_gpu_compatible: true } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': None, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} 2021-06-13 10:49:54,257 [INFO] tensorflow: Using config: {'_model_dir': '/new/media/hdd/datasets/mapillary/vistas/segmentation_training_2gpu', '_tf_random_seed': None, '_save_summary_steps': 5, '_save_checkpoints_steps': None, '_save_checkpoints_secs': None, '_session_config': intra_op_parallelism_threads: 1 inter_op_parallelism_threads: 38 gpu_options { allow_growth: true visible_device_list: "0" force_gpu_compatible: true } , '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': None, '_log_step_count_steps': None, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': , '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1} 2021-06-13 10:49:54,803 [INFO] iva.unet.model.utilities: The total number of training samples 18000 and the batch size per GPU 7 2021-06-13 10:49:54,803 [INFO] iva.unet.model.utilities: Cannot iterate over exactly 18000 samples with a batch size of 7; each epoch will therefore take one extra step. 2021-06-13 10:49:54,803 [INFO] iva.unet.model.utilities: Steps per epoch taken: 2572 Running for 30 Epochs 2021-06-13 10:49:54,803 [INFO] __main__: Running for 30 Epochs INFO:tensorflow:Create CheckpointSaverHook. 2021-06-13 10:49:54,803 [INFO] tensorflow: Create CheckpointSaverHook. WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. 2021-06-13 10:49:55,544 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead. WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code 2021-06-13 10:49:55,586 [WARNING] tensorflow: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of >. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code WARNING:tensorflow:From /home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/data_loader.py:266: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead. 2021-06-13 10:49:55,587 [WARNING] tensorflow: From /home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/utils/data_loader.py:266: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead. WARNING:tensorflow:The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical. 2021-06-13 10:49:55,631 [WARNING] tensorflow: The operation `tf.image.convert_image_dtype` will be skipped since the input and output dtypes are identical. INFO:tensorflow:Calling model_fn. 2021-06-13 10:49:55,650 [INFO] tensorflow: Calling model_fn. {'exec_mode': 'train', 'model_dir': '/new/media/hdd/datasets/mapillary/vistas/segmentation_training_2gpu', 'log_dir': None, 'batch_size': 7, 'learning_rate': 0.0005000000237487257, 'crossvalidation_idx': None, 'max_steps': None, 'weight_decay': 3.000000026176508e-09, 'log_summary_steps': 10, 'warmup_steps': 0, 'augment': True, 'use_amp': False, 'use_trt': False, 'use_xla': False, 'loss': 'cross_entropy', 'epochs': 30, 'pretrained_weights_file': None, 'unet_model': , 'key': 'nvidia_tlt', 'experiment_spec': dataset_config { augment: true dataset: "custom" input_image_type: "color" train_images_path: "/new/media/hdd/datasets/mapillary/vistas/images/train" train_masks_path: "/new/media/hdd/datasets/mapillary/vistas/masks/train" val_images_path: "/new/media/hdd/datasets/mapillary/vistas/images/val" val_masks_path: "/new/media/hdd/datasets/mapillary/vistas/masks/val" test_images_path: "/new/media/hdd/datasets/mapillary/vistas/images/val" data_class_config { target_classes { name: "animal--bird" mapping_class: "animal--bird" } target_classes { name: "animal--ground-animal" label_id: 1 mapping_class: "animal--ground-animal" } target_classes { name: "construction--barrier--ambiguous" label_id: 2 mapping_class: "construction--barrier--ambiguous" } target_classes { name: "construction--barrier--concrete-block" label_id: 3 mapping_class: "construction--barrier--concrete-block" } target_classes { name: "construction--barrier--curb" label_id: 4 mapping_class: "construction--barrier--curb" } target_classes { name: "construction--barrier--fence" label_id: 5 mapping_class: "construction--barrier--fence" } target_classes { name: "construction--barrier--guard-rail" label_id: 6 mapping_class: "construction--barrier--guard-rail" } target_classes { name: "construction--barrier--other-barrier" label_id: 7 mapping_class: "construction--barrier--other-barrier" } target_classes { name: "construction--barrier--road-median" label_id: 8 mapping_class: "construction--barrier--road-median" } target_classes { name: "construction--barrier--road-side" label_id: 9 mapping_class: "construction--barrier--road-side" } target_classes { name: "construction--barrier--separator" label_id: 10 mapping_class: "construction--barrier--separator" } target_classes { name: "construction--barrier--temporary" label_id: 11 mapping_class: "construction--barrier--temporary" } target_classes { name: "construction--barrier--wall" label_id: 12 mapping_class: "construction--barrier--wall" } target_classes { name: "construction--flat--bike-lane" label_id: 13 mapping_class: "construction--flat--bike-lane" } target_classes { name: "construction--flat--crosswalk-plain" label_id: 14 mapping_class: "construction--flat--crosswalk-plain" } target_classes { name: "construction--flat--curb-cut" label_id: 15 mapping_class: "construction--flat--curb-cut" } target_classes { name: "construction--flat--driveway" label_id: 16 mapping_class: "construction--flat--driveway" } target_classes { name: "construction--flat--parking" label_id: 17 mapping_class: "construction--flat--parking" } target_classes { name: "construction--flat--parking-aisle" label_id: 18 mapping_class: "construction--flat--parking-aisle" } target_classes { name: "construction--flat--pedestrian-area" label_id: 19 mapping_class: "construction--flat--pedestrian-area" } target_classes { name: "construction--flat--rail-track" label_id: 20 mapping_class: "construction--flat--rail-track" } target_classes { name: "construction--flat--road" label_id: 21 mapping_class: "construction--flat--road" } target_classes { name: "construction--flat--road-shoulder" label_id: 22 mapping_class: "construction--flat--road-shoulder" } target_classes { name: "construction--flat--service-lane" label_id: 23 mapping_class: "construction--flat--service-lane" } target_classes { name: "construction--flat--sidewalk" label_id: 24 mapping_class: "construction--flat--sidewalk" } target_classes { name: "construction--flat--traffic-island" label_id: 25 mapping_class: "construction--flat--traffic-island" } target_classes { name: "construction--structure--bridge" label_id: 26 mapping_class: "construction--structure--bridge" } target_classes { name: "construction--structure--building" label_id: 27 mapping_class: "construction--structure--building" } target_classes { name: "construction--structure--garage" label_id: 28 mapping_class: "construction--structure--garage" } target_classes { name: "construction--structure--tunnel" label_id: 29 mapping_class: "construction--structure--tunnel" } target_classes { name: "human--person--individual" label_id: 30 mapping_class: "human--person--individual" } target_classes { name: "human--person--person-group" label_id: 31 mapping_class: "human--person--person-group" } target_classes { name: "human--rider--bicyclist" label_id: 32 mapping_class: "human--rider--bicyclist" } target_classes { name: "human--rider--motorcyclist" label_id: 33 mapping_class: "human--rider--motorcyclist" } target_classes { name: "human--rider--other-rider" label_id: 34 mapping_class: "human--rider--other-rider" } target_classes { name: "marking--continuous--dashed" label_id: 35 mapping_class: "marking--continuous--dashed" } target_classes { name: "marking--continuous--solid" label_id: 36 mapping_class: "marking--continuous--solid" } target_classes { name: "marking--continuous--zigzag" label_id: 37 mapping_class: "marking--continuous--zigzag" } target_classes { name: "marking--discrete--ambiguous" label_id: 38 mapping_class: "marking--discrete--ambiguous" } target_classes { name: "marking--discrete--arrow--left" label_id: 39 mapping_class: "marking--discrete--arrow--left" } target_classes { name: "marking--discrete--arrow--other" label_id: 40 mapping_class: "marking--discrete--arrow--other" } target_classes { name: "marking--discrete--arrow--right" label_id: 41 mapping_class: "marking--discrete--arrow--right" } target_classes { name: "marking--discrete--arrow--split-left-or-straight" label_id: 42 mapping_class: "marking--discrete--arrow--split-left-or-straight" } target_classes { name: "marking--discrete--arrow--split-right-or-straight" label_id: 43 mapping_class: "marking--discrete--arrow--split-right-or-straight" } target_classes { name: "marking--discrete--arrow--straight" label_id: 44 mapping_class: "marking--discrete--arrow--straight" } target_classes { name: "marking--discrete--crosswalk-zebra" label_id: 45 mapping_class: "marking--discrete--crosswalk-zebra" } target_classes { name: "marking--discrete--give-way-row" label_id: 46 mapping_class: "marking--discrete--give-way-row" } target_classes { name: "marking--discrete--give-way-single" label_id: 47 mapping_class: "marking--discrete--give-way-single" } target_classes { name: "marking--discrete--hatched--chevron" label_id: 48 mapping_class: "marking--discrete--hatched--chevron" } target_classes { name: "marking--discrete--hatched--diagonal" label_id: 49 mapping_class: "marking--discrete--hatched--diagonal" } target_classes { name: "marking--discrete--other-marking" label_id: 50 mapping_class: "marking--discrete--other-marking" } target_classes { name: "marking--discrete--stop-line" label_id: 51 mapping_class: "marking--discrete--stop-line" } target_classes { name: "marking--discrete--symbol--bicycle" label_id: 52 mapping_class: "marking--discrete--symbol--bicycle" } target_classes { name: "marking--discrete--symbol--other" label_id: 53 mapping_class: "marking--discrete--symbol--other" } target_classes { name: "marking--discrete--text" label_id: 54 mapping_class: "marking--discrete--text" } target_classes { name: "marking-only--continuous--dashed" label_id: 55 mapping_class: "marking-only--continuous--dashed" } target_classes { name: "marking-only--discrete--crosswalk-zebra" label_id: 56 mapping_class: "marking-only--discrete--crosswalk-zebra" } target_classes { name: "marking-only--discrete--other-marking" label_id: 57 mapping_class: "marking-only--discrete--other-marking" } target_classes { name: "marking-only--discrete--text" label_id: 58 mapping_class: "marking-only--discrete--text" } target_classes { name: "nature--mountain" label_id: 59 mapping_class: "nature--mountain" } target_classes { name: "nature--sand" label_id: 60 mapping_class: "nature--sand" } target_classes { name: "nature--sky" label_id: 61 mapping_class: "nature--sky" } target_classes { name: "nature--snow" label_id: 62 mapping_class: "nature--snow" } target_classes { name: "nature--terrain" label_id: 63 mapping_class: "nature--terrain" } target_classes { name: "nature--vegetation" label_id: 64 mapping_class: "nature--vegetation" } target_classes { name: "nature--water" label_id: 65 mapping_class: "nature--water" } target_classes { name: "object--banner" label_id: 66 mapping_class: "object--banner" } target_classes { name: "object--bench" label_id: 67 mapping_class: "object--bench" } target_classes { name: "object--bike-rack" label_id: 68 mapping_class: "object--bike-rack" } target_classes { name: "object--catch-basin" label_id: 69 mapping_class: "object--catch-basin" } target_classes { name: "object--cctv-camera" label_id: 70 mapping_class: "object--cctv-camera" } target_classes { name: "object--fire-hydrant" label_id: 71 mapping_class: "object--fire-hydrant" } target_classes { name: "object--junction-box" label_id: 72 mapping_class: "object--junction-box" } target_classes { name: "object--mailbox" label_id: 73 mapping_class: "object--mailbox" } target_classes { name: "object--manhole" label_id: 74 mapping_class: "object--manhole" } target_classes { name: "object--parking-meter" label_id: 75 mapping_class: "object--parking-meter" } target_classes { name: "object--phone-booth" label_id: 76 mapping_class: "object--phone-booth" } target_classes { name: "object--pothole" label_id: 77 mapping_class: "object--pothole" } target_classes { name: "object--sign--advertisement" label_id: 78 mapping_class: "object--sign--advertisement" } target_classes { name: "object--sign--ambiguous" label_id: 79 mapping_class: "object--sign--ambiguous" } target_classes { name: "object--sign--back" label_id: 80 mapping_class: "object--sign--back" } target_classes { name: "object--sign--information" label_id: 81 mapping_class: "object--sign--information" } target_classes { name: "object--sign--other" label_id: 82 mapping_class: "object--sign--other" } target_classes { name: "object--sign--store" label_id: 83 mapping_class: "object--sign--store" } target_classes { name: "object--street-light" label_id: 84 mapping_class: "object--street-light" } target_classes { name: "object--support--pole" label_id: 85 mapping_class: "object--support--pole" } target_classes { name: "object--support--pole-group" label_id: 86 mapping_class: "object--support--pole-group" } target_classes { name: "object--support--traffic-sign-frame" label_id: 87 mapping_class: "object--support--traffic-sign-frame" } target_classes { name: "object--support--utility-pole" label_id: 88 mapping_class: "object--support--utility-pole" } target_classes { name: "object--traffic-cone" label_id: 89 mapping_class: "object--traffic-cone" } target_classes { name: "object--traffic-light--general-single" label_id: 90 mapping_class: "object--traffic-light--general-single" } target_classes { name: "object--traffic-light--pedestrians" label_id: 91 mapping_class: "object--traffic-light--pedestrians" } target_classes { name: "object--traffic-light--general-upright" label_id: 92 mapping_class: "object--traffic-light--general-upright" } target_classes { name: "object--traffic-light--general-horizontal" label_id: 93 mapping_class: "object--traffic-light--general-horizontal" } target_classes { name: "object--traffic-light--cyclists" label_id: 94 mapping_class: "object--traffic-light--cyclists" } target_classes { name: "object--traffic-light--other" label_id: 95 mapping_class: "object--traffic-light--other" } target_classes { name: "object--traffic-sign--ambiguous" label_id: 96 mapping_class: "object--traffic-sign--ambiguous" } target_classes { name: "object--traffic-sign--back" label_id: 97 mapping_class: "object--traffic-sign--back" } target_classes { name: "object--traffic-sign--direction-back" label_id: 98 mapping_class: "object--traffic-sign--direction-back" } target_classes { name: "object--traffic-sign--direction-front" label_id: 99 mapping_class: "object--traffic-sign--direction-front" } target_classes { name: "object--traffic-sign--front" label_id: 100 mapping_class: "object--traffic-sign--front" } target_classes { name: "object--traffic-sign--information-parking" label_id: 101 mapping_class: "object--traffic-sign--information-parking" } target_classes { name: "object--traffic-sign--temporary-back" label_id: 102 mapping_class: "object--traffic-sign--temporary-back" } target_classes { name: "object--traffic-sign--temporary-front" label_id: 103 mapping_class: "object--traffic-sign--temporary-front" } target_classes { name: "object--trash-can" label_id: 104 mapping_class: "object--trash-can" } target_classes { name: "object--vehicle--bicycle" label_id: 105 mapping_class: "object--vehicle--bicycle" } target_classes { name: "object--vehicle--boat" label_id: 106 mapping_class: "object--vehicle--boat" } target_classes { name: "object--vehicle--bus" label_id: 107 mapping_class: "object--vehicle--bus" } target_classes { name: "object--vehicle--car" label_id: 108 mapping_class: "object--vehicle--car" } target_classes { name: "object--vehicle--caravan" label_id: 109 mapping_class: "object--vehicle--caravan" } target_classes { name: "object--vehicle--motorcycle" label_id: 110 mapping_class: "object--vehicle--motorcycle" } target_classes { name: "object--vehicle--on-rails" label_id: 111 mapping_class: "object--vehicle--on-rails" } target_classes { name: "object--vehicle--other-vehicle" label_id: 112 mapping_class: "object--vehicle--other-vehicle" } target_classes { name: "object--vehicle--trailer" label_id: 113 mapping_class: "object--vehicle--trailer" } target_classes { name: "object--vehicle--truck" label_id: 114 mapping_class: "object--vehicle--truck" } target_classes { name: "object--vehicle--vehicle-group" label_id: 115 mapping_class: "object--vehicle--vehicle-group" } target_classes { name: "object--vehicle--wheeled-slow" label_id: 116 mapping_class: "object--vehicle--wheeled-slow" } target_classes { name: "object--water-valve" label_id: 117 mapping_class: "object--water-valve" } target_classes { name: "void--car-mount" label_id: 118 mapping_class: "void--car-mount" } target_classes { name: "void--dynamic" label_id: 119 mapping_class: "void--dynamic" } target_classes { name: "void--ego-vehicle" label_id: 120 mapping_class: "void--ego-vehicle" } target_classes { name: "void--ground" label_id: 121 mapping_class: "void--ground" } target_classes { name: "void--static" label_id: 122 mapping_class: "void--static" } target_classes { name: "void--unlabeled" label_id: 123 mapping_class: "void--unlabeled" } } } model_config { pretrained_model_file: "/new/media/hdd/datasets/mapillary/vistas/tlt_semantic_segmentation_vresnet10/resnet_10.hdf5" num_layers: 10 use_batch_norm: true training_precision { backend_floatx: FLOAT32 } freeze_bn: true arch: "resnet" all_projections: true model_input_height: 640 model_input_width: 640 model_input_channels: 3 } training_config { batch_size: 7 regularizer { type: L2 weight: 3.000000026176508e-09 } optimizer { adam { epsilon: 9.99999993922529e-09 beta1: 0.8999999761581421 beta2: 0.9990000128746033 } } checkpoint_interval: 1 log_summary_steps: 10 learning_rate: 0.0005000000237487257 loss: "cross_entropy" epochs: 30 } , 'seed': 0, 'benchmark': False, 'temp_dir': '/tmp/tmp9xhhc81c', 'num_classes': 124, 'start_step': 0, 'checkpoint_interval': 1, 'phase': None} __________________________________________________________________________________________________ Layer (type) Output Shape Param # Connected to ================================================================================================== input_1 (InputLayer) (None, 3, 640, 640) 0 __________________________________________________________________________________________________ conv1 (Conv2D) (None, 64, 320, 320) 9472 input_1[0][0] __________________________________________________________________________________________________ bn_conv1 (BatchNormalization) (None, 64, 320, 320) 256 conv1[0][0] __________________________________________________________________________________________________ activation_1 (Activation) (None, 64, 320, 320) 0 bn_conv1[0][0] __________________________________________________________________________________________________ block_1a_conv_1 (Conv2D) (None, 64, 160, 160) 36928 activation_1[0][0] __________________________________________________________________________________________________ block_1a_bn_1 (BatchNormalizati (None, 64, 160, 160) 256 block_1a_conv_1[0][0] __________________________________________________________________________________________________ block_1a_relu_1 (Activation) (None, 64, 160, 160) 0 block_1a_bn_1[0][0] __________________________________________________________________________________________________ block_1a_conv_2 (Conv2D) (None, 64, 160, 160) 36928 block_1a_relu_1[0][0] __________________________________________________________________________________________________ block_1a_conv_shortcut (Conv2D) (None, 64, 160, 160) 4160 activation_1[0][0] __________________________________________________________________________________________________ block_1a_bn_2 (BatchNormalizati (None, 64, 160, 160) 256 block_1a_conv_2[0][0] __________________________________________________________________________________________________ block_1a_bn_shortcut (BatchNorm (None, 64, 160, 160) 256 block_1a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_1 (Add) (None, 64, 160, 160) 0 block_1a_bn_2[0][0] block_1a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_1a_relu (Activation) (None, 64, 160, 160) 0 add_1[0][0] __________________________________________________________________________________________________ block_2a_conv_1 (Conv2D) (None, 128, 80, 80) 73856 block_1a_relu[0][0] __________________________________________________________________________________________________ block_2a_bn_1 (BatchNormalizati (None, 128, 80, 80) 512 block_2a_conv_1[0][0] __________________________________________________________________________________________________ block_2a_relu_1 (Activation) (None, 128, 80, 80) 0 block_2a_bn_1[0][0] __________________________________________________________________________________________________ block_2a_conv_2 (Conv2D) (None, 128, 80, 80) 147584 block_2a_relu_1[0][0] __________________________________________________________________________________________________ block_2a_conv_shortcut (Conv2D) (None, 128, 80, 80) 8320 block_1a_relu[0][0] __________________________________________________________________________________________________ block_2a_bn_2 (BatchNormalizati (None, 128, 80, 80) 512 block_2a_conv_2[0][0] __________________________________________________________________________________________________ block_2a_bn_shortcut (BatchNorm (None, 128, 80, 80) 512 block_2a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_2 (Add) (None, 128, 80, 80) 0 block_2a_bn_2[0][0] block_2a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_2a_relu (Activation) (None, 128, 80, 80) 0 add_2[0][0] __________________________________________________________________________________________________ block_3a_conv_1 (Conv2D) (None, 256, 40, 40) 295168 block_2a_relu[0][0] __________________________________________________________________________________________________ block_3a_bn_1 (BatchNormalizati (None, 256, 40, 40) 1024 block_3a_conv_1[0][0] __________________________________________________________________________________________________ block_3a_relu_1 (Activation) (None, 256, 40, 40) 0 block_3a_bn_1[0][0] __________________________________________________________________________________________________ block_3a_conv_2 (Conv2D) (None, 256, 40, 40) 590080 block_3a_relu_1[0][0] __________________________________________________________________________________________________ block_3a_conv_shortcut (Conv2D) (None, 256, 40, 40) 33024 block_2a_relu[0][0] __________________________________________________________________________________________________ block_3a_bn_2 (BatchNormalizati (None, 256, 40, 40) 1024 block_3a_conv_2[0][0] __________________________________________________________________________________________________ block_3a_bn_shortcut (BatchNorm (None, 256, 40, 40) 1024 block_3a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_3 (Add) (None, 256, 40, 40) 0 block_3a_bn_2[0][0] block_3a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_3a_relu (Activation) (None, 256, 40, 40) 0 add_3[0][0] __________________________________________________________________________________________________ block_4a_conv_1 (Conv2D) (None, 512, 40, 40) 1180160 block_3a_relu[0][0] __________________________________________________________________________________________________ block_4a_bn_1 (BatchNormalizati (None, 512, 40, 40) 2048 block_4a_conv_1[0][0] __________________________________________________________________________________________________ block_4a_relu_1 (Activation) (None, 512, 40, 40) 0 block_4a_bn_1[0][0] __________________________________________________________________________________________________ block_4a_conv_2 (Conv2D) (None, 512, 40, 40) 2359808 block_4a_relu_1[0][0] __________________________________________________________________________________________________ block_4a_conv_shortcut (Conv2D) (None, 512, 40, 40) 131584 block_3a_relu[0][0] __________________________________________________________________________________________________ block_4a_bn_2 (BatchNormalizati (None, 512, 40, 40) 2048 block_4a_conv_2[0][0] __________________________________________________________________________________________________ block_4a_bn_shortcut (BatchNorm (None, 512, 40, 40) 2048 block_4a_conv_shortcut[0][0] __________________________________________________________________________________________________ add_4 (Add) (None, 512, 40, 40) 0 block_4a_bn_2[0][0] block_4a_bn_shortcut[0][0] __________________________________________________________________________________________________ block_4a_relu (Activation) (None, 512, 40, 40) 0 add_4[0][0] __________________________________________________________________________________________________ conv2d_transpose_1 (Conv2DTrans (None, 256, 80, 80) 2097408 block_4a_relu[0][0] __________________________________________________________________________________________________ concatenate_1 (Concatenate) (None, 384, 80, 80) 0 conv2d_transpose_1[0][0] block_2a_relu[0][0] __________________________________________________________________________________________________ conv2d_1 (Conv2D) (None, 256, 80, 80) 884992 concatenate_1[0][0] __________________________________________________________________________________________________ conv2d_transpose_2 (Conv2DTrans (None, 128, 160, 160 524416 conv2d_1[0][0] __________________________________________________________________________________________________ concatenate_2 (Concatenate) (None, 192, 160, 160 0 conv2d_transpose_2[0][0] block_1a_relu[0][0] __________________________________________________________________________________________________ conv2d_2 (Conv2D) (None, 128, 160, 160 221312 concatenate_2[0][0] __________________________________________________________________________________________________ conv2d_transpose_3 (Conv2DTrans (None, 64, 320, 320) 131136 conv2d_2[0][0] __________________________________________________________________________________________________ concatenate_3 (Concatenate) (None, 128, 320, 320 0 conv2d_transpose_3[0][0] bn_conv1[0][0] __________________________________________________________________________________________________ conv2d_3 (Conv2D) (None, 64, 320, 320) 73792 concatenate_3[0][0] __________________________________________________________________________________________________ conv2d_transpose_4 (Conv2DTrans (None, 64, 640, 640) 65600 conv2d_3[0][0] __________________________________________________________________________________________________ concatenate_4 (Concatenate) (None, 67, 640, 640) 0 conv2d_transpose_4[0][0] input_1[0][0] __________________________________________________________________________________________________ conv2d_4 (Conv2D) (None, 64, 640, 640) 38656 concatenate_4[0][0] __________________________________________________________________________________________________ conv2d_5 (Conv2D) (None, 124, 640, 640 71548 conv2d_4[0][0] ================================================================================================== Total params: 9,027,708 Trainable params: 9,021,820 Non-trainable params: 5,888 __________________________________________________________________________________________________ INFO:tensorflow:Done calling model_fn. 2021-06-13 10:49:57,187 [INFO] tensorflow: Done calling model_fn. INFO:tensorflow:Graph was finalized. 2021-06-13 10:49:58,201 [INFO] tensorflow: Graph was finalized. INFO:tensorflow:Running local_init_op. 2021-06-13 10:49:58,710 [INFO] tensorflow: Running local_init_op. INFO:tensorflow:Done running local_init_op. 2021-06-13 10:49:58,763 [INFO] tensorflow: Done running local_init_op. [GPU] Restoring pretrained weights from: /tmp/tmpm3jqqqc3/model.ckpt-0 2021-06-13 10:50:03,835 [INFO] iva.unet.hooks.pretrained_restore_hook: Pretrained weights loaded with success... INFO:tensorflow:Saving checkpoints for step-0. 2021-06-13 10:50:06,022 [INFO] tensorflow: Saving checkpoints for step-0. WARNING:tensorflow:From /home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/hooks/training_hook.py:92: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead. 2021-06-13 10:50:09,154 [WARNING] tensorflow: From /home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/hooks/training_hook.py:92: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead. Killed Traceback (most recent call last): File "/usr/local/bin/unet", line 8, in sys.exit(main()) File "/home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/unet/entrypoint/unet.py", line 12, in main File "/home/vpraveen/.cache/dazel/_dazel_vpraveen/216c8b41e526c3295d3b802489ac2034/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/entrypoint/entrypoint.py", line 296, in launch_job AssertionError: Process run failed.