SSD Mobilenet_v2 with Kitti dataset training error

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Please provide the following information when requesting support.

• Hardware (T4/V100/Xavier/Nano/etc)
Ubuntu 20.04 with 2 Nvidia GPUs
• Network Type (Detectnet_v2/Faster_rcnn/Yolo_v4/LPRnet/Mask_rcnn/Classification/etc)
SSD Mobilenet_v2
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here)
5.0
• Training spec file(If have, please share here)
ssd_train_resnet18_kitti.txt from SSD example but with arch: “mobilenet_v2”
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)

Hi,

I’ve successfully trained an SSD ResNet-18 model and am now looking to experiment with the MobileNet_v2 backend. However, I’m encountering an issue: the training crashes after 6 or 7 epochs. Specifically, the loss spikes from 6 or 7 to over 100, and then it crashes.

I’ve attempted to troubleshoot by adjusting batch sizes (16, 8, 4) and experimenting with various min and max learning rate values. Unfortunately, these changes only seem to affect the point at which the crash occurs.

Should I consider modifying other hyperparameters? Is there a guide or best practices I can refer to for this?

Another consideration that crossed my mind is the image size. Do all backends share the same expected tile size? Should I be making any modifications to my dataset in this regard?

Thank you in advance.

3

Do you run the official 5.0 notebook successfully? You can refer to it as the best practices.
Please also share the crash log as well.

4

Hi Morganh,

I ran the official notebook succesfully (Resnet18), and this is the log:

I1023 09:53:39.369798 1 tensorrt.cc:65] TRITONBACKEND_Initialize: tensorrt
I1023 09:53:39.369819 1 tensorrt.cc:75] Triton TRITONBACKEND API version: 1.16
I1023 09:53:39.369821 1 tensorrt.cc:81] 'tensorrt' TRITONBACKEND API version: 1.16
I1023 09:53:39.369824 1 tensorrt.cc:105] backend configuration:
{"cmdline":{"auto-complete-config":"true","backend-directory":"/opt/tritonserver/backends","min-compute-capability":"6.000000","default-max-batch-size":"4"}}
I1023 09:53:39.370685 1 tensorrt.cc:231] TRITONBACKEND_ModelInitialize: TensorRT_Test1 (version 1)
I1023 09:53:39.389011 1 logging.cc:46] Loaded engine size: 14 MiB
2023-10-23 09:53:39.394727: I tensorflow/cc/saved_model/loader.cc:233] Restoring SavedModel bundle.
E1023 09:53:39.401613 1 logging.cc:40] 1: [runtime.cpp::parsePlan::314] Error Code 1: Serialization (Serialization assertion plan->header.magicTag == rt::kPLAN_MAGIC_TAG failed.)
I1023 09:53:39.402149 1 tensorrt.cc:274] TRITONBACKEND_ModelFinalize: delete model state
E1023 09:53:39.402175 1 model_lifecycle.cc:621] failed to load 'TensorRT_Test1' version 1: Internal: unable to load plan file to auto complete config: /models/TensorRT_Test1/1/model.plan
I1023 09:53:39.402188 1 model_lifecycle.cc:756] failed to load 'TensorRT_Test1'
5

You run https://github.com/NVIDIA/tao_tutorials/blob/main/notebooks/tao_launcher_starter_kit/ssd/ssd.ipynb successfully, right?

Where does the log come from? Can you share steps/commands/etc?

6

Hi,

I changed the resnet18 architecture to mobilenet_v2 in the ssd_train_resnet18_kitti.txt file. My goal is to train a mobilenet_v2 with 640x640 input images with my custom dataset.

7

Can you share the full training log?

1 Like
8

Hi,

This is the loaded specs file:

random_seed: 42
ssd_config {
  aspect_ratios_global: "[1.0, 2.0, 0.5, 3.0, 1.0/3.0]"
  scales: "[0.05, 0.1, 0.25, 0.4, 0.55, 0.7, 0.85]"
  two_boxes_for_ar1: true
  clip_boxes: false
  variances: "[0.1, 0.1, 0.2, 0.2]"
  arch: "mobilenet_v2"
  nlayers: 18
  freeze_bn: false
  freeze_blocks: 0
}
training_config {
  batch_size_per_gpu: 16
  num_epochs: 80
  enable_qat: false
  learning_rate {
  soft_start_annealing_schedule {
    min_learning_rate: 5e-1
    max_learning_rate: 1e-1
    soft_start: 0.15
    annealing: 0.8
    }
  }
  regularizer {
    type: L1
    weight: 3e-5
  }
}
eval_config {
  validation_period_during_training: 10
  average_precision_mode: SAMPLE
  batch_size: 16
  matching_iou_threshold: 0.5
}
nms_config {
  confidence_threshold: 0.01
  clustering_iou_threshold: 0.6
  top_k: 200
}
augmentation_config {
    output_width: 300
    output_height: 300
    output_channel: 3
}
dataset_config {
  data_sources: {
    tfrecords_path: "/workspace/tao-experiments/data/ssd/tfrecords/kitti_train*"
  }
  include_difficult_in_training: true
  target_class_mapping {
      key: "car"
      value: "car"
  }
  target_class_mapping {
      key: "pedestrian"
      value: "pedestrian"
  }
  target_class_mapping {
      key: "cyclist"
      value: "cyclist"
  }
  target_class_mapping {
      key: "van"
      value: "car"
  }
  target_class_mapping {
      key: "person_sitting"
      value: "pedestrian"
  }
  validation_data_sources: {
    image_directory_path: "/workspace/tao-experiments/data/kitti_split/val/image"
    label_directory_path: "/workspace/tao-experiments/data/kitti_split/val/label"
  }
}

And the full log (Depending on the learning rate it takes more or less epochs to crash, with default learning rate arround 7 epochs):

To run with multigpu, please change --gpus based on the number of available GPUs in your machine.
2023-10-20 11:36:52,683 [TAO Toolkit] [INFO] root 160: Registry: ['nvcr.io']
2023-10-20 11:36:52,772 [TAO Toolkit] [INFO] nvidia_tao_cli.components.instance_handler.local_instance 361: Running command in container: nvcr.io/nvidia/tao/tao-toolkit:5.0.0-tf1.15.5
2023-10-20 11:36:52,837 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 275: Printing tty value True
Using TensorFlow backend.
2023-10-20 09:36:53.768914: I tensorflow/stream_executor/platform/default/dso_loader.cc:50] Successfully opened dynamic library libcudart.so.12
2023-10-20 09:36:53,812 [TAO Toolkit] [WARNING] tensorflow 40: Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
2023-10-20 09:36:54,617 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use sklearn by default. This improves performance in some cases. To enable sklearn export the environment variable  TF_ALLOW_IOLIBS=1.
2023-10-20 09:36:54,646 [TAO Toolkit] [WARNING] tensorflow 42: TensorFlow will not use Dask by default. This improves performance in some cases. To enable Dask export the environment variable  TF_ALLOW_IOLIBS=1.
2023-10-20 09:36:54,650 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use Pandas by default. This improves performance in some cases. To enable Pandas export the environment variable  TF_ALLOW_IOLIBS=1.
2023-10-20 09:36:54,830 [TAO Toolkit] [WARNING] matplotlib 500: Matplotlib created a temporary config/cache directory at /tmp/matplotlib-30_ow1ga because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
2023-10-20 09:36:55,096 [TAO Toolkit] [INFO] matplotlib.font_manager 1633: generated new fontManager
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:TensorFlow will not use sklearn by default. This improves performance in some cases. To enable sklearn export the environment variable  TF_ALLOW_IOLIBS=1.
2023-10-20 09:36:57,959 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use sklearn by default. This improves performance in some cases. To enable sklearn export the environment variable  TF_ALLOW_IOLIBS=1.
WARNING:tensorflow:TensorFlow will not use sklearn by default. This improves performance in some cases. To enable sklearn export the environment variable  TF_ALLOW_IOLIBS=1.
2023-10-20 09:36:57,979 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use sklearn by default. This improves performance in some cases. To enable sklearn export the environment variable  TF_ALLOW_IOLIBS=1.
WARNING:tensorflow:TensorFlow will not use Dask by default. This improves performance in some cases. To enable Dask export the environment variable  TF_ALLOW_IOLIBS=1.
2023-10-20 09:36:57,989 [TAO Toolkit] [WARNING] tensorflow 42: TensorFlow will not use Dask by default. This improves performance in some cases. To enable Dask export the environment variable  TF_ALLOW_IOLIBS=1.
WARNING:tensorflow:TensorFlow will not use Pandas by default. This improves performance in some cases. To enable Pandas export the environment variable  TF_ALLOW_IOLIBS=1.
2023-10-20 09:36:57,992 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use Pandas by default. This improves performance in some cases. To enable Pandas export the environment variable  TF_ALLOW_IOLIBS=1.
WARNING:tensorflow:TensorFlow will not use Dask by default. This improves performance in some cases. To enable Dask export the environment variable  TF_ALLOW_IOLIBS=1.
2023-10-20 09:36:58,009 [TAO Toolkit] [WARNING] tensorflow 42: TensorFlow will not use Dask by default. This improves performance in some cases. To enable Dask export the environment variable  TF_ALLOW_IOLIBS=1.
WARNING:tensorflow:TensorFlow will not use Pandas by default. This improves performance in some cases. To enable Pandas export the environment variable  TF_ALLOW_IOLIBS=1.
2023-10-20 09:36:58,013 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use Pandas by default. This improves performance in some cases. To enable Pandas export the environment variable  TF_ALLOW_IOLIBS=1.
WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/scripts/train.py:181: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/scripts/train.py:181: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

2023-10-20 09:36:59,742 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/scripts/train.py:181: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

2023-10-20 09:36:59,742 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/scripts/train.py:181: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/scripts/train.py:184: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2023-10-20 09:36:59,742 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/scripts/train.py:184: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/scripts/train.py:184: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2023-10-20 09:36:59,742 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/scripts/train.py:184: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2023-10-20 09:36:59,964 [TAO Toolkit] [INFO] nvidia_tao_tf1.cv.common.logging.logging 197: Log file already exists at /workspace/tao-experiments/ssd/experiment_dir_unpruned_mobilenet/status.json
2023-10-20 09:36:59,965 [TAO Toolkit] [INFO] nvidia_tao_tf1.cv.ssd.utils.spec_loader 179: Merging specification from /workspace/tao-experiments/ssd/specs/ssd_train_mobilenet_v2.txt
WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

2023-10-20 09:36:59,966 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

2023-10-20 09:36:59,973 [TAO Toolkit] [INFO] __main__ 275: Loading pretrained weights. This may take a while...
WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

2023-10-20 09:36:59,973 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

2023-10-20 09:36:59,975 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.

2023-10-20 09:36:59,985 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.

2023-10-20 09:36:59,987 [TAO Toolkit] [INFO] nvidia_tao_tf1.cv.common.logging.logging 197: Log file already exists at /workspace/tao-experiments/ssd/experiment_dir_unpruned_mobilenet/status.json
2023-10-20 09:36:59,987 [TAO Toolkit] [INFO] nvidia_tao_tf1.cv.ssd.utils.spec_loader 179: Merging specification from /workspace/tao-experiments/ssd/specs/ssd_train_mobilenet_v2.txt
WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

2023-10-20 09:36:59,989 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

2023-10-20 09:36:59,995 [TAO Toolkit] [INFO] __main__ 275: Loading pretrained weights. This may take a while...
WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

2023-10-20 09:36:59,996 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

2023-10-20 09:36:59,997 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.

2023-10-20 09:37:00,007 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:4185: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

2023-10-20 09:37:00,597 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:4185: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:4185: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

2023-10-20 09:37:00,633 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:4185: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

Using DALI augmentation pipeline.
Using DALI augmentation pipeline.
WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/box_coder/input_encoder.py:511: The name tf.log is deprecated. Please use tf.math.log instead.

2023-10-20 09:37:02,566 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/box_coder/input_encoder.py:511: The name tf.log is deprecated. Please use tf.math.log instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/box_coder/input_encoder.py:511: The name tf.log is deprecated. Please use tf.math.log instead.

2023-10-20 09:37:02,649 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/box_coder/input_encoder.py:511: The name tf.log is deprecated. Please use tf.math.log instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/third_party/keras/tensorflow_backend.py:199: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

2023-10-20 09:37:05,227 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/third_party/keras/tensorflow_backend.py:199: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/third_party/keras/tensorflow_backend.py:199: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

2023-10-20 09:37:05,387 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/third_party/keras/tensorflow_backend.py:199: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

2023-10-20 09:37:05,432 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

2023-10-20 09:37:05,432 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.

2023-10-20 09:37:05,433 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

2023-10-20 09:37:05,598 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

2023-10-20 09:37:05,598 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.

2023-10-20 09:37:05,599 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.

2023-10-20 09:37:07,295 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.

2023-10-20 09:37:07,515 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

2023-10-20 09:37:09,482 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

2023-10-20 09:37:09,724 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.

2023-10-20 09:37:10,025 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

2023-10-20 09:37:10,221 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.

2023-10-20 09:37:10,268 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

2023-10-20 09:37:10,465 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

Weights for those layers can not be loaded: ['re_lu_0']
STOP trainig now and check the pre-train model if this is not expected!
Layers that load weights from the pretrained model: ['conv1', 'bn_conv1', 'expanded_conv_depthwise', 'expanded_conv_depthwise_bn', 'expanded_conv_project', 'expanded_conv_project_bn', 'block_1_expand', 'block_1_expand_bn', 'block_1_depthwise', 'block_1_depthwise_bn', 'block_1_project', 'block_1_project_bn', 'block_2_expand', 'block_2_expand_bn', 'block_2_depthwise', 'block_2_depthwise_bn', 'block_2_project', 'block_2_projected_inputs', 'block_2_project_bn', 'block_3_expand', 'block_3_expand_bn', 'block_3_depthwise', 'block_3_depthwise_bn', 'block_3_project', 'block_3_project_bn', 'block_4_expand', 'block_4_expand_bn', 'block_4_depthwise', 'block_4_depthwise_bn', 'block_4_project', 'block_4_projected_inputs', 'block_4_project_bn', 'block_5_expand', 'block_5_expand_bn', 'block_5_depthwise', 'block_5_depthwise_bn', 'block_5_project', 'block_5_projected_inputs', 'block_5_project_bn', 'block_6_expand', 'block_6_expand_bn', 'block_6_depthwise', 'block_6_depthwise_bn', 'block_6_project', 'block_6_project_bn', 'block_7_expand', 'block_7_expand_bn', 'block_7_depthwise', 'block_7_depthwise_bn', 'block_7_project', 'block_7_projected_inputs', 'block_7_project_bn', 'block_8_expand', 'block_8_expand_bn', 'block_8_depthwise', 'block_8_depthwise_bn', 'block_8_project', 'block_8_projected_inputs', 'block_8_project_bn', 'block_9_expand', 'block_9_expand_bn', 'block_9_depthwise', 'block_9_depthwise_bn', 'block_9_project', 'block_9_projected_inputs', 'block_9_project_bn', 'block_10_expand', 'block_10_expand_bn', 'block_10_depthwise', 'block_10_depthwise_bn', 'block_10_project', 'block_10_project_bn', 'block_11_expand', 'block_11_expand_bn', 'block_11_depthwise', 'block_11_depthwise_bn', 'block_11_project', 'block_11_projected_inputs', 'block_11_project_bn', 'block_12_expand', 'block_12_expand_bn', 'block_12_depthwise', 'block_12_depthwise_bn', 'block_12_project', 'block_12_projected_inputs', 'block_12_project_bn']
Initialize optimizer
Weights for those layers can not be loaded: ['re_lu_0']
STOP trainig now and check the pre-train model if this is not expected!
Layers that load weights from the pretrained model: ['conv1', 'bn_conv1', 'expanded_conv_depthwise', 'expanded_conv_depthwise_bn', 'expanded_conv_project', 'expanded_conv_project_bn', 'block_1_expand', 'block_1_expand_bn', 'block_1_depthwise', 'block_1_depthwise_bn', 'block_1_project', 'block_1_project_bn', 'block_2_expand', 'block_2_expand_bn', 'block_2_depthwise', 'block_2_depthwise_bn', 'block_2_project', 'block_2_projected_inputs', 'block_2_project_bn', 'block_3_expand', 'block_3_expand_bn', 'block_3_depthwise', 'block_3_depthwise_bn', 'block_3_project', 'block_3_project_bn', 'block_4_expand', 'block_4_expand_bn', 'block_4_depthwise', 'block_4_depthwise_bn', 'block_4_project', 'block_4_projected_inputs', 'block_4_project_bn', 'block_5_expand', 'block_5_expand_bn', 'block_5_depthwise', 'block_5_depthwise_bn', 'block_5_project', 'block_5_projected_inputs', 'block_5_project_bn', 'block_6_expand', 'block_6_expand_bn', 'block_6_depthwise', 'block_6_depthwise_bn', 'block_6_project', 'block_6_project_bn', 'block_7_expand', 'block_7_expand_bn', 'block_7_depthwise', 'block_7_depthwise_bn', 'block_7_project', 'block_7_projected_inputs', 'block_7_project_bn', 'block_8_expand', 'block_8_expand_bn', 'block_8_depthwise', 'block_8_depthwise_bn', 'block_8_project', 'block_8_projected_inputs', 'block_8_project_bn', 'block_9_expand', 'block_9_expand_bn', 'block_9_depthwise', 'block_9_depthwise_bn', 'block_9_project', 'block_9_projected_inputs', 'block_9_project_bn', 'block_10_expand', 'block_10_expand_bn', 'block_10_depthwise', 'block_10_depthwise_bn', 'block_10_project', 'block_10_project_bn', 'block_11_expand', 'block_11_expand_bn', 'block_11_depthwise', 'block_11_depthwise_bn', 'block_11_project', 'block_11_projected_inputs', 'block_11_project_bn', 'block_12_expand', 'block_12_expand_bn', 'block_12_depthwise', 'block_12_depthwise_bn', 'block_12_project', 'block_12_projected_inputs', 'block_12_project_bn']
Initialize optimizer
WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/utils/tensor_utils.py:133: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.

2023-10-20 09:38:59,642 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/utils/tensor_utils.py:133: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/utils/tensor_utils.py:134: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.

2023-10-20 09:38:59,642 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/utils/tensor_utils.py:134: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/utils/tensor_utils.py:135: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

2023-10-20 09:38:59,642 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/utils/tensor_utils.py:135: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/utils/tensor_utils.py:133: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.

2023-10-20 09:38:59,795 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/utils/tensor_utils.py:133: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/utils/tensor_utils.py:134: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.

2023-10-20 09:38:59,795 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/utils/tensor_utils.py:134: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/utils/tensor_utils.py:135: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

2023-10-20 09:38:59,796 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/utils/tensor_utils.py:135: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
Input (InputLayer)              (16, 3, 300, 300)    0                                            
__________________________________________________________________________________________________
conv1_pad (ZeroPadding2D)       (16, 3, 302, 302)    0           Input[0][0]                      
__________________________________________________________________________________________________
conv1 (Conv2D)                  (16, 32, 150, 150)   864         conv1_pad[0][0]                  
__________________________________________________________________________________________________
bn_conv1 (BatchNormalization)   (16, 32, 150, 150)   128         conv1[0][0]                      
__________________________________________________________________________________________________
re_lu_0 (ReLU)                  (16, 32, 150, 150)   0           bn_conv1[0][0]                   
__________________________________________________________________________________________________
expanded_conv_depthwise_pad (Ze (16, 32, 152, 152)   0           re_lu_0[0][0]                    
__________________________________________________________________________________________________
expanded_conv_depthwise (Depthw (16, 32, 150, 150)   288         expanded_conv_depthwise_pad[0][0]
__________________________________________________________________________________________________
expanded_conv_depthwise_bn (Bat (16, 32, 150, 150)   128         expanded_conv_depthwise[0][0]    
__________________________________________________________________________________________________
expanded_conv_relu (ReLU)       (16, 32, 150, 150)   0           expanded_conv_depthwise_bn[0][0] 
__________________________________________________________________________________________________
expanded_conv_project (Conv2D)  (16, 16, 150, 150)   512         expanded_conv_relu[0][0]         
__________________________________________________________________________________________________
expanded_conv_project_bn (Batch (16, 16, 150, 150)   64          expanded_conv_project[0][0]      
__________________________________________________________________________________________________
block_1_expand (Conv2D)         (16, 96, 150, 150)   1536        expanded_conv_project_bn[0][0]   
__________________________________________________________________________________________________
block_1_expand_bn (BatchNormali (16, 96, 150, 150)   384         block_1_expand[0][0]             
__________________________________________________________________________________________________
re_lu_2 (ReLU)                  (16, 96, 150, 150)   0           block_1_expand_bn[0][0]          
__________________________________________________________________________________________________
block_1_depthwise_pad (ZeroPadd (16, 96, 152, 152)   0           re_lu_2[0][0]                    
__________________________________________________________________________________________________
block_1_depthwise (DepthwiseCon (16, 96, 75, 75)     864         block_1_depthwise_pad[0][0]      
__________________________________________________________________________________________________
block_1_depthwise_bn (BatchNorm (16, 96, 75, 75)     384         block_1_depthwise[0][0]          
__________________________________________________________________________________________________
block_1_relu (ReLU)             (16, 96, 75, 75)     0           block_1_depthwise_bn[0][0]       
__________________________________________________________________________________________________
block_1_project (Conv2D)        (16, 24, 75, 75)     2304        block_1_relu[0][0]               
__________________________________________________________________________________________________
block_1_project_bn (BatchNormal (16, 24, 75, 75)     96          block_1_project[0][0]            
__________________________________________________________________________________________________
block_2_expand (Conv2D)         (16, 144, 75, 75)    3456        block_1_project_bn[0][0]         
__________________________________________________________________________________________________
block_2_expand_bn (BatchNormali (16, 144, 75, 75)    576         block_2_expand[0][0]             
__________________________________________________________________________________________________
re_lu_3 (ReLU)                  (16, 144, 75, 75)    0           block_2_expand_bn[0][0]          
__________________________________________________________________________________________________
block_2_depthwise_pad (ZeroPadd (16, 144, 77, 77)    0           re_lu_3[0][0]                    
__________________________________________________________________________________________________
block_2_depthwise (DepthwiseCon (16, 144, 75, 75)    1296        block_2_depthwise_pad[0][0]      
__________________________________________________________________________________________________
block_2_depthwise_bn (BatchNorm (16, 144, 75, 75)    576         block_2_depthwise[0][0]          
__________________________________________________________________________________________________
block_2_relu (ReLU)             (16, 144, 75, 75)    0           block_2_depthwise_bn[0][0]       
__________________________________________________________________________________________________
block_2_project (Conv2D)        (16, 24, 75, 75)     3456        block_2_relu[0][0]               
__________________________________________________________________________________________________
block_2_projected_inputs (Conv2 (16, 24, 75, 75)     576         block_1_project_bn[0][0]         
__________________________________________________________________________________________________
block_2_project_bn (BatchNormal (16, 24, 75, 75)     96          block_2_project[0][0]            
__________________________________________________________________________________________________
block_2_add (Add)               (16, 24, 75, 75)     0           block_2_projected_inputs[0][0]   
                                                                 block_2_project_bn[0][0]         
__________________________________________________________________________________________________
block_3_expand (Conv2D)         (16, 144, 75, 75)    3456        block_2_add[0][0]                
__________________________________________________________________________________________________
block_3_expand_bn (BatchNormali (16, 144, 75, 75)    576         block_3_expand[0][0]             
__________________________________________________________________________________________________
re_lu_4 (ReLU)                  (16, 144, 75, 75)    0           block_3_expand_bn[0][0]          
__________________________________________________________________________________________________
block_3_depthwise_pad (ZeroPadd (16, 144, 77, 77)    0           re_lu_4[0][0]                    
__________________________________________________________________________________________________
block_3_depthwise (DepthwiseCon (16, 144, 38, 38)    1296        block_3_depthwise_pad[0][0]      
__________________________________________________________________________________________________
block_3_depthwise_bn (BatchNorm (16, 144, 38, 38)    576         block_3_depthwise[0][0]          
__________________________________________________________________________________________________
block_3_relu (ReLU)             (16, 144, 38, 38)    0           block_3_depthwise_bn[0][0]       
__________________________________________________________________________________________________
block_3_project (Conv2D)        (16, 32, 38, 38)     4608        block_3_relu[0][0]               
__________________________________________________________________________________________________
block_3_project_bn (BatchNormal (16, 32, 38, 38)     128         block_3_project[0][0]            
__________________________________________________________________________________________________
block_4_expand (Conv2D)         (16, 192, 38, 38)    6144        block_3_project_bn[0][0]         
__________________________________________________________________________________________________
block_4_expand_bn (BatchNormali (16, 192, 38, 38)    768         block_4_expand[0][0]             
__________________________________________________________________________________________________
re_lu_5 (ReLU)                  (16, 192, 38, 38)    0           block_4_expand_bn[0][0]          
__________________________________________________________________________________________________
block_4_depthwise_pad (ZeroPadd (16, 192, 40, 40)    0           re_lu_5[0][0]                    
__________________________________________________________________________________________________
block_4_depthwise (DepthwiseCon (16, 192, 38, 38)    1728        block_4_depthwise_pad[0][0]      
__________________________________________________________________________________________________
block_4_depthwise_bn (BatchNorm (16, 192, 38, 38)    768         block_4_depthwise[0][0]          
__________________________________________________________________________________________________
block_4_relu (ReLU)             (16, 192, 38, 38)    0           block_4_depthwise_bn[0][0]       
__________________________________________________________________________________________________
block_4_project (Conv2D)        (16, 32, 38, 38)     6144        block_4_relu[0][0]               
__________________________________________________________________________________________________
block_4_projected_inputs (Conv2 (16, 32, 38, 38)     1024        block_3_project_bn[0][0]         
__________________________________________________________________________________________________
block_4_project_bn (BatchNormal (16, 32, 38, 38)     128         block_4_project[0][0]            
__________________________________________________________________________________________________
block_4_add (Add)               (16, 32, 38, 38)     0           block_4_projected_inputs[0][0]   
                                                                 block_4_project_bn[0][0]         
__________________________________________________________________________________________________
block_5_expand (Conv2D)         (16, 192, 38, 38)    6144        block_4_add[0][0]                
__________________________________________________________________________________________________
block_5_expand_bn (BatchNormali (16, 192, 38, 38)    768         block_5_expand[0][0]             
__________________________________________________________________________________________________
re_lu_6 (ReLU)                  (16, 192, 38, 38)    0           block_5_expand_bn[0][0]          
__________________________________________________________________________________________________
block_5_depthwise_pad (ZeroPadd (16, 192, 40, 40)    0           re_lu_6[0][0]                    
__________________________________________________________________________________________________
block_5_depthwise (DepthwiseCon (16, 192, 38, 38)    1728        block_5_depthwise_pad[0][0]      
__________________________________________________________________________________________________
block_5_depthwise_bn (BatchNorm (16, 192, 38, 38)    768         block_5_depthwise[0][0]          
__________________________________________________________________________________________________
block_5_relu (ReLU)             (16, 192, 38, 38)    0           block_5_depthwise_bn[0][0]       
__________________________________________________________________________________________________
block_5_project (Conv2D)        (16, 32, 38, 38)     6144        block_5_relu[0][0]               
__________________________________________________________________________________________________
block_5_projected_inputs (Conv2 (16, 32, 38, 38)     1024        block_4_add[0][0]                
__________________________________________________________________________________________________
block_5_project_bn (BatchNormal (16, 32, 38, 38)     128         block_5_project[0][0]            
__________________________________________________________________________________________________
block_5_add (Add)               (16, 32, 38, 38)     0           block_5_projected_inputs[0][0]   
                                                                 block_5_project_bn[0][0]         
__________________________________________________________________________________________________
block_6_expand (Conv2D)         (16, 192, 38, 38)    6144        block_5_add[0][0]                
__________________________________________________________________________________________________
block_6_expand_bn (BatchNormali (16, 192, 38, 38)    768         block_6_expand[0][0]             
__________________________________________________________________________________________________
re_lu_7 (ReLU)                  (16, 192, 38, 38)    0           block_6_expand_bn[0][0]          
__________________________________________________________________________________________________
block_6_depthwise_pad (ZeroPadd (16, 192, 40, 40)    0           re_lu_7[0][0]                    
__________________________________________________________________________________________________
block_6_depthwise (DepthwiseCon (16, 192, 19, 19)    1728        block_6_depthwise_pad[0][0]      
__________________________________________________________________________________________________
block_6_depthwise_bn (BatchNorm (16, 192, 19, 19)    768         block_6_depthwise[0][0]          
__________________________________________________________________________________________________
block_6_relu (ReLU)             (16, 192, 19, 19)    0           block_6_depthwise_bn[0][0]       
__________________________________________________________________________________________________
block_6_project (Conv2D)        (16, 64, 19, 19)     12288       block_6_relu[0][0]               
__________________________________________________________________________________________________
block_6_project_bn (BatchNormal (16, 64, 19, 19)     256         block_6_project[0][0]            
__________________________________________________________________________________________________
block_7_expand (Conv2D)         (16, 384, 19, 19)    24576       block_6_project_bn[0][0]         
__________________________________________________________________________________________________
block_7_expand_bn (BatchNormali (16, 384, 19, 19)    1536        block_7_expand[0][0]             
__________________________________________________________________________________________________
re_lu_8 (ReLU)                  (16, 384, 19, 19)    0           block_7_expand_bn[0][0]          
__________________________________________________________________________________________________
block_7_depthwise_pad (ZeroPadd (16, 384, 21, 21)    0           re_lu_8[0][0]                    
__________________________________________________________________________________________________
block_7_depthwise (DepthwiseCon (16, 384, 19, 19)    3456        block_7_depthwise_pad[0][0]      
__________________________________________________________________________________________________
block_7_depthwise_bn (BatchNorm (16, 384, 19, 19)    1536        block_7_depthwise[0][0]          
__________________________________________________________________________________________________
block_7_relu (ReLU)             (16, 384, 19, 19)    0           block_7_depthwise_bn[0][0]       
__________________________________________________________________________________________________
block_7_project (Conv2D)        (16, 64, 19, 19)     24576       block_7_relu[0][0]               
__________________________________________________________________________________________________
block_7_projected_inputs (Conv2 (16, 64, 19, 19)     4096        block_6_project_bn[0][0]         
__________________________________________________________________________________________________
block_7_project_bn (BatchNormal (16, 64, 19, 19)     256         block_7_project[0][0]            
__________________________________________________________________________________________________
block_7_add (Add)               (16, 64, 19, 19)     0           block_7_projected_inputs[0][0]   
                                                                 block_7_project_bn[0][0]         
__________________________________________________________________________________________________
block_8_expand (Conv2D)         (16, 384, 19, 19)    24576       block_7_add[0][0]                
__________________________________________________________________________________________________
block_8_expand_bn (BatchNormali (16, 384, 19, 19)    1536        block_8_expand[0][0]             
__________________________________________________________________________________________________
re_lu_9 (ReLU)                  (16, 384, 19, 19)    0           block_8_expand_bn[0][0]          
__________________________________________________________________________________________________
block_8_depthwise_pad (ZeroPadd (16, 384, 21, 21)    0           re_lu_9[0][0]                    
__________________________________________________________________________________________________
block_8_depthwise (DepthwiseCon (16, 384, 19, 19)    3456        block_8_depthwise_pad[0][0]      
__________________________________________________________________________________________________
block_8_depthwise_bn (BatchNorm (16, 384, 19, 19)    1536        block_8_depthwise[0][0]          
__________________________________________________________________________________________________
block_8_relu (ReLU)             (16, 384, 19, 19)    0           block_8_depthwise_bn[0][0]       
__________________________________________________________________________________________________
block_8_project (Conv2D)        (16, 64, 19, 19)     24576       block_8_relu[0][0]               
__________________________________________________________________________________________________
block_8_projected_inputs (Conv2 (16, 64, 19, 19)     4096        block_7_add[0][0]                
__________________________________________________________________________________________________
block_8_project_bn (BatchNormal (16, 64, 19, 19)     256         block_8_project[0][0]            
__________________________________________________________________________________________________
block_8_add (Add)               (16, 64, 19, 19)     0           block_8_projected_inputs[0][0]   
                                                                 block_8_project_bn[0][0]         
__________________________________________________________________________________________________
block_9_expand (Conv2D)         (16, 384, 19, 19)    24576       block_8_add[0][0]                
__________________________________________________________________________________________________
block_9_expand_bn (BatchNormali (16, 384, 19, 19)    1536        block_9_expand[0][0]             
__________________________________________________________________________________________________
re_lu_10 (ReLU)                 (16, 384, 19, 19)    0           block_9_expand_bn[0][0]          
__________________________________________________________________________________________________
block_9_depthwise_pad (ZeroPadd (16, 384, 21, 21)    0           re_lu_10[0][0]                   
__________________________________________________________________________________________________
block_9_depthwise (DepthwiseCon (16, 384, 19, 19)    3456        block_9_depthwise_pad[0][0]      
__________________________________________________________________________________________________
block_9_depthwise_bn (BatchNorm (16, 384, 19, 19)    1536        block_9_depthwise[0][0]          
__________________________________________________________________________________________________
block_9_relu (ReLU)             (16, 384, 19, 19)    0           block_9_depthwise_bn[0][0]       
__________________________________________________________________________________________________
block_9_project (Conv2D)        (16, 64, 19, 19)     24576       block_9_relu[0][0]               
__________________________________________________________________________________________________
block_9_projected_inputs (Conv2 (16, 64, 19, 19)     4096        block_8_add[0][0]                
__________________________________________________________________________________________________
block_9_project_bn (BatchNormal (16, 64, 19, 19)     256         block_9_project[0][0]            
__________________________________________________________________________________________________
block_9_add (Add)               (16, 64, 19, 19)     0           block_9_projected_inputs[0][0]   
                                                                 block_9_project_bn[0][0]         
__________________________________________________________________________________________________
block_10_expand (Conv2D)        (16, 384, 19, 19)    24576       block_9_add[0][0]                
__________________________________________________________________________________________________
block_10_expand_bn (BatchNormal (16, 384, 19, 19)    1536        block_10_expand[0][0]            
__________________________________________________________________________________________________
re_lu_11 (ReLU)                 (16, 384, 19, 19)    0           block_10_expand_bn[0][0]         
__________________________________________________________________________________________________
block_10_depthwise_pad (ZeroPad (16, 384, 21, 21)    0           re_lu_11[0][0]                   
__________________________________________________________________________________________________
block_10_depthwise (DepthwiseCo (16, 384, 19, 19)    3456        block_10_depthwise_pad[0][0]     
__________________________________________________________________________________________________
block_10_depthwise_bn (BatchNor (16, 384, 19, 19)    1536        block_10_depthwise[0][0]         
__________________________________________________________________________________________________
block_10_relu (ReLU)            (16, 384, 19, 19)    0           block_10_depthwise_bn[0][0]      
__________________________________________________________________________________________________
block_10_project (Conv2D)       (16, 96, 19, 19)     36864       block_10_relu[0][0]              
__________________________________________________________________________________________________
block_10_project_bn (BatchNorma (16, 96, 19, 19)     384         block_10_project[0][0]           
__________________________________________________________________________________________________
block_11_expand (Conv2D)        (16, 576, 19, 19)    55296       block_10_project_bn[0][0]        
__________________________________________________________________________________________________
block_11_expand_bn (BatchNormal (16, 576, 19, 19)    2304        block_11_expand[0][0]            
__________________________________________________________________________________________________
re_lu_12 (ReLU)                 (16, 576, 19, 19)    0           block_11_expand_bn[0][0]         
__________________________________________________________________________________________________
block_11_depthwise_pad (ZeroPad (16, 576, 21, 21)    0           re_lu_12[0][0]                   
__________________________________________________________________________________________________
block_11_depthwise (DepthwiseCo (16, 576, 19, 19)    5184        block_11_depthwise_pad[0][0]     
__________________________________________________________________________________________________
block_11_depthwise_bn (BatchNor (16, 576, 19, 19)    2304        block_11_depthwise[0][0]         
__________________________________________________________________________________________________
block_11_relu (ReLU)            (16, 576, 19, 19)    0           block_11_depthwise_bn[0][0]      
__________________________________________________________________________________________________
block_11_project (Conv2D)       (16, 96, 19, 19)     55296       block_11_relu[0][0]              
__________________________________________________________________________________________________
block_11_projected_inputs (Conv (16, 96, 19, 19)     9216        block_10_project_bn[0][0]        
__________________________________________________________________________________________________
block_11_project_bn (BatchNorma (16, 96, 19, 19)     384         block_11_project[0][0]           
__________________________________________________________________________________________________
block_11_add (Add)              (16, 96, 19, 19)     0           block_11_projected_inputs[0][0]  
                                                                 block_11_project_bn[0][0]        
__________________________________________________________________________________________________
block_12_expand (Conv2D)        (16, 576, 19, 19)    55296       block_11_add[0][0]               
__________________________________________________________________________________________________
block_12_expand_bn (BatchNormal (16, 576, 19, 19)    2304        block_12_expand[0][0]            
__________________________________________________________________________________________________
re_lu_13 (ReLU)                 (16, 576, 19, 19)    0           block_12_expand_bn[0][0]         
__________________________________________________________________________________________________
block_12_depthwise_pad (ZeroPad (16, 576, 21, 21)    0           re_lu_13[0][0]                   
__________________________________________________________________________________________________
block_12_depthwise (DepthwiseCo (16, 576, 19, 19)    5184        block_12_depthwise_pad[0][0]     
__________________________________________________________________________________________________
block_12_depthwise_bn (BatchNor (16, 576, 19, 19)    2304        block_12_depthwise[0][0]         
__________________________________________________________________________________________________
block_12_relu (ReLU)            (16, 576, 19, 19)    0           block_12_depthwise_bn[0][0]      
__________________________________________________________________________________________________
block_12_project (Conv2D)       (16, 96, 19, 19)     55296       block_12_relu[0][0]              
__________________________________________________________________________________________________
block_12_projected_inputs (Conv (16, 96, 19, 19)     9216        block_11_add[0][0]               
__________________________________________________________________________________________________
block_12_project_bn (BatchNorma (16, 96, 19, 19)     384         block_12_project[0][0]           
__________________________________________________________________________________________________
block_12_add (Add)              (16, 96, 19, 19)     0           block_12_projected_inputs[0][0]  
                                                                 block_12_project_bn[0][0]        
__________________________________________________________________________________________________
ssd_expand_block_1_conv_0 (Conv (16, 64, 19, 19)     6208        block_12_add[0][0]               
__________________________________________________________________________________________________
ssd_expand_block_1_relu_0 (ReLU (16, 64, 19, 19)     0           ssd_expand_block_1_conv_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_1_conv_1 (Conv (16, 128, 10, 10)    73728       ssd_expand_block_1_relu_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_1_bn_1 (BatchN (16, 128, 10, 10)    512         ssd_expand_block_1_conv_1[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_1_relu_1 (ReLU (16, 128, 10, 10)    0           ssd_expand_block_1_bn_1[0][0]    
__________________________________________________________________________________________________
ssd_expand_block_2_conv_0 (Conv (16, 64, 10, 10)     8256        ssd_expand_block_1_relu_1[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_2_relu_0 (ReLU (16, 64, 10, 10)     0           ssd_expand_block_2_conv_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_2_conv_1 (Conv (16, 128, 5, 5)      73728       ssd_expand_block_2_relu_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_2_bn_1 (BatchN (16, 128, 5, 5)      512         ssd_expand_block_2_conv_1[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_2_relu_1 (ReLU (16, 128, 5, 5)      0           ssd_expand_block_2_bn_1[0][0]    
__________________________________________________________________________________________________
ssd_expand_block_3_conv_0 (Conv (16, 64, 5, 5)       8256        ssd_expand_block_2_relu_1[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_3_relu_0 (ReLU (16, 64, 5, 5)       0           ssd_expand_block_3_conv_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_3_conv_1 (Conv (16, 128, 3, 3)      73728       ssd_expand_block_3_relu_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_3_bn_1 (BatchN (16, 128, 3, 3)      512         ssd_expand_block_3_conv_1[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_3_relu_1 (ReLU (16, 128, 3, 3)      0           ssd_expand_block_3_bn_1[0][0]    
__________________________________________________________________________________________________
ssd_expand_block_4_conv_0 (Conv (16, 64, 3, 3)       8256        ssd_expand_block_3_relu_1[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_4_relu_0 (ReLU (16, 64, 3, 3)       0           ssd_expand_block_4_conv_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_4_conv_1 (Conv (16, 128, 2, 2)      73728       ssd_expand_block_4_relu_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_4_bn_1 (BatchN (16, 128, 2, 2)      512         ssd_expand_block_4_conv_1[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_4_relu_1 (ReLU (16, 128, 2, 2)      0           ssd_expand_block_4_bn_1[0][0]    
__________________________________________________________________________________________________
ssd_conf_0 (Conv2D)             (16, 24, 38, 38)     41496       re_lu_7[0][0]                    
__________________________________________________________________________________________________
ssd_conf_1 (Conv2D)             (16, 24, 19, 19)     20760       block_12_add[0][0]               
__________________________________________________________________________________________________
ssd_conf_2 (Conv2D)             (16, 24, 10, 10)     27672       ssd_expand_block_1_relu_1[0][0]  
__________________________________________________________________________________________________
ssd_conf_3 (Conv2D)             (16, 24, 5, 5)       27672       ssd_expand_block_2_relu_1[0][0]  
__________________________________________________________________________________________________
ssd_conf_4 (Conv2D)             (16, 24, 3, 3)       27672       ssd_expand_block_3_relu_1[0][0]  
__________________________________________________________________________________________________
ssd_conf_5 (Conv2D)             (16, 24, 2, 2)       27672       ssd_expand_block_4_relu_1[0][0]  
__________________________________________________________________________________________________
permute_13 (Permute)            (16, 38, 38, 24)     0           ssd_conf_0[0][0]                 
__________________________________________________________________________________________________
permute_14 (Permute)            (16, 19, 19, 24)     0           ssd_conf_1[0][0]                 
__________________________________________________________________________________________________
permute_15 (Permute)            (16, 10, 10, 24)     0           ssd_conf_2[0][0]                 
__________________________________________________________________________________________________
permute_16 (Permute)            (16, 5, 5, 24)       0           ssd_conf_3[0][0]                 
__________________________________________________________________________________________________
permute_17 (Permute)            (16, 3, 3, 24)       0           ssd_conf_4[0][0]                 
__________________________________________________________________________________________________
permute_18 (Permute)            (16, 2, 2, 24)       0           ssd_conf_5[0][0]                 
__________________________________________________________________________________________________
conf_reshape_0 (Reshape)        (16, 8664, 1, 4)     0           permute_13[0][0]                 
__________________________________________________________________________________________________
conf_reshape_1 (Reshape)        (16, 2166, 1, 4)     0           permute_14[0][0]                 
__________________________________________________________________________________________________
conf_reshape_2 (Reshape)        (16, 600, 1, 4)      0           permute_15[0][0]                 
__________________________________________________________________________________________________
conf_reshape_3 (Reshape)        (16, 150, 1, 4)      0           permute_16[0][0]                 
__________________________________________________________________________________________________
conf_reshape_4 (Reshape)        (16, 54, 1, 4)       0           permute_17[0][0]                 
__________________________________________________________________________________________________
conf_reshape_5 (Reshape)        (16, 24, 1, 4)       0           permute_18[0][0]                 
__________________________________________________________________________________________________
mbox_conf (Concatenate)         (16, 11658, 1, 4)    0           conf_reshape_0[0][0]             
                                                                 conf_reshape_1[0][0]             
                                                                 conf_reshape_2[0][0]             
                                                                 conf_reshape_3[0][0]             
                                                                 conf_reshape_4[0][0]             
                                                                 conf_reshape_5[0][0]             
__________________________________________________________________________________________________
ssd_loc_0 (Conv2D)              (16, 24, 38, 38)     41496       re_lu_7[0][0]                    
__________________________________________________________________________________________________
ssd_loc_1 (Conv2D)              (16, 24, 19, 19)     20760       block_12_add[0][0]               
__________________________________________________________________________________________________
ssd_loc_2 (Conv2D)              (16, 24, 10, 10)     27672       ssd_expand_block_1_relu_1[0][0]  
__________________________________________________________________________________________________
ssd_loc_3 (Conv2D)              (16, 24, 5, 5)       27672       ssd_expand_block_2_relu_1[0][0]  
__________________________________________________________________________________________________
ssd_loc_4 (Conv2D)              (16, 24, 3, 3)       27672       ssd_expand_block_3_relu_1[0][0]  
__________________________________________________________________________________________________
ssd_loc_5 (Conv2D)              (16, 24, 2, 2)       27672       ssd_expand_block_4_relu_1[0][0]  
__________________________________________________________________________________________________
before_softmax_permute (Permute (16, 4, 1, 11658)    0           mbox_conf[0][0]                  
__________________________________________________________________________________________________
permute_19 (Permute)            (16, 38, 38, 24)     0           ssd_loc_0[0][0]                  
__________________________________________________________________________________________________
permute_20 (Permute)            (16, 19, 19, 24)     0           ssd_loc_1[0][0]                  
__________________________________________________________________________________________________
permute_21 (Permute)            (16, 10, 10, 24)     0           ssd_loc_2[0][0]                  
__________________________________________________________________________________________________
permute_22 (Permute)            (16, 5, 5, 24)       0           ssd_loc_3[0][0]                  
__________________________________________________________________________________________________
permute_23 (Permute)            (16, 3, 3, 24)       0           ssd_loc_4[0][0]                  
__________________________________________________________________________________________________
permute_24 (Permute)            (16, 2, 2, 24)       0           ssd_loc_5[0][0]                  
__________________________________________________________________________________________________
ssd_anchor_0 (AnchorBoxes)      (16, 1444, 6, 8)     0           ssd_loc_0[0][0]                  
__________________________________________________________________________________________________
ssd_anchor_1 (AnchorBoxes)      (16, 361, 6, 8)      0           ssd_loc_1[0][0]                  
__________________________________________________________________________________________________
ssd_anchor_2 (AnchorBoxes)      (16, 100, 6, 8)      0           ssd_loc_2[0][0]                  
__________________________________________________________________________________________________
ssd_anchor_3 (AnchorBoxes)      (16, 25, 6, 8)       0           ssd_loc_3[0][0]                  
__________________________________________________________________________________________________
ssd_anchor_4 (AnchorBoxes)      (16, 9, 6, 8)        0           ssd_loc_4[0][0]                  
__________________________________________________________________________________________________
ssd_anchor_5 (AnchorBoxes)      (16, 4, 6, 8)        0           ssd_loc_5[0][0]                  
__________________________________________________________________________________________________
mbox_conf_softmax_ (Softmax)    (16, 4, 1, 11658)    0           before_softmax_permute[0][0]     
__________________________________________________________________________________________________
loc_reshape_0 (Reshape)         (16, 8664, 1, 4)     0           permute_19[0][0]                 
__________________________________________________________________________________________________
loc_reshape_1 (Reshape)         (16, 2166, 1, 4)     0           permute_20[0][0]                 
__________________________________________________________________________________________________
loc_reshape_2 (Reshape)         (16, 600, 1, 4)      0           permute_21[0][0]                 
__________________________________________________________________________________________________
loc_reshape_3 (Reshape)         (16, 150, 1, 4)      0           permute_22[0][0]                 
__________________________________________________________________________________________________
loc_reshape_4 (Reshape)         (16, 54, 1, 4)       0           permute_23[0][0]                 
__________________________________________________________________________________________________
loc_reshape_5 (Reshape)         (16, 24, 1, 4)       0           permute_24[0][0]                 
__________________________________________________________________________________________________
anchor_reshape_0 (Reshape)      (16, 8664, 1, 8)     0           ssd_anchor_0[0][0]               
__________________________________________________________________________________________________
anchor_reshape_1 (Reshape)      (16, 2166, 1, 8)     0           ssd_anchor_1[0][0]               
__________________________________________________________________________________________________
anchor_reshape_2 (Reshape)      (16, 600, 1, 8)      0           ssd_anchor_2[0][0]               
__________________________________________________________________________________________________
anchor_reshape_3 (Reshape)      (16, 150, 1, 8)      0           ssd_anchor_3[0][0]               
__________________________________________________________________________________________________
anchor_reshape_4 (Reshape)      (16, 54, 1, 8)       0           ssd_anchor_4[0][0]               
__________________________________________________________________________________________________
anchor_reshape_5 (Reshape)      (16, 24, 1, 8)       0           ssd_anchor_5[0][0]               
__________________________________________________________________________________________________
mbox_conf_softmax (Permute)     (16, 11658, 1, 4)    0           mbox_conf_softmax_[0][0]         
__________________________________________________________________________________________________
mbox_loc (Concatenate)          (16, 11658, 1, 4)    0           loc_reshape_0[0][0]              
                                                                 loc_reshape_1[0][0]              
                                                                 loc_reshape_2[0][0]              
                                                                 loc_reshape_3[0][0]              
                                                                 loc_reshape_4[0][0]              
                                                                 loc_reshape_5[0][0]              
__________________________________________________________________________________________________
mbox_priorbox (Concatenate)     (16, 11658, 1, 8)    0           anchor_reshape_0[0][0]           
                                                                 anchor_reshape_1[0][0]           
                                                                 anchor_reshape_2[0][0]           
                                                                 anchor_reshape_3[0][0]           
                                                                 anchor_reshape_4[0][0]           
                                                                 anchor_reshape_5[0][0]           
__________________________________________________________________________________________________
concatenate_2 (Concatenate)     (16, 11658, 1, 16)   0           mbox_conf_softmax[0][0]          
                                                                 mbox_loc[0][0]                   
                                                                 mbox_priorbox[0][0]              
__________________________________________________________________________________________________
ssd_predictions (Reshape)       (16, 11658, 16)      0           concatenate_2[0][0]              
==================================================================================================
Total params: 1,265,824
Trainable params: 1,247,808
Non-trainable params: 18,016
__________________________________________________________________________________________________
2023-10-20 09:39:00,491 [TAO Toolkit] [INFO] __main__ 356: Number of images in the training dataset:	  6733
2023-10-20 09:39:00,491 [TAO Toolkit] [INFO] __main__ 358: Number of images in the validation dataset:	   748

2023-10-20 09:39:01,080 [TAO Toolkit] [INFO] nvidia_tao_tf1.cv.common.logging.logging 197: Log file already exists at /workspace/tao-experiments/ssd/experiment_dir_unpruned_mobilenet/status.json
2023-10-20 09:39:05,971 [TAO Toolkit] [INFO] root 2102: Starting Training Loop.
Epoch 1/80
[1697794753.720439] [3b22f7efe2f5:245  :f]        vfs_fuse.c:424  UCX  WARN  failed to connect to vfs socket '': Invalid argument
[1697794754.158449] [3b22f7efe2f5:246  :f]        vfs_fuse.c:424  UCX  WARN  failed to connect to vfs socket '': Invalid argument
  6/211 [..............................] - ETA: 8:52 - loss: nan                             Batch 5: Invalid loss, terminating training
Batch 5: Invalid loss, terminating training
/usr/local/lib/python3.8/dist-packages/keras/callbacks.py:120: UserWarning: Method on_batch_end() is slow compared to the batch update (2.297638). Check your callbacks.
  warnings.warn('Method on_batch_end() is slow compared '
2023-10-20 09:39:25,752 [TAO Toolkit] [INFO] root 2102: Training finished successfully.
2023-10-20 09:39:25,753 [TAO Toolkit] [INFO] __main__ 569: Training finished successfully.

Epoch 00001: saving model to /workspace/tao-experiments/ssd/experiment_dir_unpruned_mobilenet/weights/ssd_mobilenet_v2_epoch_001.hdf5
/usr/local/lib/python3.8/dist-packages/keras/callbacks.py:120: UserWarning: Method on_batch_end() is slow compared to the batch update (2.301577). Check your callbacks.
  warnings.warn('Method on_batch_end() is slow compared '
2023-10-20 09:39:37,905 [TAO Toolkit] [INFO] root 2102: Training loop in progress
2023-10-20 09:39:37,906 [TAO Toolkit] [INFO] root 2102: Training loop complete.
2023-10-20 09:39:37,906 [TAO Toolkit] [INFO] root 2102: Training finished successfully.
2023-10-20 09:39:37,906 [TAO Toolkit] [INFO] __main__ 569: Training finished successfully.
Execution status: PASS
2023-10-20 11:39:43,746 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 337: Stopping container.
9

It is nan issue.

Please modify max_learning_rate and min_learning_rate. It is not expected to set min_learning_rate which is larger than max_learning_rate.

Also, please set a lower max_learning_rate and retry.

10

Sorry, this log is more representative (I did a lot of tests):

To run with multigpu, please change --gpus based on the number of available GPUs in your machine.
2023-10-24 07:56:30,922 [TAO Toolkit] [INFO] root 160: Registry: ['nvcr.io']
2023-10-24 07:56:31,015 [TAO Toolkit] [INFO] nvidia_tao_cli.components.instance_handler.local_instance 361: Running command in container: nvcr.io/nvidia/tao/tao-toolkit:5.0.0-tf1.15.5
2023-10-24 07:56:31,081 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 275: Printing tty value True
Using TensorFlow backend.
2023-10-24 05:56:31.992379: I tensorflow/stream_executor/platform/default/dso_loader.cc:50] Successfully opened dynamic library libcudart.so.12
2023-10-24 05:56:32,032 [TAO Toolkit] [WARNING] tensorflow 40: Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
2023-10-24 05:56:32,841 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use sklearn by default. This improves performance in some cases. To enable sklearn export the environment variable  TF_ALLOW_IOLIBS=1.
2023-10-24 05:56:32,881 [TAO Toolkit] [WARNING] tensorflow 42: TensorFlow will not use Dask by default. This improves performance in some cases. To enable Dask export the environment variable  TF_ALLOW_IOLIBS=1.
2023-10-24 05:56:32,885 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use Pandas by default. This improves performance in some cases. To enable Pandas export the environment variable  TF_ALLOW_IOLIBS=1.
2023-10-24 05:56:33,066 [TAO Toolkit] [WARNING] matplotlib 500: Matplotlib created a temporary config/cache directory at /tmp/matplotlib-dy5lq54_ because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
2023-10-24 05:56:33,348 [TAO Toolkit] [INFO] matplotlib.font_manager 1633: generated new fontManager
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
WARNING:tensorflow:TensorFlow will not use sklearn by default. This improves performance in some cases. To enable sklearn export the environment variable  TF_ALLOW_IOLIBS=1.
2023-10-24 05:56:36,114 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use sklearn by default. This improves performance in some cases. To enable sklearn export the environment variable  TF_ALLOW_IOLIBS=1.
WARNING:tensorflow:TensorFlow will not use Dask by default. This improves performance in some cases. To enable Dask export the environment variable  TF_ALLOW_IOLIBS=1.
2023-10-24 05:56:36,144 [TAO Toolkit] [WARNING] tensorflow 42: TensorFlow will not use Dask by default. This improves performance in some cases. To enable Dask export the environment variable  TF_ALLOW_IOLIBS=1.
WARNING:tensorflow:TensorFlow will not use Pandas by default. This improves performance in some cases. To enable Pandas export the environment variable  TF_ALLOW_IOLIBS=1.
2023-10-24 05:56:36,147 [TAO Toolkit] [WARNING] tensorflow 43: TensorFlow will not use Pandas by default. This improves performance in some cases. To enable Pandas export the environment variable  TF_ALLOW_IOLIBS=1.
WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/scripts/train.py:181: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

2023-10-24 05:56:37,901 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/scripts/train.py:181: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/scripts/train.py:184: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2023-10-24 05:56:37,902 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/scripts/train.py:184: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2023-10-24 05:56:38,058 [TAO Toolkit] [INFO] nvidia_tao_tf1.cv.common.logging.logging 197: Log file already exists at /workspace/tao-experiments/ssd/experiment_dir_unpruned_mobilenet/status.json
2023-10-24 05:56:38,058 [TAO Toolkit] [INFO] nvidia_tao_tf1.cv.ssd.utils.spec_loader 179: Merging specification from /workspace/tao-experiments/ssd/specs/ssd_train_mobilenet_v2.txt
WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

2023-10-24 05:56:38,061 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:153: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.

2023-10-24 05:56:38,068 [TAO Toolkit] [INFO] __main__ 275: Loading pretrained weights. This may take a while...
WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

2023-10-24 05:56:38,069 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:517: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

2023-10-24 05:56:38,071 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:4138: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.

2023-10-24 05:56:38,081 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:1834: The name tf.nn.fused_batch_norm is deprecated. Please use tf.compat.v1.nn.fused_batch_norm instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:4185: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

2023-10-24 05:56:38,702 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:4185: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.

Using DALI augmentation pipeline.
WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/box_coder/input_encoder.py:511: The name tf.log is deprecated. Please use tf.math.log instead.

2023-10-24 05:56:40,762 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/box_coder/input_encoder.py:511: The name tf.log is deprecated. Please use tf.math.log instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/third_party/keras/tensorflow_backend.py:199: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

2023-10-24 05:56:43,396 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/third_party/keras/tensorflow_backend.py:199: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

2023-10-24 05:56:43,598 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:174: The name tf.get_default_session is deprecated. Please use tf.compat.v1.get_default_session instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

2023-10-24 05:56:43,598 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:190: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.

2023-10-24 05:56:43,599 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:199: The name tf.is_variable_initialized is deprecated. Please use tf.compat.v1.is_variable_initialized instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.

2023-10-24 05:56:45,500 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:206: The name tf.variables_initializer is deprecated. Please use tf.compat.v1.variables_initializer instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

2023-10-24 05:56:47,645 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/optimizers.py:790: The name tf.train.Optimizer is deprecated. Please use tf.compat.v1.train.Optimizer instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.

2023-10-24 05:56:48,186 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:986: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

2023-10-24 05:56:48,380 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/keras/backend/tensorflow_backend.py:973: The name tf.assign is deprecated. Please use tf.compat.v1.assign instead.

Weights for those layers can not be loaded: ['re_lu_0']
STOP trainig now and check the pre-train model if this is not expected!
Layers that load weights from the pretrained model: ['conv1', 'bn_conv1', 'expanded_conv_depthwise', 'expanded_conv_depthwise_bn', 'expanded_conv_project', 'expanded_conv_project_bn', 'block_1_expand', 'block_1_expand_bn', 'block_1_depthwise', 'block_1_depthwise_bn', 'block_1_project', 'block_1_project_bn', 'block_2_expand', 'block_2_expand_bn', 'block_2_depthwise', 'block_2_depthwise_bn', 'block_2_project', 'block_2_projected_inputs', 'block_2_project_bn', 'block_3_expand', 'block_3_expand_bn', 'block_3_depthwise', 'block_3_depthwise_bn', 'block_3_project', 'block_3_project_bn', 'block_4_expand', 'block_4_expand_bn', 'block_4_depthwise', 'block_4_depthwise_bn', 'block_4_project', 'block_4_projected_inputs', 'block_4_project_bn', 'block_5_expand', 'block_5_expand_bn', 'block_5_depthwise', 'block_5_depthwise_bn', 'block_5_project', 'block_5_projected_inputs', 'block_5_project_bn', 'block_6_expand', 'block_6_expand_bn', 'block_6_depthwise', 'block_6_depthwise_bn', 'block_6_project', 'block_6_project_bn', 'block_7_expand', 'block_7_expand_bn', 'block_7_depthwise', 'block_7_depthwise_bn', 'block_7_project', 'block_7_projected_inputs', 'block_7_project_bn', 'block_8_expand', 'block_8_expand_bn', 'block_8_depthwise', 'block_8_depthwise_bn', 'block_8_project', 'block_8_projected_inputs', 'block_8_project_bn', 'block_9_expand', 'block_9_expand_bn', 'block_9_depthwise', 'block_9_depthwise_bn', 'block_9_project', 'block_9_projected_inputs', 'block_9_project_bn', 'block_10_expand', 'block_10_expand_bn', 'block_10_depthwise', 'block_10_depthwise_bn', 'block_10_project', 'block_10_project_bn', 'block_11_expand', 'block_11_expand_bn', 'block_11_depthwise', 'block_11_depthwise_bn', 'block_11_project', 'block_11_projected_inputs', 'block_11_project_bn', 'block_12_expand', 'block_12_expand_bn', 'block_12_depthwise', 'block_12_depthwise_bn', 'block_12_project', 'block_12_projected_inputs', 'block_12_project_bn']
Initialize optimizer
WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/utils/tensor_utils.py:133: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.

2023-10-24 05:58:35,785 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/utils/tensor_utils.py:133: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/utils/tensor_utils.py:134: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.

2023-10-24 05:58:35,786 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/utils/tensor_utils.py:134: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.

WARNING:tensorflow:From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/utils/tensor_utils.py:135: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

2023-10-24 05:58:35,786 [TAO Toolkit] [WARNING] tensorflow 137: From /usr/local/lib/python3.8/dist-packages/nvidia_tao_tf1/cv/ssd/utils/tensor_utils.py:135: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

__________________________________________________________________________________________________
Layer (type)                    Output Shape         Param #     Connected to                     
==================================================================================================
Input (InputLayer)              (16, 3, 300, 300)    0                                            
__________________________________________________________________________________________________
conv1_pad (ZeroPadding2D)       (16, 3, 302, 302)    0           Input[0][0]                      
__________________________________________________________________________________________________
conv1 (Conv2D)                  (16, 32, 150, 150)   864         conv1_pad[0][0]                  
__________________________________________________________________________________________________
bn_conv1 (BatchNormalization)   (16, 32, 150, 150)   128         conv1[0][0]                      
__________________________________________________________________________________________________
re_lu_0 (ReLU)                  (16, 32, 150, 150)   0           bn_conv1[0][0]                   
__________________________________________________________________________________________________
expanded_conv_depthwise_pad (Ze (16, 32, 152, 152)   0           re_lu_0[0][0]                    
__________________________________________________________________________________________________
expanded_conv_depthwise (Depthw (16, 32, 150, 150)   288         expanded_conv_depthwise_pad[0][0]
__________________________________________________________________________________________________
expanded_conv_depthwise_bn (Bat (16, 32, 150, 150)   128         expanded_conv_depthwise[0][0]    
__________________________________________________________________________________________________
expanded_conv_relu (ReLU)       (16, 32, 150, 150)   0           expanded_conv_depthwise_bn[0][0] 
__________________________________________________________________________________________________
expanded_conv_project (Conv2D)  (16, 16, 150, 150)   512         expanded_conv_relu[0][0]         
__________________________________________________________________________________________________
expanded_conv_project_bn (Batch (16, 16, 150, 150)   64          expanded_conv_project[0][0]      
__________________________________________________________________________________________________
block_1_expand (Conv2D)         (16, 96, 150, 150)   1536        expanded_conv_project_bn[0][0]   
__________________________________________________________________________________________________
block_1_expand_bn (BatchNormali (16, 96, 150, 150)   384         block_1_expand[0][0]             
__________________________________________________________________________________________________
re_lu_2 (ReLU)                  (16, 96, 150, 150)   0           block_1_expand_bn[0][0]          
__________________________________________________________________________________________________
block_1_depthwise_pad (ZeroPadd (16, 96, 152, 152)   0           re_lu_2[0][0]                    
__________________________________________________________________________________________________
block_1_depthwise (DepthwiseCon (16, 96, 75, 75)     864         block_1_depthwise_pad[0][0]      
__________________________________________________________________________________________________
block_1_depthwise_bn (BatchNorm (16, 96, 75, 75)     384         block_1_depthwise[0][0]          
__________________________________________________________________________________________________
block_1_relu (ReLU)             (16, 96, 75, 75)     0           block_1_depthwise_bn[0][0]       
__________________________________________________________________________________________________
block_1_project (Conv2D)        (16, 24, 75, 75)     2304        block_1_relu[0][0]               
__________________________________________________________________________________________________
block_1_project_bn (BatchNormal (16, 24, 75, 75)     96          block_1_project[0][0]            
__________________________________________________________________________________________________
block_2_expand (Conv2D)         (16, 144, 75, 75)    3456        block_1_project_bn[0][0]         
__________________________________________________________________________________________________
block_2_expand_bn (BatchNormali (16, 144, 75, 75)    576         block_2_expand[0][0]             
__________________________________________________________________________________________________
re_lu_3 (ReLU)                  (16, 144, 75, 75)    0           block_2_expand_bn[0][0]          
__________________________________________________________________________________________________
block_2_depthwise_pad (ZeroPadd (16, 144, 77, 77)    0           re_lu_3[0][0]                    
__________________________________________________________________________________________________
block_2_depthwise (DepthwiseCon (16, 144, 75, 75)    1296        block_2_depthwise_pad[0][0]      
__________________________________________________________________________________________________
block_2_depthwise_bn (BatchNorm (16, 144, 75, 75)    576         block_2_depthwise[0][0]          
__________________________________________________________________________________________________
block_2_relu (ReLU)             (16, 144, 75, 75)    0           block_2_depthwise_bn[0][0]       
__________________________________________________________________________________________________
block_2_project (Conv2D)        (16, 24, 75, 75)     3456        block_2_relu[0][0]               
__________________________________________________________________________________________________
block_2_projected_inputs (Conv2 (16, 24, 75, 75)     576         block_1_project_bn[0][0]         
__________________________________________________________________________________________________
block_2_project_bn (BatchNormal (16, 24, 75, 75)     96          block_2_project[0][0]            
__________________________________________________________________________________________________
block_2_add (Add)               (16, 24, 75, 75)     0           block_2_projected_inputs[0][0]   
                                                                 block_2_project_bn[0][0]         
__________________________________________________________________________________________________
block_3_expand (Conv2D)         (16, 144, 75, 75)    3456        block_2_add[0][0]                
__________________________________________________________________________________________________
block_3_expand_bn (BatchNormali (16, 144, 75, 75)    576         block_3_expand[0][0]             
__________________________________________________________________________________________________
re_lu_4 (ReLU)                  (16, 144, 75, 75)    0           block_3_expand_bn[0][0]          
__________________________________________________________________________________________________
block_3_depthwise_pad (ZeroPadd (16, 144, 77, 77)    0           re_lu_4[0][0]                    
__________________________________________________________________________________________________
block_3_depthwise (DepthwiseCon (16, 144, 38, 38)    1296        block_3_depthwise_pad[0][0]      
__________________________________________________________________________________________________
block_3_depthwise_bn (BatchNorm (16, 144, 38, 38)    576         block_3_depthwise[0][0]          
__________________________________________________________________________________________________
block_3_relu (ReLU)             (16, 144, 38, 38)    0           block_3_depthwise_bn[0][0]       
__________________________________________________________________________________________________
block_3_project (Conv2D)        (16, 32, 38, 38)     4608        block_3_relu[0][0]               
__________________________________________________________________________________________________
block_3_project_bn (BatchNormal (16, 32, 38, 38)     128         block_3_project[0][0]            
__________________________________________________________________________________________________
block_4_expand (Conv2D)         (16, 192, 38, 38)    6144        block_3_project_bn[0][0]         
__________________________________________________________________________________________________
block_4_expand_bn (BatchNormali (16, 192, 38, 38)    768         block_4_expand[0][0]             
__________________________________________________________________________________________________
re_lu_5 (ReLU)                  (16, 192, 38, 38)    0           block_4_expand_bn[0][0]          
__________________________________________________________________________________________________
block_4_depthwise_pad (ZeroPadd (16, 192, 40, 40)    0           re_lu_5[0][0]                    
__________________________________________________________________________________________________
block_4_depthwise (DepthwiseCon (16, 192, 38, 38)    1728        block_4_depthwise_pad[0][0]      
__________________________________________________________________________________________________
block_4_depthwise_bn (BatchNorm (16, 192, 38, 38)    768         block_4_depthwise[0][0]          
__________________________________________________________________________________________________
block_4_relu (ReLU)             (16, 192, 38, 38)    0           block_4_depthwise_bn[0][0]       
__________________________________________________________________________________________________
block_4_project (Conv2D)        (16, 32, 38, 38)     6144        block_4_relu[0][0]               
__________________________________________________________________________________________________
block_4_projected_inputs (Conv2 (16, 32, 38, 38)     1024        block_3_project_bn[0][0]         
__________________________________________________________________________________________________
block_4_project_bn (BatchNormal (16, 32, 38, 38)     128         block_4_project[0][0]            
__________________________________________________________________________________________________
block_4_add (Add)               (16, 32, 38, 38)     0           block_4_projected_inputs[0][0]   
                                                                 block_4_project_bn[0][0]         
__________________________________________________________________________________________________
block_5_expand (Conv2D)         (16, 192, 38, 38)    6144        block_4_add[0][0]                
__________________________________________________________________________________________________
block_5_expand_bn (BatchNormali (16, 192, 38, 38)    768         block_5_expand[0][0]             
__________________________________________________________________________________________________
re_lu_6 (ReLU)                  (16, 192, 38, 38)    0           block_5_expand_bn[0][0]          
__________________________________________________________________________________________________
block_5_depthwise_pad (ZeroPadd (16, 192, 40, 40)    0           re_lu_6[0][0]                    
__________________________________________________________________________________________________
block_5_depthwise (DepthwiseCon (16, 192, 38, 38)    1728        block_5_depthwise_pad[0][0]      
__________________________________________________________________________________________________
block_5_depthwise_bn (BatchNorm (16, 192, 38, 38)    768         block_5_depthwise[0][0]          
__________________________________________________________________________________________________
block_5_relu (ReLU)             (16, 192, 38, 38)    0           block_5_depthwise_bn[0][0]       
__________________________________________________________________________________________________
block_5_project (Conv2D)        (16, 32, 38, 38)     6144        block_5_relu[0][0]               
__________________________________________________________________________________________________
block_5_projected_inputs (Conv2 (16, 32, 38, 38)     1024        block_4_add[0][0]                
__________________________________________________________________________________________________
block_5_project_bn (BatchNormal (16, 32, 38, 38)     128         block_5_project[0][0]            
__________________________________________________________________________________________________
block_5_add (Add)               (16, 32, 38, 38)     0           block_5_projected_inputs[0][0]   
                                                                 block_5_project_bn[0][0]         
__________________________________________________________________________________________________
block_6_expand (Conv2D)         (16, 192, 38, 38)    6144        block_5_add[0][0]                
__________________________________________________________________________________________________
block_6_expand_bn (BatchNormali (16, 192, 38, 38)    768         block_6_expand[0][0]             
__________________________________________________________________________________________________
re_lu_7 (ReLU)                  (16, 192, 38, 38)    0           block_6_expand_bn[0][0]          
__________________________________________________________________________________________________
block_6_depthwise_pad (ZeroPadd (16, 192, 40, 40)    0           re_lu_7[0][0]                    
__________________________________________________________________________________________________
block_6_depthwise (DepthwiseCon (16, 192, 19, 19)    1728        block_6_depthwise_pad[0][0]      
__________________________________________________________________________________________________
block_6_depthwise_bn (BatchNorm (16, 192, 19, 19)    768         block_6_depthwise[0][0]          
__________________________________________________________________________________________________
block_6_relu (ReLU)             (16, 192, 19, 19)    0           block_6_depthwise_bn[0][0]       
__________________________________________________________________________________________________
block_6_project (Conv2D)        (16, 64, 19, 19)     12288       block_6_relu[0][0]               
__________________________________________________________________________________________________
block_6_project_bn (BatchNormal (16, 64, 19, 19)     256         block_6_project[0][0]            
__________________________________________________________________________________________________
block_7_expand (Conv2D)         (16, 384, 19, 19)    24576       block_6_project_bn[0][0]         
__________________________________________________________________________________________________
block_7_expand_bn (BatchNormali (16, 384, 19, 19)    1536        block_7_expand[0][0]             
__________________________________________________________________________________________________
re_lu_8 (ReLU)                  (16, 384, 19, 19)    0           block_7_expand_bn[0][0]          
__________________________________________________________________________________________________
block_7_depthwise_pad (ZeroPadd (16, 384, 21, 21)    0           re_lu_8[0][0]                    
__________________________________________________________________________________________________
block_7_depthwise (DepthwiseCon (16, 384, 19, 19)    3456        block_7_depthwise_pad[0][0]      
__________________________________________________________________________________________________
block_7_depthwise_bn (BatchNorm (16, 384, 19, 19)    1536        block_7_depthwise[0][0]          
__________________________________________________________________________________________________
block_7_relu (ReLU)             (16, 384, 19, 19)    0           block_7_depthwise_bn[0][0]       
__________________________________________________________________________________________________
block_7_project (Conv2D)        (16, 64, 19, 19)     24576       block_7_relu[0][0]               
__________________________________________________________________________________________________
block_7_projected_inputs (Conv2 (16, 64, 19, 19)     4096        block_6_project_bn[0][0]         
__________________________________________________________________________________________________
block_7_project_bn (BatchNormal (16, 64, 19, 19)     256         block_7_project[0][0]            
__________________________________________________________________________________________________
block_7_add (Add)               (16, 64, 19, 19)     0           block_7_projected_inputs[0][0]   
                                                                 block_7_project_bn[0][0]         
__________________________________________________________________________________________________
block_8_expand (Conv2D)         (16, 384, 19, 19)    24576       block_7_add[0][0]                
__________________________________________________________________________________________________
block_8_expand_bn (BatchNormali (16, 384, 19, 19)    1536        block_8_expand[0][0]             
__________________________________________________________________________________________________
re_lu_9 (ReLU)                  (16, 384, 19, 19)    0           block_8_expand_bn[0][0]          
__________________________________________________________________________________________________
block_8_depthwise_pad (ZeroPadd (16, 384, 21, 21)    0           re_lu_9[0][0]                    
__________________________________________________________________________________________________
block_8_depthwise (DepthwiseCon (16, 384, 19, 19)    3456        block_8_depthwise_pad[0][0]      
__________________________________________________________________________________________________
block_8_depthwise_bn (BatchNorm (16, 384, 19, 19)    1536        block_8_depthwise[0][0]          
__________________________________________________________________________________________________
block_8_relu (ReLU)             (16, 384, 19, 19)    0           block_8_depthwise_bn[0][0]       
__________________________________________________________________________________________________
block_8_project (Conv2D)        (16, 64, 19, 19)     24576       block_8_relu[0][0]               
__________________________________________________________________________________________________
block_8_projected_inputs (Conv2 (16, 64, 19, 19)     4096        block_7_add[0][0]                
__________________________________________________________________________________________________
block_8_project_bn (BatchNormal (16, 64, 19, 19)     256         block_8_project[0][0]            
__________________________________________________________________________________________________
block_8_add (Add)               (16, 64, 19, 19)     0           block_8_projected_inputs[0][0]   
                                                                 block_8_project_bn[0][0]         
__________________________________________________________________________________________________
block_9_expand (Conv2D)         (16, 384, 19, 19)    24576       block_8_add[0][0]                
__________________________________________________________________________________________________
block_9_expand_bn (BatchNormali (16, 384, 19, 19)    1536        block_9_expand[0][0]             
__________________________________________________________________________________________________
re_lu_10 (ReLU)                 (16, 384, 19, 19)    0           block_9_expand_bn[0][0]          
__________________________________________________________________________________________________
block_9_depthwise_pad (ZeroPadd (16, 384, 21, 21)    0           re_lu_10[0][0]                   
__________________________________________________________________________________________________
block_9_depthwise (DepthwiseCon (16, 384, 19, 19)    3456        block_9_depthwise_pad[0][0]      
__________________________________________________________________________________________________
block_9_depthwise_bn (BatchNorm (16, 384, 19, 19)    1536        block_9_depthwise[0][0]          
__________________________________________________________________________________________________
block_9_relu (ReLU)             (16, 384, 19, 19)    0           block_9_depthwise_bn[0][0]       
__________________________________________________________________________________________________
block_9_project (Conv2D)        (16, 64, 19, 19)     24576       block_9_relu[0][0]               
__________________________________________________________________________________________________
block_9_projected_inputs (Conv2 (16, 64, 19, 19)     4096        block_8_add[0][0]                
__________________________________________________________________________________________________
block_9_project_bn (BatchNormal (16, 64, 19, 19)     256         block_9_project[0][0]            
__________________________________________________________________________________________________
block_9_add (Add)               (16, 64, 19, 19)     0           block_9_projected_inputs[0][0]   
                                                                 block_9_project_bn[0][0]         
__________________________________________________________________________________________________
block_10_expand (Conv2D)        (16, 384, 19, 19)    24576       block_9_add[0][0]                
__________________________________________________________________________________________________
block_10_expand_bn (BatchNormal (16, 384, 19, 19)    1536        block_10_expand[0][0]            
__________________________________________________________________________________________________
re_lu_11 (ReLU)                 (16, 384, 19, 19)    0           block_10_expand_bn[0][0]         
__________________________________________________________________________________________________
block_10_depthwise_pad (ZeroPad (16, 384, 21, 21)    0           re_lu_11[0][0]                   
__________________________________________________________________________________________________
block_10_depthwise (DepthwiseCo (16, 384, 19, 19)    3456        block_10_depthwise_pad[0][0]     
__________________________________________________________________________________________________
block_10_depthwise_bn (BatchNor (16, 384, 19, 19)    1536        block_10_depthwise[0][0]         
__________________________________________________________________________________________________
block_10_relu (ReLU)            (16, 384, 19, 19)    0           block_10_depthwise_bn[0][0]      
__________________________________________________________________________________________________
block_10_project (Conv2D)       (16, 96, 19, 19)     36864       block_10_relu[0][0]              
__________________________________________________________________________________________________
block_10_project_bn (BatchNorma (16, 96, 19, 19)     384         block_10_project[0][0]           
__________________________________________________________________________________________________
block_11_expand (Conv2D)        (16, 576, 19, 19)    55296       block_10_project_bn[0][0]        
__________________________________________________________________________________________________
block_11_expand_bn (BatchNormal (16, 576, 19, 19)    2304        block_11_expand[0][0]            
__________________________________________________________________________________________________
re_lu_12 (ReLU)                 (16, 576, 19, 19)    0           block_11_expand_bn[0][0]         
__________________________________________________________________________________________________
block_11_depthwise_pad (ZeroPad (16, 576, 21, 21)    0           re_lu_12[0][0]                   
__________________________________________________________________________________________________
block_11_depthwise (DepthwiseCo (16, 576, 19, 19)    5184        block_11_depthwise_pad[0][0]     
__________________________________________________________________________________________________
block_11_depthwise_bn (BatchNor (16, 576, 19, 19)    2304        block_11_depthwise[0][0]         
__________________________________________________________________________________________________
block_11_relu (ReLU)            (16, 576, 19, 19)    0           block_11_depthwise_bn[0][0]      
__________________________________________________________________________________________________
block_11_project (Conv2D)       (16, 96, 19, 19)     55296       block_11_relu[0][0]              
__________________________________________________________________________________________________
block_11_projected_inputs (Conv (16, 96, 19, 19)     9216        block_10_project_bn[0][0]        
__________________________________________________________________________________________________
block_11_project_bn (BatchNorma (16, 96, 19, 19)     384         block_11_project[0][0]           
__________________________________________________________________________________________________
block_11_add (Add)              (16, 96, 19, 19)     0           block_11_projected_inputs[0][0]  
                                                                 block_11_project_bn[0][0]        
__________________________________________________________________________________________________
block_12_expand (Conv2D)        (16, 576, 19, 19)    55296       block_11_add[0][0]               
__________________________________________________________________________________________________
block_12_expand_bn (BatchNormal (16, 576, 19, 19)    2304        block_12_expand[0][0]            
__________________________________________________________________________________________________
re_lu_13 (ReLU)                 (16, 576, 19, 19)    0           block_12_expand_bn[0][0]         
__________________________________________________________________________________________________
block_12_depthwise_pad (ZeroPad (16, 576, 21, 21)    0           re_lu_13[0][0]                   
__________________________________________________________________________________________________
block_12_depthwise (DepthwiseCo (16, 576, 19, 19)    5184        block_12_depthwise_pad[0][0]     
__________________________________________________________________________________________________
block_12_depthwise_bn (BatchNor (16, 576, 19, 19)    2304        block_12_depthwise[0][0]         
__________________________________________________________________________________________________
block_12_relu (ReLU)            (16, 576, 19, 19)    0           block_12_depthwise_bn[0][0]      
__________________________________________________________________________________________________
block_12_project (Conv2D)       (16, 96, 19, 19)     55296       block_12_relu[0][0]              
__________________________________________________________________________________________________
block_12_projected_inputs (Conv (16, 96, 19, 19)     9216        block_11_add[0][0]               
__________________________________________________________________________________________________
block_12_project_bn (BatchNorma (16, 96, 19, 19)     384         block_12_project[0][0]           
__________________________________________________________________________________________________
block_12_add (Add)              (16, 96, 19, 19)     0           block_12_projected_inputs[0][0]  
                                                                 block_12_project_bn[0][0]        
__________________________________________________________________________________________________
ssd_expand_block_1_conv_0 (Conv (16, 64, 19, 19)     6208        block_12_add[0][0]               
__________________________________________________________________________________________________
ssd_expand_block_1_relu_0 (ReLU (16, 64, 19, 19)     0           ssd_expand_block_1_conv_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_1_conv_1 (Conv (16, 128, 10, 10)    73728       ssd_expand_block_1_relu_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_1_bn_1 (BatchN (16, 128, 10, 10)    512         ssd_expand_block_1_conv_1[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_1_relu_1 (ReLU (16, 128, 10, 10)    0           ssd_expand_block_1_bn_1[0][0]    
__________________________________________________________________________________________________
ssd_expand_block_2_conv_0 (Conv (16, 64, 10, 10)     8256        ssd_expand_block_1_relu_1[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_2_relu_0 (ReLU (16, 64, 10, 10)     0           ssd_expand_block_2_conv_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_2_conv_1 (Conv (16, 128, 5, 5)      73728       ssd_expand_block_2_relu_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_2_bn_1 (BatchN (16, 128, 5, 5)      512         ssd_expand_block_2_conv_1[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_2_relu_1 (ReLU (16, 128, 5, 5)      0           ssd_expand_block_2_bn_1[0][0]    
__________________________________________________________________________________________________
ssd_expand_block_3_conv_0 (Conv (16, 64, 5, 5)       8256        ssd_expand_block_2_relu_1[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_3_relu_0 (ReLU (16, 64, 5, 5)       0           ssd_expand_block_3_conv_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_3_conv_1 (Conv (16, 128, 3, 3)      73728       ssd_expand_block_3_relu_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_3_bn_1 (BatchN (16, 128, 3, 3)      512         ssd_expand_block_3_conv_1[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_3_relu_1 (ReLU (16, 128, 3, 3)      0           ssd_expand_block_3_bn_1[0][0]    
__________________________________________________________________________________________________
ssd_expand_block_4_conv_0 (Conv (16, 64, 3, 3)       8256        ssd_expand_block_3_relu_1[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_4_relu_0 (ReLU (16, 64, 3, 3)       0           ssd_expand_block_4_conv_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_4_conv_1 (Conv (16, 128, 2, 2)      73728       ssd_expand_block_4_relu_0[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_4_bn_1 (BatchN (16, 128, 2, 2)      512         ssd_expand_block_4_conv_1[0][0]  
__________________________________________________________________________________________________
ssd_expand_block_4_relu_1 (ReLU (16, 128, 2, 2)      0           ssd_expand_block_4_bn_1[0][0]    
__________________________________________________________________________________________________
ssd_conf_0 (Conv2D)             (16, 24, 38, 38)     41496       re_lu_7[0][0]                    
__________________________________________________________________________________________________
ssd_conf_1 (Conv2D)             (16, 24, 19, 19)     20760       block_12_add[0][0]               
__________________________________________________________________________________________________
ssd_conf_2 (Conv2D)             (16, 24, 10, 10)     27672       ssd_expand_block_1_relu_1[0][0]  
__________________________________________________________________________________________________
ssd_conf_3 (Conv2D)             (16, 24, 5, 5)       27672       ssd_expand_block_2_relu_1[0][0]  
__________________________________________________________________________________________________
ssd_conf_4 (Conv2D)             (16, 24, 3, 3)       27672       ssd_expand_block_3_relu_1[0][0]  
__________________________________________________________________________________________________
ssd_conf_5 (Conv2D)             (16, 24, 2, 2)       27672       ssd_expand_block_4_relu_1[0][0]  
__________________________________________________________________________________________________
permute_13 (Permute)            (16, 38, 38, 24)     0           ssd_conf_0[0][0]                 
__________________________________________________________________________________________________
permute_14 (Permute)            (16, 19, 19, 24)     0           ssd_conf_1[0][0]                 
__________________________________________________________________________________________________
permute_15 (Permute)            (16, 10, 10, 24)     0           ssd_conf_2[0][0]                 
__________________________________________________________________________________________________
permute_16 (Permute)            (16, 5, 5, 24)       0           ssd_conf_3[0][0]                 
__________________________________________________________________________________________________
permute_17 (Permute)            (16, 3, 3, 24)       0           ssd_conf_4[0][0]                 
__________________________________________________________________________________________________
permute_18 (Permute)            (16, 2, 2, 24)       0           ssd_conf_5[0][0]                 
__________________________________________________________________________________________________
conf_reshape_0 (Reshape)        (16, 8664, 1, 4)     0           permute_13[0][0]                 
__________________________________________________________________________________________________
conf_reshape_1 (Reshape)        (16, 2166, 1, 4)     0           permute_14[0][0]                 
__________________________________________________________________________________________________
conf_reshape_2 (Reshape)        (16, 600, 1, 4)      0           permute_15[0][0]                 
__________________________________________________________________________________________________
conf_reshape_3 (Reshape)        (16, 150, 1, 4)      0           permute_16[0][0]                 
__________________________________________________________________________________________________
conf_reshape_4 (Reshape)        (16, 54, 1, 4)       0           permute_17[0][0]                 
__________________________________________________________________________________________________
conf_reshape_5 (Reshape)        (16, 24, 1, 4)       0           permute_18[0][0]                 
__________________________________________________________________________________________________
mbox_conf (Concatenate)         (16, 11658, 1, 4)    0           conf_reshape_0[0][0]             
                                                                 conf_reshape_1[0][0]             
                                                                 conf_reshape_2[0][0]             
                                                                 conf_reshape_3[0][0]             
                                                                 conf_reshape_4[0][0]             
                                                                 conf_reshape_5[0][0]             
__________________________________________________________________________________________________
ssd_loc_0 (Conv2D)              (16, 24, 38, 38)     41496       re_lu_7[0][0]                    
__________________________________________________________________________________________________
ssd_loc_1 (Conv2D)              (16, 24, 19, 19)     20760       block_12_add[0][0]               
__________________________________________________________________________________________________
ssd_loc_2 (Conv2D)              (16, 24, 10, 10)     27672       ssd_expand_block_1_relu_1[0][0]  
__________________________________________________________________________________________________
ssd_loc_3 (Conv2D)              (16, 24, 5, 5)       27672       ssd_expand_block_2_relu_1[0][0]  
__________________________________________________________________________________________________
ssd_loc_4 (Conv2D)              (16, 24, 3, 3)       27672       ssd_expand_block_3_relu_1[0][0]  
__________________________________________________________________________________________________
ssd_loc_5 (Conv2D)              (16, 24, 2, 2)       27672       ssd_expand_block_4_relu_1[0][0]  
__________________________________________________________________________________________________
before_softmax_permute (Permute (16, 4, 1, 11658)    0           mbox_conf[0][0]                  
__________________________________________________________________________________________________
permute_19 (Permute)            (16, 38, 38, 24)     0           ssd_loc_0[0][0]                  
__________________________________________________________________________________________________
permute_20 (Permute)            (16, 19, 19, 24)     0           ssd_loc_1[0][0]                  
__________________________________________________________________________________________________
permute_21 (Permute)            (16, 10, 10, 24)     0           ssd_loc_2[0][0]                  
__________________________________________________________________________________________________
permute_22 (Permute)            (16, 5, 5, 24)       0           ssd_loc_3[0][0]                  
__________________________________________________________________________________________________
permute_23 (Permute)            (16, 3, 3, 24)       0           ssd_loc_4[0][0]                  
__________________________________________________________________________________________________
permute_24 (Permute)            (16, 2, 2, 24)       0           ssd_loc_5[0][0]                  
__________________________________________________________________________________________________
ssd_anchor_0 (AnchorBoxes)      (16, 1444, 6, 8)     0           ssd_loc_0[0][0]                  
__________________________________________________________________________________________________
ssd_anchor_1 (AnchorBoxes)      (16, 361, 6, 8)      0           ssd_loc_1[0][0]                  
__________________________________________________________________________________________________
ssd_anchor_2 (AnchorBoxes)      (16, 100, 6, 8)      0           ssd_loc_2[0][0]                  
__________________________________________________________________________________________________
ssd_anchor_3 (AnchorBoxes)      (16, 25, 6, 8)       0           ssd_loc_3[0][0]                  
__________________________________________________________________________________________________
ssd_anchor_4 (AnchorBoxes)      (16, 9, 6, 8)        0           ssd_loc_4[0][0]                  
__________________________________________________________________________________________________
ssd_anchor_5 (AnchorBoxes)      (16, 4, 6, 8)        0           ssd_loc_5[0][0]                  
__________________________________________________________________________________________________
mbox_conf_softmax_ (Softmax)    (16, 4, 1, 11658)    0           before_softmax_permute[0][0]     
__________________________________________________________________________________________________
loc_reshape_0 (Reshape)         (16, 8664, 1, 4)     0           permute_19[0][0]                 
__________________________________________________________________________________________________
loc_reshape_1 (Reshape)         (16, 2166, 1, 4)     0           permute_20[0][0]                 
__________________________________________________________________________________________________
loc_reshape_2 (Reshape)         (16, 600, 1, 4)      0           permute_21[0][0]                 
__________________________________________________________________________________________________
loc_reshape_3 (Reshape)         (16, 150, 1, 4)      0           permute_22[0][0]                 
__________________________________________________________________________________________________
loc_reshape_4 (Reshape)         (16, 54, 1, 4)       0           permute_23[0][0]                 
__________________________________________________________________________________________________
loc_reshape_5 (Reshape)         (16, 24, 1, 4)       0           permute_24[0][0]                 
__________________________________________________________________________________________________
anchor_reshape_0 (Reshape)      (16, 8664, 1, 8)     0           ssd_anchor_0[0][0]               
__________________________________________________________________________________________________
anchor_reshape_1 (Reshape)      (16, 2166, 1, 8)     0           ssd_anchor_1[0][0]               
__________________________________________________________________________________________________
anchor_reshape_2 (Reshape)      (16, 600, 1, 8)      0           ssd_anchor_2[0][0]               
__________________________________________________________________________________________________
anchor_reshape_3 (Reshape)      (16, 150, 1, 8)      0           ssd_anchor_3[0][0]               
__________________________________________________________________________________________________
anchor_reshape_4 (Reshape)      (16, 54, 1, 8)       0           ssd_anchor_4[0][0]               
__________________________________________________________________________________________________
anchor_reshape_5 (Reshape)      (16, 24, 1, 8)       0           ssd_anchor_5[0][0]               
__________________________________________________________________________________________________
mbox_conf_softmax (Permute)     (16, 11658, 1, 4)    0           mbox_conf_softmax_[0][0]         
__________________________________________________________________________________________________
mbox_loc (Concatenate)          (16, 11658, 1, 4)    0           loc_reshape_0[0][0]              
                                                                 loc_reshape_1[0][0]              
                                                                 loc_reshape_2[0][0]              
                                                                 loc_reshape_3[0][0]              
                                                                 loc_reshape_4[0][0]              
                                                                 loc_reshape_5[0][0]              
__________________________________________________________________________________________________
mbox_priorbox (Concatenate)     (16, 11658, 1, 8)    0           anchor_reshape_0[0][0]           
                                                                 anchor_reshape_1[0][0]           
                                                                 anchor_reshape_2[0][0]           
                                                                 anchor_reshape_3[0][0]           
                                                                 anchor_reshape_4[0][0]           
                                                                 anchor_reshape_5[0][0]           
__________________________________________________________________________________________________
concatenate_2 (Concatenate)     (16, 11658, 1, 16)   0           mbox_conf_softmax[0][0]          
                                                                 mbox_loc[0][0]                   
                                                                 mbox_priorbox[0][0]              
__________________________________________________________________________________________________
ssd_predictions (Reshape)       (16, 11658, 16)      0           concatenate_2[0][0]              
==================================================================================================
Total params: 1,265,824
Trainable params: 1,247,808
Non-trainable params: 18,016
__________________________________________________________________________________________________
2023-10-24 05:58:36,455 [TAO Toolkit] [INFO] __main__ 356: Number of images in the training dataset:	  6733
2023-10-24 05:58:36,455 [TAO Toolkit] [INFO] __main__ 358: Number of images in the validation dataset:	   748

2023-10-24 05:58:37,017 [TAO Toolkit] [INFO] nvidia_tao_tf1.cv.common.logging.logging 197: Log file already exists at /workspace/tao-experiments/ssd/experiment_dir_unpruned_mobilenet/status.json
2023-10-24 05:58:41,686 [TAO Toolkit] [INFO] root 2102: Starting Training Loop.
Epoch 1/80
  2/421 [..............................] - ETA: 49:53 - loss: 112.3073  /usr/local/lib/python3.8/dist-packages/keras/callbacks.py:120: UserWarning: Method on_batch_end() is slow compared to the batch update (2.125494). Check your callbacks.
  warnings.warn('Method on_batch_end() is slow compared '
421/421 [==============================] - 49s 116ms/step - loss: 43.1540
[1698127174.677111] [290444ac5bd1:240  :f]        vfs_fuse.c:424  UCX  WARN  failed to connect to vfs socket '': Invalid argument

Epoch 00001: saving model to /workspace/tao-experiments/ssd/experiment_dir_unpruned_mobilenet/weights/ssd_mobilenet_v2_epoch_001.hdf5
2023-10-24 05:59:47,440 [TAO Toolkit] [INFO] root 2102: Training loop in progress
Epoch 2/80
421/421 [==============================] - 35s 83ms/step - loss: 12.0487

Epoch 00002: saving model to /workspace/tao-experiments/ssd/experiment_dir_unpruned_mobilenet/weights/ssd_mobilenet_v2_epoch_002.hdf5
2023-10-24 06:00:23,925 [TAO Toolkit] [INFO] root 2102: Training loop in progress
Epoch 3/80
421/421 [==============================] - 35s 82ms/step - loss: 8.7104

Epoch 00003: saving model to /workspace/tao-experiments/ssd/experiment_dir_unpruned_mobilenet/weights/ssd_mobilenet_v2_epoch_003.hdf5
2023-10-24 06:00:58,823 [TAO Toolkit] [INFO] root 2102: Training loop in progress
Epoch 4/80
421/421 [==============================] - 35s 82ms/step - loss: 7.8193

Epoch 00004: saving model to /workspace/tao-experiments/ssd/experiment_dir_unpruned_mobilenet/weights/ssd_mobilenet_v2_epoch_004.hdf5
2023-10-24 06:01:33,848 [TAO Toolkit] [INFO] root 2102: Training loop in progress
Epoch 5/80
421/421 [==============================] - 34s 82ms/step - loss: 7.3644

Epoch 00005: saving model to /workspace/tao-experiments/ssd/experiment_dir_unpruned_mobilenet/weights/ssd_mobilenet_v2_epoch_005.hdf5
2023-10-24 06:02:08,564 [TAO Toolkit] [INFO] root 2102: Training loop in progress
Epoch 6/80
421/421 [==============================] - 34s 82ms/step - loss: 7.0480

Epoch 00006: saving model to /workspace/tao-experiments/ssd/experiment_dir_unpruned_mobilenet/weights/ssd_mobilenet_v2_epoch_006.hdf5
2023-10-24 06:02:43,243 [TAO Toolkit] [INFO] root 2102: Training loop in progress
Epoch 7/80
421/421 [==============================] - 35s 82ms/step - loss: 6.7710

Epoch 00007: saving model to /workspace/tao-experiments/ssd/experiment_dir_unpruned_mobilenet/weights/ssd_mobilenet_v2_epoch_007.hdf5
2023-10-24 06:03:18,095 [TAO Toolkit] [INFO] root 2102: Training loop in progress
Epoch 8/80
421/421 [==============================] - 34s 82ms/step - loss: 12.0559

Epoch 00008: saving model to /workspace/tao-experiments/ssd/experiment_dir_unpruned_mobilenet/weights/ssd_mobilenet_v2_epoch_008.hdf5
2023-10-24 06:03:52,815 [TAO Toolkit] [INFO] root 2102: Training loop in progress
Epoch 9/80
421/421 [==============================] - 35s 82ms/step - loss: 17.2156

Epoch 00009: saving model to /workspace/tao-experiments/ssd/experiment_dir_unpruned_mobilenet/weights/ssd_mobilenet_v2_epoch_009.hdf5
2023-10-24 06:04:27,644 [TAO Toolkit] [INFO] root 2102: Training loop in progress
Epoch 10/80
421/421 [==============================] - 35s 82ms/step - loss: 17.2336

Epoch 00010: saving model to /workspace/tao-experiments/ssd/experiment_dir_unpruned_mobilenet/weights/ssd_mobilenet_v2_epoch_010.hdf5
Producing predictions: 100%|████████████████████| 47/47 [00:10<00:00,  4.65it/s]
2023-10-24 06:05:15,089 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:15,206 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:15,295 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:15,495 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:15,540 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:15,572 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:15,796 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:15,847 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:15,864 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:15,865 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:15,931 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:15,937 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:16,172 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:16,267 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:16,518 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:16,523 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:16,558 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.

2023-10-24 06:05:16,621 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:16,633 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:16,666 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:16,860 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:17,153 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:17,220 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:17,314 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:17,320 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:17,495 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:17,500 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:17,583 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:17,634 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:17,676 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:17,789 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:17,799 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:17,912 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:17,967 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:18,000 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:18,027 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:18,136 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:18,212 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:18,263 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:18,336 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:18,347 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:18,350 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:18,372 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:18,378 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:18,419 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:18,424 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:18,458 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:18,662 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.

2023-10-24 06:05:18,759 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:18,764 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:18,812 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:18,846 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:18,962 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:19,037 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:19,085 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:19,093 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:19,130 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:19,267 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:19,275 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:19,293 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:19,316 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:19,324 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
2023-10-24 06:05:19,332 [TAO Toolkit] [WARNING] nvidia_tao_tf1.cv.common.evaluator.ap_evaluator 124: Got label marked as difficult(occlusion > 0), please set occlusion field in KITTI label to 0, if you want to include it in mAP calculation during validation/evaluation.
Start to calculate AP for each class
*******************************
car           AP    0.0001
cyclist       AP    0.0
pedestrian    AP    0.0
              mAP   3e-05
*******************************
Validation loss: 13606.16414605615
2023-10-24 06:05:19,699 [TAO Toolkit] [INFO] root 2102: Evaluation metrics generated.
2023-10-24 06:05:19,702 [TAO Toolkit] [INFO] root 2102: Training loop in progress
Epoch 11/80
 50/421 [==>...........................] - ETA: 32s - loss: nan              Batch 49: Invalid loss, terminating training

Epoch 00011: saving model to /workspace/tao-experiments/ssd/experiment_dir_unpruned_mobilenet/weights/ssd_mobilenet_v2_epoch_011.hdf5
2023-10-24 06:05:24,384 [TAO Toolkit] [INFO] root 2102: Training loop in progress
2023-10-24 06:05:24,384 [TAO Toolkit] [INFO] root 2102: Training loop complete.
2023-10-24 06:05:24,385 [TAO Toolkit] [INFO] root 2102: Training finished successfully.
2023-10-24 06:05:24,389 [TAO Toolkit] [INFO] __main__ 569: Training finished successfully.
Execution status: PASS
2023-10-24 08:05:31,296 [TAO Toolkit] [INFO] nvidia_tao_cli.components.docker_handler.docker_handler 337: Stopping container.
11

Yes, it is nan issue.
Please set a lower max_learning_rate or lower batch_size_per_gpu and retry.

1 Like
12

Thanks!

It finally worked with:

min_learning_rate: 1e-5
max_learning_rate: 1e-3

1 Like
13

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.