Accuracy goes to 0% when Pruning PeopleNet 2.6

I encounter a problem with the pruning of the PeopleNet_2.6 model, I trained the model with a dataset of person, the model after training has an accuracy of 88%.

tao detectnet_v2 evaluate -e /raid/train_experiment/tao_experiments/person/person-interieur-fevrier-with-v2.6/model_pruned/result-after-pruning/experiment_spec.txt -m /raid/train_experiment/tao_experiments/person/person-interieur-fevrier-with-v2.6/model.step-9345.tlt -k tlt_encode

Total params: 22,240,663
Trainable params: 13,661,445
Non-trainable params: 8,579,218
__________________________________________________________________________________________________
2023-02-17 15:41:49,874 [INFO] root: Model constructed.
2023-02-17 15:41:49,875 [INFO] root: {
    "size": 85.49764251708984,
    "param_count": 22.240663
}

INFO:tensorflow:Graph was finalized.
2023-02-17 15:41:51,906 [INFO] tensorflow: Graph was finalized.
INFO:tensorflow:Running local_init_op.
2023-02-17 15:41:54,850 [INFO] tensorflow: Running local_init_op.
INFO:tensorflow:Done running local_init_op.
2023-02-17 15:41:55,577 [INFO] tensorflow: Done running local_init_op.
2023-02-17 15:41:57,866 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 43, 0.00s/step
2023-02-17 15:42:09,880 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 43, 1.20s/step
2023-02-17 15:42:15,135 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 43, 0.53s/step
2023-02-17 15:42:20,726 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 43, 0.56s/step
2023-02-17 15:42:26,841 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 43, 0.61s/step
Matching predictions to ground truth, class 1/1.: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20856/20856 [00:05<00:00, 3798.46it/s]
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

2023-02-17 15:42:35,811 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.

2023-02-17 15:42:35,812 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.


Validation cost: 0.000890
Mean average_precision (in %): 85.6119

class name      average precision (in %)
------------  --------------------------
person                           85.6119

Median Inference Time: 0.011561
2023-02-17 15:42:36,067 [INFO] __main__: Evaluation complete.
Time taken to run __main__:main: 0:01:04.549897.
2023-02-17 16:42:39,510 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

But when I prune it with different threshold, as soon as it loses a few parameters its accuracy goes to 0%.
With a Pruning ratio = 0.903 i had an accuracy of 0.00250412% and with a Pruning ratio = 0.9612 i had an accuracy of 1.27705%

tao detectnet_v2 evaluate -e /raid/train_experiment/tao_experiments/person/person-interieur-fevrier-with-v2.6/model_pruned/result-after-pruning/experiment_spec.txt -m /raid/train_experiment/tao_experiments/person/person-interieur-fevrier-with-v2.6/model_pruned/model_v1_pruned_0.3.tlt -k tlt_encode
 
==================================================================================================
Total params: 20,104,975
Trainable params: 11,527,293
Non-trainable params: 8,577,682
__________________________________________________________________________________________________
2023-02-17 16:05:03,074 [INFO] root: Model constructed.
2023-02-17 16:05:03,074 [INFO] root: {
    "size": 77.35216522216797,
    "param_count": 20.104975
}

INFO:tensorflow:Graph was finalized.
2023-02-17 16:05:04,676 [INFO] tensorflow: Graph was finalized.
INFO:tensorflow:Running local_init_op.
2023-02-17 16:05:06,658 [INFO] tensorflow: Running local_init_op.
INFO:tensorflow:Done running local_init_op.
2023-02-17 16:05:07,262 [INFO] tensorflow: Done running local_init_op.
2023-02-17 16:05:09,576 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 43, 0.00s/step
2023-02-17 16:06:17,605 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 43, 6.80s/step
2023-02-17 16:07:18,499 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 43, 6.09s/step
2023-02-17 16:08:19,844 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 43, 6.13s/step
2023-02-17 16:09:20,409 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 43, 6.06s/step
Matching predictions to ground truth, class 1/1.: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 220636/220636 [00:19<00:00, 11130.96it/s]
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

2023-02-17 16:10:05,874 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.

2023-02-17 16:10:05,875 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.


Validation cost: 0.009478
Mean average_precision (in %): 0.0025

class name      average precision (in %)
------------  --------------------------
person                        0.00250412

Median Inference Time: 0.010796
2023-02-17 16:10:06,016 [INFO] __main__: Evaluation complete.
Time taken to run __main__:main: 0:05:15.901036.
2023-02-17 17:10:08,466 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.
tao detectnet_v2 evaluate -e /raid/train_experiment/tao_experiments/person/person-interieur-fevrier-with-v2.6/model_pruned/result-after-pruning/experiment_spec.txt -m /raid/train_experiment/tao_experiments/person/person-interieur-fevrier-with-v2.6/model_pruned/model_v1_pruned_0.2.tlt -k tlt_encode

Total params: 21,377,983
Trainable params: 12,799,437
Non-trainable params: 8,578,546
__________________________________________________________________________________________________

INFO:tensorflow:Graph was finalized.
2023-02-17 16:15:44,754 [INFO] tensorflow: Graph was finalized.
INFO:tensorflow:Running local_init_op.
2023-02-17 16:15:46,777 [INFO] tensorflow: Running local_init_op.
INFO:tensorflow:Done running local_init_op.
2023-02-17 16:15:47,425 [INFO] tensorflow: Done running local_init_op.
2023-02-17 16:15:49,875 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 43, 0.00s/step
2023-02-17 16:16:04,048 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 43, 1.42s/step
2023-02-17 16:16:11,245 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 43, 0.72s/step
2023-02-17 16:16:18,388 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 43, 0.71s/step
2023-02-17 16:16:25,761 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 43, 0.74s/step
Matching predictions to ground truth, class 1/1.: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 42311/42311 [00:05<00:00, 8077.39it/s]
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

2023-02-17 16:16:35,157 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.

2023-02-17 16:16:35,158 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.


Validation cost: 0.001681
Mean average_precision (in %): 1.2771

class name      average precision (in %)
------------  --------------------------
person                           1.27705

Median Inference Time: 0.010996
2023-02-17 16:16:35,301 [INFO] __main__: Evaluation complete.
Time taken to run __main__:main: 0:01:05.641826.
2023-02-17 17:16:37,855 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

I don’t have this problem by training the Peoplenet 2.1 model that I was using before. I trained it with the same dataset and the same spec file, I just changed the path of the pretrained model, I only lose 3.2% accuracy with a Pruning ratio = 0.098.

model_config {
  arch: "resnet"
  pretrained_model_file: "/home/tao_experiments/nvidia_pretrained/peopleNet/v2.1/unpruned/resnet34_peoplenet.tlt"
  freeze_blocks: 0
  freeze_blocks: 1
  freeze_blocks: 2
  freeze_blocks: 3
  all_projections: True
  num_layers: 34
  use_pooling: False
  use_batch_norm: True
  dropout_rate: 0.15
  training_precision: {
    backend_floatx: FLOAT32 
  }
  objective_set: {
    cov {}
    bbox {
      scale: 35.0
      offset: 0.5
    }
  }
}
############################################################
bbox_rasterizer_config {
  target_class_config {
    key: "person"
    value: {
      cov_center_x: 0.5
      cov_center_y: 0.5
      cov_radius_x: 0.4
      cov_radius_y: 0.4
      bbox_min_radius: 1.0
    }
  }
  deadzone_radius: 0.67
}
############################################################
postprocessing_config {
  target_class_config {
    key: "person"
    value: {
      clustering_config {
        coverage_threshold: 0.005
        dbscan_eps: 0.15
        dbscan_min_samples: 0.05
        minimum_bounding_box_height: 20
      }
    }
  }
}
############################################################
cost_function_config {
  target_classes {
    name: "person"
    class_weight: 1.0
    coverage_foreground_weight: 0.05
    objectives {
      name: "cov"
      initial_weight: 1.0
      weight_target: 1.0
    }
    objectives {
      name: "bbox"
      initial_weight: 10.0
      weight_target: 1.0
    }
  }
  enable_autoweighting: True
  max_objective_weight: 0.9999
  min_objective_weight: 0.0001
}
############################################################
training_config {
  batch_size_per_gpu: 32
  num_epochs: 80
  enable_qat: True
  checkpoint_interval: 5
  learning_rate {
    soft_start_annealing_schedule {
      min_learning_rate: 2e-6#7e-7
      max_learning_rate: 7e-4#7e-5
      soft_start: 0.1
      annealing: 0.7
    }
  }
  regularizer {
    type: L1
    weight: 3e-9
  }
  optimizer {
    adam {
      epsilon: 1e-08
      beta1: 0.9
      beta2: 0.999
    }
  }
  cost_scaling {
    enabled: False
    initial_exponent: 20.0
    increment: 0.005
    decrement: 1.0
  }
}
############################################################
augmentation_config {
  preprocessing {
    output_image_width: 960
    output_image_height: 544
    output_image_channel: 3
    min_bbox_width: 1.0
    min_bbox_height: 1.0
  }
  spatial_augmentation {

    hflip_probability: 0.5
    vflip_probability: 0.0
    zoom_min: 1.0
    zoom_max: 1.0
    translate_max_x: 8.0
    translate_max_y: 8.0
  }
  color_augmentation {
    color_shift_stddev: 0.0
    hue_rotation_max: 25.0
    saturation_shift_max: 0.2
    contrast_scale_max: 0.1
    contrast_center: 0.5
  }
}
############################################################
evaluation_config {
  average_precision_mode: INTEGRATE
  validation_period_during_training: 5
  first_validation_epoch: 5
  minimum_detection_ground_truth_overlap {
    key: "person"
    value: 0.6
  }
  evaluation_box_config {
    key: "person"
    value {
      minimum_height: 4
      maximum_height: 9999
      minimum_width: 4
      maximum_width: 9999
    }
  }
}
############################################################
dataset_config {
  data_sources: {
    tfrecords_path: "/raid/dataset/dataset_for_tao2/Personne_interieur/all/tfrecords/trainval*"
    image_directory_path: "/raid/dataset/dataset_for_tao2/Personne_interieur/all/"
  }
  image_extension: "jpg"
  target_class_mapping {
      key: "person"
      value: "person"
  }
  validation_fold: 0
}
tao detectnet_v2 evaluate -e /raid/train_experiment/tao_experiments/person/person-interieur-fevrier-with-v2.6/model_pruned/result-after-pruning/experiment_spec.txt -m /raid/train_experiment/tao_experiments/person/person-interieur-fevrier-with-v2.1/model.step-9345.tlt -k tlt_encode

Total params: 22,240,663
Trainable params: 13,661,445
Non-trainable params: 8,579,218
__________________________________________________________________________________________________
2023-02-17 13:41:24,420 [INFO] root: Model constructed.
2023-02-17 13:41:24,420 [INFO] root: {
    "size": 85.49764251708984,
    "param_count": 22.240663
}


INFO:tensorflow:Graph was finalized.
2023-02-17 13:41:25,920 [INFO] tensorflow: Graph was finalized.
INFO:tensorflow:Running local_init_op.
2023-02-17 13:41:27,790 [INFO] tensorflow: Running local_init_op.
INFO:tensorflow:Done running local_init_op.
2023-02-17 13:41:28,346 [INFO] tensorflow: Done running local_init_op.
2023-02-17 13:41:30,479 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 43, 0.00s/step
2023-02-17 13:41:40,638 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 43, 1.02s/step
2023-02-17 13:41:45,206 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 43, 0.46s/step
2023-02-17 13:41:50,073 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 43, 0.49s/step
2023-02-17 13:41:54,853 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 43, 0.48s/step
Matching predictions to ground truth, class 1/1.: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16973/16973 [00:02<00:00, 7503.15it/s]
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

2023-02-17 13:41:59,424 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.

2023-02-17 13:41:59,425 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.


Validation cost: 0.000859
Mean average_precision (in %): 86.0512

class name      average precision (in %)
------------  --------------------------
person                           86.0512

Median Inference Time: 0.010335
2023-02-17 13:41:59,566 [INFO] __main__: Evaluation complete.
Time taken to run __main__:main: 0:00:47.573606.
2023-02-17 14:42:01,819 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.
tao detectnet_v2 evaluate -e /raid/train_experiment/tao_experiments/person/person-interieur-fevrier-with-v2.6/model_pruned/result-after-pruning/experiment_spec.txt -m /raid/train_experiment/tao_experiments/person/person-interieur-fevrier-with-v2.1/pruned/model_v1_pruned_0.1.tlt -k tlt_encode

Total params: 2,180,711
Trainable params: 598,053
Non-trainable params: 1,582,658
__________________________________________________________________________________________________
2023-02-17 13:50:36,967 [INFO] root: Model constructed.
2023-02-17 13:50:36,968 [INFO] root: {
    "size": 8.975570678710938,
    "param_count": 2.180711
}


INFO:tensorflow:Graph was finalized.
2023-02-17 13:50:38,474 [INFO] tensorflow: Graph was finalized.
INFO:tensorflow:Running local_init_op.
2023-02-17 13:50:40,304 [INFO] tensorflow: Running local_init_op.
INFO:tensorflow:Done running local_init_op.
2023-02-17 13:50:40,898 [INFO] tensorflow: Done running local_init_op.
2023-02-17 13:50:43,135 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 43, 0.00s/step
2023-02-17 13:50:52,871 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 43, 0.97s/step
2023-02-17 13:50:56,329 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 43, 0.35s/step
2023-02-17 13:50:59,782 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 43, 0.35s/step
2023-02-17 13:51:03,381 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 43, 0.36s/step
Matching predictions to ground truth, class 1/1.: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 16504/16504 [00:02<00:00, 7493.61it/s]
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

2023-02-17 13:51:07,549 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.

2023-02-17 13:51:07,549 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.


Validation cost: 0.000928
Mean average_precision (in %): 82.8259

class name      average precision (in %)
------------  --------------------------
person                           82.8259

Median Inference Time: 0.006414
2023-02-17 13:51:07,680 [INFO] __main__: Evaluation complete.
Time taken to run __main__:main: 0:00:42.552095.
2023-02-17 14:51:10,042 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

Is there another method to prune the peopleNet 2.6 model?

Thank you !

May I know that if you run retraining after pruning?

Why you run with pruning ratio=0.098 in Peoplenet 2.1 but use much different pruning ratio in peoplenet 2.6?

Thank you for your answer, yes of course I retrained the model after the pruning I had an accuracy of 33% which is very different from the model before the prune 88%.
The problem is not the training after the pruning pruning but it’s the prune itself, even if it is not retrained the accuracy should not go to 0% after the prune. The PeopleNet 2.1 model when I prune it and even if it is not retrained it keeps an accuracy of 82%.

I tested the pruning of the two models with different ratios, the people net 2.1 model even if it keeps only 9% of its parameters it keeps an accuracy of 82% (without retraining) while the peopleNet 2.6 model even if it keeps 98% of its weights, its accuracy drops to 0% , so even when I change the pruning ratio of peopleNet 2.6, the accuracy stays at 0%.

Can you run inference directly with the pruned 2.6 model to check the inference result?

Thank you for your answer, yes I launched the “tao detectnet_v2 evaluate” method with the model 2.6 pruned and retrained here are the results:

INFO:tensorflow:Graph was finalized.
2023-02-28 21:22:14,560 [INFO] tensorflow: Graph was finalized.
INFO:tensorflow:Running local_init_op.
2023-02-28 21:22:17,386 [INFO] tensorflow: Running local_init_op.
INFO:tensorflow:Done running local_init_op.
2023-02-28 21:22:18,253 [INFO] tensorflow: Done running local_init_op.
2023-02-28 21:22:21,866 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 43, 0.00s/step
2023-02-28 21:22:43,733 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 43, 2.19s/step
2023-02-28 21:22:53,295 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 43, 0.96s/step
2023-02-28 21:23:02,991 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 43, 0.97s/step
2023-02-28 21:23:13,473 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 43, 1.05s/step
Matching predictions to ground truth, class 1/1.: 100%|█████████████████████████████████████████| 73217/73217 [00:16<00:00, 4360.86it/s]
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

2023-02-28 21:23:39,193 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.

2023-02-28 21:23:39,193 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.


Validation cost: 0.001666
Mean average_precision (in %): 32.1748

class name      average precision (in %)
------------  --------------------------
person                           32.1748

Median Inference Time: 0.008847
2023-02-28 21:23:39,378 [INFO] __main__: Evaluation complete.
Time taken to run __main__:main: 0:01:48.015205.
2023-02-28 22:23:42,942 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

And I also launched the method with the model 2.1 pruned and retrained here are the results:

INFO:tensorflow:Graph was finalized.
2023-02-28 21:25:43,946 [INFO] tensorflow: Graph was finalized.
INFO:tensorflow:Running local_init_op.
2023-02-28 21:25:48,088 [INFO] tensorflow: Running local_init_op.
INFO:tensorflow:Done running local_init_op.
2023-02-28 21:25:49,067 [INFO] tensorflow: Done running local_init_op.
2023-02-28 21:25:52,944 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 43, 0.00s/step
2023-02-28 21:26:10,512 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 43, 1.76s/step
2023-02-28 21:26:16,496 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 43, 0.60s/step
2023-02-28 21:26:22,345 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 43, 0.58s/step
2023-02-28 21:26:28,434 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 43, 0.61s/step
Matching predictions to ground truth, class 1/1.: 100%|█████████████████████████████████████████| 16507/16507 [00:04<00:00, 3756.76it/s]
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

2023-02-28 21:26:36,347 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.

2023-02-28 21:26:36,348 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.


Validation cost: 0.000928
Mean average_precision (in %): 82.8377

class name      average precision (in %)
------------  --------------------------
person                           82.8377

Median Inference Time: 0.009809
2023-02-28 21:26:36,553 [INFO] __main__: Evaluation complete.
Time taken to run __main__:main: 0:01:13.800811.
2023-02-28 22:26:40,428 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

Both models are trained with the same dataset and the same parmeters.

From the model info during training or evaluation, we can find that Peoplenet2.6 is a QAT-trained model. Peoplenet2.1 and Peoplenet2.5 is not trained with QAT-enabled.
For 2.6 version, please follow https://developer.nvidia.com/blog/improving-int8-accuracy-using-quantization-aware-training-and-tao-toolkit/ to run retraining.

  • Include the pruned model as the pretrained weights.
  • Set the load_graph flag to true, so that the newly pruned model structure is imported.
  • update the training_config component of the re-training spec file to set the enable_qat flag.

More, you can also use v2.5 model(wget ‘https://api.ngc.nvidia.com/v2/models/nvidia/tao/peoplenet/versions/trainable_v2.5/files/resnet34_peoplenet.tlt’) which has similar accuracy as v2.6.

**Hi , **
Thank you for your answer, I tested the 2.6 model without “qat”, I have an accuracy before pruning of 93.198%

2023-03-21 15:51:19,754 [INFO] root: Model constructed.
2023-03-21 15:51:19,754 [INFO] root: {
    "size": 85.27497863769531,
    "param_count": 22.240581
}
WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:139: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.

2023-03-21 15:51:19,755 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:139: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.

WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.

2023-03-21 15:51:19,755 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.

WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.

2023-03-21 15:51:19,756 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.

WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

2023-03-21 15:51:19,757 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead.

2023-03-21 15:51:19,757 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead.

INFO:tensorflow:Graph was finalized.
2023-03-21 15:51:21,285 [INFO] tensorflow: Graph was finalized.
INFO:tensorflow:Running local_init_op.
2023-03-21 15:51:23,319 [INFO] tensorflow: Running local_init_op.
INFO:tensorflow:Done running local_init_op.
2023-03-21 15:51:23,970 [INFO] tensorflow: Done running local_init_op.
2023-03-21 15:51:26,402 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 43, 0.00s/step
2023-03-21 15:51:39,360 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 43, 1.30s/step
2023-03-21 15:51:43,766 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 43, 0.44s/step
2023-03-21 15:51:48,254 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 43, 0.45s/step
2023-03-21 15:51:52,871 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 43, 0.46s/step
Matching predictions to ground truth, class 1/1.: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 12613/12613 [00:02<00:00, 5114.34it/s]
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

2023-03-21 15:51:57,655 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.

2023-03-21 15:51:57,655 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.


Validation cost: 0.000747
Mean average_precision (in %): 93.1980

class name      average precision (in %)
------------  --------------------------
person                            93.198

Median Inference Time: 0.009030
2023-03-21 15:51:57,772 [INFO] __main__: Evaluation complete.
Time taken to run __main__:main: 0:00:53.198784.
2023-03-21 16:52:01,445 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

after pruning with a “pruning_ratio”: 0.77 , the accuracy dropped to 64.6609%

Total params: 17,237,325
Trainable params: 8,660,509
Non-trainable params: 8,576,816
__________________________________________________________________________________________________
2023-03-21 15:16:16,766 [INFO] root: Model constructed.
2023-03-21 15:16:16,766 [INFO] root: {
    "size": 66.19320678710938,
    "param_count": 17.237325
}
WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:139: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.

2023-03-21 15:16:16,768 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:139: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.

WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.

2023-03-21 15:16:16,768 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.

WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.

2023-03-21 15:16:16,768 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.

WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

2023-03-21 15:16:16,769 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead.

2023-03-21 15:16:16,770 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead.

INFO:tensorflow:Graph was finalized.
2023-03-21 15:16:18,129 [INFO] tensorflow: Graph was finalized.
INFO:tensorflow:Running local_init_op.
2023-03-21 15:16:20,231 [INFO] tensorflow: Running local_init_op.
INFO:tensorflow:Done running local_init_op.
2023-03-21 15:16:20,929 [INFO] tensorflow: Done running local_init_op.
2023-03-21 15:16:23,662 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 43, 0.00s/step
2023-03-21 15:17:32,393 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 43, 6.87s/step
2023-03-21 15:18:39,451 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 43, 6.71s/step
2023-03-21 15:19:46,517 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 43, 6.71s/step
2023-03-21 15:20:51,876 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 43, 6.54s/step
Matching predictions to ground truth, class 1/1.: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 438603/438603 [01:04<00:00, 6849.07it/s]
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

2023-03-21 15:22:40,554 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.

2023-03-21 15:22:40,555 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.


Validation cost: 0.006821
Mean average_precision (in %): 64.6609

class name      average precision (in %)
------------  --------------------------
person                           64.6609

Median Inference Time: 0.009428
2023-03-21 15:22:40,705 [INFO] __main__: Evaluation complete.
Time taken to run __main__:main: 0:06:38.482676.
2023-03-21 16:22:44,772 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

and with a “pruning_ratio”: 0.3923 the accurate dropped to 0.00154355%

2023-03-21 15:27:54,899 [INFO] root: Model constructed.
2023-03-21 15:27:54,899 [INFO] root: {
    "size": 33.724693298339844,
    "param_count": 8.726653
}
WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:139: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.

2023-03-21 15:27:54,901 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:139: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.

WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.

2023-03-21 15:27:54,902 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.

WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.

2023-03-21 15:27:54,903 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.

WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

2023-03-21 15:27:54,904 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead.

2023-03-21 15:27:54,905 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead.

INFO:tensorflow:Graph was finalized.
2023-03-21 15:27:56,709 [INFO] tensorflow: Graph was finalized.
INFO:tensorflow:Running local_init_op.
2023-03-21 15:27:58,612 [INFO] tensorflow: Running local_init_op.
INFO:tensorflow:Done running local_init_op.
2023-03-21 15:27:59,227 [INFO] tensorflow: Done running local_init_op.
2023-03-21 15:28:01,254 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 43, 0.00s/step
2023-03-21 15:30:46,272 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 43, 16.50s/step
2023-03-21 15:33:26,734 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 43, 16.05s/step
2023-03-21 15:35:56,554 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 43, 14.98s/step
2023-03-21 15:38:24,866 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 43, 14.83s/step
Matching predictions to ground truth, class 1/1.: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2795615/2795615 [05:00<00:00, 9314.17it/s]
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

2023-03-21 15:45:43,416 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.

2023-03-21 15:45:43,416 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.


Validation cost: 0.014070
Mean average_precision (in %): 0.0015

class name      average precision (in %)
------------  --------------------------
person                        0.00154355

Median Inference Time: 0.006761
2023-03-21 15:45:43,548 [INFO] __main__: Evaluation complete.
Time taken to run __main__:main: 0:18:05.823615.
2023-03-21 16:45:47,431 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

I also tested the 2.5 model, I had almost the same results. it was good after a “pruning_ratio”: 0.826 (accuracy of 89.6411) but with “pruning_ratio”: 0.39282 accuracy dropped to 0.0294864%


2023-03-22 15:19:48,171 [INFO] root: Model constructed.
2023-03-22 15:19:48,172 [INFO] root: {
    "size": 70.81352996826172,
    "param_count": 18.389871
}
WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:139: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.

2023-03-22 15:19:48,173 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:139: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.

WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.

2023-03-22 15:19:48,173 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.

WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.

2023-03-22 15:19:48,174 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.

WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

2023-03-22 15:19:48,175 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead.

2023-03-22 15:19:48,175 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead.

INFO:tensorflow:Graph was finalized.
2023-03-22 15:19:50,290 [INFO] tensorflow: Graph was finalized.
INFO:tensorflow:Running local_init_op.
2023-03-22 15:19:52,611 [INFO] tensorflow: Running local_init_op.
INFO:tensorflow:Done running local_init_op.
2023-03-22 15:19:53,251 [INFO] tensorflow: Done running local_init_op.
2023-03-22 15:19:56,309 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 43, 0.00s/step
2023-03-22 15:20:12,775 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 43, 1.65s/step
2023-03-22 15:20:20,496 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 43, 0.77s/step
2023-03-22 15:20:28,456 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 43, 0.80s/step
2023-03-22 15:20:36,311 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 43, 0.79s/step
Matching predictions to ground truth, class 1/1.: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 39319/39319 [00:04<00:00, 8012.61it/s]
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

2023-03-22 15:20:45,221 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.

2023-03-22 15:20:45,222 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.


Validation cost: 0.001016
Mean average_precision (in %): 89.6411

class name      average precision (in %)
------------  --------------------------
person                           89.6411

Median Inference Time: 0.010570
2023-03-22 15:20:45,362 [INFO] __main__: Evaluation complete.
Time taken to run __main__:main: 0:01:12.929661.
2023-03-22 16:20:49,253 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.
----------------------------------------
Total params: 8,736,461
Trainable params: 165,693
Non-trainable params: 8,570,768
__________________________________________________________________________________________________
2023-03-22 16:09:08,300 [INFO] root: Model constructed.
2023-03-22 16:09:08,300 [INFO] root: {
    "size": 33.761802673339844,
    "param_count": 8.736461
}
WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:139: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.

2023-03-22 16:09:08,302 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:139: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.

WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.

2023-03-22 16:09:08,303 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.

WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.

2023-03-22 16:09:08,303 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.

WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

2023-03-22 16:09:08,304 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.

WARNING:tensorflow:From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead.

2023-03-22 16:09:08,305 [WARNING] tensorflow: From /opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead.

INFO:tensorflow:Graph was finalized.
2023-03-22 16:09:09,869 [INFO] tensorflow: Graph was finalized.
INFO:tensorflow:Running local_init_op.
2023-03-22 16:09:12,186 [INFO] tensorflow: Running local_init_op.
INFO:tensorflow:Done running local_init_op.
2023-03-22 16:09:12,955 [INFO] tensorflow: Done running local_init_op.
2023-03-22 16:09:15,539 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 43, 0.00s/step
2023-03-22 16:12:31,812 [INFO] iva.detectnet_v2.evaluation.evaluation: step 10 / 43, 19.63s/step
2023-03-22 16:15:41,224 [INFO] iva.detectnet_v2.evaluation.evaluation: step 20 / 43, 18.94s/step
2023-03-22 16:18:35,981 [INFO] iva.detectnet_v2.evaluation.evaluation: step 30 / 43, 17.48s/step
2023-03-22 16:21:31,941 [INFO] iva.detectnet_v2.evaluation.evaluation: step 40 / 43, 17.60s/step
Matching predictions to ground truth, class 1/1.: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 2759208/2759208 [06:02<00:00, 7605.10it/s]
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

2023-03-22 16:30:21,257 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.

WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.

2023-03-22 16:30:21,257 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.


Validation cost: 0.045465
Mean average_precision (in %): 0.0295

class name      average precision (in %)
------------  --------------------------
person                         0.0294864

Median Inference Time: 0.006763
2023-03-22 16:30:21,386 [INFO] __main__: Evaluation complete.
Time taken to run __main__:main: 0:21:29.635331.
2023-03-22 17:30:25,188 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

Are the 2.6 and 2.5 models different from the others and shouldn’t be pruned too much like the others? because the others when we used a pruning ratio of 0.10, we kept the accuracy above 65% and the model only has a size of 8.54MB ? should we prune the 2.6 and 2.5 it partially by stage ?

Thank you

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

Difference models have different distribution of weights. So the same pruning ration may have different result during different models. Yes, for the first time, suggest to not prune too much, then run retraining. Then prune again, then retrain, etc.

For how to retrain the peopelent2.6 after pruning, please follow Improving INT8 Accuracy Using Quantization Aware Training and the NVIDIA TAO Toolkit | NVIDIA Technical Blog to run retraining.

  • Include the pruned model as the pretrained weights.
  • Set the load_graph flag to true, so that the newly pruned model structure is imported.
  • update the training_config component of the re-training spec file to set the enable_qat flag.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.