I can use the “tao-converter-jp46-trt8.0.1.6” to generate an engine that runs in deepstream-app 6.0. When I try and build an engine with the dla -u flag set to 0 or 1. The engine build fails.
I managed to download and convert the official yoloV4 model and it works fine. With that in mind, I noticed that both the official yoloV4 model and the peoplesegnet_resnet50 I mentioned above both use int8 with calibration file. So I exported my custom trained yoloV4 model as int8 and then I managed to create the engine file.
I can run it in the deepstream-app but it doesn’t produce bounding boxes. For the GPU or DLA version. Is there a common mistake? I can use int8 models trained on detectNet with no issues?
I have retrained the model and it works perfectly with int32. However I have the same issue with int8, there are no bboxes being displayed. I see this post had the same issue
As you suggested I can run the app successfully wit the video file in the sample.
“The ONNX operator number change on the optimization: 483 → 232
2022-02-04 12:29:02,398 [INFO] keras2onnx: The ONNX operator number change on the optimization: 483 → 232
2022-02-04 12:29:03,855 [INFO] iva.common.export.base_exporter: Generating a tensorfile with random tensor images. This may work well as a profiling tool, however, it may result in inaccurate results at inference. Please generate a tensorfile using the tlt-int8-tensorfile, or provide a custom directory of images for best performance.”
I have images in the directory. I will continue testing, any suggestions would be appreciated. I have around 10k images and batches set to 100. Do you think this could be the issue? I will try with 1000.
looks like I had an issue with the below. I have resolved it and now I dont get the same message with “provide a custom directory of images for best performance.” i will provide feedback on Monday.
If there is the log of "iva.common.export.base_exporter: Generating a tensorfile with random tensor images. ” , then there is something wrong in the commands “cal_image_dir” , “batch_size” and “batches”.
If after int-8 calibration the accuracy of the int-8 inferences seem to degrade, it could be because that there wasn’t enough data in the calibration tensorfile used to calibrate the model or, the training data is not entirely representative of your test images, and the calibration maybe incorrect. Therefore, you may either regenerate the calibration tensorfile with more batches of the training data and recalibrate the model, or calibrate the model on a few images from the test set.
2022-02-07 10:31:57,210 [INFO] root: Registry: [‘nvcr.io’]
2022-02-07 10:31:57,286 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3
2022-02-07 10:31:57,294 [WARNING] tlt.components.docker_handler.docker_handler:
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the “user”:“UID:GID” in the
DockerOptions portion of the “/home/ubuntu/.tao_mounts.json” file. You can obtain your
users UID and GID by using the “id -u” and “id -g” commands on the
terminal.
Using TensorFlow backend.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
2022-02-07 10:32:03,884 [INFO] root: Building exporter object.
2022-02-07 10:32:12,825 [INFO] root: Exporting the model.
2022-02-07 10:32:12,825 [INFO] root: Using input nodes: [‘Input’]
2022-02-07 10:32:12,825 [INFO] root: Using output nodes: [‘BatchedNMS’]
2022-02-07 10:32:12,825 [INFO] iva.common.export.keras_exporter: Using input nodes: [‘Input’]
2022-02-07 10:32:12,825 [INFO] iva.common.export.keras_exporter: Using output nodes: [‘BatchedNMS’]
The ONNX operator number change on the optimization: 483 → 232
2022-02-07 10:32:57,201 [INFO] keras2onnx: The ONNX operator number change on the optimization: 483 → 232
2022-02-07 10:32:58,620 [INFO] iva.common.export.base_exporter: Generating a tensorfile with random tensor images. This may work well as a profiling tool, however, it may result in inaccurate results at inference. Please generate a tensorfile using the tlt-int8-tensorfile, or provide a custom directory of images for best performance.
100%|█████████████████████████████████████| 1000/1000 [2:10:30<00:00, 7.83s/it]
2022-02-07 12:43:28,850 [INFO] iva.common.export.keras_exporter: Calibration takes time especially if number of batches is large.
2022-02-07 12:43:28,850 [INFO] root: Calibration takes time especially if number of batches is large.
2022-02-07 13:15:06,252 [INFO] iva.common.export.base_calibrator: Saving calibration cache (size 7084) to /workspace/tao-experiments/yolo_v4/export/cal.bin
2022-02-07 13:17:43,761 [INFO] root: Export complete.
2022-02-07 13:17:43,761 [INFO] root: {
“param_count”: 31.572441,
“size”: 121.07246398925781
}
2022-02-07 13:17:46,306 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.
My images are 1280x720 and I have the output set to
output_width: 960
output_height: 544
Could this be causing issues? I see from this post Network Image Input Resizing - #6 by Morganh That the dataloader either pads with zeros, or crops to fit to the output resolution.
Yes, the cal.bin is generated by training dataset.
If after int-8 calibration the accuracy of the int-8 inferences seem to degrade, it could be because there wasn’t enough data in the calibration tensorfile used to calibrate thee model or, the training data is not entirely representative of your test images, and the calibration maybe incorrect. Therefore, you may either regenerate the calibration tensorfile with more batches of the training data and recalibrate the model, or calibrate the model on a few images from the test set.