INT8 Yolo model conversion led to accuracy drop in deepstream

Hi,

Thanks for sharing the data with us.

Have you tried the model with TensorRT API directly?
If not, would you mind to give it a try?

This will help us to distinguish the issue is from Deepstream or TensorRT.

Thanks.

I tried. Two methods give very similar results with a tiny difference in mAP for less than 0.5%.

Hi,

Thanks for sharing the subset with us.
Could you also share the source for generating calibration file with us?

Thanks.

Here is the tensorrt API source, and the Deepstream source.

Hi,

We are checking this issue internally.
Will update more with you later.

Thanks.

Thanks. Looking forward to your update

Hi,

We check the calibration shared in this comment.

In general, TensorRT will merge/combine several layers together for acceleration (ex. conv+scale+activation).
However, the layer are calibrated without merging in your cache file.

Not sure if this causes some unexpected accuracy drop.
Would you mind to try the calibration tool shared in the below GitHub again:

We have verified that the cache files in the GitHub can output the detections correctly.

Thanks.

Thanks for your feedback!

I’ll give it a try.

After using the cache file generated from the recommended repo in the DS yolo app, the performance speed dropped significantly, to about 8fps.

It seems that tensorrt did not know how to perform INT8 quantization based on the give calibration cache so it ended up making a FLOAT32 or 16 engine.

I might have misunderstood some of your statements. So when you say

did you test it out in the deepstream yolo-app?

Thanks again for your help.

Hi,

Could you share your detailed procedure with us?

The INT8 is a configuration set by the user.
So the model will inference in INT8 mode if configure and cache provided correctly.

Ex.
config_infer_primary_yoloV3_tiny.txt

[property]
...
int8-calib-file=[the cache file generated above]
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=1
...

Thanks.

I followed the demo#5 to create an onnx file, and followed demo#6 to calibrate and get a calibration cache. Then I used the cache in the deepstream yolo-app as int8-calib-file.

Hi,

Could you share the .cfg, .weight, .onnx and the corresponding cache file with us?

More, we test the default YoloV3 Tiny model cache.
And can get the expected output result.

Please validate if this also works on your side.
Thanks.

Here are the files.

When I used tensorrt-demo generated caches within the repo, they all worked fine. When I moved the cache into deepstream, I got the following:

ERROR: [TRT]: Calibration failure occurred with no scaling factors detected. This could be due to no int8 calibrator or insufficient custom scales for network layers. Please see int8 sample to setup calibration correctly.
ERROR: [TRT]: Builder failed while configuring INT8 mode.
Building engine failed!

I also tried the yolov3-tiny cache as you suggested, and the same thing happened – it only works for the given repo and cannot be transferred to Deepstream. The error is the same as mentioned above.

Hi,

Thanks for sharing the model and cache.

We can reproduce this issue internally, and is checking.
Will get back to you late.

Hi,

We change the layer name 000_net into data in calib_yolov3-int8-608.bin.

TRT-7103-EntropyCalibration2
data: 3c010a14
...

And Deepstream can run the model with cache successfully.
Could you also give it a try?

Thanks.

Of course. I’ll work on it as soon as I can.

Thanks.

Hi,
It did work. However the accuracy only increased for about 1.2 percent, which means the INT8 quantization still caused about 6 percent accuracy drop. Is there any other way to improve this?

Hi,

Sorry for the late update.

To further improve INT8 accuracy, please check the below page for the Quantization Aware Training (QAT):

Thanks.

Hi,

Sorry for the late update.

To further improve INT8 accuracy, please check the below page for the Quantization Aware Training (QAT):

Thanks.

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.