Why mess if do INT8 optimizing with low-rank approximation SSD

haifengli · February 27, 2018, 10:08am

We implemented a SSD in TensorRT 3.0.2 with plugins(deploy_gie.prototxt), the performance is 46 fps with 704x448 input and INT8 optimized running at Tesla P4. But the result is mess and the bbox is useless once we do low-rank approximation together(deploy_gie_low_rank.prototxt). Could it be INT8 optimizing doesn’t work with 1x3 convolution kernel?
deploy_gie.proto.txt (15.3 KB)
deploy_gie_low_rank.proto.txt (33.7 KB)

AastaLLL · March 1, 2018, 7:19am

Hi,

Could you help to check if the result of low rank model is good on FLOAT mode?
Thanks.

haifengli · March 1, 2018, 7:30am

Hi,
AastaLLL, we are running the low rank model without INT8 optimizing and the result is correct, it is a detect cnn visible by deepstream playback module. Can we provide a demo program with .cpp, .caffemodel, labels.txt and demo.jpg and upload them to this topic?

Thanks.

AastaLLL · March 5, 2018, 7:13am

Hi,

Have you implemented the calibration for INT8 mode?

Thanks.

haifengli · March 5, 2018, 11:43am

Hi,

We have implemented the calibration for INT8 mode with 2 times, and it is very fast even IPlugin layer present.
I will discuss with Neo how to provide the extra information to get more help from you.

Thanks.

haifengli · March 14, 2018, 1:44am

Hi,

We are keep updating our .caffemodel file, when should we update the INT8 Calibration Table file, and how many images needed for doing the INT8 Calibration? We using batch 6 and 1000 images now.

Thanks.

AastaLLL · March 14, 2018, 9:28am

Hi,

We have received your information internally and are checking.
Will update to you later.

Thanks.

haifengli · March 14, 2018, 10:02am

Thanks for you help.

AastaLLL · March 15, 2018, 7:52am

Hi,

Could you please share the following information?

1. Which calibration type do you use? Legacy or entropy?
2. Could you share the calibration score for original/optimized VGG networks?

Thanks.

haifengli · March 15, 2018, 8:46am

Hi,

1.Entropy.
2.We used calibration with SSD network, we visualized bbox result with deepstream video live detect sample code, and it is not so obviously to calculate score. We need do some work to finish this.

Thanks.

AastaLLL · March 19, 2018, 7:21am

Hi,

Is there any alternative to check the accuracy of original and optimized calibration?
We want to make sure the calibration for optimized VGG is good first.

Thanks.

haifengli · March 19, 2018, 7:24pm

Hi,

We think the optimized ones lost accuracy, it generate meanless bboxs. We optimized SSD with pruning and the result is good.
Is there any small and common CNN can do INT8 and low-rank optimization together?

Thanks.

NeoSong · March 20, 2018, 9:22am

Dear AastaLLL,

We have done the calibration on non-optimized VGG network, and the INT8 network output is correct.
We did the same process on the low rank optimized VGG network, the INT8 network output is wrong.

According to the two procedure, we believe there is no problem with our calibration procedure, and we doubt:

maybe there is some problem in the INT8 calibration tool when dealing with Nx1 or 1xN conv layer.
maybe there is some problem in the TensorRT INT8 forward calculation when dealing with Nx1 or 1xN conv layer.

We did not do extra experiments on public dataset, so we can not supply you more evidence.

AastaLLL · March 21, 2018, 3:31am

Hi,

Thanks for your message.
We are trying to reproduce this issue with a simpler model to narrow down the root cause.

Will update information with you later.

haifengli · March 22, 2018, 3:53pm

Hi,

We hope there is some news information soon.

Thanks.

AastaLLL · March 23, 2018, 7:32am

Hi,

We found that the INT8 calibration is sensitive to the scale but still need some time to check.
Will update information with you later.

Thanks.

NeoSong · March 27, 2018, 7:17am

Dear AastaLLL,

Is there any updates?

AastaLLL · March 27, 2018, 7:27am

Hi,

Could you help us to reproduce this issue with LeNet and MNIST use case?

LeNet is a tiny model; it can help us to monitor the behavior of calibrator.
MNIST is a simpler use case without bounding box handler; it allows us to check the network output directly.

A: Test a standard MNIST LeNet model with FP32 mode ---------------> should be correct
B: Apply INT8 calibration on step-A ------------------------------------> should be correct
C: Test a low-rank optimized MNIST LeNet model with FP32 mode ----> should be correct
D: Apply INT8 calibration on step-C ------------------------------------> shoud reproduce the error

Thanks.

NeoSong · April 2, 2018, 11:09am

Dear AastaLLL,

OK, we provide you the new package in private, which contains state C and state D, accompany with full dataset, which you can re-do the INT8 calibration if you think the problem lies in INT8 calibration process.

AastaLLL · April 3, 2018, 8:23am

Hi,

Thanks for your help.
Could you share the data from the forum with a private message directly?

Thanks.