Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU) Jetson NX• DeepStream Version 5.0• JetPack Version (valid for Jetson only) 4.4 DP• TensorRT Version 7.1• NVIDIA GPU Driver Version (valid for GPU only)
I am running the sample in /opt/nvidia/deepstream/deepstream-5.0/sources/objectDetector_SSD
Both of the detection results are correct. I expect the int8 performance will be higher than fp16.
However, I found int8 and fp16 shows the similar performance, both is around 30fps.
Could you let me know why int8 has the same perf with fp16? and how to achieve higher fps with int8 than fp16?
mchi
May 18, 2020, 2:44pm
#3
Hi @andy.linluo
NOTE: To use INT8 mode, the INT8 calibration file for the SSD model needs to be
provided along with changing the network-mode to 1 in config_infer_primary_ssd.txt.
Refer to sampleUffSSD for running the sample in INT8 mode. The sample writes the
calibration cache to file "CalibrationTableSSD".
If you did’t provide INT8 calib, it will fall back to FP16, you should be able to find the related log in the output
@mchi , I did generate int8 calib file with
I randomly selected 500 images from coco validation dataset for calibration and successfully generate int8 calib cache. I believe it is correct because the output with sample images is correct.
Then I used the following command to generate the int8 engine file.
I found the size of the int8 engine file is a little larger than the fp16 engine file generated by deepstream. Is it normal? I assume int8 engine file should be much smaller than fp16 engine file.
Is there anything I did wrong?
mchi
May 28, 2020, 4:06am
#5
Hi @andy.linluo
$ ./trtexec --uff=sample_ssd_relu6.uff --output=NMS --uffInput=Input,3,300,300 --fp16 --workspace=450 --batch=18.09466 ms (end to end 8.10616 ms)
./trtexec --uff=sample_ssd_relu6.uff --output=NMS --uffInput=Input,3,300,300 --int8 --workspace=450 --batch=17.68774 ms (end to end 7.69861 ms)
mchi
May 28, 2020, 4:13am
#6
With “–dumpProfile” in the trtexec command as below, from the profile output, we can find that most of the inference time is on NMS plugin which time should be the same for FP16 and INT8, so we got the similar perf with FP16 and INT8.
$ ./trtexec --uff=sample_ssd_relu6.uff --output=NMS --uffInput=Input,3,300,300 --int8 --workspace=450 --batch=1 --dumpProfile
@mchi , thanks.
mchi
May 28, 2020, 2:55pm
#8
From TRT6, trt plugin supports to be implemented to support INT8, FP16 and fp32.
How many batch do you want to run? You could check the INT8 and FP16 perf with the real batch you want to run.
Thanks!
@mchi , where can I find the detailed info about trt plugins such as precision support?
mchi
May 28, 2020, 3:34pm
#10
@andy.linluo
Platform: Tesla T4
TRT verson: 7.0.0.11
Batch Size: 32
Int8 one iteration fp16 one iteration
total 20.18ms 27.40ms
NMS 7.22ms 7.78ms
Without NMS 12.96ms 19.62ms
I have verified large batch scenarios like batch=32, and find ratio of FPS for int8 and fp16 is about int8/fp16 = 1.5 (excluding NMS), int8 is 50% faster.
So, consider using larger batch size if possible.
1 Like