Batch Size Failure in Custom YOLOv3 INT8

Hi,

I have been trying to use the examples at /opt/nvidia/deepstream/deepstream-4.0/sources/objectDetector_Yolo/ and was able to successfully load a custom YOLOv3 model in all 3 precision modes (FP32, FP16 & INT8) when batch-size=1. When I tried changing the deepstream_app_config_yoloV3.txt & config_infer_primary_yoloV3.txt to set batch-size=4 with 4 input streams, the deepstream app builds model_b4_fp32.engine & model_b4_fp16.engine and runs when precision is set to FP32 & FP16. But when I set the precision to INT8 I see the below error:

Building the TensorRT Engine…
0:00:07.041273045 11552 0xf81eb90 ERROR nvinfer gstnvinfer.cpp:511:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:log(): Assertion failed: !empty()
…/builder/intInterval.cpp:25
Aborting…
0:00:07.044651085 11552 0xf81eb90 ERROR nvinfer gstnvinfer.cpp:511:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:log(): …/builder/intInterval.cpp (25) - Assertion Error in max: 0 (!empty())
Building engine failed!

I couldn’t find much on the web with the above error messages. I will appreciate any thoughts on what the problem could be and how it can be fixed.

Thanks,
Dilip.

Hi,
I can not repro your issue, here is my environments on Jetson Xavier:
1 get yolov3 files.
wget https://raw.githubusercontent.com/pjreddie/darknet/master/cfg/yolov3.cfg -q --show-progress
wget https://pjreddie.com/media/files/yolov3.weights -q --show-progress
2 build yolo library
export CUDA_VER=10.0
make -C nvdsinfer_custom_impl_Yolo/
3 modify deppstream config batch-size of source, streammux, pgie to 4,
4 run
deepstream-app -c deepstream_app_config_yoloV3.txt

build success log:
(102) conv-bn-leaky 256 x 76 x 76 128 x 76 x 76 61311070
(103) conv-bn-leaky 128 x 76 x 76 256 x 76 x 76 61607006
(104) conv-bn-leaky 256 x 76 x 76 128 x 76 x 76 61640286
(105) conv-bn-leaky 128 x 76 x 76 256 x 76 x 76 61936222
(106) conv-linear 256 x 76 x 76 255 x 76 x 76 62001757
(107) yolo 255 x 76 x 76 255 x 76 x 76 62001757
Output blob names :
yolo_83
yolo_95
yolo_107
Total number of layers: 257
Total number of layers on DLA: 0
Building the TensorRT Engine…
Building complete!
0:07:08.359672621 8988 0x1b0b8950 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:generateTRTModel(): Storing the serialized cuda engine to file at /opt/nvidia/deepstream/deepstream-4.0/sources/objectDetector_Yolo/model_b4_int8.engine

can you specify how can we repro your issue?

1 Like

Hello @amycao,

Thank you so much for your reply. I was able to make some headway with your comments:

  • You were right with regards to INT8 batch-size4 model for yolov3. I was able to build this successfully as well. The error I reported earlier was happening with a custom yolov3 model.
  • I verified that I was unable to build my INT8 model with custom calibration table even with batch size = 1. I think I can rule out batch size as a factor.
  • Like I said earlier, I was able to build FP32 & FP16 modes of my custom model with batch size=4.

With the above information I tried building my custom yolov3 model with INT8 precision and batch size 4, but used the default yolov3-calibration.table.trt5.1 and it worked. So it appears my custom calibration table was somehow incompatible with the yolo deepstream example.

I used a modified version of onnx_to_tensorrt.py in /usr/src/tensorrt/samples/python/yolov3_onnx to produce an INT8 trt engine and a custom calibration table. I couldn’t use that int8 engine in the yolo deepstream sample and it looks like I can’t use the calibration table either. The documentation provided in https://docs.nvidia.com/metropolis/deepstream/Custom_YOLO_Model_in_the_DeepStream_YOLO_App.pdf says we can use our own calibration table for custom model. But I don’t know why I was seeing the above error message with a custom calibration table, the model (yolov3 config+weights) seems to be fine.

I noticed the default calibration table was generated by TensortRT 5.1 but the calibration table I created was with TensorRT 6.0. I can see the formats being different:

TensorRT 5.1 calibration table:
TRT-5105-EntropyCalibration2
data: 40000889
(Unnamed Layer* 0) [Constant]_output: 4000890a
(Unnamed Layer* 1) [ElementWise]_output: 3c008912
(Unnamed Layer* 2) [Convolution]_output: 3c543871…

TensorRT 6.0 calibration table:
TRT-6001-EntropyCalibration
000_net: 3c010a14
001_convolutional_bn: 3e51ceba
001_convolutional_lrelu: 3dd30ae2
002_convolutional_bn: 3da8337a…

Can you confirm if a calibration table produced by tensorRT 6.0 is compatible with the yolo deepstream sample ?

Thanks,
Dilip.

Hi @dilip.s,
You need to use the calibration table generated by the TensorRT release which is supported by DeepStream.
You can find which TensorRT is supported by DS in DS release note.

Thanks!

1 Like

Hello mchi,

Thanks for the clarification. I will refer the TRT version compatibility with DeepStream in the release notes.

-Dilip.