I have been trying to use the examples at /opt/nvidia/deepstream/deepstream-4.0/sources/objectDetector_Yolo/ and was able to successfully load a custom YOLOv3 model in all 3 precision modes (FP32, FP16 & INT8) when batch-size=1. When I tried changing the deepstream_app_config_yoloV3.txt & config_infer_primary_yoloV3.txt to set batch-size=4 with 4 input streams, the deepstream app builds model_b4_fp32.engine & model_b4_fp16.engine and runs when precision is set to FP32 & FP16. But when I set the precision to INT8 I see the below error:
Building the TensorRT Engine…
0:00:07.041273045 11552 0xf81eb90 ERROR nvinfer gstnvinfer.cpp:511:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:log(): Assertion failed: !empty()
…/builder/intInterval.cpp:25
Aborting…
0:00:07.044651085 11552 0xf81eb90 ERROR nvinfer gstnvinfer.cpp:511:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:log(): …/builder/intInterval.cpp (25) - Assertion Error in max: 0 (!empty())
Building engine failed!
I couldn’t find much on the web with the above error messages. I will appreciate any thoughts on what the problem could be and how it can be fixed.
build success log:
(102) conv-bn-leaky 256 x 76 x 76 128 x 76 x 76 61311070
(103) conv-bn-leaky 128 x 76 x 76 256 x 76 x 76 61607006
(104) conv-bn-leaky 256 x 76 x 76 128 x 76 x 76 61640286
(105) conv-bn-leaky 128 x 76 x 76 256 x 76 x 76 61936222
(106) conv-linear 256 x 76 x 76 255 x 76 x 76 62001757
(107) yolo 255 x 76 x 76 255 x 76 x 76 62001757
Output blob names :
yolo_83
yolo_95
yolo_107
Total number of layers: 257
Total number of layers on DLA: 0
Building the TensorRT Engine…
Building complete!
0:07:08.359672621 8988 0x1b0b8950 INFO nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary_gie_classifier> NvDsInferContext[UID 1]:generateTRTModel(): Storing the serialized cuda engine to file at /opt/nvidia/deepstream/deepstream-4.0/sources/objectDetector_Yolo/model_b4_int8.engine
Thank you so much for your reply. I was able to make some headway with your comments:
You were right with regards to INT8 batch-size4 model for yolov3. I was able to build this successfully as well. The error I reported earlier was happening with a custom yolov3 model.
I verified that I was unable to build my INT8 model with custom calibration table even with batch size = 1. I think I can rule out batch size as a factor.
Like I said earlier, I was able to build FP32 & FP16 modes of my custom model with batch size=4.
With the above information I tried building my custom yolov3 model with INT8 precision and batch size 4, but used the default yolov3-calibration.table.trt5.1 and it worked. So it appears my custom calibration table was somehow incompatible with the yolo deepstream example.
I used a modified version of onnx_to_tensorrt.py in /usr/src/tensorrt/samples/python/yolov3_onnx to produce an INT8 trt engine and a custom calibration table. I couldn’t use that int8 engine in the yolo deepstream sample and it looks like I can’t use the calibration table either. The documentation provided in https://docs.nvidia.com/metropolis/deepstream/Custom_YOLO_Model_in_the_DeepStream_YOLO_App.pdf says we can use our own calibration table for custom model. But I don’t know why I was seeing the above error message with a custom calibration table, the model (yolov3 config+weights) seems to be fine.
I noticed the default calibration table was generated by TensortRT 5.1 but the calibration table I created was with TensorRT 6.0. I can see the formats being different:
Hi @dilip.s,
You need to use the calibration table generated by the TensorRT release which is supported by DeepStream.
You can find which TensorRT is supported by DS in DS release note.