mAP of engine model in Deepstream with JP5.1.1 is much better than the one in Deepstream with JP5.1.2?

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

I am using Xavier NX. And I am testing 2 JP version 5.1.1, 5.1.2

I had qat onnx model. I converted this qat onnx model to qat engine for JP5.1.1 and 5.1.2.
I run inference and evaluate 2 qat engine with Python API, mAP of 2 qat engine was same.

But when I run inference and evaluate engine with Deepstream (DS6.2, TRT8.5.2 in both JP version, same source code), there was big mAP (~10%) between 2 engine models. mAP with JP5.1.1 is good, but mAP with JP5.1.2 is bad.
From my investigation, the issue is inference part with Deepstream (DS6.2, TRT8.5.2).

I also checked release note of JP5.1.2 and did not find any strange points.
Sorry, I can not share my models.

3

please refer to this link Jetson model Platform and OS Compatibility. DeepStream version needs to be consistent with Jetpack’s version.

4

@fanzh
Thanks for quick response.
What does GA means in 5.1 GA?
I dont see DS version for JP5.1.1 in the above link.

In this link, DS6.2 is compatible with JP5.1.2 JetPack SDK 5.1.2 | NVIDIA Developer, but in the above link DS6.3 is compatible with JP5.1.2. It makes me confused.

5
  1. 5.1 GA is one version of 5.1.
  2. please install the corresponding version of DS.
  3. DS6.2 is compatible with 5.1 GA.
6

@fanzh
I checked DS6.3 with JP5.1.2. mAP is bad. I am not clear the root cause here.

7

do you mean the same code will get different results on DS6.2+JP5.1.1 and DS6.2+JP5.1.2? from the doc 511 and 512, the TensorRT version is the same.

8

Yes. DS6.2+JP5.1.1 gives good mAP. DS6.2+JP5.1.2 and DS6.3+JP5.1.2 give bad mAP for COCO dataset. The mAP gap is about 10%. For other dataset, the big mAP gap is not.

9

what is the whole pipeline? how did you use DeepStream? it is the custom code? can you use deepstream sample to reproduce this issue?

10

I used Deepstream-app https://github.com/NVIDIA-AI-IOT/yolo_deepstream/tree/main/deepstream_yolo
Model is yolov7. Could you check it again please?. Sorry I can not share my model.

11

did you modify the code? can the issue be reproduced without code and model modification?

12

It is almost same. Config for yolov7 int8 engine. I run on Jetson Xavier board.

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
#0=RGB, 1=BGR
model-color-format=0
onnx-file=yolov7.onnx
model-engine-file=yolov7.onnx_b1_gpu0_int8.engine
labelfile-path=labels.txt
## 0=FP32, 1=INT8, 2=FP16 mode
network-mode=1
num-detected-classes=80
gie-unique-id=1
network-type=0
is-classifier=0
## 1=DBSCAN, 2=NMS, 3= DBSCAN+NMS Hybrid, 4 = None(No clustering)
cluster-mode=2
maintain-aspect-ratio=0
symmetric-padding=1
## Bilinear Interpolation
scaling-filter=1
#parse-bbox-func-name=NvDsInferParseCustomYoloV7
parse-bbox-func-name=NvDsInferParseCustomYoloV7_cuda
#disable-output-host-copy=0
disable-output-host-copy=1
custom-lib-path=nvdsinfer_custom_impl_Yolo/libnvdsinfer_custom_impl_Yolo.so
#scaling-compute-hw=0
## start from DS6.2
#crop-objects-to-roi-boundary=1


[class-attrs-all]
#nms-iou-threshold=0.3
#threshold=0.7
nms-iou-threshold=0.65
pre-cluster-threshold=0.001
topk=300

When running inference with JP5.1.2 + DS6.2 or JP5.1.2 + DS6.3, it outputs addional info compared JP5.1.1 + DS6.2

NvMMLiteBlockCreate : Block : BlockType = 256 
NvMMLiteBlockCreate : Block : BlockType = 256 
NvMMLiteBlockCreate : Block : BlockType = 256 
NvMMLiteBlockCreate : Block : BlockType = 256 
[JPEG Decode] BeginSequence Display WidthxHeight 640x480
NvMMLiteBlockCreate : Block : BlockType = 256 
[JPEG Decode] BeginSequence Display WidthxHeight 640x425
NvMMLiteBlockCreate : Block : BlockType = 256 
NvMMLiteBlockCreate : Block : BlockType = 256 
NvMMLiteBlockCreate : Block : BlockType = 256 
[JPEG Decode] BeginSequence Display WidthxHeight 427x640
[JPEG Decode] BeginSequence Display WidthxHeight 500x332
[JPEG Decode] BeginSequence Display WidthxHeight 640x480
[JPEG Decode] BeginSequence Display WidthxHeight 480x640
[JPEG Decode] BeginSequence Display WidthxHeight 640x427
[JPEG Decode] BeginSequence Display WidthxHeight 480x640
[JPEG Decode] BeginSequence Display WidthxHeight 640x425
NvMMLiteBlockCreate : Block : BlockType = 256 
[JPEG Decode] BeginSequence Display WidthxHeight 640x424

I used COCO images with YUV420 format to be supported in DS.
Do you think image decoding can be the root cause?

13

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

  1. can you try network-mode=0? maybe in8 calibration file is bonding to specific low-level.
  2. you can use this method to dump the Inference Input, then you will know if the inference inputs are the same.
14

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.