• Hardware Platform Jetson
• DeepStream Version 6.3
• JetPack Version 5.1
• TensorRT Version 5.1
• Issue Type questions
Using pycocotools I calculated average precision of my YoloV5 model on a test set:
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.855
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.971
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.967
However using the engine file of the same model on Jetson on the same set gives these results:
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.747
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.924
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.889
Is such drop in precision expected when running on Jetson or should there be no changes in the results?
Is this calculation based on pytorch?
Do you mean the TensorRT engine? What is the TensorRT engine type? Is the test based on the same inferencing pipeline?
nikola2
4
Yes
I get same results using FP16 and FP32 engine.
YoloV5 uses symmetric padding and maintains aspect ratio, which was missing from my pipeline, now the results have improved:
Average Precision (AP) @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.833
Average Precision (AP) @[ IoU=0.50 | area= all | maxDets=100 ] = 0.972
Average Precision (AP) @[ IoU=0.75 | area= all | maxDets=100 ] = 0.964
There’s still ~2% difference for mAP@[0.5:0.95], is this expected?
You may check the nvinfer configurations as DeepStream SDK FAQ - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums
If the dataset is PNG/JPEG files, the extra RGB to NV12 format conversion will introduce difficiation into the pipeline.
Pytorch use OpenCV and Pytorch functions as decoding and processing. It is not guranteed the DeepStream is bit aligned as Pytorch and OpenCV.
nikola2
6
I have tested and confirmed this difference doesn’t affect my use case.