Low scores after converting working TAO yolov4 model to .plan

[joel42]

Please provide the following information when requesting support.

• Hardware : Train: T4, inference: Orin
• Network Type yolov4
• TLT Version :toolkit_version: 5.3.0
• Training spec file:
yolo_v4_train_resnet18_kitti_planes.txt (2.0 KB)

• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)

I have successfully trained a yolov4 model on my own data. When I do tao model yolo_v4 inference... using the resultant hdf5 file, I am getting good detections with probabilites in the [0,1] range. I am trying to serialize these models for use on a Jetson Orin. Because tao deploy does not work. It is my understanding that I need to first export as an onnx file, and use trtexec to convert the onnx file to a .plan file. To export as an onnx file I ran this on the T4 I used for training: tao model yolo_v4 export -m /workspace/tao-models/yolov4_resnet18_epoch_150-planes.hdf5 -o /workspace/tao-models/yolov4_resnet18_epoch_150-planes.onnx -e /workspace/tao-models/yolo_v4_train_resnet18_kitti_planes.txt --target_opset 12 --gen_ds_config. The onnx file exported, albeit with the last layer as a TensorRT node. So I can’t actually check the fidelity of this export using the onnx runtime. Then I used trtexec to convert the onnx file to a .plan: ./trtexec --onnx=/models/yolov4_resnet18_epoch_150-planes.onnx --maxShapes=Input:16x3x512x512 --minShapes=Input:1x3x512x512 --optShapes=Input:8x3x512x512 --saveEngine=/models/planes-model-2-fp32.plan --verbose . I then used this config.pbtxt for triton inference server (extension changed to .txt for uploading):
config.txt (522 Bytes)
The model loads into Triton without any problems. When I actually send requests to the Orin for inference from another machine, the fun begins. All of the scores are now [0.0, 0.04] roughly. (This is after disaggregating triton server outputs using BatchedNMS = number of detections, BatchedNMS1 = boxes, BatchedNMS2 = scores , BatchedNMS3 = labels)

Any ideas why this might be the case? I have also tried using tao converter to convert the .onnx to a .plan. Same result. Low scores .

[Morganh]

@joel42

Because tao deploy does not work.

Could you refer to following link GitHub - NVIDIA/tao_deploy: Package for deploying deep learning models from TAO Toolkit to run tao-deploy on Jetson devices? You can use this way to narrow down. To check if tao-deploy engine-generation/evaluation/inference can work.

[yingliu]

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks