We have been using Darknet framework for a while and are thinking on migrating to TLT 3.0 or (TAO now). We are doing an accuracy comparison with our previous YOLOv4 models trained on our person dataset.
In order to compare trainings, framework etc… We have a python script which compute PASCAL2010 mAP. This script has been well tested and gives the exact same mAP than Darknet official repo (darknet detector map command) with the same weights.
We wanted to compute TLT accuracy using our script as well (to make a fair comparison with our previous results, we need to use the same tool).
To do so we first run tlt inference, then use the bounding boxes coordinates generated in the labels folder (KITTI format) and converted them to the Darknet format. Finally compute the Pascal2010 mAP using our script.
In addition we also have computed the mAP using TRT. We have converted TLT file into TRT, run the inferences, scale and converted bounding boxes and finally compute the mAP using our script.
We also get the same score than tlt evaluate
To summarise, the problem is we are unable to get the same mAP by running tlt evaluate or trt inference + script VS tlt inference + script.
- tlt evaluate => mAP@0.5 = 84.5%
- trt inference + script => mAP@0.5 = 84.5%
- tlt inference + script => mAP@0.5 = 77.8%
To remove possible errors, we have so far confirmed:
- mAP computed by our script is the same than Darknet official repo
- Predicted bounding boxes conversion from KITTI to Darknet format is correct, we have checked is visually.
- Compare tlt evaluate vs trt inference + script => accuracy is equal
Do you have any insights on why we would get such a difference? Is there any post processing done is tlt inference not present in tlt evaluate and trt inference o vice-versa?
Thank you for your help.