Seems that you misunderstand the time of tlt-infer. The tlt-infer will write bbox to images and also write label files.
It does not mean the inference time.
For how to check the inference time, you can run trtexec.
Reference: Measurement model speed - #4 by Morganh
With this way, the fp32 and fp16 should be different at inference speed.