What happens when I call --noTF32 when using trtexec on the Jetson Orin?

Hi,

From what I understand, the Jetson Orin will use TF32 format by default. I see a noticeable decrease in performance when using --noTF32 which is expected. Is this actually allowing me to see the performance using FP32? Thank you!

Sorry for the late response, our team will do the investigation and provide suggestions soon. Thanks

Hi,

By default, the trtexec runs the model in fp32 mode.
You can find the detailed instructions below:

Thanks.

Hi AastaLLL,

This isn’t consistent with what I’ve observed. I see a noticeable decrease in performance when I use the --noTF32 flag. This is consistent with switching from TF32 to FP32. Are both modes supported on the Jetson Orin? What explains this drop in performance? I see the decrease in performance in all power modes and all emulation modes. Thank you.

Hi,

First, please make sure you do the benchmark in the full performance mode.

$ sudo nvpmodel -m 0
$ sudo jetson_clocks

Do you test it with the default TensorRT model? For example, mnist.onnx.
If yes, could you share the result you got in the different settings with us first?

Thanks.