Jetson Thor - INT8 quantization show no performance gain over FP16

[AastaLLL]

Hi,

There is a report that FP8 doesn’t have the expected performance with TensorRT on Thor:

Low ViT Performance Gain on Jetson Thor Using FP8 vs FP16

Hello, Looking at the documentation, to enable fp8 operations you need some onnx surgery (inserting Q/DQ at specific locations) to trigger the right MHA (Multi-Head Attention) fusion in conjunction with fp8 precision. However, the performance improvement is quite low for base ViT model (~20% latency reduction). It is even worse on the EfficientSAM encoder with basically no gain. By looking at the profiling and layer info from TensorRT the FP8 seems there (even though some tactics are quite cr…

We will check if the same perf regression issue for the INT8 case.
Thanks.