Jetson Thor AGX - Poor INT8 performance

[AastaLLL]

Hi,

Below is a similar topic for your reference:

Low ViT Performance Gain on Jetson Thor Using FP8 vs FP16

Hello, Looking at the documentation, to enable fp8 operations you need some onnx surgery (inserting Q/DQ at specific locations) to trigger the right MHA (Multi-Head Attention) fusion in conjunction with fp8 precision. However, the performance improvement is quite low for base ViT model (~20% latency reduction). It is even worse on the EfficientSAM encoder with basically no gain. By looking at the profiling and layer info from TensorRT the FP8 seems there (even though some tactics are quite cr…

Could you test with --dumpLayerInfo --dumpProfile --profilingVerbosity=detailed --separateProfileRun --useCudaGraph --noDataTransfers argument and share the output with us?

Thanks.