Models running in Cuda Cores or Tensor Cores

AI & Data Science Deep Learning (Training & Inference) TensorRT
,
1

Description

I want to know if my TensorRT engines are running in Cuda Cores or Tensor Cores. I profiled TensorRT models (ResNet, Inception, MobileNet, SqueezeNet, RegNet, ConvNeXt) across FP32/FP16/INT8 on RTX 5080 using Nsight Compute. Results show all models use Tensor Cores via xmma kernels, with FP16 achieving ~88% TC utilization, FP32 ~81% (TF32), INT8 ~81%.
My question is: I did this in the right way? I was expecting that FP32 models was running in Cuda Cores, but the analysis shows that this models are using TF32 instead, are this corect too?

Environment

TensorRT Version: 10.13.3.9
GPU Type: NVIDIA RTX 5080
Nvidia Driver Version: 580.95.05
CUDA Version: 13.0
CUDNN Version: 9.1.0 (91002)
Operating System + Version: Ubuntu 20.04.3
Python Version (if applicable): 3.10.12
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

2

*** Please note that this reply is generated by LLM automatically ***

Based on the provided information, it appears that you have profiled your TensorRT models (ResNet, Inception, MobileNet, SqueezeNet, RegNet, ConvNeXt) across different precision modes (FP32, FP16, INT8) using Nsight Compute on an NVIDIA RTX 5080 GPU.

The results show that all models use Tensor Cores, with FP16 achieving 88% Tensor Core utilization, FP32 achieving 81% TF32 utilization, and INT8 achieving 81% utilization.

Your expectation was that FP32 models would run on CUDA Cores, but the analysis shows that they are using TF32 instead. This is correct.

TF32 (Tensor Float 32) is a format that allows for faster execution of certain operations on Tensor Cores, while still maintaining the accuracy of FP32. TF32 is a feature of NVIDIA’s Tensor Cores, which are designed to accelerate matrix operations, such as those found in deep learning models.

When you profiled your FP32 models, the results showed that they were using TF32, which means that the Tensor Cores were being utilized to accelerate the computations. This is a good thing, as it can lead to improved performance.

It’s worth noting that TF32 is not the same as FP32. TF32 is a format that is optimized for Tensor Cores, while FP32 is a more general-purpose floating-point format. However, TF32 is designed to be compatible with FP32, so you can expect similar accuracy and results.

In summary, your analysis is correct, and the results show that your FP32 models are indeed using TF32 on the Tensor Cores, which is a good thing for performance.

Here’s a comparison of the results:

  • FP16: 88% Tensor Core utilization
  • FP32: 81% TF32 utilization (on Tensor Cores)
  • INT8: 81% utilization (on Tensor Cores)

All models are using Tensor Cores, with FP16 achieving the highest utilization. FP32 models are using TF32, which is a format optimized for Tensor Cores, and INT8 models are also using Tensor Cores.

Overall, your analysis suggests that your models are well-optimized for the NVIDIA RTX 5080 GPU, and that the Tensor Cores are being effectively utilized to accelerate the computations.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***