How to monitor tensor cores utilization?

kirpasaccessory · August 2, 2021, 1:16pm

Is it possible to monitor tensor cores usage like “nvidia-smi dmon”?

NVES · August 2, 2021, 1:37pm

Hi ,
We recommend you to check the supported features from the below link.

You can refer below link for all the supported operators list.
For unsupported operators, you need to create a custom plugin to support the operation

github.com

onnx/onnx-tensorrt/blob/main/docs/operators.md

<!--- SPDX-License-Identifier: Apache-2.0 -->

# Supported ONNX Operators

TensorRT 8.4 supports operators up to Opset 17. Latest information of ONNX operators can be found [here](https://github.com/onnx/onnx/blob/master/docs/Operators.md)

TensorRT supports the following ONNX data types: DOUBLE, FLOAT32, FLOAT16, INT8, and BOOL

> Note: There is limited support for INT32, INT64, and DOUBLE types. TensorRT will attempt to cast down INT64 to INT32 and DOUBLE down to FLOAT, clamping values to `+-INT_MAX` or `+-FLT_MAX` if necessary.

See below for the support matrix of ONNX operators in ONNX-TensorRT.

## Operator Support Matrix

| Operator                  | Supported  | Supported Types | Restrictions                                                                                                           |
|---------------------------|------------|-----------------|------------------------------------------------------------------------------------------------------------------------|
| Abs                       | Y          | FP32, FP16, INT32 |
| Acos                      | Y          | FP32, FP16 |
| Acosh                     | Y          | FP32, FP16 |
| Add                       | Y          | FP32, FP16, INT32 |

This file has been truncated. show original

Thanks!

kirpasaccessory · August 2, 2021, 1:39pm

I need to monitoring of utilization of tensor cores, is it possible?

spolisetty · August 3, 2021, 10:27am

Hi @kirpasaccessory,

You can use profiler to monitor utilization, Please refer
https://docs.nvidia.com/deeplearning/frameworks/dlprof-user-guide/index.html
https://docs.nvidia.com/deeplearning/tensorrt/best-practices/index.html#nvprof

kirpasaccessory · August 3, 2021, 1:01pm

So without profiler is it impossible?
For example I want to check if my tensorflow model actualy used tensor cores under training…

spolisetty · August 5, 2021, 11:29am

Yes, to monitor Tensor cores utilization we have to use profiler. Please refer above docs for more details.

Topic		Replies	Views
How to measure Tensor core utilization using NVIDIA profiling tools such as Nsight System, DLProf, nvprof etc TensorRT cudnn	4	1362	January 31, 2024
GPU Utilization TensorRT tensorrt	3	700	August 29, 2023
TensorRT Engine TensorRT tensorrt	3	451	April 27, 2022
Does TensorRT support GeForce and Quadro boards TensorRT	1	605	April 6, 2023
Get tensor core usage through nvml System Management and Monitoring (NVML)	4	2153	December 17, 2022
GPU for molecular dynamics (Desmond) TensorRT	3	1497	October 18, 2021
TensorRT batch inference - How to be sure one kernel does use all the GPU ressources? TensorRT tensorrt , nsight	3	772	May 18, 2021
Android for NX module TensorRT board-design	1	350	September 15, 2021
Comaptibility Of TensorRT with Windows 11 Professional TensorRT	2	2273	December 6, 2021
nVidia release versions compatibility TensorRT	4	915	September 20, 2023

How to monitor tensor cores utilization?

Related topics