How to monitor tensor cores utilization?

kirpasaccessory · August 2, 2021, 1:16pm

Is it possible to monitor tensor cores usage like “nvidia-smi dmon”?

NVES · August 2, 2021, 1:37pm

Hi ,
We recommend you to check the supported features from the below link.

You can refer below link for all the supported operators list.
For unsupported operators, you need to create a custom plugin to support the operation

github.com

onnx/onnx-tensorrt/blob/main/docs/operators.md

<!--- SPDX-License-Identifier: Apache-2.0 -->

# Supported ONNX Operators

TensorRT 8.4 supports operators up to Opset 17. Latest information of ONNX operators can be found [here](https://github.com/onnx/onnx/blob/master/docs/Operators.md)

TensorRT supports the following ONNX data types: DOUBLE, FLOAT32, FLOAT16, INT8, and BOOL

> Note: There is limited support for INT32, INT64, and DOUBLE types. TensorRT will attempt to cast down INT64 to INT32 and DOUBLE down to FLOAT, clamping values to `+-INT_MAX` or `+-FLT_MAX` if necessary.

See below for the support matrix of ONNX operators in ONNX-TensorRT.

## Operator Support Matrix

| Operator                  | Supported  | Supported Types | Restrictions                                                                                                           |
|---------------------------|------------|-----------------|------------------------------------------------------------------------------------------------------------------------|
| Abs                       | Y          | FP32, FP16, INT32 |
| Acos                      | Y          | FP32, FP16 |
| Acosh                     | Y          | FP32, FP16 |
| Add                       | Y          | FP32, FP16, INT32 |

This file has been truncated. show original

Thanks!

kirpasaccessory · August 2, 2021, 1:39pm

I need to monitoring of utilization of tensor cores, is it possible?

spolisetty · August 3, 2021, 10:27am

Hi @kirpasaccessory,

You can use profiler to monitor utilization, Please refer
https://docs.nvidia.com/deeplearning/frameworks/dlprof-user-guide/index.html
https://docs.nvidia.com/deeplearning/tensorrt/best-practices/index.html#nvprof

kirpasaccessory · August 3, 2021, 1:01pm

So without profiler is it impossible?
For example I want to check if my tensorflow model actualy used tensor cores under training…

spolisetty · August 5, 2021, 11:29am

Yes, to monitor Tensor cores utilization we have to use profiler. Please refer above docs for more details.