NvInfer not utilizing Tensor Cores

[IvensaMDH]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
GPU
• DeepStream Version
6.0
• JetPack Version (valid for Jetson only)
• TensorRT Version
8
• NVIDIA GPU Driver Version (valid for GPU only)
470.57.02
• Issue Type( questions, new requirements, bugs)
question
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

I am unable to get NvInfer to utilize the Tensor Cores (T4). Tested with FP32 and FP16 without any luck.

We are using the DCGM exporter link and have tested that it is able to export tensor core utilization with dcgmproftester successfully.

Sample pipeline:

gst-launch-1.0 uridecodebin uri=“file://test.mp4” ! nvvideoconvert ! “video/x-raw(memory:NVMM), format=NV12, width=128, height=256” ! m.sink_0 nvstreammux name=“m” batch-size=1 ! nvinfer config-file-path=“config.txt” ! fakesink

Config file:

[property]
gpu-id=0
model-color-format=0
onnx-file=model_fp32.onnx
batch-size=1
network-mode=0
interval=0
gie-unique-id=1
process-mode=1
network-type=100
output-tensor-meta=1

Am i missing something…?

Thanks,

[mchi]

I don’t think tensorrt wrapped by nvinfer plugin supports Prometheus, so this is expected

[IvensaMDH]

All right,

Is there any other way that I can verify how the hardware is being utilized correctly?

Thanks,

[IvensaMDH]

@mchi my apologies - i’ve lied to you:

While the tensor core utilization is not impressive (yet) - we are indeed able to monitor it.

first two spikes are engine builds - third is inference.

/M