TensorRT output full of NaN

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Hello,

I am trying to convert the embedding from pyannote3.0 to TensorRT.

Environment

TensorRT Version: Docker → nvcr.io/nvidia/tensorrt:23.01-py3
GPU Type: A30
Nvidia Driver Version: 535.86.10
CUDA Version: 12.2

Explication

1) Load Model and Convert Model into tensorRT

The model “hbredin/wespeaker-voxceleb-resnet34-LM” is in ONNX format.

onnx_model_path= hf_hub_download(
                    repo_id= "hbredin/wespeaker-voxceleb-resnet34-LM",
                    filename="speaker-embedding.onnx",
                )
trt_engine_path = onnx_model_path.replace('.onnx', '.trt')
command = ["trtexec", f"--onnx={onnx_model_path}", f"--saveEngine={trt_engine_path}", "--explicitBatch"]
process = subprocess.run(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)

2) Load data

waves = torch.load("data_waveforms.pt")
x = test.prepare_input(waves)
x.shape, x.nbytes

((1, 998, 80), 319360)

x is my input. I made an inference with the model to see the size of the output and to get an idea of my embedding.

embeddings = model.run(output_names=["embs"], input_feed={"feats": x})[0]
embeddings.shape, embeddings.nbytes

((1, 256), 1024)

x_shape = (1, 998, 80)
output_shape = (1, 256)

batch_size = 1
d_input = cuda.mem_alloc(batch_size * x.nbytes)
d_output = cuda.mem_alloc(batch_size * output_shape[1] * 4)  # 4 bytes pour chaque float32
bindings = [int(d_input), int(d_output)]
cuda.memcpy_htod(d_input, x.ravel())
test.context.execute_v2(bindings)
output_data = np.empty(output_shape, dtype=np.float32)
cuda.memcpy_dtoh(output_data, d_output)
output_data

array([[nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
nan, nan, nan, nan, nan, nan, …]]

Does anyone have an idea about the problem?

2

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.

In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!