Why trt model returns bunch of Nan values?

I am transferring my onnx model to trt model, after testing the outputs of the models i found that trtmodels returns plenty of NaN values.

Here are the outputs from both models. The part values are fine, but I still got plenty of NaN values.


[array([[[3.6024790e+00, 1.9787143e+00, 2.4665287e+00, 2.4537530e+00,
          6.6367686e-03, 5.0844550e-03],
         [1.2190340e+00, 1.4738089e+00, 1.2516123e+00, 1.9088440e+00,
          8.5598230e-03, 5.2941740e-03],
         [2.4905281e+00, 1.4704772e+00, 1.7078139e+00, 1.3909163e+00,
          1.7221481e-02, 8.3563328e-03],
         [5.0809355e+00, 4.6628089e+00, 5.4717007e+00, 2.9312749e+00,
          6.7818165e-03, 2.8427649e-01],
         [5.8280797e+00, 4.6378379e+00, 3.4127855e+00, 2.3493521e+00,
          2.3016751e-02, 4.6690524e-02],
         [4.9988794e+00, 4.4283361e+00, 5.8102098e+00, 2.5537195e+00,
          1.4624715e-02, 5.5993736e-02]]], dtype=float32)]


>>> trt_infer = TensorRTInfer(trtpath)
>>> trt_infer.infer(b)
array([[[3.625     , 1.9707031 , 2.4804688 , 2.5175781 , 0.00651169,
        [1.2158203 , 1.4755859 , 1.2509766 , 1.9121094 , 0.00841522,
        [2.4882812 , 1.4541016 , 1.7050781 , 1.3740234 , 0.01737976,
        [       nan,        nan,        nan,        nan,        nan,
        [       nan,        nan,        nan,        nan,        nan,
        [       nan,        nan,        nan,        nan,        nan,
                nan]]], dtype=float32)

Here is the command i used to transferring model:

./trtexec --onnx=sample_test_fixed_batch_2.onnx --saveEngine=sample_test_fixed_batch_2.trt --workspace=4096 --fp16 --verbose

I also uploaded both of the models
sample_test_fixed_batch_2.onnx (4.0 MB)
sample_test_fixed_batch_2.trt (3.1 MB)

Dear @QQQQ,
Could you share the jetpack version and used python code snippet for reproduce on our end.

HI @SivaRamaKrishnaNV
Jetpack 4.4 and attach is the code
nvidia_sample.py (4.9 KB)

Dear @QQQQ,
Is it possible to test on latest Jetpack release and confirm?

No, i haven’t tried it on jep 4.6 , since all the other service all build based on 4.4

Please try with the latest version. Thanks

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.