Cuda Error in findFastestTactic: 700 when using tensorrt in fp16 mode on Xavier NX with Jetpack4.4

Description

When I converted an onnx-model to trt-model in fp16 model on Xavier NX with Jetpack4.4, the cuda error came out like:
[06/05/2020-17:10:15] [E] [TRT] …/builder/cudnnBuilderUtils.cpp (423) - Cuda Error in findFastestTactic: 700 (an illegal memory access was encountered)
[06/05/2020-17:10:15] [E] [TRT] …/rtSafe/safeRuntime.cpp (32) - Cuda Error in free: 700 (an illegal memory access was encountered)
terminate called after throwing an instance of ‘nvinfer1::CudaError’
what(): std::exception

But I can do it in fp16 mode on TX2 with Jetpack4.4 successfully.

Environment

ONNX IR version: 0.0.3
JetPack Version: 4.4
GPU Type: Jetson Xavier

Moving to Jetson Xavier NX forum so that Jetson team can take a look.

Thanks

Hi,

Thanks for reporting this to us.
Do you have an onnx model so we can reproduce this in our environment?

Thanks.


The network consists of two identical parts. If I only convert the first part on in fp16 model on Xavier NX with Jetpack4.4, it can be successfully converted. But the whole network fails.

Hi,

We can reproduce this issue in our environment and will pass it to our internal team.
Just in case you don’t know, the float32 version can be inferenced without issues.

$ /usr/src/tensorrt/bin/trtexec --onnx=./checkpoints_0.9_ghost_320_780_part_v5.onnx

Thanks.

Thank you for your reply. I know that the float32 version is normal.
Waiting for your good news.

Hi,

This issue is fixed in our internal TensorRT package.
We will let you know once it is available for the public.

Thanks.

Thanks.

Hi, is the JetPack 4.4 - L4T R32.4.3 production for NX the new version that the issue is fixed?

No, it’s not fixed at this version, but in the coming release,

OK, thanks.

Hi,
I am facing the same problem.
the newest TensorRT is 7.2. Was it fixed ?
thanks