ONNX to TensorRT model conversion failure on Jetson Nano

Hi,

We’re currently looking into different model input sizes for our Yolo-based AI model. However, we’re running into trouble when converting the ONNX model for any input size larger than 640x640.

I’ve attached a log of the conversion, which also includes the exact command we ran. We’ve tried different workspace sizes and have played with the trtexec command-line parameters but haven’t had any success yet.

Our OS is currently based on the Yocto layers for L4T R32.7.3. Our TensorRT version is 8.2.1.

Thanks in advance for your help.

Best regards,
Niels Avonds

trtexec.log (2.1 MB)

Hi,

Could you monitor the system with tegrastats at the same and share the log with us?
Which batchsize do you use?

Thanks.

Hi,

I’ve attached the tegrastats log. We’re using batchsize 1.

Best regards,
Niels
tegrastats.log (115.6 KB)

Do the logs differ when the workspace size is changed? Can you provide a log that shows the verbose build without providing a workspace size to trtexec? This will default to the workspace being the same size as the device memory.

Are you running any other processes that utilizes the GPU when building the engine?

Hi,

We’re not running anything else on the GPU when building the engine. I’ve attached a verbose log when no workspace parameter is set.

Best regards,
Niels
trtexec.log (3.9 MB)

Thanks for the response Niels.

Looking at the log, the following jumps out to me:

[10/20/2023-14:11:27] [E] Error[1]: [resizingAllocator.cpp::deallocate::100] Error Code 1: Cuda Runtime (the launch timed out and was terminated)
[10/20/2023-14:11:27] [W] [TRT] GPU error during getBestTactic: Reshape_232 : the launch timed out and was terminated

According to Kernel launch timeout, the Jetson Nano has a 5 second timeout watchdog for cuda launches, and it seems that using larger input sizes is causing TensorRT to time kernels that exceed this limit. There are some commands in that post to disable the timer, can you try those and see if it allows you to build the engine?

Hi,

Thanks a lot, this has resolved the model conversion issue for us. We’re now able to convert the model on a Jetson Nano.

Thanks again!
Best regards,
Niels

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.