Jetson Orin Nano TensorRT

Robotics & Edge Computing Jetson Systems Jetson Orin Nano
1

Hi,
I currently have a Jetson Orin Nano with:

Jetpack - 6.0
CUDA - 12.2
Python - 3.10
Torch Version - 2.4.0a0+07cecf4168.nv24.05
TensorRT - 10.2
Torch2trt - 0.5.0

At the moment I have a pytorch model which runs slower than expected. I want to use a TensorRT engine to speed up. I tried both:

  1. Torch2trt: which throws me the below error
    File “/home/miniconda3/envs/new_env/lib/python3.10/site-packages/torch/nn/modules/module.py”, line 1918, in state_dict
    hook_result = hook(self, destination, prefix, local_metadata)
    File “/home/miniconda3/envs/new_env/lib/python3.10/site-packages/torch2trt-0.5.0-py3.10.egg/torch2trt/trt_module.py”, line 60, in _on_state_dict
    state_dict[prefix + “engine”] = bytearray(self.engine.serialize())
    AttributeError: ‘NoneType’ object has no attribute ‘serialize’

  2. Torch → ONNX → TensorRT: Throws me:

[07/22/2024-19:37:20] [E] Error[10]: IBuilder::buildSerializedNetwork: Error Code 10: Internal Error (Could not find any implementation for node /tcn/network/network.0/net1/conv1/Conv.)
[07/22/2024-19:37:20] [E] Error[1]: [checkMacros.cpp::catchCudaError::205] Error Code 1: Cuda Runtime (no kernel image is available for execution on the device)
[07/22/2024-19:37:20] [E] Engine could not be created from network
[07/22/2024-19:37:20] [E] Building engine failed
[07/22/2024-19:37:20] [E] Failed to create engine from model or file.
[07/22/2024-19:37:20] [E] Engine set up failed

It would be great if someone could help,
Thanks!

4

Dear @prp852,
Can you share the onnx model for repro?

5

Hi,
Thank you so much for your reply!

Long story short, something messed up and I ended up with this error.

I am now using another GPU (CUDA 12.5) and can convert ONNX to TensorRT
Is there a code or documentation I can refer on how to use the “.trt” file to run inference?

Thanks again :))

6

Dear @prp852,
You can use trtexec to quickly test inference timings with TRT model.

7

Hi,

I think the test worked, thanks for the suggestion. I am looking for a Python code that will allow me to run inference using the TensorRT file I have.

To explain in detail, I am reading CAN messages in real-time at 100Hz using Python. Now, I want to run inference on the TensorRT file using Python (or C++) and send the prediction message. The receiving and sending CAN messages part in Jetson is done, but running inference took a lot of time when I used a PyTorch model. Now that I have a TensorRT file, is there a way (or template code) to run TensorRT inference using Python?

Thank you so much!

8

Dear @prp852 ,

Please search(you can use grep -lir deserializeCudaEngine * under TRT sample directory)for the usage of deserializeCudaEngine API in TensorRT samples.

9

Hi,
I was able to write a code from the TRT documentation!

Thanks for your support.