Hello there,
I’m currently working in a school project, where we’re using machine learning methods in order to develop autonomous driving toy cars. These are equipped with lots of hardware & sensors which communicate through ros packages. My task inside this team is to optimize the inference time of an already trained (tiny-)yolo tensorflow model, which is used for object detection on our newly purchased hardware (NVIDIA Jetson Xavier NX). This model already runs with an inference of about 30ms on x86 hardware (without TensorRT). However, on our new ARM based hardware the inference time is about 100ms, which is too bad and expected to be a lot better. The object detection runs inside a docker container (using the nvcr.io/nvidia/l4t-tensorflow:r32.7.1-tf2.7-py3 image from NGC) as a ros package. I already ensured, that the docker container uses the NVIDIA container runtime and therefore has access to CUDA.
So my next step was to make use of tensorRT to optimize the inference time. Only Problem was that the tensorflow version of the used docker image was built without tensorRT. Currently I’m trying to do the conversion to the tensorRT model outside the docker container.
I was just wondering, if I’m missing something or you have any ideas/resources that could help me achieving my goal.
Output of $jtop:
NVIDIA Jetson Xavier NX (Developer Kit Version) - Jetpack UNKNOWN [L4T 32.7.2]
- Type: Xavier NX (Developer Kit Version)
- SOC Family: tegra194 ID: 25
- Module: P3668 Board: P3509-000
- Code Name: jakku
- Cuda ARCH: 7.2
- Serial Number: 1423421015790
- Libraries: - Hostname: xavier-nx
- CUDA: 10.2.300 - Interfaces:
- OpenCV: 4.1.1 compiled CUDA: NO * wlan0: 192.168.178.155
- TensorRT: 8.2.1.8 * docker0: 172.17.0.1
- VPI: ii libnvvpi1 1.2.3 arm64 NVIDIA Vision Programming Interface library * br-6999ad3e172.19.0.1
- VisionWorks: 1.6.0.501 * br-d7446ff7172.18.0.1
- Vulkan: 1.2.70
- cuDNN: 8.2.1.32
Thanks already
Alex