Run TensorRT 8 on Jetpack 4.5

I have a system with Jetpack 4.5. It is set up by a third party so I can not easily change the Jetpack version:

# R32 (release), REVISION: 5.2, GCID: 27767740, BOARD: t186ref, EABI: aarch64, DATE: Fri Jul  9 16:05:07 UTC 2021

But we require TensorRT 8 for some of the neural networks we like to run. I thought that would be easy to do with getting the right docker. However if I run:

nvidia-docker run -it nvcr.io/nvidia/l4t-tensorrt:r8.0.1-runtime bash

It gives me the following error:

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #1: error running hook: exit status 1, stdout: , stderr: exec command: [/usr/bin/nvidia-container-cli --load-kmods configure --ldconfig=@/sbin/ldconfig.real --device=all --compute --utility --require=base-only --require=cuda>=10.2 --pid=3245 /var/lib/docker/overlay2/57a897f188601eaf4838f056d7b274f82048ec5ef2e5992c36d96e8f21a5cb06/merged]
nvidia-container-cli: requirement error: invalid expression: unknown.
ERRO[0001] error waiting for container: context canceled 

I tried also docker run --gpus all -it nvcr.io/nvidia/l4t-tensorrt:r8.0.1-runtime bash with the same result.

Any ideas how to go about this? How can I use TensorRT on an old Jetpack version?

Hi @cosama, that newer container is incompatible with the older JetPack 4.5. You won’t be able to install newer TensorRT without upgrading the JetPack version, as TensorRT has underlying dependencies on CUDA and the GPU driver in the L4T kernel. Recommend contacting the third party for instructions to upgrade your system.

@dusty_nv Thank you so much for the quick reply, much appreciated.

Unfortunately, we need to run this on systems that are deployed in the field. The third party controls the devices and the deployment and they can not be upgraded to a more recent Jetpack version.

Jetpack 4.5 features CUDA 10.2, so does Jetpack 4.6, however the later has TensorRT8.0.1 available. Considering the CUDA version is the same I would assume it should be possible to run TensorRT8.0.1 on the older Jetpack release?

My understanding is you can install whatever software in your container, as long as the GPU is properly exposed. Or do you need to talk to the host GPU drivers instead? Are those incompatible between Jetpack 4.5 and 4.6?

I suspect there should be a solution along the lines of: Disable mount plugins - #3 by AastaLLL?

On JetPack 4.x, CUDA/cuDNN/TensorRT are mounted from the host and not installed inside the containers. It’s not officially supported to independently upgrade/change the TensorRT version separate from the JetPack version. I suppose you could always try it by taking the libraries from a JetPack 4.6 install, however I have no idea if it’ll work or not.

BTW on JetPack 5.x, CUDA/cuDNN/TensorRT get installed inside the containers themselves, so your understanding of containers is correct for JetPack 5.x. However on JetPack 5.x, there’s still the issue of GPU driver support in the L4T kernel, so you can’t just run arbitrarily new versions of CUDA/ect if your JetPack-L4T version is significantly outdated to the point that it’s incompatible with the newer CUDA/TensorRT that you want to run. This is not dissimilar to x86 where the GPU containers have a minimum NVIDIA driver version required sometimes.

@dusty_nv I finally figured this out. It is indeed possible, as I expected to run pretty much any jetpack version in a docker on the jetson devices.

I used this Dockerfile here as a base, with the platform and l4t version that I needed. After connecting it to the nvidia package repo, you can install tensorrt with apt.

In order to expose the GPU, without mounting all the host libraries I used the solution from here. I ended up using:

docker run -it --rm --device=/dev/nvhost-ctrl --device=/dev/nvhost-ctrl-gpu \
    --device=/dev/nvhost-prof-gpu --device=/dev/nvmap --device=/dev/nvhost-gpu \
    --device=/dev/nvhost-as-gpu --device=/dev/nvhost-vic \
    -v /usr/lib/aarch64-linux-gnu/tegra:/usr/lib/aarch64-linux-gnu/tegra --network host 
    ...

The last volume needs to be mounted because currently some of the packages from the nvidia package repo do not install in a docker because /proc/device_tree can not be accessed in the building process (see docker issue).

It would be great if that last step mount would not be necessary and /proc/device_tree would not be checked during package installation, but to go around this it would be a very ugly hack.

Hope that helps somebody else.

2 Likes

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.