I’m trying to use Onnxruntime inside a Docker container. The base image is l4t-r32 (from docker hub /r/stereolabs/zed/, Cuda 10.2) and so I installed onnxruntime 1.6.0 using binaries from Jetson Zoo. However, when trying to import onnxruntime, I get the following error:
ImportError: cannot import name ‘get_all_providers’
I also tried with onnxruntime 1.10.0, it installs and imports fine but when I run rt.get_device() it says GPU and rt.get_available_providers() returns ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider']. However, when I create an inference session, session.get_providers() returns only CPUExecutionProvider.
My best guess is that the issue has something to do with cuDNN, but from my understanding the base image of l4t-r32.4 should come with cuDNN installed?
TensorRT Version: None. Can it be installed in an arm64 container? CUDA Version: 10.2 CUDNN Version: Operating System + Version: Ubuntu 18.04 Python Version (if applicable): 3.6 Baremetal or Container (if container which image + tag): 3.8-py-devel-l4t-r32.4
My ONNX model works fine (with onnxruntime GPU inference) on my host machine, it’s only in the docker container that it is restricted to CPU EP. So unfortunately I don’t think that sharing the model will help. As for scripts, when using onnxruntime version 1.6 (as at should be compatible with Cuda 10.2) the code breaks at
with error code as stated above, not much of a script unfortunately. When using onnxruntime version 1.10, a script to reproduce the error would be
import onnxruntime as rt
rt.get_device() # 'GPU'
model = rt.InferenceSession(model.onnx, None, providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])
# Note: results in warning, Failed to create CUDAExecutionProvider. Please
reference https://onnxruntime.ai/docs/execution-providers/TensorRT-ExecutionProvider.html#requirements to ensure all dependencies are met.
model.get_providers() # 'CPUExecutionProvider'
I’m assuming that the latter breaks because onnxruntime 1.10 is not the correct version for Cuda 10.2. However, I don’t know how to fix the ImportError when using 1.6.
As for trtexec/TensorRT, the docker environment does not have TensorRT and all the binaries to install it appear to be for amd64, while the container is of arm64.