Onnxruntime in Docker

AI & Data Science Deep Learning (Training & Inference) cuDNN
1

Description

I’m trying to use Onnxruntime inside a Docker container. The base image is l4t-r32 (from docker hub /r/stereolabs/zed/, Cuda 10.2) and so I installed onnxruntime 1.6.0 using binaries from Jetson Zoo. However, when trying to import onnxruntime, I get the following error:

ImportError: cannot import name ‘get_all_providers’

I also tried with onnxruntime 1.10.0, it installs and imports fine but when I run rt.get_device() it says GPU and rt.get_available_providers() returns ['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'CPUExecutionProvider']. However, when I create an inference session, session.get_providers() returns only CPUExecutionProvider.

My best guess is that the issue has something to do with cuDNN, but from my understanding the base image of l4t-r32.4 should come with cuDNN installed?

Environment

TensorRT Version: None. Can it be installed in an arm64 container?
CUDA Version: 10.2
CUDNN Version:
Operating System + Version: Ubuntu 18.04
Python Version (if applicable): 3.6
Baremetal or Container (if container which image + tag): 3.8-py-devel-l4t-r32.4

2

Hi,
Request you to share the ONNX model and the script if not shared already so that we can assist you better.
Alongside you can try few things:

  1. validating your model with the below snippet

check_model.py

import sys
import onnx
filename = yourONNXmodel
model = onnx.load(filename)
onnx.checker.check_model(model).
2) Try running your model with trtexec command.

In case you are still facing issue, request you to share the trtexec “”–verbose"" log for further debugging
Thanks!

4

Hi,

My ONNX model works fine (with onnxruntime GPU inference) on my host machine, it’s only in the docker container that it is restricted to CPU EP. So unfortunately I don’t think that sharing the model will help. As for scripts, when using onnxruntime version 1.6 (as at should be compatible with Cuda 10.2) the code breaks at

import onnxruntime

with error code as stated above, not much of a script unfortunately. When using onnxruntime version 1.10, a script to reproduce the error would be

import onnxruntime as rt
rt.get_device()   # 'GPU'

model = rt.InferenceSession(model.onnx, None, providers=['CUDAExecutionProvider', 'CPUExecutionProvider'])  
# Note: results in warning, Failed to create CUDAExecutionProvider. Please 
reference https://onnxruntime.ai/docs/execution-providers/TensorRT-ExecutionProvider.html#requirements to ensure all dependencies are met.

model.get_providers()  # 'CPUExecutionProvider'

I’m assuming that the latter breaks because onnxruntime 1.10 is not the correct version for Cuda 10.2. However, I don’t know how to fix the ImportError when using 1.6.

As for trtexec/TensorRT, the docker environment does not have TensorRT and all the binaries to install it appear to be for amd64, while the container is of arm64.