Docker Issue - Error response from daemon: OCI runtime create failed

p.lafferty · March 11, 2021, 5:57pm

Hello

I understand this may be a duplicate question but I’m looking for some advice.
I currently have a docker container that allows me to run an inference script using my TensorFlow PB model.

After flashing my Jetson Agx Xavier, I am getting the following error:

I am starting my docker container with the following command:
xhost + && sudo docker run --privileged
–runtime nvidia --network host
-v $(pwd):/raptor
–env DISPLAY=unix$DISPLAY
-v $XAUTH:/root/.Xauthority
-v /tmp/.X11-unix:/tmp/.X11-unix
-ti raptor/model_inference:latest bash -i

And I’ve discovered that the docker container runs perfectly without the “–runtime nvidia” parameter.
I didn’t create the docker container myself so if you need to see, for example, the DockerFile or any other docker related files, let me know and I’ll attach them.

Because of my inexperience in this software, I am only clutching at straws for a solution and could be doing with a gentle nudge in the right direction.
My guess is that the problem lies with CUDA and the version installed on my Jetson Xavier but I’m not sure.

I have tried the following forums and have tested many of the suggested solutions but have had no success:

Thanks in advance for any help or advice as its much appreciated.

Kind Regards
Patrick

dusty_nv · March 22, 2021, 2:02pm

Hi @p.lafferty,

When you re-flashed your Jetson, was it with a different version of JetPack-L4T? If so, did you rebuild your container against the version of l4t-base that matches the new L4T version you re-flashed with? Otherwise you will get an error.

For example, if you updated your Jetson to L4T R32.5.0, you must also rebuild your container against l4t-base:r32.5.0 (or against another base container that derives from l4t-base:r32.5.0)

Also, see if you can just run l4t-base:r32.5.0 (or whatever your current L4T version is) with --runtime nvidia

p.lafferty · March 23, 2021, 10:00am

Thanks @dusty_nv , I’ve managed to fix my docker container.

Couple of tips for anyone experiencing the same issue:

if you happen to be using TensorFlow check out this (NVIDIA L4T TensorFlow | NVIDIA NGC) site for the version to declare in your DockerFile. Pay particular attention to the TensorFlow version - I had to build my docker container twice because tf.saved_model.load only works in TensorFlow 2.X and I had 1.15.
if your struggling to find out your L4T version you can find it using jetsonInfo.py from this (GitHub - jetsonhacks/jetsonUtilities: Get information about the NVIDIA Jetson OS environment. Lists L4T and JetPack versions, along with major libraries.) repository.

This can be a painful error to fix and I endured a few weeks of pain but stick at it you’ll figure it out.