Docker run error why trying to run Tensorflow Image

Good morning eveybody.
Me and my supervisor are working on a project and we are trying to run Tensorflow inside a Docker image on our Jetson TX2 devevelopment board. We are running the L4T 32.4.4. The output of the jetsonInfo.py script from jetsonhacks is the following:

vwm@sva-ms:~/$ python jetsonInfo.py 
NVIDIA Jetson TX2
 L4T 32.4.4 [ JetPack UNKNOWN ]
   Ubuntu 18.04.5 LTS
   Kernel Version: 4.9.140-tegra
 CUDA 10.2.89
   CUDA Architecture: 6.2
 OpenCV version: 4.1.1
   OpenCV Cuda: NO
 CUDNN: 8.0.0.180
 TensorRT: 7.1.3.0
 Vision Works: 1.6.0.501
 VPI: 0.4.4

Weirdly enough, I don’t get the Jetpack version. A run with apt-get gives me this:

vwm@sva-ms:~/drive$ sudo apt show nvidia-jetpack
Package: nvidia-jetpack
Version: 4.4.1-b50
Priority: standard
Section: metapackages
Maintainer: NVIDIA Corporation
Installed-Size: 199 kB
Depends: nvidia-cuda (= 4.4.1-b50), nvidia-opencv (= 4.4.1-b50), nvidia-cudnn8 (= 4.4.1-b50), nvidia-tensorrt (= 4.4.1-b50), nvidia-visionworks (= 4.4.1-b50), nvidia-container (= 4.4.1-b50), nvidia-vpi (= 4.4.1-b50), nvidia-l4t-jetson-multimedia-api (>> 32.4-0), nvidia-l4t-jetson-multimedia-api (<< 32.5-0)
Homepage: http://developer.nvidia.com/jetson
Download-Size: 29,4 kB
APT-Sources: https://repo.download.nvidia.com/jetson/t186 r32.4/main arm64 Packages
Description: NVIDIA Jetpack Meta Package

N: There are 2 additional records. Please use the '-a' switch to see them

Based on this, we pulled the image nvcr.io/nvidia/l4t-tensorflow:r32.4.4-tf2.3-py3 with the command:
sudo docker pull nvcr.io/nvidia/l4t-tensorflow:r32.4.4-tf2.3-py3
And we tried to run it with the command:
sudo docker run -it --rm --runtime nvidia --network host nvcr.io/nvidia/l4t-tensorflow:r32.4.4-tf2.3-py3

Only to get the following error:

vwm@sva-ms:~$ sudo docker run -it --rm --runtime nvidia --network host nvcr.io/nvidia/l4t-tensorflow:r32.4.4-tf2.3-py3
docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: 
starting container process caused: process_linux.go:459: container init caused: 
Running hook #0:: error running hook: exit status 1, stdout: , stderr: 
nvidia-container-cli: initialization error: driver error: failed to process request: unknown.

We also tried the L4T base image nvcr.io/nvidia/l4t-base:r32.4.4 only to get the exact same result:

vwm@sva-ms:~/drive$ sudo docker run -it --rm --runtime nvidia --network host nvcr.io/nvidia/l4t-base:r32.4.4
docker: Error response from daemon: OCI runtime create failed: container_linux.go:370:
starting container process caused: process_linux.go:459: container init caused: 
Running hook #0:: error running hook: exit status 1, stdout: , stderr:
nvidia-container-cli: initialization error: driver error: failed to process request: unknown.

At this point, I’m not sure what is going on. I suspect something is wrong with the JetPack install, from the jetsonInfo.py script output. But I don’t know what is wrong and how to proceed.

The outputs from docker info:

vwm@sva-ms:~/drive$ sudo docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 20.10.3
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux nvidia
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 269548fa27e0089a8b8278fc4fc781d7f65a939b
 runc version: ff819c7e9184c13b7c2607fe6c30ae19403a7aff
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.9.140-tegra
 Operating System: Ubuntu 18.04.5 LTS
 OSType: linux
 Architecture: aarch64
 CPUs: 4
 Total Memory: 7.676GiB
 Name: sva-ms
 ID: 5F4R:RSXN:BDZP:O2HN:JA7H:2Q3M:F3EP:7EAO:LRIN:WYK6:GSUP:2RDH
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio weight support
WARNING: No blkio weight_device support

We also tried to update the Jetpack packages with this and it also didn’t work:

vwm@sva-ms:~/drive$ sudo apt install nvidia-jetpack
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 nvidia-jetpack : Depends: nvidia-container (= 4.4.1-b50) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

We are in the second day trying to make this thing work and nothing. I just keep receiving that Docker error driver error and etc.
What is it not working?
Thanks in advance for any help.

Hi @Lucas_Red, I’m not sure what is causing the issue with your broken nvidia-container package, sorry about that. It may be easiest to backup your work and re-flash the device. Otherwise try uninstalling/purging/reinstalling the nvidia-container package to see if that helps.