Docker broken upon update to v28 - jetson-containers won't run, reinstalling ubuntu with SDK manager seems to end wit broken installs

let me post the screenshots first from the reinstalling a fresh installation.

i did not recall the reinstall process to be so… difficult.

i think it is due to using WSL2 and thus additional steps, the previous time installation was with a host device running ubuntu 22.04.

  1. need to remember that when installing from WSL2, the Jetson device will constantly disconnect even though the USB cable is still connected to the AGX Orin.

  1. need to remember to check that “usbipd attach” command is available.



usbipd attach --busid <BUSID> --wsl --auto-attach

this command is the one that prevents the agx orin from disconnecting at many of the steps during the installation. not sure if i recall seeing this anywhere. so hopefully i will remember this the next time i need to reinstall again. since reliance on docker is a very painful experience.


installation continues…

i think the setup restart or something… and the device cannot be found.

so I unplugged and replugged the USB cable, and run the command again.


but turns out, something was not installed properly???



so a few items just won’t reinstall.
okay… we retry the installation again…

but the same errors again.

looks like for now have to install this version of docker.

kkk@ORIN:~$ ls
Desktop    Music                                      Public     Videos
Documents  NV_L4T_DOCKER_TARGET_POST_INSTALL_COMP.sh  snap
Downloads  Pictures                                   Templates
kkk@ORIN:~$ ./NV_L4T_DOCKER_TARGET_POST_INSTALL_COMP.sh
[sudo] password for kkk: 
INFO[0000] Loading config from /etc/docker/daemon.json  
INFO[0000] Wrote updated config to /etc/docker/daemon.json 
INFO[0000] It is recommended that docker daemon be restarted. 
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xeu docker.service" for details.
kkk@ORIN:~$ sudo apt install docker-ce=5:27.5.1-1~ubuntu.22.04~jammy \
                 docker-ce-cli=5:27.5.1-1~ubuntu.22.04~jammy \
                 docker-compose-plugin=2.32.4-1~ubuntu.22.04~jammy \
                 docker-buildx-plugin=0.20.0-1~ubuntu.22.04~jammy \
                 docker-ce-rootless-extras=5:27.5.1-1~ubuntu.22.04~jammy
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Suggested packages:
  cgroupfs-mount | cgroup-lite
The following packages will be DOWNGRADED:
  docker-buildx-plugin docker-ce docker-ce-cli docker-ce-rootless-extras docker-compose-plugin
0 upgraded, 0 newly installed, 5 downgraded, 0 to remove and 22 not upgraded.
Need to get 80,3 MB of archives.
After this operation, 8.998 kB disk space will be freed.
Do you want to continue? [Y/n] y
Get:1 https://download.docker.com/linux/ubuntu jammy/stable arm64 docker-buildx-plugin arm64 0.20.0-1~ubuntu.22.04~jammy [30,7 MB]
Get:2 https://download.docker.com/linux/ubuntu jammy/stable arm64 docker-ce-cli arm64 5:27.5.1-1~ubuntu.22.04~jammy [13,8 MB]
Get:3 https://download.docker.com/linux/ubuntu jammy/stable arm64 docker-ce arm64 5:27.5.1-1~ubuntu.22.04~jammy [16,0 MB]
Get:4 https://download.docker.com/linux/ubuntu jammy/stable arm64 docker-ce-rootless-extras arm64 5:27.5.1-1~ubuntu.22.04~jammy [8.657 kB]
Get:5 https://download.docker.com/linux/ubuntu jammy/stable arm64 docker-compose-plugin arm64 2.32.4-1~ubuntu.22.04~jammy [11,1 MB]
Fetched 80,3 MB in 7s (10,8 MB/s)                                                                                                   
dpkg: warning: downgrading docker-buildx-plugin from 0.21.0-1~ubuntu.22.04~jammy to 0.20.0-1~ubuntu.22.04~jammy
Progress: [  5%] [#####..........................................................................................................] 
Progress: [  5%] [#####..........................................................................................................] 
Unpacking docker-buildx-plugin (0.20.0-1~ubuntu.22.04~jammy) over (0.21.0-1~ubuntu.22.04~jammy) ...
dpkg: warning: downgrading docker-ce-cli from 5:28.0.0-1~ubuntu.22.04~jammy to 5:27.5.1-1~ubuntu.22.04~jammy
Preparing to unpack .../docker-ce-cli_5%3a27.5.1-1~ubuntu.22.04~jammy_arm64.deb ...
Unpacking docker-ce-cli (5:27.5.1-1~ubuntu.22.04~jammy) over (5:28.0.0-1~ubuntu.22.04~jammy) ....................................] 
Replaced by files in installed package docker-ce (5:28.0.0-1~ubuntu.22.04~jammy) ...
dpkg: warning: downgrading docker-ce from 5:28.0.0-1~ubuntu.22.04~jammy to 5:27.5.1-1~ubuntu.22.04~jammy
Preparing to unpack .../docker-ce_5%3a27.5.1-1~ubuntu.22.04~jammy_arm64.deb .....................................................] 
Unpacking docker-ce (5:27.5.1-1~ubuntu.22.04~jammy) over (5:28.0.0-1~ubuntu.22.04~jammy) ........................................] 
dpkg: warning: downgrading docker-ce-rootless-extras from 5:28.0.0-1~ubuntu.22.04~jammy to 5:27.5.1-1~ubuntu.22.04~jammy.........] 
Preparing to unpack .../docker-ce-rootless-extras_5%3a27.5.1-1~ubuntu.22.04~jammy_arm64.deb .....................................] 
Unpacking docker-ce-rootless-extras (5:27.5.1-1~ubuntu.22.04~jammy) over (5:28.0.0-1~ubuntu.22.04~jammy) ...........................] 
dpkg: warning: downgrading docker-compose-plugin from 2.33.0-1~ubuntu.22.04~jammy to 2.32.4-1~ubuntu.22.04~jammy
Preparing to unpack .../docker-compose-plugin_2.32.4-1~ubuntu.22.04~jammy_arm64.deb ...
Unpacking docker-compose-plugin (2.32.4-1~ubuntu.22.04~jammy) over (2.33.0-1~ubuntu.22.04~jammy) ...
Setting up docker-buildx-plugin (0.20.0-1~ubuntu.22.04~jammy) ...
Setting up docker-compose-plugin (2.32.4-1~ubuntu.22.04~jammy) ...
Setting up docker-ce-cli (5:27.5.1-1~ubuntu.22.04~jammy) ...
Setting up docker-ce-rootless-extras (5:27.5.1-1~ubuntu.22.04~jammy) ...
Setting up docker-ce (5:27.5.1-1~ubuntu.22.04~jammy) ...
Processing triggers for man-db (2.10.2-1) ...
kkk@ORIN:~$ 

from the reply in this thread, proceeded with downgrading docker.

$ sudo apt install docker-ce=5:27.5.1-1~ubuntu.22.04~jammy \
                 docker-ce-cli=5:27.5.1-1~ubuntu.22.04~jammy \
                 docker-compose-plugin=2.32.4-1~ubuntu.22.04~jammy \
                 docker-buildx-plugin=0.20.0-1~ubuntu.22.04~jammy \
                 docker-ce-rootless-extras=5:27.5.1-1~ubuntu.22.04~jammy

and then you realised that it will be upgraded the next time you do an apt update upgrade:

kkk@ORIN:~$ sudo apt update
Hit:1 https://download.docker.com/linux/ubuntu jammy InRelease
Hit:2 https://esm.ubuntu.com/apps/ubuntu jammy-apps-security InRelease                                
Hit:3 http://ports.ubuntu.com/ubuntu-ports jammy InRelease                                            
Hit:4 https://esm.ubuntu.com/apps/ubuntu jammy-apps-updates InRelease           
Get:5 http://ports.ubuntu.com/ubuntu-ports jammy-updates InRelease [128 kB]
Hit:6 https://esm.ubuntu.com/infra/ubuntu jammy-infra-security InRelease      
Hit:7 https://esm.ubuntu.com/infra/ubuntu jammy-infra-updates InRelease        
Hit:8 https://repo.download.nvidia.com/jetson/common r36.4 InRelease           
Hit:9 http://ports.ubuntu.com/ubuntu-ports jammy-backports InRelease                       
Get:10 http://ports.ubuntu.com/ubuntu-ports jammy-security InRelease [129 kB]   
Hit:11 https://repo.download.nvidia.com/jetson/t234 r36.4 InRelease
Hit:12 https://repo.download.nvidia.com/jetson/ffmpeg r36.4 InRelease
Fetched 257 kB in 4s (71,6 kB/s)
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
27 packages can be upgraded. Run 'apt list --upgradable' to see them.
kkk@ORIN:~$ apt list --upgradable
Listing... Done
docker-buildx-plugin/jammy 0.21.0-1~ubuntu.22.04~jammy arm64 [upgradable from: 0.20.0-1~ubuntu.22.04~jammy]
docker-ce-cli/jammy 5:28.0.0-1~ubuntu.22.04~jammy arm64 [upgradable from: 5:27.5.1-1~ubuntu.22.04~jammy]
docker-ce-rootless-extras/jammy 5:28.0.0-1~ubuntu.22.04~jammy arm64 [upgradable from: 5:27.5.1-1~ubuntu.22.04~jammy]
docker-ce/jammy 5:28.0.0-1~ubuntu.22.04~jammy arm64 [upgradable from: 5:27.5.1-1~ubuntu.22.04~jammy]
docker-compose-plugin/jammy 2.33.0-1~ubuntu.22.04~jammy arm64 [upgradable from: 2.32.4-1~ubuntu.22.04~jammy]
iperf3/jammy-apps-security 3.9-1+deb11u1ubuntu0.1~esm1 arm64 [upgradable from: 3.9-1+deb11u1build0.22.04.1]
libavcodec58/jammy-apps-security 7:4.4.2-0ubuntu0.22.04.1+esm6 arm64 [upgradable from: 7:4.4.2-0ubuntu0.22.04.1]
libavfilter7/jammy-apps-security 7:4.4.2-0ubuntu0.22.04.1+esm6 arm64 [upgradable from: 7:4.4.2-0ubuntu0.22.04.1]
libavformat58/jammy-apps-security 7:4.4.2-0ubuntu0.22.04.1+esm6 arm64 [upgradable from: 7:4.4.2-0ubuntu0.22.04.1]
libavutil56/jammy-apps-security 7:4.4.2-0ubuntu0.22.04.1+esm6 arm64 [upgradable from: 7:4.4.2-0ubuntu0.22.04.1]
libde265-0/jammy-apps-security 1.0.8-1ubuntu0.3+esm1 arm64 [upgradable from: 1.0.8-1ubuntu0.3]
libiperf0/jammy-apps-security 3.9-1+deb11u1ubuntu0.1~esm1 arm64 [upgradable from: 3.9-1+deb11u1build0.22.04.1]
libjs-jquery-ui/jammy-apps-security 1.13.1+dfsg-1ubuntu0.1~esm1 all [upgradable from: 1.13.1+dfsg-1]
libopencv-core4.5d/jammy-apps-security 4.5.4+dfsg-9ubuntu4+esm1 arm64 [upgradable from: 4.5.4+dfsg-9ubuntu4]
libopenexr25/jammy-apps-security 2.5.7-1ubuntu0.1~esm1 arm64 [upgradable from: 2.5.7-1]
libpmix-dev/jammy-apps-security 4.1.2-2ubuntu1+esm1 arm64 [upgradable from: 4.1.2-2ubuntu1]
libpmix2/jammy-apps-security 4.1.2-2ubuntu1+esm1 arm64 [upgradable from: 4.1.2-2ubuntu1]
libpostproc55/jammy-apps-security 7:4.4.2-0ubuntu0.22.04.1+esm6 arm64 [upgradable from: 7:4.4.2-0ubuntu0.22.04.1]
libpython2.7-minimal/jammy-apps-security 2.7.18-13ubuntu1.5+esm3 arm64 [upgradable from: 2.7.18-13ubuntu1.5]
libpython2.7-stdlib/jammy-apps-security 2.7.18-13ubuntu1.5+esm3 arm64 [upgradable from: 2.7.18-13ubuntu1.5]
libpython2.7/jammy-apps-security 2.7.18-13ubuntu1.5+esm3 arm64 [upgradable from: 2.7.18-13ubuntu1.5]
libswresample3/jammy-apps-security 7:4.4.2-0ubuntu0.22.04.1+esm6 arm64 [upgradable from: 7:4.4.2-0ubuntu0.22.04.1]
libswscale5/jammy-apps-security 7:4.4.2-0ubuntu0.22.04.1+esm6 arm64 [upgradable from: 7:4.4.2-0ubuntu0.22.04.1]
libzbar0/jammy-apps-security 0.23.92-4ubuntu0.1~esm1 arm64 [upgradable from: 0.23.92-4build2]
python2.7-minimal/jammy-apps-security 2.7.18-13ubuntu1.5+esm3 arm64 [upgradable from: 2.7.18-13ubuntu1.5]
python2.7/jammy-apps-security 2.7.18-13ubuntu1.5+esm3 arm64 [upgradable from: 2.7.18-13ubuntu1.5]
python3-scipy/jammy-apps-security 1.8.0-1exp2ubuntu1+esm1 arm64 [upgradable from: 1.8.0-1exp2ubuntu1]
kkk@ORIN:~$ 

use aptitude to help visually hold back the docker related files from upgrading.

safely update and upgrade everything else.

kkk@ORIN:~$ sudo apt update
Hit:1 http://ports.ubuntu.com/ubuntu-ports jammy InRelease
Hit:2 http://ports.ubuntu.com/ubuntu-ports jammy-updates InRelease                                                                                                             
Hit:3 https://download.docker.com/linux/ubuntu jammy InRelease                                                                                                                 
Hit:4 http://ports.ubuntu.com/ubuntu-ports jammy-backports InRelease                                                                                                           
Hit:5 http://ports.ubuntu.com/ubuntu-ports jammy-security InRelease                                                          
Hit:6 https://esm.ubuntu.com/apps/ubuntu jammy-apps-security InRelease                                
Hit:7 https://repo.download.nvidia.com/jetson/common r36.4 InRelease
Hit:8 https://esm.ubuntu.com/apps/ubuntu jammy-apps-updates InRelease           
Hit:9 https://esm.ubuntu.com/infra/ubuntu jammy-infra-security InRelease        
Hit:10 https://esm.ubuntu.com/infra/ubuntu jammy-infra-updates InRelease        
Hit:11 https://repo.download.nvidia.com/jetson/t234 r36.4 InRelease             
Hit:12 https://repo.download.nvidia.com/jetson/ffmpeg r36.4 InRelease
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
7 packages can be upgraded. Run 'apt list --upgradable' to see them.
kkk@ORIN:~$ sudo apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages have been kept back:
  docker-buildx-plugin docker-ce docker-ce-cli docker-ce-rootless-extras docker-compose-plugin libpmix-dev libpmix2
0 upgraded, 0 newly installed, 0 to remove and 7 not upgraded.
kkk@ORIN:~$ 

wait… what now???

kkk@ORIN:~$ sudo dockerd
[sudo] password for kkk: 
INFO[2025-02-23T17:47:57.569523504+07:00] Starting up                                  
failed to start daemon, ensure docker is not running or delete /var/run/docker.pid: process with PID 1515 is still running
kkk@ORIN:~$ docker ps -a
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get "http://%2Fvar%2Frun%2Fdocker.sock/v1.47/containers/json?all=1": dial unix /var/run/docker.sock: connect: permission denied
kkk@ORIN:~$ docker images
permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Head "http://%2Fvar%2Frun%2Fdocker.sock/_ping": dial unix /var/run/docker.sock: connect: permission denied
kkk@ORIN:~$ 
kkk@ORIN:~$ sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
c9c5fd25a1bd: Pull complete 
Digest: sha256:e0b569a5163a5e6be84e210a2587e7d447e08f87a0e90798363fa44a0464a1e8
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (arm64v8)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

kkk@ORIN:~$ 

okay docker is running now after downgrading.

continue with the other setup options like adding user to docker group and all that. Ubuntu | Docker Docs

Hi,

Now docker v28.0.0 is not working on r36.4.3. Downgrading to v27.5.1 can fix this issue.
Another workaround is to build a custom kernel with the required configuration enabled.

You can find more info in the below comment:

If kernel compiling doesn’t work on WSL2, please try our cross-compiling container below:

Thanks.