Any software update to JetPack 6.0 causes docker to fail

After using SDK-Manager to create a boot NVMe drive I hade everything working well, using dusty-nv docker containers. However I made the mistake of accepting the software update that appears every time you boot. Any update as all will corrupt ubuntu 22.04 and docker will not run and is unrepairable. You have to go through the 1 hour long rebuild of the NVMe driver back to a clean copy of JetPack 6.0

Hi Simon. I am experiencing the exact same issue now. I boot my Orin Nano from an SD Card and the update just kicked off by itself this morning. Can you outline you steps for fixing this issue? Thanks.

Dusty, I followed the same link, and it does not work. I still get a bridge error in docker.

Hi @simon.bunn ,

Could you just test if the following works? I also had similar problems with JP 6.0 and trying see if they are related.

sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

sudo systemctl restart docker
4 Likes

@doruk.sonmez1 Worked first time! Did not have to run any of the bridge commands.

Hi @simon.bunn,

That’s great to hear! Here is the explanation if you are interested. If there is any wrong in my answer, we should also see it here.

I started by looking at /var/log/syslog just after doing sudo systemctl restart docker, and it tells a little bit about the issue. Here is the starting point of the issue:

Feb 27 14:09:42 ubuntu systemd[1]: Starting Docker Application Container Engine...
Feb 27 14:09:42 ubuntu dockerd[7854]: time="2024-02-27T14:09:42.137289371+03:00" level=info msg="Starting up"
Feb 27 14:09:42 ubuntu dockerd[7854]: time="2024-02-27T14:09:42.138304422+03:00" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
Feb 27 14:09:42 ubuntu systemd[1]: var-lib-docker-overlay2-metacopy\x2dcheck3567031384-merged.mount: Deactivated successfully.
Feb 27 14:09:42 ubuntu dockerd[7854]: time="2024-02-27T14:09:42.182083583+03:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Feb 27 14:09:42 ubuntu dockerd[7854]: time="2024-02-27T14:09:42.184203734+03:00" level=info msg="Loading containers: start."
Feb 27 14:09:42 ubuntu dockerd[7854]: time="2024-02-27T14:09:42.410130016+03:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
Feb 27 14:09:42 ubuntu dockerd[7854]: failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to register "bridge" driver: unable to add return rule in DOCKER-ISOLATION-STAGE-1 chain:  (iptables failed: iptables --wait -A DOCKER-ISOLATION-STAGE-1 -j RETURN: iptables v1.8.7 (nf_tables):  RULE_APPEND failed (No such file or directory): rule in chain DOCKER-ISOLATION-STAGE-1
Feb 27 14:09:42 ubuntu dockerd[7854]:  (exit status 4))
Feb 27 14:09:42 ubuntu systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Feb 27 14:09:42 ubuntu systemd[1]: docker.service: Failed with result 'exit-code'.

We can see that Docker fails to start because it can not "register bridge driver". Next, it complains about some iptables command that is running in the background and having an error: iptables failed: iptables --wait -A DOCKER-ISOLATION-STAGE-1 -j RETURN: iptables v1.8.7 (nf_tables).

Now, according to this post (Chris's Wiki :: blog/linux/NftablesUbuntu2204Experience), “iptables” command started to use nftables for creating the rules starting with Ubuntu 22.04 (iptables-nft package). And once I checked the nftables configuration (/etc/nftables.conf), it had a parameter: flush ruleset. This basically means that every time you accept system upgrades and/or do reboots on Jetsons with JetPack 6.0 DP, you will end up with “flushed iptables rules” because nftables overwrites the rules. This will cause Docker not to be able to start because the service is not able to create iptables rules, all over again. So as an alternative, using the iptables-legacy package with the commands I have shared in my earlier comment should do the trick.

1 Like

I have now updated twice, and also added Ubuntu Pro. All still working after many reboots. Looks like this fix is permanent

2 Likes

That’s great! Sorry for the late response, I didn’t have any chance to check it on the weekend.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.