Problems with Docker version 28.0.1 on Jetson Orin NX

christopher.avila · March 14, 2025, 9:53pm

After everything else this fixed it for me, thanks.
I get these are dev kits but man it’s been nothing but a pain using this Orin.

bashaku · March 14, 2025, 11:44pm

Docker 28.0.2 is expected to be released next week.
This release will fix the problem.

Adding

[Service]
Environment="DOCKER_INSECURE_NO_IPTABLES_RAW=1"

to docker.service unit file will make it work even with kernels that do not have CONFIG_IP_NF_RAW.

github.com/moby/moby

Add an opt-out for iptables 'raw' rules

master ← robmry:skip_raw_rules

opened 04:01PM - 10 Mar 25 UTC

robmry

+97 -9

**- What I did** - fix https://github.com/moby/moby/issues/49557 For kerne…ls that don't have `CONFIG_IP_NF_RAW`, if the env var `DOCKER_INSECURE_NO_IPTABLES_RAW` is set to `"1"`, don't try to create raw rules. **_Warning:_** When the environment variable is set, direct routing to published ports is possible from other hosts on the local network, even if the port is published to a loopback address. It un-does some of the security hardening described at https://www.docker.com/blog/docker-engine-28-hardening-container-networking-by-default/ **- How I did it** The env var is `DOCKER_INSECURE_NO_IPTABLES_RAW` ... because that's why the workaround is needed. Alternatively, it could be called something like `DOCKER_INSECURE_ALLOW_DIRECT_ROUTING` ... because that's currently the effect. Then it'd need to do the same thing for an nftables/firewalld implementation. If we want to add a more "feature-y" way to allow direct routing at some point - it should be via a new "gateway mode" with well defined semantics, something less drastic than `nat-unprotected` (allowing access from remote hosts, but only to published ports, and maybe not to ports published to `127.0.0.1`). That'd work per network rather than globally, and it'd need some different regression testing and more documentation. So, I went for this simple workaround for kernels without the required module. **- How to verify it** New integration test. Also, checked a container with host port mapping started on a host without `CONFIG_IP_NF_RAW` ... ``` # cat /etc/systemd/system/docker.service.d/insecure_direct_routing.conf [Service] Environment="DOCKER_INSECURE_NO_IPTABLES_RAW=1" # systemctl daemon-reload # systemctl restart docker # docker run --rm -ti -p 127.0.0.1:8080:80 alpine ``` In docker's log ... ``` Mar 10 15:38:50 debian2 dockerd[7232]: time="2025-03-10T15:38:50.830162607Z" level=debug msg="DOCKER_INSECURE_NO_IPTABLES_RAW=1 - skipping raw rules" eid=3c070182713114fcf3b58f606cd0d2dc773516360c0bd9eed60bb4899d90d467 ep=elated_keldysh net=bridge nid=b30950846c3a84d5e17e8cf0406ff435a76323e87d41d3fdf47262032e670af6 Mar 10 15:38:50 debian2 dockerd[7232]: time="2025-03-10T15:38:50.830260399Z" level=debug msg="DOCKER_INSECURE_NO_IPTABLES_RAW=1 - skipping raw rules" eid=3c070182713114fcf3b58f606cd0d2dc773516360c0bd9eed60bb4899d90d467 ep=elated_keldysh net=bridge nid=b30950846c3a84d5e17e8cf0406ff435a76323e87d41d3fdf47262032e670af6 ``` (And, no rules created in the iptables/ip6tables `raw` table.) **- Human readable description for the release notes** ```markdown changelog Add environment variable `DOCKER_INSECURE_NO_IPTABLES_RAW=1` to allow Docker to run on systems where the Linux kernel can't provide `CONFIG_IP_NF_RAW` support. When enabled, Docker will not create rules in the iptables `raw` table. Warning: This is not recommended for production environments as it reduces security by allowing other hosts on the local network to route to ports published to host addresses, even when they are published to `127.0.0.1.` This option bypasses some of the security hardening introduced in Docker Engine 28.0.0. ```

There is one thing to note :

Warning: This is not recommended for production environments as it reduces security by allowing other hosts on the local network to route to ports published to host addresses, even when they are published to 127.0.0.1. This option bypasses some of the security hardening introduced in Docker Engine 28.0.0.

yash24 · March 20, 2025, 1:01pm

i am till facing the same issue in Docker version 28.0.2

AastaLLL · March 24, 2025, 6:50am

Hi,

Could you downgrade the docker to v27.5.1 to avoid this issue?

$ sudo apt-get install -y docker-ce=5:27.5* docker-ce-cli=5:27.5* --allow-downgrades

We have enabled the kernel in our internal branch so the module will exist by default in our future release.

Thanks.

yash24 · March 24, 2025, 8:51am

+Satish Rathod

Please check and revert.

Regards,
Yash

crobisaur · March 25, 2025, 6:27pm

Hello,

We have tried this on our AGX orin with no success, containerd failed to start using systemctl.

[1]: Starting Docker Socket for the API...
Mar 26 02:23:04 tegra-ubuntu systemd[1]: Listening on Docker Socket for the API.
Mar 26 02:23:04 tegra-ubuntu systemd[1]: docker.service: Start request repeated too quickly.
Mar 26 02:23:04 tegra-ubuntu systemd[1]: docker.service: Failed with result 'exit-code'.
Mar 26 02:23:04 tegra-ubuntu systemd[1]: Failed to start Docker Application Container Engine.
Mar 26 02:23:04 tegra-ubuntu systemd[1]: docker.socket: Failed with result 'service-start-limit-hit'.

I’m going to try building a kernel in the meantime

AastaLLL · March 26, 2025, 6:27am

Hi, @crobisaur

Which docker version do you use?
For the latest docker v28.0.2, you will need to enable the below config:

CONFIG_IP_NF_RAW=m

If downgrading docker is an option for you, docker v27.5.1 should work normally without updating the kernel.

$ sudo apt-get install -y docker-ce=5:27.5* docker-ce-cli=5:27.5* --allow-downgrades

Thanks.

crobisaur · March 26, 2025, 8:35pm

Found a solution,

We found that containerd and docker.io from the ubuntu repositories are compatible with eachother so long that we just disable iptables in the docker daemon.json file
"iptables": false

After that docker worked for our use case. If you still need iptables for your setup then this might not be as helpful.

cheers.

tbrinks · April 2, 2025, 4:01pm

I’m having the same issue as everyone previously described. As of April 2, Docker version 28.0.4 does not solve the issue.

Apr 02 09:44:55 ubuntu dockerd[10634]: time="2025-04-02T09:44:55.436052112-06:00" level=error msg="Handler for POST /v1.48/containers/be7bf9599ffd3a4b6d368c74b5f2916b21017f9351c2efd6ac52712ad1b349d5/start returned error: failed to set up container networking: driver failed programming external connectivity on endpoint push-to-talk (6dcc3523466e7495750bce771d748da297e51dac2670fe115bcf14550667911d): Unable to enable DIRECT ACCESS FILTERING - DROP rule: (iptables failed: iptables --wait -t raw -A PREROUTING -p tcp -d 172.18.0.2 --dport 5235 ! -i br-fb442e96f9f5 -j DROP: iptables v1.8.7 (legacy): can't initialize iptables table raw: Table does not exist (do you need to insmod?)\nPerhaps iptables or your kernel needs to be upgraded.\n (exit status 3))" spanID=23f4bab18d020a7e traceID=94dc82cf7a3a4536e6dc354b9d895612

Client: Docker Engine - Community
 Version:    28.0.4
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.22.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.34.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 6
 Server Version: 28.0.4
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 05044ec0a9a75232cad458027ca83437aae3f4da
 runc version: v1.2.5-0-g59923ef
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 5.15.148-tegra
 Operating System: Ubuntu 22.04.5 LTS
 OSType: linux
 Architecture: aarch64
 CPUs: 8
 Total Memory: 61.37GiB
 Name: ubuntu
 ID: cf97503d-95d0-4b08-9ea8-72ccfd46aec0
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false

pzolotykh · April 3, 2025, 10:24am

Downgrade to 27.5.1 solved for me for JetPack 6.2

AastaLLL · April 7, 2025, 6:00am

Hi, @tbrinks

Please try to downgrade the docker to v27 or update the kernel.

Thanks.

Topic		Replies	Views
Error with “Nvidia Container Runtime with Docker Integration” on AGX Orin with JP6.2 Jetson AGX Orin containers	35	4190	March 17, 2025
Docker 28 still fails on JetPack 6.2.1 (L4T 36.4.4) Jetson AGX Orin docker	17	1189	August 28, 2025
Docker broken upon update to v28 - jetson-containers won't run, reinstalling ubuntu with SDK manager seems to end wit broken installs Jetson AGX Orin sdkm , docker	7	690	February 24, 2025
Another solution with iptables raw issue Jetson Orin Nano docker	3	1221	June 16, 2025
Iptables error message Jetson Orin Nano kernel , generative_ai	2	1112	May 13, 2025
Jetson - Docker - Failed to set up container networking Jetson Orin Nano docker	1	2022	April 28, 2025
Docker In Docker Error in Nvidia Jetpack Environment Jetson Orin Nano docker	3	429	March 31, 2025
Iptables NAT rules on L4T 36.4.3 Jetson Orin NX jetpack , kernel , jetson , jetson-orin	2	119	December 23, 2025
JetPack 6.0 docker integration broken exposing ports Jetson AGX Xavier docker , containers	1	374	March 11, 2025
Any software update to JetPack 6.0 causes docker to fail Jetson Orin Nano docker	8	2243	March 4, 2024

Problems with Docker version 28.0.1 on Jetson Orin NX

Related topics