Jetson AGX ORIN + k3s + network policies

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
1

Context

Jetson Orin AGX developper kit 64G
jetpack 6.2
ubuntu 22.04
Secutrity issues : Need to apply network policies inside kubernetes cluster

Install k3s
curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC=“–write-kubeconfig-mode 644 --disable=traefik --disable servicelb --node-ip=203.0.113.254 --advertise-address=203.0.113.254 --flannel-backend=host-gw --flannel-iface=dummy0” sh -

systemctl status k3s ==>
E0708 network_policy_controller.go:295] Aborting sync. Failed to sync network policy chains: failed to perform ipset restore: ipset v7.16: Error in line 1: Kernel error received: set type not supported

NetworkPolicies (kube-router in K3s) needs following kernel modules:

  • ipset : ip_set + usuel types (ip_set_hash_ip, ip_set_hash_net, ip_set_hash_ipport, ip_set_hash_netport, …)
  • xtables “set” : xt_set
  • nfnetlink

sudo modprobe ip_set ip_set_hash_ip ip_set_hash_net ip_set_hash_ipport ip_set_hash_netport xt_set nfnetlink
modprobe: FATAL: Module ip_set not found in directory /lib/modules/5.15.148-tegra
lsmod | egrep ‘ip_set|xt_set|nfnetlink’
nfnetlink 20480 1 nf_tables
sudo ipset create test hash:ip
ipset v7.15: Kernel error received: Invalid argument

Goal: compile and load missing modules:
cd ~
wget https://developer.nvidia.com/embedded/L4T/r36_Release_v4.0/sources/public_sources.tbz2
tar xf public_sources.tbz2
cd Linux_for_Tegra/source/
tar xf kernel_src.tbz2
cd kernel/kernel-jammy-src
zcat /proc/config.gz > .config
scripts/config --set-str CONFIG_LOCALVERSION “-tegra”

scripts/config --module CONFIG_IP_SET_HASH_IP
scripts/config --module CONFIG_IP_SET_HASH_IPPORT
scripts/config --module CONFIG_IP_SET_HASH_IPPORTIP
scripts/config --module CONFIG_IP_SET_HASH_IPPORTNET
scripts/config --module CONFIG_IP_SET_HASH_NETPORT
scripts/config --module CONFIG_IP_SET_HASH_NETNET
scripts/config --module CONFIG_IP_SET_HASH_NETIFACE
scripts/config --module CONFIG_IP_SET_HASH_IPMARK
scripts/config --module CONFIG_IP_SET_HASH_MAC
scripts/config --module CONFIG_IP_SET_BITMAP_IP
scripts/config --module CONFIG_IP_SET_BITMAP_IPMAC
scripts/config --module CONFIG_IP_SET_BITMAP_PORT
make olddefconfig
make clean
make prepare
make modules_prepare
make -j"$(nproc)" modules
make LOCALVERSION=-tegra M=net/netfilter/ipset clean
make LOCALVERSION=-tegra M=net/netfilter/ipset modules -j"$(nproc)"

sudo cp net/netfilter/ipset/ip_set*.ko /lib/modules/$(uname -r)/kernel/net/netfilter/ipset/
sudo depmod -a

==> Load and check:
sudo modprobe ip_set

sudo modprobe ip_set_hash_ip ip_set_hash_net ip_set_hash_ipport ip_set_hash_netport

sudo ipset create test hash:ip && sudo ipset add test 1.2.3.4 && sudo ipset destroy test ==> OK , no more error

Expected behavior : network policy should deny trafic
sudo k3s kubectl -n test run nginx --image=nginx --restart=Never --port=80
sudo k3s kubectl -n test run client --image=busybox:1.35 --restart=Never – sleep 3600
sudo k3s kubectl -n test exec client – wget -qO- http://$NGINX_IP:80 | head -n 2
==>
Default Deny :
cat <<‘EOF’ | sudo k3s kubectl -n test apply -f -
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
spec:
podSelector: {}
policyTypes:
Ingress
EOF
sudo k3s kubectl -n test exec client – wget -qO- http://$NGINX_IP:80 | head -n 2
==>
Deny is not applied !

any explanation please?

5

Hi,
Please try the solution:
Kubernetes on Jetson Orin Nanos - #15 by AastaLLL

6

Thank you for your response.
The goal in my case is to apply NetworkPolicies in the k3s cluster.

First scenario:
With the default installation of k3s (using containerd) → the cluster works fine for deploying applications in different namespaces.

  • Case 1: With Flannel → the “NetworkPolicy” object is created in the cluster, but it’s not actually enforced.
  • Case 2: Disable Flannel and install Calico →
kubectl -n kube-system get po | grep calico-node
kube-system calico-node-8mznq    0/1      Running 

kubectl -n kube-system logs daemonset/calico-node --tail=20

2025-10-21 21:02:25.992 [WARNING][228508] felix/ipsets.go 367: Failed to update IP sets. Marking dataplane for resync. error=failed to write one or more IP set: exit status 1 family="inet"
2025-10-21 21:02:26.248 [INFO][228508] felix/ipsets.go 344: Retrying after an ipsets update failure... family="inet"
2025-10-21 21:02:26.253 [INFO][228508] felix/ipsets.go 614: Resync found members missing from dataplane. family="inet" numMissing=1 setName="cali40all-hosts-net"
2025-10-21 21:02:26.262 [WARNING][228508] felix/ipsets.go 795: Failed to complete ipset restore, IP sets may be out-of-sync. closeErr=<nil> commitErr=<nil> family="inet" flushErr=<nil> input="create cali40this-host hash:ip family inet maxelem 1048576\nadd cali40this-host 172.17.0.1\nadd cali40this-host 10.42.0.1\nadd cali40this-host 127.0.0.0\nadd cali40this-host 127.0.0.1\nadd cali40this-host 192.168.1.166\nadd cali40this-host 203.0.113.254\nadd cali40all-hosts-net 203.0.113.254/32\nCOMMIT\n" processErr=exit status 1 stderr="ipset v7.11: Error in line 1: Kernel error received: set type not supported\n" stdout="" writeErr=<nil>
2025-10-21 21:02:26.262 [WARNING][228508] felix/ipsets.go 367: Failed to update IP sets. Marking dataplane for resync. error=failed to write one or more IP set: exit status 1 family="inet"
2025-10-21 21:02:26.459 [INFO][228508] felix/ipip_mgr.go 107: Failed to get IPIP tunnel device, assuming it isn't present error=Link not found
2025-10-21 21:02:26.466 [WARNING][228508] felix/ipip_mgr.go 113: Failed to add IPIP tunnel device error=exit status 1
2025-10-21 21:02:26.466 [WARNING][228508] felix/ipip_mgr.go 90: Failed configure IPIP tunnel device, retrying... error=exit status 1

These messages keep repeating.

Second scenario:
Reinstall k3s with Docker as you suggested →
The same result and the same ‘error’ message, whether with Flannel or with Calico!
Switching from Containerd to Docker didn’t change anything at all.

Would you have an explanation for this?
The only remaining possibility seems to be recompiling the kernel to add the missing modules?

7

Hi,
I saw here (Kubernetes on Jetson Orin Nanos - #11 by AastaLLL) according to @AastaLLL response, that “Currently, we are not able to use iptables-nft on JetPack 6 due to some kernel configuration issues”
So I think it’s not worth compiling the kernel !
Do you have any idea if this issue is resolved on JetPack 7 (not necessarily on ORIN in that case)?
Thanks

8

Hi,
I just tested on Jetson THOR with JetPack 7.0, and indeed it’s the same modules that are missing:

zgrep -E 'CONFIG_IP_SET|CONFIG_IP_SET_HASH' /proc/config.gz || \ grep -E 'CONFIG_IP_SET|CONFIG_IP_SET_HASH' /boot/config-$(uname -r) 
CONFIG_IP_SET=m 
CONFIG_IP_SET_MAX=256 
# CONFIG_IP_SET_BITMAP_IP is not set 
# CONFIG_IP_SET_BITMAP_IPMAC is not set 
# CONFIG_IP_SET_BITMAP_PORT is not set 
# CONFIG_IP_SET_HASH_IP is not set 
# CONFIG_IP_SET_HASH_IPMARK is not set 
# CONFIG_IP_SET_HASH_IPPORT is not set 
# CONFIG_IP_SET_HASH_IPPORTIP is not set 
# CONFIG_IP_SET_HASH_IPPORTNET is not set 
# CONFIG_IP_SET_HASH_IPMAC is not set 
# CONFIG_IP_SET_HASH_MAC is not set 
# CONFIG_IP_SET_HASH_NETPORTNET is not set 
# CONFIG_IP_SET_HASH_NET is not set 
# CONFIG_IP_SET_HASH_NETNET is not set 
# CONFIG_IP_SET_HASH_NETPORT is not set 
# CONFIG_IP_SET_HASH_NETIFACE is not set 
# CONFIG_IP_SET_LIST_SET is not set

I deduce that the firewall part (network policies) in k3s won’t work either.
Any confirmations? Explanations?

Thanks.

9

Hi,

I tried to build a new kernel (for JEtson AGX Orin jetpack 6.2) integrating missing modules using @Jetsonhacks scripts ( jetsonhacks/jetson-orin-kernel-builder: Build the Linux kernel and modules on board the Jetson AGX Orin, Orin Nano or Orin NX)

Steps described here : Reboot fails after kernel building & installing · Issue #4 · jetsonhacks/jetson-orin-kernel-builder

The reboot of the node fails when setting

CONFIG_IP_MULTIPLE_TABLES=y

can’t explane it !
Any help ?

Thanks

10

Hi,
Please refer to kernel customization section in developer guide. And this post:
No data from Joystick Logitech-f710 - #10 by DaneLLL