K3s pods CrashLoopBackOff on Jetson Xavier NX

Hi NVIDIA team,

I’m starting to work with k3s and I’m trying to run a k3s cluster on a Jetson Xavier NX board with JetPack 4.6:

$ uname -srvmpio
Linux 4.9.253-tegra #1 SMP PREEMPT Mon Jul 26 12:19:28 PDT 2021 aarch64 aarch64 aarch64 GNU/Linux

This is the example I’ve been trying to replicate: The first experience with k3s (lightweight Kubernetes). Deploy your first app! - DEV Community 👩‍💻👨‍💻

Every command seems to execute properly, but when I check the pods status, it shows CrashLoopBackOff:

$ k3s kubectl get ingress,svc,pods -n retail-project-dev
NAME                                                   CLASS    HOSTS   ADDRESS        PORTS   AGE
ingress.networking.k8s.io/simple-rest-golang-ingress   <none>   *       192.168.0.17   80      21m

NAME                                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
service/simple-rest-golang-service   ClusterIP   10.43.205.169   <none>        80/TCP    21m

NAME                                                READY   STATUS             RESTARTS   AGE
pod/simple-rest-golang-deployment-d96cb9d78-54k6v   0/1     CrashLoopBackOff   8          21m
pod/simple-rest-golang-deployment-d96cb9d78-224vf   0/1     CrashLoopBackOff   8          21m

I followed the same steps on my laptop:

$ uname -srvmpio
Linux 5.4.0-90-generic #101~18.04.1-Ubuntu SMP Fri Oct 22 09:25:04 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

And the pods get to running state without any issue:

$ k3s kubectl get ingress,svc,pods -n retail-project-dev
NAME                                                   CLASS    HOSTS   ADDRESS        PORTS   AGE
ingress.networking.k8s.io/simple-rest-golang-ingress   <none>   *       192.168.0.27   80      2m51s

NAME                                 TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)   AGE
service/simple-rest-golang-service   ClusterIP   10.43.128.18   <none>        80/TCP    2m51s

NAME                                                READY   STATUS    RESTARTS   AGE
pod/simple-rest-golang-deployment-d96cb9d78-d8cn5   1/1     Running   0          2m51s
pod/simple-rest-golang-deployment-d96cb9d78-dmddt   1/1     Running   0          2m51s

I also used a script that is supposed to verify if the kernel configuration meet the requirements to run kubernetes. You can find the script here: moby/check-config.sh at master · moby/moby · GitHub

This is the result of running the script on the Jetson board:

$ ./check-config.sh 
info: reading kernel config from /proc/config.gz ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled (as module)
- CONFIG_BRIDGE: enabled
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
- CONFIG_NETFILTER_XT_MARK: enabled (as module)
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_POSIX_MQUEUE: enabled
- CONFIG_NF_NAT_IPV4: enabled (as module)
- CONFIG_NF_NAT_NEEDED: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_SECCOMP_FILTER: enabled
- CONFIG_CGROUP_PIDS: enabled
- CONFIG_MEMCG_SWAP: enabled
- CONFIG_MEMCG_SWAP_ENABLED: enabled
    (cgroup swap accounting is currently enabled)
- CONFIG_IOSCHED_CFQ: enabled
- CONFIG_CFQ_GROUP_IOSCHED: missing
- CONFIG_BLK_CGROUP: enabled
- CONFIG_BLK_DEV_THROTTLING: enabled
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: enabled
- CONFIG_NET_CLS_CGROUP: enabled
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: missing
- CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_PROTO_TCP: enabled
- CONFIG_IP_VS_PROTO_UDP: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_SECURITY_SELINUX: missing
- CONFIG_SECURITY_APPARMOR: missing
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: enabled
    - CONFIG_BRIDGE_VLAN_FILTERING: enabled
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: enabled
      - CONFIG_CRYPTO_GCM: enabled
      - CONFIG_CRYPTO_SEQIV: enabled
      - CONFIG_CRYPTO_GHASH: enabled
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled
      - CONFIG_XFRM_ALGO: enabled
      - CONFIG_INET_ESP: enabled (as module)
      - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled
  - "ipvlan":
    - CONFIG_IPVLAN: enabled
  - "macvlan":
    - CONFIG_MACVLAN: enabled (as module)
    - CONFIG_DUMMY: enabled
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_FTP: enabled (as module)
    - CONFIG_NF_NAT_TFTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: enabled (as module)
    - CONFIG_BTRFS_FS_POSIX_ACL: enabled
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: enabled
    - CONFIG_DM_THIN_PROVISIONING: missing
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000

This shows that all the “Generally Necessary” options are enabled.

Is there any known issues with k3s and the Jetson boards or maybe related to JetPack 4.6 that may cause this behavior?

Thank you!

Have a try this version from below link. K3s: v1.19.7+k3s1 (arm64 version)

Thank you very much for your help,
It turns out that the issue was not the version of k3s, but the example itself.
The image used to deploy the k3s pods was made for x86-64, so I had to search for another one made for arm and it worked fine.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.