K3S installation on Jetson AGX Orin with Jetpack 6

madiaza · February 19, 2024, 9:00am

Hello,

I’ve been doing some tests to try to get K3S running on a Jetson AGX Orin with Jetpack 6, and I did not managed to have it working. I tried all the flannel options, as the default (vxlan) was not working as observed on the K3S log. With none of the different network options K3S default containers starts. Depending on the option, their status keep pending forever or enter into a crash loop.
After this, I tried to use docker instead of containerd as runtime with no luck, same result.

Has anyone managed to install K3S on a Jetson AGX with Jetpack 6? I tested on a Jetson nano with Jetpack 4 and on a Jetson Orin Nano with Jetpack 5.11 and on both of them works fine and GPU can be mapped to the docker containers, but no luck with Jetpack 6 on an AGX.

Thanks a lot for the help!

AastaLLL · February 20, 2024, 3:15am

Hi,

As you mentioned, K3S should work on JetPack 4 and JetPack 5.

We don’t notice that it is not working with JetPack 6.
Is there any error log or message about the failure so we can know more about the issue?

Thanks.

madiaza · February 21, 2024, 2:16pm

Hello. This is a partial log just after the K3S installation:

Jan 27 23:16:29 ubuntu k3s[9030]: I0127 23:16:29.678750 9030 serving.go:355] Generated self-signed cert in-memory
Jan 27 23:16:29 ubuntu k3s[9030]: time=“2024-01-27T23:16:29+01:00” level=info msg=“Stopped tunnel to 127.0.0.1:6443”
Jan 27 23:16:29 ubuntu k3s[9030]: time=“2024-01-27T23:16:29+01:00” level=info msg=“Proxy done” err=“context canceled” url=“wss://127.0.0.1:6443/v1-k3s/connect”
Jan 27 23:16:29 ubuntu k3s[9030]: time=“2024-01-27T23:16:29+01:00” level=info msg=“Connecting to proxy” url=“wss://192.168.2.110:6443/v1-k3s/connect”
Jan 27 23:16:29 ubuntu k3s[9030]: time=“2024-01-27T23:16:29+01:00” level=info msg=“error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF”
Jan 27 23:16:29 ubuntu k3s[9030]: I0127 23:16:29.701143 9030 kube.go:152] Node controller sync successful
Jan 27 23:16:29 ubuntu k3s[9030]: I0127 23:16:29.701235 9030 vxlan.go:141] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false
Jan 27 23:16:29 ubuntu k3s[9030]: time=“2024-01-27T23:16:29+01:00” level=info msg=“Handling backend connection request [ubuntu]”
Jan 27 23:16:29 ubuntu k3s[9030]: time=“2024-01-27T23:16:29+01:00” level=fatal msg=“flannel exited: failed to register flannel network: operation not supported”
Jan 27 23:16:29 ubuntu systemd[1]: k3s.service: Main process exited, code=exited, status=1/FAILURE
Jan 27 23:16:29 ubuntu systemd[1]: k3s.service: Failed with result ‘exit-code’.
Jan 27 23:16:29 ubuntu systemd[1]: k3s.service: Consumed 12.528s CPU time.

Thanks for your help!

AastaLLL · February 22, 2024, 4:42am

Hi,

Based on the below link, it looks like the issue comes from the k3s rather than the Jetson environment.
Is the k3s version of JetPack 5 and 6 identical?

github.com/rancher/rancher

msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"

opened 01:35PM - 22 Feb 20 UTC

firaskafri

internal

Clusters keeps disconnecting on Rancher HA installation with the below log that …shows repeatedly on cluster-agent on all clusters: ``` time="2020-02-22T13:31:39Z" level=info msg="Connecting to wss://rancher.website.com/v3/connect with token *****" time="2020-02-22T13:31:39Z" level=info msg="Connecting to proxy" url="wss://rancher.website.com/v3/connect" time="2020-02-22T13:33:17Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF" ``` Clusters themselves are healthy and operational, but connection to rancher is continuously lost every few mins. gz#13407

Thanks

mygapula.sudheer · February 26, 2024, 9:42am

Hello ,
Am facing same issue but looks to me issue is vxlan missing , Error registering network: operation not supported · Issue #1028 · flannel-io/flannel · GitHub .

Could you please suggest how to install vxlan kernal module .

Thanks ,
sudheer.

logs:

10:43 AM

k3s.service: Failed with result ‘exit-code’.

systemd

10:43 AM

k3s.service: Main process exited, code=exited, status=1/FAILURE

systemd

10:43 AM

time=“2024-02-26T10:43:01+01:00” level=fatal msg=“flannel exited: failed to register flannel network: operation not supported”

k3s

10:43 AM

I0226 10:43:01.408777 133075 vxlan.go:141] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false

AastaLLL · February 27, 2024, 6:05am

Hi,

Do you also meet the issue on JetPack 6?
Thanks.

mygapula.sudheer · February 27, 2024, 6:18am

yes with JetPack 6

Note: i have not tested with lower versions of JetPack .

Thanks,
sudheer.

madiaza · February 27, 2024, 8:57am

Hello,

Yes, it was the exact same k3s version

madiaza · February 27, 2024, 9:01am

I also tested all the different network installation options, described here: Network Options | K3s and none of them worked. I also tried to use docker instead containerd as the runtime, and no luck either.

AastaLLL · March 1, 2024, 6:39am

Hi, both

Thanks for this info.

We will check this internally and update more info with you later.
At the same time, would you mind sharing the installation steps that work on JetPack 5 but fail on JetPack 6 with us?

Thanks.

madiaza · March 1, 2024, 3:26pm

Hello,

The command I used to install K3S is the exact same on JetPack 5 and JetPack 6:
curl -sfL https://get.k3s.io | sh -s - --docker, as described here:https://docs.k3s.io/advanced
When the install is successful, pods should look like this:

$ sudo k3s kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system local-path-provisioner-6d59f47c7-lncxn 1/1 Running 0 51s
kube-system metrics-server-7566d596c8-9tnck 1/1 Running 0 51s
kube-system helm-install-traefik-mbkn9 0/1 Completed 1 51s
kube-system coredns-8655855d6-rtbnb 1/1 Running 0 51s
kube-system svclb-traefik-jbmvl 2/2 Running 0 43s
kube-system traefik-758cd5fc85-2wz97 1/1 Running 0 43s

AastaLLL · March 4, 2024, 6:35am

Hi,

Thanks for the reply.
We will test this on JetPack 6 DP and then update more info with you.

AastaLLL · March 4, 2024, 7:40am

Hi,

We tested k3s on our internal JetPack 6 GA release and it can work correctly.
Please wait for the GA release.

$ sudo k3s kubectl get pods --all-namespaces
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   local-path-provisioner-6c86858495-nsrbt   1/1     Running     0          10m
kube-system   coredns-6799fbcd5-9c5q2                   1/1     Running     0          10m
kube-system   metrics-server-67c658944b-5xnjv           1/1     Running     0          10m
kube-system   helm-install-traefik-crd-24xlb            0/1     Completed   0          10m
kube-system   helm-install-traefik-mv9h7                0/1     Completed   1          10m
kube-system   svclb-traefik-18656cd4-sg4n4              2/2     Running     0          10m
kube-system   traefik-f4564c4f4-lcsdf                   1/1     Running     0          10m

Thanks.

madiaza · March 6, 2024, 7:32am

Thanks a lot for the testing, we will wait for the GA release. Do you know when it will be available?

Regards.

AastaLLL · March 7, 2024, 5:30am

Hi,

Based on the roadmap below, JetPack 6 GA targets for March 2024 which is very close.

But please wait for our announcement of the real release date.
Thanks.

AastaLLL · March 12, 2024, 7:14am

Hi,

If you want to test this on r36.2 (JetPack 6DP), please try to build the kernel from source with CONFIG_CFS_BANDWIDTH=y

https://docs.nvidia.com/jetson/archives/r36.2/DeveloperGuide/SD/Kernel.html

Thanks.

madiaza · March 15, 2024, 3:54pm

Hello,

I’ll definitively take a look to the Kernel customization options, as it looks that some of the modules will be required for our operation with the Jetson AGX boards. Regarding that, could you please confirm that the following kernel modules will be included on the JetPack 6.0 GA release? :

CONFIG_CFS_BANDWIDTH
CONFIG_NFT_CT
CONFIG_NFT_COUNTER
CONFIG_NFT_MASQ
CONFIG_NFT_REDIR
CONFIG_NFT_NAT
CONFIG_NFT_REJECT
CONFIG_WIREGUARD
CONFIG_VXLAN
CONFIG_NET_SCH_HFSC
CONFIG_NET_SCH_FQ_CODEL

Thanks a lot for your help!

system · March 29, 2024, 3:54pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Fail to install K3S and customize kernel on Jetson AGX ORIN Jetson AGX Orin kernel	10	228	February 10, 2025
Cannot passthrough GPU to Kubernetes pod on the Jetson AGX Orin dev kit Jetson AGX Orin gpu , kubernetes	15	187	April 20, 2025
Jetson Orin Nano Dev Board Pods Stuck in ContainersCreating State Jetson Orin Nano docker , kubernetes	7	282	July 30, 2024
Issue Running K3s on Jetson Orin NX 16GB with JetPack 6.0 Jetson Orin NX kernel , ubuntu , jetson	5	64	March 12, 2025
Trouble flashing Orin AGX 32GB for jetpack 6.0 using system docker. Failed to find 'dtc' Jetson AGX Orin reflash	16	299	July 15, 2024
K3s on Jetson Nano 4GB with Jetson MATE \| (NVIDIA Container Toolkit) upgrade Jetson Nano containers , gpu	15	2897	September 25, 2023
Nvidia-container runtime with containerd error on Jetpack 6 Jetson AGX Orin containers	5	862	April 10, 2024
Failed to flash Jetson Linux -- Debug Log posted Jetson AGX Orin reflash	5	695	June 14, 2023
Help, I bricked my AGX orin 64G Jetson AGX Orin reflash	19	1337	February 5, 2023
JetPack 6.0 Developer Preview - Release Announcement Jetson AGX Orin	53	8121	May 3, 2024

K3S installation on Jetson AGX Orin with Jetpack 6

Related topics