Hello Team,
I want to setup k3s on Jetson MATE with one Master and one Worker…
Device & Versions Used:
- Jetson Nano 4GB SoM (2[Master & Worker])
- JetPack 4.6 on both
- Comes with [CUDA 10.2, NVIDIA Container Runtime 1.0.3, Docker 20.x]
My setup :
- I Configured the docker to use default runtime as NVIDIA on
/etc/docker/daemon.json
- Installed k3s with
--docker to use current docker runtime which i configured on step 1.
- Installed k8s-device-plugin on k3s using k8s-device-plugin
- My Worker node is using the k3s default runtime which is containerd with nvidia-gpu has configured (Just FYI)
Problem I’m Facing
I deployed a container/pod on docker/k3s with deviceQuery program separately to test whether both of them are recognize the GPU devices. It results PASS. But whenever I deploying a pod with below pasted nvidia parameters in my YAML file, Pods stating that nodes doesn’t have GPU devices 0/0.
Sample YAML
apiVersion: v1
kind: Pod
metadata:
name: gpu-pod
spec:
restartPolicy: Never
containers:
- name: cuda-container
image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2
resources:
limits:
nvidia.com/gpu: 1 # requesting 1 GPU
tolerations:
-key: nvidia.com/gpu
operator: Exists
effect: NoSchedule
I am not sure what is the problem with my configuration and setup. Help me to run GPU enabled program on k3s with 2 x Jetson Nano 4GB SoM’s
How to upgrade nvidia-container-toolkit on JetPack 4.6 ?
Hi @vesmanmartin, I’m not personally familiar with installing/using k3s, but to have GPU access in container on Jetson, you should only need to start the docker container with --runtime nvidia (or have docker’s default-runtime set to nvidia, like you said)
limits:
nvidia.com/gpu: 1 # requesting 1 GPU
tolerations:
-key: nvidia.com/gpu
I’d try taking this out, and then maybe k3s won’t try to request GPU (it’s possible that feature in k3s relies on nvidia-smi or some other non-ARM tool to detect GPU). And since you have your default-runtime set to nvidia, docker should already make Jetson’s GPU available.
- name: cuda-container
image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2
This is an x86 container and not an aarch64 container build for Jetson/JetPack. I’d start with nvcr.io/nvidia/l4t-base:r32.7.1 and try running vectorAdd/deviceQuery/ect in that.
On JetPack 4, CUDA/cuDNN/TensorRT are mounted into the container from the host device. Meaning, that l4t-base has CUDA Toolkit mounted under /usr/local/cuda, and your Jetson needs CUDA Toolkit installed on it. If you are using the SD card image, that already comes with full JetPack and CUDA/ect. If you flashed your Jetson SoM’s with SDK Manager, that would install CUDA/ect in a post-flashing step. Otherwise you should be able to install it from apt with cuda-toolkit-10-2
I’m not sure if you could build/install https://github.com/NVIDIA/nvidia-container-toolkit from source or not, but I’d not recommend changing the nvidia runtime setup that comes with JetPack 4 due to the mounting of the libraries from the system (these are specified under /etc/nvidia-container-runtime/host-files-for-container.d/) as it is likely to stop working and then you won’t be able to use CUDA in your containers correctly. (on JetPack 5 it was changed to have CUDA/cuDNN/TensorRT/ect installed into the containers themselves)
Thanks @dusty_nv for the quick response and time :)
Actually for k8s-device-plugin(Exposing GPU on k3s Cluster), there is prerequisites to be consider which has nvidia-container-toolkit >= 1.11.0 for intergrated GPU tegra based systems as mentioned in this k8s-device-plugin also pasted below:
Prerequisites
The list of prerequisites for running the NVIDIA device plugin is described below:
NVIDIA drivers ~= 384.81
nvidia-docker >= 2.0 || nvidia-container-toolkit >= 1.7.0 (>= 1.11.0 to use integrated GPUs on Tegra-based systems)
nvidia-container-runtime configured as the default low-level runtime
Kubernetes version >= 1.10
But in JetPack 4.6 it comes with nvidia-container-toolkit = 1.0.1-1. Therefore I want to upgrade the version to make plugin compatible.
@vesmanmartin the version of nvidia-container-toolkit that comes with JetPack 5 also happens to be 1.11, so what they probably mean is that plugin is compatible with JetPack 5. On JetPack 4, I would try working around it by not even needing the plugin. Of course, you can manually try upgrading the nvidia-container-runtime but YMMV with respect to it continuing to work on JetPack 4.
Thanks, But we have to have plugin deployed on k3s so that it can expose the GPU inside the cluster. Thus I am attempting to upgrade Nvidia Container Toolkit to the stated version.
Else, Let me try with JetPack 5.x to check whether it matters.
@vesmanmartin your Jetson’s GPU should already be exposed inside docker because you set your default docker-runtime to nvidia. I don’t know if the k3s plugin is needed for other reasons or not besides that. The original Jetson Nano doesn’t support JetPack 5.
So Is there anyway to run k3s with GPU enabled on Jetson Nano with JetPack 4.6
As mentioned above…I’m not personally familiar with K3S, but your pod seems to get hung-up on loading the k8s-device-plugin which is primarily intended to allocate discrete GPUs on servers. On Jetson, I don’t believe any such allocation should be needed since there is just the one integrated GPU onboard, and this GPU is automatically exposed to docker containers anytime --runtime nvidia is used (which you have set to your default)
Try removing all the GPU stuff out of your YAML and perhaps it won’t try using that plugin. Run one of the l4t containers built for Jetson not the x86 containers.
Here is a similar article that uses K8S on Nano:
https://medium.com/jit-team/building-a-gpu-enabled-kubernets-cluster-for-machine-learning-with-nvidia-jetson-nano-7b67de74172a
note that it doesn’t use the device plugin but can still use Jetson GPU
I have already studied this medium and plenty of other blogs and articles related to this. Also in the above article you mentioned it stated at last to use plugin for use other available GPU’s.
So, you are saying due to the integrated Tegra GPU we don’t need to use resource parameters on the YAML files ???
Again, I have not use K3S/K8S so I can’t say for sure, but in that blog they were able to use Jetson’s GPU just fine without the plugin. It says the plugin is desired for resource monitoring. Have you just tried removing the resource parameters from your YAML file yet?
Yeah removing the resource parameters is working fine. But, the thing is that I just remote with SSH and run Jtop on my master node for monitor GPU and CPU resource meanwhile running the TensorFlow programs through k3s. GPU is not using instead 4 CPU’s are getting exhausted and brick my whole system !!!
This is what my problem exactly. I even tried different GPU TensorFlow programs but none of them are using GPU instead CPU getting exhausted.
Thanks
Which container are you running? Was the TensorFlow in it built with CUDA support for JetPack? Are you able to run deviceQuery or vectorAdd CUDA samples through nvcr.io/nvidia/l4t-base:r32.7.1 container?
I am using the NVIDIA official L4T image nvcr.io/nvidia/l4t-tensorflow:r32.6.1-tf2.5-py3 which having CUDA support
Yes, I can run both the deviceQuery and vectorAdd on the above mentioned image with tag r32.6.1
OK great @vesmanmartin, this is good! 👍
If you run this TensorFlow script, does it show the GPUs available as 1?
import tensorflow as tf
print('TensorFlow version: ' + str(tf.__version__))
print('Num GPUs Available: ', len(tf.config.experimental.list_physical_devices('GPU')))
Here’s the output when I run it in nvcr.io/nvidia/l4t-tensorflow:r32.7.1-tf2.7-py3
TensorFlow version: 2.7.0
2023-08-31 12:55:18.535698: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1019] ARM64 does not support NUMA - returning NUMA node zero
2023-08-31 12:55:18.548388: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1019] ARM64 does not support NUMA - returning NUMA node zero
2023-08-31 12:55:18.548641: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1019] ARM64 does not support NUMA - returning NUMA node zero
Num GPUs Available: 1
I got the solutions from @byer !!!
To upgrade and install nvidia-container-toolkit==1.13.0 Just run the following commands:
- Add The NVIDIA Docker Repository Key
- Add the NVIDIA Docker Repository To APT
- Install NVIDIA Docker Container Files And Docker Compose
curl -s -L https://nvidia.github.io/nvidia-docker/ubuntu18.04/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list && \
curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add - && \
sudo apt update && sudo apt install -y nvidia-docker2* docker-compose nvidia-container-toolkit nvidia-container-toolkit-base
NOTE: If you haven’t upgrade the system. please do sudo apt upgrade to get newest version of nvidia-container-runtime and docker
Just restart the Docker and K3s services:
sudo systemctl restart docker
sudo systemctl restart k3s
To verify you can run dpkg -l | grep nvidia-container- to list all the NVIDIA packages with the version number.
Now k3s-device-plugin will not have the compatibility issue anymore. Below mentioned version is needed to deploy k8s-device-plugin.
kubectl describe nodes command will show you the information of the nodes. Now we are able to see the k3s nodes are exposing the GPU device as nvidia.com/gpu:1
Capacity:
cpu: 4
ephemeral-storage: 239254916Ki
hugepages-2Mi: 0
memory: 4057736Ki
nvidia.com/gpu: 1
pods: 110
Allocatable:
cpu: 4
ephemeral-storage: 232747182103
hugepages-2Mi: 0
memory: 4057736Ki
nvidia.com/gpu: 1
pods: 110
Thanks @byer for your kind support and solutions and @dusty_nv for your quick reply’s and solutions.