Microk8s on Spark

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Looking to spin up microk8s on my Spark (for development/deployment testing) so thought I’d try the steps at: MicroK8s - MicroK8s on NVIDIA DGX

sudo snap install microk8s --classic
sudo microk8s enable gpu

Snap installed microk8s but got the following error for the GPU:

Addon gpu was not found in any repository

Is support for the GPU under microk8s on the Spark on the roadmap?

3

I was able to get this working, though the gpu addon is not enabled on arm64 by default “because the gpu addon is only tested on amd64 for new releases” as per MicroK8s contributor Angelos Kolaitis. (see Cannot enable gpu addon on aarch64 instances (aws g5g.metal) · Issue #4454 · canonical/microk8s · GitHub )

Procedure followed:

  1. Install microk8s via sudo snap install microk8s --classic
  2. Add arm64 to the supported_architectures of the nvidia and gpu addons in /var/snap/microk8s/common/addons/core/addons.yaml
  3. Enable the gpu addon via sudo microk8s enable gpu
  4. Deploy the NVIDIA device plugin for ARM64 manually by running sudo microk8s kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/refs/tags/v0.18.0/deployments/static/nvidia-device-plugin.yml
  5. Update the cuda-vector-add container image in the “MicroK8s on NVIDIA DGX” example to nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04

After running sudo microk8s kubectl apply on the updated cuda-vector-add manifest, the pod was scheduled and deployed:

$ sudo microk8s kubectl logs cuda-vector-add
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done
4

To deploy GPU Operator, please follow our documentation here: Installing the NVIDIA GPU Operator — NVIDIA GPU Operator

We also have a section dedicated to deploying with MicroK8s: Installing the NVIDIA GPU Operator — NVIDIA GPU Operator

The instructions on the microk8s website is not supported by NVIDIA

5

We also recommend checking out our Cloud-Native-Stack solution

The most recent version, 16.0, is supported on DGX Spark

6

Thanks - will use CNS.

Would you recommend a MicroK8s or a kubeadm installation?

7

Depends on your experience and context. I personally use the kubeadm install as that is what I am familiar with.

8

Hi @graemes ,

I installed microk8s on DGX-Spark plus GPU Operator - managed to RUN NIMs as well.
I wanted something quick and supported by the official GPU operator documentation

I installed microk8s using snap install –classic + dns using microk8s enable dns - > nothing fancy.

I installed the GPU Operator:

snap install helm --classic
snap install kubectl --classic 
microk8s config > ~root/.kube/config 

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia \
    && helm repo update



helm install gpu-operator -n gpu-operator --create-namespace \
  nvidia/gpu-operator $HELM_OPTIONS \
    --version=v25.3.4 \
    --set toolkit.env[0].name=CONTAINERD_CONFIG \
    --set toolkit.env[0].value=/var/snap/microk8s/current/args/containerd-template.toml \
    --set toolkit.env[1].name=CONTAINERD_SOCKET \
    --set toolkit.env[1].value=/var/snap/microk8s/common/run/containerd.sock \
    --set toolkit.env[2].name=CONTAINERD_RUNTIME_CLASS \
    --set toolkit.env[2].value=nvidia \
    --set toolkit.env[3].name=CONTAINERD_SET_AS_DEFAULT \
    --set-string toolkit.env[3].value=true


root@spark-1a3f:/home/kirson# kubectl describe node |grep NVIDIA
                    nvidia.com/gpu.machine=NVIDIA_DGX_Spark
                    nvidia.com/gpu.product=NVIDIA-GB10

For my needs - I installed prometheus & NGINX helm charts.

There are many ways to instsall k8s on DGX-Spark - however i found this method faster then kubeadm , kubespray ( Worry for the footprint ) , k3s

9

@aniculescu @ekirson @GradientDescent

Thanks for your guidance. I decided to use use kubeadm via CNS as it should be more closely aligned with BCM K8s deployments.

10

Great ! good Luck