Recently, I have successfully gotten CUDA running in Kubernetes with containerd CRI. Everything works properly, but I need to see processes that are working in the GPU with nvidia-smi inside the pod or container. When I enter nvidia-smi inside the container, it doesn’t show me anything when I enter nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory. It shows me host PIDs, not namespace PID or container PID .
nvidia-smi
Thu Aug 28 05:56:32 2025
±----------------------------------------------------------------------------------------+
| NVIDIA-SMI 575.57.08 Driver Version: 575.57.08 CUDA Version: 12.9 |
|-----------------------------------------±-----------------------±---------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA RTX A4000 On | 00000000:84:00.0 Off | Off |
| 53% 72C P2 68W / 140W | 2518MiB / 16376MiB | 0% Default |
| | | N/A |
±----------------------------------------±-----------------------±---------------------+
±----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
±----------------------------------------------------------------------------------------+
nvidia-smi --query-compute-apps=pid,process_name,used_gpu_memory
pid, process_name, used_gpu_memory [MiB]
1824294, [Not Found], 1323 MiB
52964, [Not Found], 1153 MiB
These commands were run inside the pod i checked all package like NVIDIA container toolkit and others but i cant resolve this Another weird subject is in a of the node that some of my pod is running i dont have this issue and inside the container, nvidia-smi give me processes with container pid.i can’t find the misconfiguration and difference between these nodes.