Hi
Error:
[core@master0 ~]$ oc logs nvidia-cuda-validator-8km4d -c cuda-validation -n nvidia-gpu-operator
Failed to allocate device vector A (error code CUDA-capable device(s) is/are busy or unavailable)!
Hardware information:
GH200
[core@master0 ~]$ oc get pods -n nvidia-gpu-operator
NAME READY STATUS RESTARTS AGE
gpu-feature-discovery-qbch8 1/1 Running 0 12h
gpu-operator-bcd4b9c6b-8znxv 1/1 Running 0 14h
nvidia-container-toolkit-daemonset-k8hcd 1/1 Running 0 13h
nvidia-cuda-validator-8km4d 0/1 Init:Error 3 (24s ago) 43s
nvidia-dcgm-exporter-dhbbn 1/1 Running 0 12h
nvidia-dcgm-pkmg7 1/1 Running 0 12h
nvidia-device-plugin-daemonset-7d5bj 1/1 Running 0 12h
nvidia-driver-daemonset-416.94.202503252048-0-mfgvf 2/2 Running 0 13h
nvidia-node-status-exporter-6s5w7 1/1 Running 0 14h
nvidia-operator-validator-x4dfq 0/1 Init:2/4 114 (93s ago) 13h
software version;
±----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.148.08 Driver Version: 570.148.08 CUDA Version: 12.8 |
|-----------------------------------------±-----------------------±---------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GH200 480GB On | 00000009:01:00.0 Off | 0 |
| N/A 33C P0 74W / 700W | 9MiB / 97871MiB | 0% Default |
| | | Disabled |
±----------------------------------------±-----------------------±---------------------+
±----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
±----------------------------------------------------------------------------------------+
[core@master0 ~]$ oc get csv -n nvidia-gpu-operator
NAME DISPLAY VERSION REPLACES PHASE
gpu-operator-certified.v25.3.1 NVIDIA GPU Operator 25.3.1 gpu-operator-certified.v25.3.0 Succeeded
Kernel version: [core@master0 ~]$ uname -r
5.14.0-427.62.1.el9_4.aarch64
openshift version:
[core@master0 ~]$ oc version
Client Version: 4.16.0-202503050504.p0.gee354f6.assembly.stream-ee354f6
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: 4.16.38
Kubernetes Version: v1.29.14+7cf4c05
Gang Chen