Got Xid 109 error in nvidia-device-plugin-daemonset pod

Graphics / Linux Linux
,
1

We deployed gpu operator in our kubernetes cluster, but we got error in nvidia-device-plugin-daemonset pod as follows:

NVIDIA_DRIVER_ROOT=/
CONTAINER_DRIVER_ROOT=/host
Starting nvidia-device-plugin
I0818 08:41:43.982755       1 main.go:178] Starting FS watcher.
I0818 08:41:43.982843       1 main.go:185] Starting OS watcher.
I0818 08:41:43.983115       1 main.go:200] Starting Plugins.
I0818 08:41:43.983138       1 main.go:257] Loading configuration.
I0818 08:41:43.983649       1 main.go:265] Updating config with default resource matching patterns.
I0818 08:41:43.984017       1 main.go:276] 
Running with config:
{
  "version": "v1",
  "flags": {
    "migStrategy": "single",
    "failOnInitError": true,
    "mpsRoot": "/run/nvidia/mps",
    "nvidiaDriverRoot": "/",
    "gdsEnabled": false,
    "mofedEnabled": false,
    "useNodeFeatureAPI": null,
    "plugin": {
      "passDeviceSpecs": true,
      "deviceListStrategy": [
        "envvar"
      ],
      "deviceIDStrategy": "uuid",
      "cdiAnnotationPrefix": "cdi.k8s.io/",
      "nvidiaCTKPath": "/usr/bin/nvidia-ctk",
      "containerDriverRoot": "/host"
    }
  },
  "resources": {
    "gpus": [
      {
        "pattern": "*",
        "name": "nvidia.com/gpu"
      }
    ],
    "mig": [
      {
        "pattern": "*",
        "name": "nvidia.com/gpu"
      }
    ]
  },
  "sharing": {
    "timeSlicing": {}
  }
}
I0818 08:41:43.984026       1 main.go:279] Retrieving plugins.
I0818 08:41:43.984795       1 factory.go:104] Detected NVML platform: found NVML library
I0818 08:41:43.984844       1 factory.go:104] Detected non-Tegra platform: /sys/devices/soc0/family file not found
I0818 08:41:44.582786       1 server.go:216] Starting GRPC server for 'nvidia.com/gpu'
I0818 08:41:44.584199       1 server.go:147] Starting to serve 'nvidia.com/gpu' on /var/lib/kubelet/device-plugins/nvidia-gpu.sock
I0818 08:41:44.587762       1 server.go:154] Registered device plugin for 'nvidia.com/gpu' with Kubelet
I0901 18:44:03.359506       1 health.go:159] Processing event {Device:{Handle:0x7d05c72301b8} EventType:8 EventData:109 GpuInstanceId:4294967295 ComputeInstanceId:4294967295}
I0901 18:44:03.359611       1 health.go:185] XidCriticalError: Xid=109 on Device=GPU-d063a73b-a898-a1eb-0cf4-13a5ad144bcb; marking device as unhealthy.
I0901 18:44:03.359689       1 server.go:292] 'nvidia.com/gpu' device marked unhealthy: GPU-d063a73b-a898-a1eb-0cf4-13a5ad144bcb

One gpu has XidCriticalError: Xid=109 error, the workaround solution is to delete the nvidia-device-plugin-daemonset pod, but it may happen after some time.

Environment:

  • GPU: RTX Pro 6000 Blackwell
  • GPU driver: 580.159.03
  • OS: Ubuntu 24.04
  • GPU operator: 24.3.0

Here is the bug report nvidia-bug-report.log.gz,

thank you for any assistance!

nvidia-bug-report.log.gz.zip (9.7 MB)