DCGM-Exporter: Missing Process-level Attribution for GPU Time-Slicing on Blackwell GB10

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Hello,

I am currently benchmarking GPU Time-Slicing on a Blackwell GB10 (Driver 580.95) within a Kubernetes environment.

I have successfully enabled KUBERNETES_VIRTUAL_GPUS=true in the dcgm-exporter. While this correctly exposes metrics with the vgpu label for multiple pods sharing the same physical GPU, I’ve observed that all telemetry values (Utilization, Power, Temperature) are identical across all virtual devices.

For example, when two pods are running a matrix multiplication stress test:

  • Physical GPU Utilization is 95%.

  • dcgm_gpu_utilization{vgpu="0"} reports 95%.

  • dcgm_gpu_utilization{vgpu="1"} reports 95%.

It appears that DCGM is simply mirroring the global physical state to all virtual handles instead of providing per-process or per-container attribution (similar to what nvidia-smi pmon or nvidia-smi dmon can show).

My questions are:

  1. Is this a fundamental architectural limitation of DCGM telemetry when using Time-Slicing (non-MIG)?

  2. Is there any plan to support actual process-level attribution within the exporter’s metrics so we can distinguish “noisy neighbors”?

  3. Are there specific DCGM profiling metrics (like DCGM_FI_PROF_GR_ENGINE_ACTIVE) that should be used instead for Blackwell when time-slicing is active?

Thanks in advance for the support!

2

Unfortunately, DCGM is not supported on GB10 so I cannot say what configuration will work

3

Thank you for the clarification regarding DCGM support on the Blackwell GB10.

Since DCGM is the standard for monitoring in Kubernetes via dcgm-exporter, the lack of support explains why we are seeing mirrored/identical metrics across all pods when using GPU Time-Slicing.

I have two follow-up questions:

  1. Is there a roadmap or an estimated date for when DCGM will officially support the Blackwell GB10 architecture (specifically for memory and process-level attribution)?

  2. In the meantime, are there any alternative NVIDIA tools or APIs (perhaps via NVML or specific profiling flags) that would allow us to differentiate GPU utilization between containers sharing the same GB10 via time-slicing?

We are currently looking into 3rd party virtualization layers like Tensor Fusion to solve this, but we would prefer an official NVIDIA path if possible

4
  1. There are no plans to support DCGM on Spark
  2. You may not need the vGPU option since we have the driver on the host shared into k8s pods. nvidia-smi will likely show GPU utilization