[Jetson AGX Thor][Kubernetes][MIG] NVIDIA k8s-device-plugin exposes only nvidia.com/gpu instead of MIG resources

Robotics & Edge Computing Jetson Systems Jetson Thor
, ,
1

Hi NVIDIA team,

I am trying to expose MIG instances on a Jetson AGX Thor as schedulable Kubernetes resources.

MIG itself works correctly on the host, and CUDA workloads can run on both MIG instances when the required /dev/ nvidia-caps/* devices are manually exposed to privileged containers. However, the official NVIDIA Kubernetes Device
Plugin currently exposes only one full GPU resource:

  nvidia.com/gpu: 1

It does not expose the two MIG instances as resources such as:

  nvidia.com/mig-1g.0gb: 1
  nvidia.com/mig-2g.0gb: 1

Environment

Device:
NVIDIA Jetson AGX Thor

Jetson Linux:
R39.2.1
GCID: 46758480

Architecture:
aarch64

Kernel:
6.8.12-1021-tegra

NVIDIA driver:
595.78

CUDA:
13.2

NVIDIA Container Toolkit:
1.19.1

Kubernetes:
K3s v1.36.3+k3s1

Container runtime:
containerd 2.3.2-k3s2

NVIDIA Kubernetes Device Plugin:

nvcr.io/nvidia/k8s-device-plugin:v0.18.2

Current Device Plugin configuration

  version: v1
  flags:
    deviceDiscoveryStrategy: tegra
    plugin:
      deviceIDStrategy: index

With this configuration, the plugin starts successfully and the node reports:

  Capacity:
    nvidia.com/gpu: 1

  Allocatable:
    nvidia.com/gpu: 1

MIG configuration

nvidia-smi -L reports:

  GPU 0: NVIDIA Thor
    MIG 2g.0gb Device 0:
      MIG-04c21d66-5860-5f1c-a8e7-4d91cf2c5e2e

    MIG 1g.0gb Device 1:
      MIG-efdd418f-43bb-5070-bf32-28fbcee542a7

The MIG capability minors are:

  GPU instance 1 / compute instance 0:
    /dev/nvidia-caps/nvidia-cap12
    /dev/nvidia-caps/nvidia-cap13

  GPU instance 2 / compute instance 0:
    /dev/nvidia-caps/nvidia-cap21
    /dev/nvidia-caps/nvidia-cap22

The mapping is also present in:

  /proc/driver/nvidia-caps/mig-minors

Observed behavior

When using the Tegra device discovery strategy, the plugin recognizes the
integrated Thor GPU but exposes it only as:

  nvidia.com/gpu: 1

It does not expose individual MIG profiles.

When trying the MIG mixed strategy, the expected resources are not
registered:

  nvidia.com/mig-1g.0gb
  nvidia.com/mig-2g.0gb

The MIG instances themselves are functional. As a workaround, I can run two
Kubernetes workloads using:

  runtimeClassName: nvidia

  securityContext:
    privileged: true

  env:
    - name: NVIDIA_VISIBLE_DEVICES
      value: "all"
    - name: CUDA_VISIBLE_DEVICES
      value: "<MIG UUID>"

With this workaround:

  • One workload runs successfully on the 1g MIG instance.
  • Another workload runs successfully on the 2g MIG instance.
  • Both workloads report CUDA as available.
  • nvidia-smi shows each Python process assigned to the expected GI/CI.

I also verified that explicitly exposing the matching
/dev/nvidia-caps/nvidia-cap* devices with device cgroup permissions allows
the containers to use the corresponding MIG instance.

Questions

  1. Is MIG resource discovery on Jetson AGX Thor officially supported by the
    NVIDIA Kubernetes Device Plugin?

  2. Does the tegra device discovery strategy currently support enumerating
    Thor MIG devices, or does MIG discovery still require the NVML discovery
    path?

  3. Is there a supported configuration using:

    deviceDiscoveryStrategy: tegra
migStrategy: mixed

    that exposes Thor resources as nvidia.com/mig-1g.0gb and
    nvidia.com/mig-2g.0gb?

  4. Is there an official way for the Device Plugin or NVIDIA Container
    Toolkit to inject the required /dev/nvidia-caps/nvidia-cap* device nodes
    and device-cgroup permissions into a non-privileged Pod?

Our desired Kubernetes resources are

  nvidia.com/mig-1g.0gb: 1
  nvidia.com/mig-2g.0gb: 1

The goal is to remove privileged: true and let kubelet grant only the
corresponding MIG capability devices to each Pod.

Thanks.

3

Hi,

Thanks for your feedback.
This is a known issue and we are working on a fix in nvidia- container toolkit.

Thanks