MIGs do not show, despite being created

Accelerated Computing CUDA CUDA Setup and Installation
1

Hello,
I am struggeling to create MIG instances on our A100 GPUs - it has worked in the past but now I am unable to split the GPUs into MIGs. The command to create a MIG instance seems to work fine (“Successfully created …” and when I check the configuration with nvidia-smi mig -lgi, it looks good as well. But the MIGs just won’t show up in nvidia-smi (-L) and GPU jobs are not being submitted (“CUDA Query failure: no CUDA-capable device is detected”).

I have tried the following without success:

  • restarting the nvidia-persistenced deamon
  • reinstalling cuda
  • rebooting

Please see the example below:

(base) milue@karla:~$ sudo nvidia-smi mig -i 0 -cgi 19
Successfully created GPU instance ID 10 on GPU 0 using profile MIG 1g.10gb (ID 19)

(base) milue@karla:~$ nvidia-smi -L
GPU 0: NVIDIA A100 80GB PCIe (UUID: GPU-cbdf5bd8-4147-9aa0-188b-e0d4b621d7e6)
GPU 1: NVIDIA A100 80GB PCIe (UUID: GPU-9ebc9e35-8017-1abd-fecd-b8740f76987a)
GPU 2: NVIDIA A100 80GB PCIe (UUID: GPU-5165e738-15f3-b35e-065a-44405c151d3e)
GPU 3: NVIDIA A100 80GB PCIe (UUID: GPU-c29cb785-577d-ba52-6388-af42454fb36e)

(base) milue@karla:~$ nvidia-smi
Mon Oct 14 13:44:16 2024
±----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03 Driver Version: 560.35.03 CUDA Version: 12.6 |
|-----------------------------------------±-----------------------±---------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA A100 80GB PCIe On | 00000000:31:00.0 Off | On |
| N/A 39C P0 42W / 300W | 88MiB / 81920MiB | N/A Default |
| | | Enabled |
±----------------------------------------±-----------------------±---------------------+
| 1 NVIDIA A100 80GB PCIe On | 00000000:4B:00.0 Off | On |
| N/A 39C P0 43W / 300W | 1MiB / 81920MiB | N/A Default |
| | | Enabled |
±----------------------------------------±-----------------------±---------------------+
| 2 NVIDIA A100 80GB PCIe On | 00000000:B1:00.0 Off | On |
| N/A 41C P0 48W / 300W | 1MiB / 81920MiB | N/A Default |
| | | Enabled |
±----------------------------------------±-----------------------±---------------------+
| 3 NVIDIA A100 80GB PCIe On | 00000000:CA:00.0 Off | On |
| N/A 39C P0 44W / 300W | 1MiB / 81920MiB | N/A Default |
| | | Enabled |
±----------------------------------------±-----------------------±---------------------+

±----------------------------------------------------------------------------------------+
| MIG devices: |
±-----------------±---------------------------------±----------±----------------------+
| GPU GI CI MIG | Memory-Usage | Vol| Shared |
| ID ID Dev | BAR1-Usage | SM Unc| CE ENC DEC OFA JPG |
| | | ECC| |
|==================+==================================+===========+=======================|
| No MIG devices found |
±----------------------------------------------------------------------------------------+

±----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| No running processes found |
±----------------------------------------------------------------------------------------+

(base) milue@karla:~$ sudo nvidia-smi mig -lgi
±------------------------------------------------------+
| GPU instances: |
| GPU Name Profile Instance Placement |
| ID ID Start:Size |
|=======================================================|
| 0 MIG 1g.10gb 19 7 0:1 |
±------------------------------------------------------+
| 0 MIG 1g.10gb 19 8 1:1 |
±------------------------------------------------------+
| 0 MIG 1g.10gb 19 9 2:1 |
±------------------------------------------------------+
| 0 MIG 1g.10gb 19 10 3:1 |
±------------------------------------------------------+
| 0 MIG 1g.10gb 19 11 4:1 |
±------------------------------------------------------+
| 0 MIG 1g.10gb 19 12 5:1 |
±------------------------------------------------------+
| 0 MIG 1g.10gb 19 13 6:1 |
±------------------------------------------------------+

Any advice is appreciated.

2

mig creation involves multiple steps. you are creating GI’s but don’t appear to be creating CI’s. see here. I suggest following that example sequence.

3

Thanks, that solved it! I somehow forgot to include the -C flag at the end (sudo nvidia-smi mig -i 2 -cgi 19 -C).