MIG support in BCM , enablement failed and clients auto reset

Infrastructure & Networking Nvidia Bright Cluster Manager
1

After enabling Mig on NVidia GPU client nodes and running below command , the client auto resets and after coming back online , it again goes to Disabled state of MIG .
Please let me know if you need more info

[root@node004 ~]# nvidia-smi mig -cgi 9,3g.40gb -i 0
Successfully created GPU instance ID  1 on GPU  0 using profile MIG 3g.40gb (ID  9)
Successfully created GPU instance ID  2 on GPU  0 using profile MIG 3g.40gb (ID  9)

[root@node004 ~]# nvidia-smi
Wed Aug  2 14:41:03 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 515.65.01    Driver Version: 515.65.01    CUDA Version: 11.7     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA A100 80G...  On   | 00000000:17:00.0 Off |                   On |
| N/A   38C    P0    45W / 300W |      0MiB / 81920MiB |     N/A      Default |
|                               |                      |              Enabled |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA A100 80G...  On   | 00000000:65:00.0 Off |                   On |
| N/A   38C    P0    43W / 300W |      0MiB / 81920MiB |     N/A      Default |
|                               |                      |              Enabled |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA A100 80G...  On   | 00000000:CA:00.0 Off |                   On |
| N/A   42C    P0    44W / 300W |      0MiB / 81920MiB |     N/A      Default |
|                               |                      |              Enabled |
+-------------------------------+----------------------+----------------------+
|   3  NVIDIA A100 80G...  On   | 00000000:E3:00.0 Off |                   On |
| N/A   40C    P0    43W / 300W |      0MiB / 81920MiB |     N/A      Default |
|                               |                      |              Enabled |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| MIG devices:                                                                |
+------------------+----------------------+-----------+-----------------------+
| GPU  GI  CI  MIG |         Memory-Usage |        Vol|         Shared        |
|      ID  ID  Dev |           BAR1-Usage | SM     Unc| CE  ENC  DEC  OFA  JPG|
|                  |                      |        ECC|                       |
|==================+======================+===========+=======================|
|  No MIG devices found                                                       |
+-----------------------------------------------------------------------------+
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
2

More output commands

[root@node004 ~]# nvidia-smi mig -lgip
+-----------------------------------------------------------------------------+
| GPU instance profiles:                                                      |
| GPU   Name             ID    Instances   Memory     P2P    SM    DEC   ENC  |
|                              Free/Total   GiB              CE    JPEG  OFA  |
|=============================================================================|
|   0  MIG 1g.10gb       19     7/7        9.50       No     14     0     0   |
|                                                             1     0     0   |
+-----------------------------------------------------------------------------+
|   0  MIG 1g.10gb+me    20     1/1        9.50       No     14     1     0   |
|                                                             1     1     1   |
+-----------------------------------------------------------------------------+
|   0  MIG 2g.20gb       14     3/3        19.50      No     28     1     0   |
|                                                             2     0     0   |
+-----------------------------------------------------------------------------+
|   0  MIG 3g.40gb        9     2/2        39.50      No     42     2     0   |
|                                                             3     0     0   |
+-----------------------------------------------------------------------------+
|   0  MIG 4g.40gb        5     1/1        39.50      No     56     2     0   |
|                                                             4     0     0   |
+-----------------------------------------------------------------------------+
|   0  MIG 7g.80gb        0     1/1        79.25      No     98     5     0   |
|                                                             7     1     1   |
+-----------------------------------------------------------------------------+
|   1  MIG 1g.10gb       19     7/7        9.50       No     14     0     0   |
|                                                             1     0     0   |
+-----------------------------------------------------------------------------+
|   1  MIG 1g.10gb+me    20     1/1        9.50       No     14     1     0   |
|                                                             1     1     1   |
+-----------------------------------------------------------------------------+
|   1  MIG 2g.20gb       14     3/3        19.50      No     28     1     0   |
|                                                             2     0     0   |
+-----------------------------------------------------------------------------+
|   1  MIG 3g.40gb        9     2/2        39.50      No     42     2     0   |
|                                                             3     0     0   |
+-----------------------------------------------------------------------------+
|   1  MIG 4g.40gb        5     1/1        39.50      No     56     2     0   |
|                                                             4     0     0   |
+-----------------------------------------------------------------------------+
|   1  MIG 7g.80gb        0     1/1        79.25      No     98     5     0   |
|                                                             7     1     1   |
+-----------------------------------------------------------------------------+
|   2  MIG 1g.10gb       19     7/7        9.50       No     14     0     0   |
|                                                             1     0     0   |
+-----------------------------------------------------------------------------+
|   2  MIG 1g.10gb+me    20     1/1        9.50       No     14     1     0   |
|                                                             1     1     1   |
+-----------------------------------------------------------------------------+
|   2  MIG 2g.20gb       14     3/3        19.50      No     28     1     0   |
|                                                             2     0     0   |
+-----------------------------------------------------------------------------+
|   2  MIG 3g.40gb        9     2/2        39.50      No     42     2     0   |
|                                                             3     0     0   |
+-----------------------------------------------------------------------------+
|   2  MIG 4g.40gb        5     1/1        39.50      No     56     2     0   |
|                                                             4     0     0   |
+-----------------------------------------------------------------------------+
|   2  MIG 7g.80gb        0     1/1        79.25      No     98     5     0   |
|                                                             7     1     1   |
+-----------------------------------------------------------------------------+
|   3  MIG 1g.10gb       19     7/7        9.50       No     14     0     0   |
|                                                             1     0     0   |
+-----------------------------------------------------------------------------+
|   3  MIG 1g.10gb+me    20     1/1        9.50       No     14     1     0   |
|                                                             1     1     1   |
+-----------------------------------------------------------------------------+
|   3  MIG 2g.20gb       14     3/3        19.50      No     28     1     0   |
|                                                             2     0     0   |
+-----------------------------------------------------------------------------+
|   3  MIG 3g.40gb        9     2/2        39.50      No     42     2     0   |
|                                                             3     0     0   |
+-----------------------------------------------------------------------------+
|   3  MIG 4g.40gb        5     1/1        39.50      No     56     2     0   |
|                                                             4     0     0   |
+-----------------------------------------------------------------------------+
|   3  MIG 7g.80gb        0     1/1        79.25      No     98     5     0   |
|                                                             7     1     1   |
+-----------------------------------------------------------------------------+
[root@node004 ~]# nvidia-smi mig -lgipp
GPU  0 Profile ID 19 Placements: {0,1,2,3,4,5,6}:1
GPU  0 Profile ID 20 Placements: {0,1,2,3,4,5,6}:1
GPU  0 Profile ID 14 Placements: {0,2,4}:2
GPU  0 Profile ID  9 Placements: {0,4}:4
GPU  0 Profile ID  5 Placement : {0}:4
GPU  0 Profile ID  0 Placement : {0}:8
GPU  1 Profile ID 19 Placements: {0,1,2,3,4,5,6}:1
GPU  1 Profile ID 20 Placements: {0,1,2,3,4,5,6}:1
GPU  1 Profile ID 14 Placements: {0,2,4}:2
GPU  1 Profile ID  9 Placements: {0,4}:4
GPU  1 Profile ID  5 Placement : {0}:4
GPU  1 Profile ID  0 Placement : {0}:8
GPU  2 Profile ID 19 Placements: {0,1,2,3,4,5,6}:1
GPU  2 Profile ID 20 Placements: {0,1,2,3,4,5,6}:1
GPU  2 Profile ID 14 Placements: {0,2,4}:2
GPU  2 Profile ID  9 Placements: {0,4}:4
GPU  2 Profile ID  5 Placement : {0}:4
GPU  2 Profile ID  0 Placement : {0}:8
GPU  3 Profile ID 19 Placements: {0,1,2,3,4,5,6}:1
GPU  3 Profile ID 20 Placements: {0,1,2,3,4,5,6}:1
GPU  3 Profile ID 14 Placements: {0,2,4}:2
GPU  3 Profile ID  9 Placements: {0,4}:4
GPU  3 Profile ID  5 Placement : {0}:4
GPU  3 Profile ID  0 Placement : {0}:8
3

@mdevries1 @gkloosterman , anything you guys can share on above issue ?

4

I would offer that issues like this need to be brought to the support team.
This forum is designed to be user-supported, not a substitute for commercial support.

5