AGX Thor - MIG 1g instances enumerate as CUDA devices but hang on the first kernel

Robotics & Edge Computing Jetson Systems Jetson Thor
1

Hi,

On AGX Thor with MIG enabled, a 1g compute instance enumerates correctly and then the

first real GPU operation never returns.

Not an error, not a slow result: no return

value, no exception, no output. Every run had to be killed on a timeout.

2g instances on the same board, in the same session, work perfectly, this blocks us from running both SONIC and GR00T on the same Thor.

from using MIG at all, because every two-way split this board allows puts a 1g instance on one side.

System

Jetson AGX Thor

L4T R39.2.0  (/etc/nv_tegra_release)       -> JetPack 7.2

nvidia-l4t-core  39.2.0-20260601141651

Driver 595.78, CUDA 13.2

Ubuntu 24.04.4

20 SMs, 125748 MB, num_DLA_cores = 0

MIG mode: Enabled, persistence mode Enabled

Reproducer


import torch

torch.cuda.init()

print(torch.cuda.get_device_name(0))                               # fine

p = torch.cuda.get_device_properties(0)

print(p.multi_processor_count, p.total_memory)                     # fine

x = torch.randn(2048, 2048, device="cuda", dtype=torch.bfloat16)   # NEVER RETURNS

torch.cuda.synchronize()

print("ok")

Create a 1g instance and point CUDA at it:


$ sudo nvidia-smi mig -i 0 -cgi 80 -C

Successfully created GPU instance ID  2 on GPU  0 using profile MIG 1g.0gb (ID 80)

Successfully created compute instance ID  0 on GPU  0 GPU instance ID  2 using profile MIG 1g.0gb (ID  0)



$ CUDA_VISIBLE_DEVICES=MIG-84de54e1-f9d8-5bd7-a199-7b0603c64a82 timeout 180 python probe.py

import torch

cuda.init()  name=NVIDIA Thor MIG 1g.0gb

SMs=8 mem=125748MB

<nothing further; killed at 180 s>

The allocation is 8 MB and the workload is one kernel, so this is not memory pressure.

Scope

Every MIG instance we have created on this board, all probed with the script above:
| Profile | SMs reported | Device opens | First kernel |

| 2g.0gb (82) | 14 | yes | 0.145 ms, 118 TFLOP/s |

| 2g.0gb+gfx (83) | 12 | yes | 0.136 ms, 127 TFLOP/s |

| 1g.0gb (80) | 8 | yes | never returns (180 s) |

| 1g.0gb+gfx (81) | 6 | yes | never returns (180 s) |

Both 1g variants hang. Both 2g variants work.

Question that I have regarding this:

1. Is 1g expected to be functional on Thor, or is it enumerated by the driver but not

actually supported on this silicon? If it is not supported, we would much rather know

that than keep testing.

2. If it is expected to work is this a known issue, and is there a JetPack or driver

level where it is fixed?

3. Is there any diagnostic we can collect that would be more useful than a timeout?

There is no error, no log line and no exit code.
The process simply sits at 100% of one core with no GPU activity.

Please any support to solve this issue.

4

Hi,

Which 1g profile do you use?
Based on our document below, we only recommended profile 78 and 83 on AGX Thor.
https://docs.nvidia.com/jetson/archives/r39.2.1/ReleaseNotes/Jetson_Linux_Release_Notes_r39.2.1.pdf#page=13

Could you try the profile 78 to see if it can work correctly?

Thanks.

5

retested from a cold boot and your recommendation is correct.

sudo reboot
sudo systemctl stop gdm.service
sudo nvidia-smi -i 0 -mig 1
sudo nvidia-smi mig -i 0 -cgi 83 -C 
sudo nvidia-smi mig -i 0 -cgi 78 -C

Both services then came up immediately on the 83 + 78 pair. our TensorRT service on 78 bound its port, and a 3B PyTorch model on 83.

Thanks!