Can we fix the number of GPU Cores used during a TensorRT inference execution? & is (Multi-instance GPU) MIG Available on Jetson AGX Thor

Robotics & Edge Computing Jetson Systems Jetson Thor
,
1

Can we fix the number of GPU Cores used during a TensorRT inference execution? I am running an inference code on Jetson AGX thor, can i control the number of GPU cores used during the particular execution?

Also is (Multi-instance GPU) MIG Available on Jetson AGX Thor Developer Kit.?

4

Hi,

MIG support for Thor is added in Jetpack 7.2.
Please find the corresponding document below:

Thanks.

5

I have set up two Multi-Instance GPU (MIG) instances on a Jetson AGX Thor system.

  1. Is there a specific command or tool to monitor the exact memory used by each MIG instance on Jetson Thor? I have looked into tegrastats, but it seems to aggregate overall system/GPU usage.

  2. When examining the available MIG profiles, the profile layouts (such as profile 78 and 83) allocate 12 SMs and 6 SMs respectively. Combined, this accounts for 18 SMs.

    What happens to the remaining 2 SMs?

6

Hello @neethujoseph02,

Currently, seems that there is no built-in tool on Jetson Thor that reports the memory usage of individual MIG instances. tegrastats only reports aggregate system/GPU usage, and nvidia-smi does not provide a per-MIG memory breakdown on Jetson Thor.

Regarding the SM count, the 12 + 6 split reported by nvidia-smi is a known issue. NVIDIA has acknowledged that the output is incorrect:

“There is a known issue 6162096 — Output of nvidia-smi is incorrect when using MIG. This gives you incorrect information of the SM split being 12 and 6.”

The discussion is available here.

In summary, the remaining 2 SMs are not lost; the discrepancy is due to the current nvidia-smi reporting issue rather than the actual hardware partitioning.

Regards,
Daniel R.
Embedded SW Engineer at RidgeRun
Contact us: support@ridgerun.com
Developers wiki: https://developer.ridgerun.com
Website: www.ridgerun.com

7

Hi,

1.
If you run with trtexec, it by default contains the memory information:

[06/29/2026-07:35:04] [I] [TRT] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +9, now: CPU 0, GPU 106 (MiB)
[06/29/2026-07:35:04] [I] Setting persistentCacheLimit to 0 bytes.
[06/29/2026-07:35:04] [I] Created execution context with device memory size: 8.42383 MiB

2.
Please use deviceQuery to check the resource details.
The output of nvidia-smi is incorrect for MIG currently.

$ sudo nvidia-smi -L
GPU 0: NVIDIA Thor (UUID: GPU-a7c66ad2-6dbb-0ab8-c1a2-37ba6dba3600)
  MIG 2g.0gb      Device  0: (UUID: MIG-c465d416-67ab-59e7-b780-1a8f1a762326)
  MIG 1g.0gb      Device  1: (UUID: MIG-b59006b0-f2c1-5668-b295-053748350196)

$ CUDA_VISIBLE_DEVICES=MIG-c465d416-67ab-59e7-b780-1a8f1a762326 ./deviceQuery 
./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "NVIDIA Thor MIG 2g.0gb"
  CUDA Driver Version / Runtime Version          13.2 / 13.2
  CUDA Capability Major/Minor version number:    11.0
  Total amount of global memory:                 125809 MBytes (131920637952 bytes)
  (012) Multiprocessors, (128) CUDA Cores/MP:    1536 CUDA Cores
...

$ CUDA_VISIBLE_DEVICES=MIG-b59006b0-f2c1-5668-b295-053748350196 ./deviceQuery 
./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "NVIDIA Thor MIG 1g.0gb"
  CUDA Driver Version / Runtime Version          13.2 / 13.2
  CUDA Capability Major/Minor version number:    11.0
  Total amount of global memory:                 125809 MBytes (131920637952 bytes)
  (008) Multiprocessors, (128) CUDA Cores/MP:    1024 CUDA Cores
...

Thanks.