SoC Engines Don't Work When MIG Is Enabled on Jetson Thor

Robotics & Edge Computing Jetson Systems Jetson Thor
,
1

Hardware: Jetson Thor T5000

Software:

  • nvidia-jetpack: 7.2-b187

  • nv_tegra_release: R39 (release), REVISION: 2.0

I enabled MIG by following the official documentation and created two GPU Instances (GIs):

$ nvidia-smi -L

GPU 0: NVIDIA Thor (UUID: GPU-a7c66ad2-6dbb-0ab8-c1a2-37ba6dba3600)

MIG 2g.0gb Device 0: (UUID: MIG-64c6ac3f-05fa-5ae0-8c2e-ec6fb3d6f79e)
MIG 1g.0gb Device 1: (UUID: MIG-e08de158-303a-5b82-b48c-49d023899e20)

I exported CUDA_VISIBLE_DEVICES in two separate terminals:

export CUDA_VISIBLE_DEVICES=MIG-64c6ac3f-05fa-5ae0-8c2e-ec6fb3d6f79e


export CUDA_VISIBLE_DEVICES=MIG-e08de158-303a-5b82-b48c-49d023899e20

CUDA applications work correctly on both MIG instances. For example:

1. CV-CUDA Resize

Profile Mean Time (ms)
2g.0gb (+gfx) 0.0733
1g.0gb (+me) 0.1214

The performance difference is roughly consistent with the expected compute resource ratio.

2. VIC

The application hangs when using the VIC backend.

3. VPI CUDA Resize

Profile Mean Time (ms)
2g.0gb (+gfx) 0.0909
1g.0gb (+me) 0.1099

4. OFA (02-stereo_disparity)

2g.0gb (+gfx):

line 203: VPI_ERROR_INTERNAL: (:NV_OF_ERR_GENERIC)

1g.0gb (+me):

line 207: VPI_ERROR_INTERNAL: VIC|OFA backend(s) error: (:NvSciError_BadParameter)

5. PVA (01-convolve_2d)

2g.0gb (+gfx):

VPI_ERROR_INTERNAL: VIC|OFA backend(s) error: (:NvSciError_BadParameter)

1g.0gb (+me):

VPI_ERROR_INTERNAL: VIC|OFA backend(s) error: (:NvSciError_BadParameter)

It appears that none of the SoC engines (VIC, PVA, or OFA) work after MIG is enabled.

I’d like to know whether I’m using MIG incorrectly, or if the current MIG implementation simply does not support these SoC engines (VIC, PVA, and OFA). If they are supported, are there any additional configuration steps required?

4

Hi,

We are checking this issue internally.
Will get back to you later.

Thanks.

5

Hi,

Thanks for reporting this issue.
Confirmed that we can reproduce the OFA and PVA error and is now checking with our internal team.

Could you also share the VIC testing you used?

Thanks.

7

Thank you for your answer. First, I would like to confirm whether these SoC engines (VIC, PVA, and OFA) are officially supported when MIG mode is enabled on Jetson Thor.Here is my VIC test code:

src_code.zip (7.3 KB)

8

Hi,

We locate a VPI issue when using MIG, and it is fixed internally.
Thanks

9

Thank you for your answer. I would like to confirm whether these SoC engines are officially supported in MIG mode.

If they are officially supported, could you please let me know how I can obtain a fixed version or the corresponding software update?

If they are not officially supported, I would like to understand the reason and the current limitations.

10

Hi,

There is a bug in VPI in handling of MIG-based UUIDs.
The SoC is working fine (verified via GStreamer), but VPI currently doesn’t work with MIG because of this bug.

Thanks

12

Hi,

Let’s keep this topic for the VPI + MIG.
For cudaMalloc, please file the new topic you created above for more information.

Thanks.