Multi Instance GPU C++ Sample Reference

Autonomous Vehicles DRIVE AGX Orin DRIVE AGX Orin General
1

Dear Team,

Hardware :: Drive AGX Orin
I would like to request you to provide any reference to the MIG( Multi Instance GPU) mode implementation with C++ API.
So that I can use this mode to split GPU for multiple Deep Learning Model.

I will be thankful for your support.

Thanks and Regards,
Vyom Mishra

3

Dear @vyom.mishra,
Do you mean to run multiple TRT model on a single GPU ?

4

Dear @SivaRamaKrishnaNV

Yes Sir, we will be running multiple TRT model on single GPU with few other modules in the pipeline of each model.

Thanks and Regards,
Vyom Mishra

5

Dear @vyom.mishra,
We don’t have a sample demonstrating this use case.
Do you notice the GPU is under utilized when running a single TRT model? If so, you can use CUDA streams to launch CUDA taks in parallel(TRT model) to use GPU efficiently.

6

Dear @SivaRamaKrishnaNV

Thanks for valuable response.

I will try the same approach for implementation.

Can you please help me with the below queries as my development environment is QNX.

  1. If I am creating parallel CUDA stream for running TRT model, which extension I should
    use for development either “.cu” or “.cpp”.
  2. How to cross- compile CUDA code for QNX OS?

Thanks and Regards,
Vyom Mishra

8

Dear @vyom.mishra,
You can use .cu to write a CUDA sample. Please check Makefile in CUDA samples as a reference. Also, please check CUDA Samples :: CUDA Toolkit Documentation for cross compilation of CUDA samples.

10