Streaming with GPU and DLA

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
,
1

Hello,
I created a plan where the first few layers are run in DLA and the remaining layers are run in GPU. I wanted to know how I can use this plan for streaming data in parallel. What I mean is this:
The first image is processed by DLA then GPU. When the first image is in the GPU, the second image is in the DLA. When the second image reaches the GPU, the third image enters the DLA, and so on. Essentially, after the first instance, at any point, I am using the DLA and GPU at the same time. Like the image below:

image
image414×124 13 KB

How can I do this, and which application should I use to prove that it is working in parallel? (Note that both the output and input are images)

I tried using nsys to see the output for the plan. Following the documentation (Developer Guide :: NVIDIA Deep Learning TensorRT Documentation), I do not see the other accelerators API in the output. I have added the link to the files as well.
Link to files

Thank you.

3

Hi,

You will need to use TensorRT API to specify certain layers run on DLA:

Or use cuDLA library:

Thanks.

4

Hello,
Thank you for the response.
I have used TensorRT Python library to set the device type to DLA for the first few layers and created the plan from this. What I would like to know is how to achieve the streaming step where I am able to execute the model in parallel. My input and output are images.

5

Hi,

Please run the multiple models within the same process but with different threads and CUDA streams.
GPU scheduler will optimize the tasks based on resources.

Thanks.

6

I am not sure I understood. Do you have a sample file that I could follow to do this?

7

Hi,

Please find the sample below:

Thanks.

8

Thank you for the sample. I followed the documentation to get the profiling data along with the sample and got the following results: (I used a plan that had the first ten layers executed in DLA and the remaining in GPU.)

  1. Without streaming.

    Screenshot from 2025-01-17 10-21-59
    Screenshot from 2025-01-17 10-21-591040×377 55.4 KB

    no_streaming.txt (23.4 KB)

  2. With streaming of 2.

    Screenshot from 2025-01-17 10-14-50
    Screenshot from 2025-01-17 10-14-501039×287 37.5 KB

    with_streaming.txt (28.5 KB)

I have also attached the logs from execution.

I can see an improvement in the throughput, but I cannot see how it is executing in parallel like I showed previously. The Nsight Systems shows a serial execution in both cases. Is there something I am missing?

9

Also is it possible to use a folder containing all the images as input for the model during while using trtexec? From what I understood from the sample, I can only use a single file that has been converted to a binary format.