Unified Memory management for trtexec

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin Nano
1

Hello,

I have a ResNet50 TensorRT model that I converted from a ResNet50 PyTorch model using the trtexec tool. Currently, I execute inference and log the timing information for each query into a JSON file using the following command:

$TRTEXEC --useRuntime=full --useSpinWait --duration=10 --dumpProfile --exportProfile=profile.json --verbose --exportTimes=time.log --separateProfileRun --shapes=images:1x3x224x224 --infStreams=1 --loadEngine=int8.engine

Upon examining the time.json file (I renamed the file extension from .json to .log to facilitate uploading), I observed that each entry contains startH2dMs, endH2dMs, startD2h and endD2h timestamps and their corresponding differences. time.log (1.4 MB)
However, to my understanding, the Jetson Orin Nano features a unified physical memory architecture between the GPU and CPU, eliminating the need for explicit communication between the CPU and GPU. Consequently, I would expect that the startH2d, endH2d, startD2hMs, and endD2hMs values would either be absent or their corresponding differences would be zero. This expectation is not reflected in the data.

Could you please provide an explanation for this discrepancy?

Note: I am using Jetson Orin Nano Dev Board

4

Dear @abhinaba.chakraborty ,
Though the physical memory is shared across CPU and GPU, the type of memory is different based on memory allocation API. Could you please take a look at Memory management section in CUDA for Tegra

5

Hi @SivaRamaKrishnaNV

Thank you for the prompt reply. I had a quick look at the documentation you shared. It seems that a quick solution could be to add the --noDataTransfers option while executing the trtexec tool via the command line for Tegra architectures.

I ran the tool with the mentioned flag and noticed that the following pattern appears above the mentioned timestamps:
startH2dMs = endH2dMs = startComputeMs and startD2hMs = endD2hMs = endComputeMs. This actually makes sense if there is no data transfer overhead between the host and device.

What are your thoughts on this? Is there any more detail I should be aware of?

Thanks.

6

Your understanding is correct

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.