Wraping Buffer Data using Pointers to Avoid Data Copy

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

Hi,

I am using TensorRT C++ API, and I want to avoid data copy using for loops to store the image pixels to TensorRT Buffer.

If I know the CPU pointer of the lowest address pixel (i.e., the first pixel) of the image, and I know the image is stored
in a contiguous memory block, is there a way to wrap the data with TensorRT buffer directly?

I have the same question for GPU pointer. I want to wrap image data stored on GPU with a TensorRT directly without any data copy given that I have the GPU pointer to the first pixel.

libtorch has a function doing this called: torch::from_blob. Is there a similar function in TensorRT?

Environment

TensorRT Version: TensorRT-7.0.0.11.Windows10.x86_64.cuda-10.2.cudnn7.6
GPU Type: GPU 1NVIDIA GeForce GTX 1660 Ti with Max-Q
Nvidia Driver Version:
CUDA Version: 10.2
CUDNN Version: 7.6
Operating System + Version: Windows 10
Python Version (if applicable):
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

Please attach or include links to any models, data, files, or scripts necessary to reproduce your issue. (Github repo, Google Drive, Dropbox, etc.)

Steps To Reproduce

Please include:

  • Exact steps/commands to build your repro
  • Exact steps/commands to run your repro
  • Full traceback of errors encountered
2

Hi @oelgendy,

TensorRT doesnt support that.

But a work around solution could be first copy all images to GPU and set the pointer of each image to TensorRT input buffer.
This can avoid copy memory in the for loop.

Thanks!

4

Thanks @AakankshaS for your answer. I used the solution in this link and it is working. I just needed to store the GPU or CPU pointers in the 2-element array buf, and pass the array to executeV2.

If the input image has a higher resolution than the ONNX input resolution, is there any helping code for splitting the image into patches like tf.image.extract_patches or torch.unfold/torch.fold?

5

Hi @oelgendy,
It is something that needs to be handled as part of preprocessing.

Thanks!