Does TensorRT support multi inputs/outputs in different location(DEVICE & HOST)

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Description

Hi, I am writing a trt plugin with IPluginV2DynamicExt. For my business, I want to give 3 outputs for my plugin, the first&second stored in DEVICE(GPU) memory, and last one stored in HOST(CPU). The last output just holds a flag information which produced by the algorithm. I don’t want to put it in GPU because in the next layer enqueue, I need to copy it back to CPU memory which is time consuming.

What I wanna know is:

  1. Does tensorRT support this situation I mentioned?
  2. If it does, how should I do?

Environment

TensorRT Version: 7.1.3.4
GPU Type: xavier
Nvidia Driver Version: 440.33
CUDA Version: 10.2
CUDNN Version: 8.0
Operating System + Version: ubuntu 18.04
Python Version (if applicable): 3.7
PyTorch Version (if applicable): 1.4

2

Hi @leemo,
Your query has been noted.
Please allow me some time to check on this.
Thanks!

3

Hi @leemo,

I am afraid, this is not supported by TensorRT.

Thanks!

4

@AakankshaS OK, thx.