Input / Output Pointer Type Passed Into Plugin's enqueue function

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

What type of memory is associated with the input / output pointers passed into the enqueue method of a plugin?

virtual int nvinfer1::IPlugin::enqueue(int batchSize,
    const void *const *inputs,
    void **outputs,
    void *workspace,
    cudaStream_t stream 
)

It seems like the memory pointer can be accessed both by CPU and GPU. So are they managed memory?

2

Hi,

Enqueue uses device memory for all the operations (input, output).
Input formats:

  • (N, C, H, W) for 2D Convolution.
  • (N, C, D, H, W) for 3D convolution.

Thanks

3

Hi,

Thanks for the reply. That is my initial guess. However, the below enqueue implementation runs without any memory error and comes out with expected outputs. Does that mean the memory is accessible through CPU also?

class SomePlugin : IPlugin... {
    ...
    int enqueue(int batchSize, const void* const* inputs, void** outputs, void* workspace,                                                                                       
                cudaStream_t stream) override {
        *(outputs[0]) = 10;
    }
    ...
}
1 Like