Description
Normally when I work with TensorRT, I have been storing data in linear CUDA memory and passing the pointer to the deserialized TensorRT engine. For example (assume “gpuTensorInput” and “gpuTensorOutput” are pointers to CUDA memory):
int inputIndex = mEngine->getBindingIndex("Input_Tensor");
int outputIndex = mEngine->getBindingIndex("Output_Tensor");
// Double check that indices are 0 or 1 (can be -1 is the layer named is not a binding)
assert(inputIndex == 0 || inputIndex == 1);
assert(outputIndex == 0 || outputIndex == 1);
// Double check that the named layers above are really input and output
assert(mEngine->bindingIsInput(inputIndex) == true);
assert(mEngine->bindingIsInput(outputIndex) == false);
void* buf[2];
buf[inputIndex] = gpuTensorInput;
buf[outputIndex] = gpuTensorOutput;
bool status = context->executeV2(buf);
However, I am in a slightly different situation this time. I have a Python regression model I am working with and want to convert it to C++ / TensorRT. What is the recommended way to convert an OpenCV Mat structure to a TensorRT tensor before running inference?
For example, let’s say I want to read from an image file:
cv::Mat image = cv::imread("C:\\RegressionModelImages\\cat.png", cv::IMREAD_COLOR);
cv::cvtColor(image, image, cv::COLOR_BGR2RGB);
This can be converted to a GpuMat if necessary:
cv::cuda::GpuMat gpu_image(image.size(), image.type());
I am not sure what to do next, because to my understanding a GPU Mat is a strided array and not a linear buffer. In other situations, I would have my tensor input inside a linear CUDA memory buffer and would pass in the address of this memory element to the deserialized engine as the input. However, I am not sure what to do from here.
Additionally, correct me if I’m wrong, but I believe that there would have to be some reordering of nchw and nhwc. Is this correct?
Finally upon getting the tensor output, how would one convert it back to OpenCV format?
Environment
TensorRT Version: 7
CUDA Version: 10.2
CUDNN Version: 7.6
TensorRT API: C++
Operating System + Version: Windows 10 64-bit