We did try using gstreamer plugins in a similar fashion as your diagram above shows.
As per per this post, we had some problems with nvinfer so at the end of that dialogue, we were advised to use the libary Yolov7 to do the inference.
This library takes a vector of cv::Mat as input. That suited us just fine since we make other use of the cv::Mat’s ourselves.