Ok. After spending about a day, I figured out the problem. Since others may come across the same issue, I try to explain it here. Please correct me if you see any misinformation.
When you decide to use cv::cuda::GpuMat, you assume that the initial data is from among the acceptable formats. However, the initial NvBufSurface (which is obtained from the input buffer), uses NV12 format. According to this post.
Therefore, it seems that we have no option but to go through the transform procedure.
So the code in the transform section should look like this:
- Get NvBufSurface from the original buffer
- Have an additional NvBufSurface as your element properties. Make sure to specify
NVBUF_COLOR_FORMAT_RGBA
in theNvBufSurfaceCreateParams - Do a transformation from part 1 to 2 with
NvBufSurfTransform. You can add some crop/scale/resize as well.
You can also just do the conversion for the sake of color format and have same rectangles. - Get the GpuMat as instructed in the above code or the code here.
- When you are done with the mat, you can now do a reversed format conversion (from RGBA, RGB, etc. back to NV12) as it is done at the end of this code. You can use the same transformation config just swap the input and output surfaces and their rectangles (to match their size).
The transformations are both done either with GPU for dGPU or VIC for Jetson. I guess they do not add too much overhead.
Best.
EDIT: Thanks to Blard.Theophile’s post, you can also use nvvideconvert for the conversion. This way, the plugin does not have to use NvBufSurfaceTransform.