Hi,
It is constraint in hardware converter. Please refer to discussion in
[Gstreamer] nvvidconv, BGR as INPUT
Please also check VPI:
VPI - Vision Programming Interface: Main Page
If you see the functions which can be applied to your use-case, please try it as a substitute of using OpenCV.