Hardware-accelerated gstreamer pipeline with RTX GPU

Hi all, I have a Linux 20.04 Desktop with a RTX GPU.

I am trying to use the NVIDIA VIDEO CODEC SDK (hardware-accelerated encoder/decoder library) for decoding video in my gstreamer pipeline. I’m using OpenCV to do further processing after decoding the video stream, subsequently, I need to convert the video to BGR after decoding. This is what my gstreamer receiver looks like right now:

                "udpsrc port=<Port number> \
                ! application/x-rtp, media=video, clock-rate=90000, encoding-name=H264 \
                ! rtpjitterbuffer \
                ! rtph264depay \
                ! h264parse \
                ! nvh264dec \
                ! nvvidconv \
                ! video/x-raw, format=BGR \
                ! appsink",

When I build & run, the code halts for a while at the .open(...) process and eventually prints a few warnings/errors:

[ WARN:0@30.017] global cap_gstreamer.cpp:1697 open OpenCV | GStreamer warning: unable to query duration of stream
GStreamer-CRITICAL **: gst_caps_get_structure: assertion ‘GST_IS_CAPS (caps)’ failed
GStreamer-CRITICAL **: gst_structure_get_int: assertion ‘structure != NULL’ failed
[ WARN:0@30.017] global cap_gstreamer.cpp:1707 open OpenCV | GStreamer warning: cannot query video width/height
GStreamer-CRITICAL **: gst_structure_get_fraction: assertion ‘structure != NULL’ failed

I also tried nvh264sldec but it didn’t work either. I am suspecting that I may not have built OpenCV with -D WITH_CUDA=ON, could this be why the pipeline doesn’t work? I did a lot of research but I couldn’t find any example pipelines.
Your help is appreciated.