GPU/HW acceleration with GStreamer nvvidconv

ben.jacobson · September 27, 2023, 2:05am

I’m a newbie here with GStreamer, running a Jetson Orin Nano (4GB) with Jetpack 5.1.1 and GStreamer version 1.16.3. I can run this pipeline up to 60 fps without problems:

gst-launch-1.0 videotestsrc ! video/x-raw,width=800,height=800,format=YUY2,framerate=60/1 ! videoconvert ! video/x-raw,format=RGB ! queue ! ccm800x800cv ! queue ! videoconvert ! queue ! fpsdisplaysink

where ccm800x800cv is a custom plugin we’re developing that inputs and outputs RGB. I wanted to experiment with using Nvidia plug-ins to accelerate the format conversion from YUY2 to RGB. Following suggestions in this forum, I tried this next pipeline. It works up to 70 fps:

gst-launch-1.0 videotestsrc ! video/x-raw,width=800,height=800,format=YUY2,framerate=70/1 ! nvvidconv ! "video/x-raw(memory:NVMM),format=RGBA" ! nvvidconv ! videoconvert ! video/x-raw,format=RGB ! queue ! ccm800x800cv ! queue ! videoconvert ! queue ! fpsdisplaysink

My questions:

Is that all the acceleration I should expect (60 fps to 70 fps)?
Do I really need the 2nd nvvidconv plugin? Seems like a lot of steps…
Is there a more optimal sequence of plugins I should be using?

When I push the framerate higher, I get dropped frames and output like:

gst-launch-1.0 videotestsrc ! video/x-raw,width=800,height=800,format=YUY2,framerate=80/1 ! nvvidconv ! "video/x-raw(memory:NVMM),format=RGBA" ! nvvidconv ! videoconvert ! video/x-raw,format=RGB ! queue ! ccm800x800cv ! queue ! videoconvert ! queue ! fpsdisplaysink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
WARNING: from element /GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstAutoVideoSink:fps-display-video_sink/GstXvImageSink:fps-display-video_sink-actual-sink-xvimage: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(3003): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstAutoVideoSink:fps-display-video_sink/GstXvImageSink:fps-display-video_sink-actual-sink-xvimage:
There may be a timestamping problem, or this computer is too slow.

Adding the -v option for more info (sorry, this gets long):

gst-launch-1.0 -v videotestsrc ! video/x-raw,width=800,height=800,format=YUY2,framerate=80/1 ! nvvidconv ! "video/x-raw(memory:NVMM),format=RGBA" ! nvvidconv ! videoconvert ! video/x-raw,format=RGB ! queue ! ccm800x800cv ! queue ! videoconvert ! queue ! fpsdisplaysink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstAutoVideoSink:fps-display-video_sink/GstXvImageSink:fps-display-video_sink-actual-sink-xvimage: sync = true
/GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw, format=(string)YUY2, width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)RGBA
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)RGBA
/GstPipeline:pipeline0/Gstnvvconv:nvvconv1.GstPad:src: caps = video/x-raw, width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)RGBA
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:src: caps = video/x-raw, width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)RGB
/GstPipeline:pipeline0/GstCapsFilter:capsfilter2.GstPad:src: caps = video/x-raw, width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)RGB
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:sink: caps = video/x-raw, width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)RGB
/GstPipeline:pipeline0/GstQueue:queue0.GstPad:src: caps = video/x-raw, width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)RGB
/GstPipeline:pipeline0/gstccm800x800cv+CCM800x800CV:gstccm800x800cv+ccm800x800cv0.GstPad:src: caps = video/x-raw, width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)RGB
/GstPipeline:pipeline0/GstQueue:queue1.GstPad:sink: caps = video/x-raw, width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)RGB
/GstPipeline:pipeline0/gstccm800x800cv+CCM800x800CV:gstccm800x800cv+ccm800x800cv0.GstPad:sink: caps = video/x-raw, width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)RGB
/GstPipeline:pipeline0/gstccm800x800cv+CCM800x800CV:gstccm800x800cv+ccm800x800cv0.GstPad:sink: caps = video/x-raw, width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)RGB
/GstPipeline:pipeline0/GstVideoConvert:videoconvert1.GstPad:src: caps = video/x-raw, width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)YUY2
/GstPipeline:pipeline0/GstQueue:queue2.GstPad:sink: caps = video/x-raw, width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)YUY2
/GstPipeline:pipeline0/GstQueue:queue2.GstPad:sink: caps = video/x-raw, width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)YUY2
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-raw, width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)YUY2
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay.GstPad:src: caps = video/x-raw, width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)YUY2
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstAutoVideoSink:fps-display-video_sink.GstGhostPad:sink.GstProxyPad:proxypad1: caps = video/x-raw, width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)YUY2
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstAutoVideoSink:fps-display-video_sink/GstXvImageSink:fps-display-video_sink-actual-sink-xvimage.GstPad:sink: caps = video/x-raw, width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)YUY2
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstAutoVideoSink:fps-display-video_sink.GstGhostPad:sink: caps = video/x-raw, width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)YUY2
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay.GstPad:video_sink: caps = video/x-raw, width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)YUY2
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink: caps = video/x-raw, width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)YUY2
/GstPipeline:pipeline0/GstVideoConvert:videoconvert1.GstPad:sink: caps = video/x-raw, width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)RGB
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:sink: caps = video/x-raw, width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)RGBA
/GstPipeline:pipeline0/Gstnvvconv:nvvconv1.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)RGBA
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive, format=(string)RGBA
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw, format=(string)YUY2, width=(int)800, height=(int)800, framerate=(fraction)80/1, multiview-mode=(string)mono, pixel-aspect-ratio=(fraction)1/1, interlace-mode=(string)progressive
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstAutoVideoSink:fps-display-video_sink/GstXvImageSink:fps-display-video_sink-actual-sink-xvimage: sync = true
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 18, dropped: 7, fps: 35.80, drop rate: 13.92
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 18, dropped: 7, fps: 35.80, drop rate: 13.92
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 38, dropped: 9, fps: 38.64, drop rate: 3.86
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 38, dropped: 9, fps: 38.64, drop rate: 3.86
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 39, dropped: 28, fps: 2.00, drop rate: 37.97
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 39, dropped: 28, fps: 2.00, drop rate: 37.97
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 54, dropped: 34, fps: 28.60, drop rate: 11.44
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 54, dropped: 34, fps: 28.60, drop rate: 11.44
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 64, dropped: 45, fps: 19.09, drop rate: 21.00
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 64, dropped: 45, fps: 19.09, drop rate: 21.00

Note: the videotestsrc is emulating a v4l2src camera that’s limited to YUY2.

DaneLLL · September 27, 2023, 2:37am

Hi,
The hardware converter does not support 24-bit BGR and RGB. You would need to convert to RGBA, copy data from NVMM buffer to CPU buffer, and re-sample to RGB through software converter. Certain CPU usage is expected for the use-case. Please execute sudo jetson_clocks to run CPU cores at maximum clock. And check if there is further improvement.

ben.jacobson · September 27, 2023, 3:15am

Thank you for the fast response! Yes, I understand that the HW converter must convert to RGBA first, and then resample to RGB in the CPU software. That’s exactly my goal in the 2nd pipeline in my original post–here’s the relevant part: [quote=“ben.jacobson, post:1, topic:267713”]
nvvidconv ! "video/x-raw(memory:NVMM),format=RGBA" ! nvvidconv ! videoconvert ! video/x-raw,format=RGB
[/quote]
Is this incorrect? Is there a better way to accomplish this goal?

DaneLLL · September 27, 2023, 3:32am

Hi,
The pipeline looks good and is optimal for this use-case.

Honey_Patouceul · September 28, 2023, 5:20pm

Also note that videotestsrc is CPU implementation. Not tried on Orin Nano, but on some other low range Jetsons I’ve faced bottlenecks because of that at high framerates.
You may try specifying a low resolution for videotestsrc and use nvvidconv for scaling into your target resolution.

Also for same reason disable text-overlay property of fpsdisplaysink.
Also as video-sink try fakesink first, and for any display be sure that this display can support 80 fps.

You may try:

# Boost clocks
sudo jetson_clocks
sudo su
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vic/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vic/state
cat /sys/kernel/debug/bpmp/debug/clk/vic/max_rate > /sys/kernel/debug/bpmp/debug/clk/vic/rate
exit

# Without your custom element
gst-launch-1.0 videotestsrc ! video/x-raw,width=200,height=200,format=YUY2,framerate=80/1 ! nvvidconv ! "video/x-raw(memory:NVMM),format=RGBA,width=800,height=800" ! nvvidconv ! videoconvert ! video/x-raw,format=RGB ! queue ! videoconvert ! queue ! fpsdisplaysink text-overlay=0 video-sink=fakesink -v

# With your custom element
gst-launch-1.0 videotestsrc ! video/x-raw,width=200,height=200,format=YUY2,framerate=80/1 ! nvvidconv ! "video/x-raw(memory:NVMM),format=RGBA,width=800,height=800" ! nvvidconv ! videoconvert ! video/x-raw,format=RGB ! queue ! ccm800x800cv ! queue ! videoconvert ! queue ! fpsdisplaysink text-overlay=0 video-sink=fakesink -v

nvvidconv would not use GPU but VIC by default. You may also try to use GPU instead setting its compute-hw property such as:

... ! nvvidconv compute-hw=GPU ! ...

ben.jacobson · October 2, 2023, 7:06pm

@Honey_Patouceul these are all great suggestions, thanks for the quick reply!!

system · October 25, 2023, 3:43am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.