TX1 gstreamer nvvidconv will not pass out of NVMM memory

Autonomous Machines Jetson & Embedded Systems Jetson TX1
1

I am trying to using the nvvidconv plugin to perform efficient colorspace conversion between I420 and RGBA from a USB webcam producing JPEG frames. The output will eventually be sent to an opencv ROS plugin. Right now I’m just trying to get the image into non NVMM memory after using nvvidconv for the colorspace conversion.

Example pipeline of what I’d like to do (that does not work):

gst-launch-1.0 -vvv -e v4l2src device=/dev/video0 do-time-stamp=true ! image/jpeg, width=1280, height=720, framerate=60/1 ! nvjpegdec ! nvvidconv ! video/x-raw, format=(string)RGBA ! videoconvert ! xvimagesink sync=false

The only way I’v been able to get something to work is by placing an “video/x-raw” between nvjpegdec and nvvidconv and using a gpu based video sink such as nvoverlaysink or “nvegltransform ! nveglglessink”

The commands below render images from the camera however I don’t know how to get them from there into my application.

nvoverlaysink:

gst-launch-1.0 -vvv -e v4l2src device=/dev/video0 do-time-stamp=true ! image/jpeg, width=1280, height=720, framerate=60/1 ! nvjpegdec ! video/x-raw ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)RGBA ! nvoverlaysink sync=false

egl:

gst-launch-1.0 -vvv -e v4l2src device=/dev/video0 do-time-stamp=true ! image/jpeg, width=1280, height=720, framerate=60/1 ! nvjpegdec ! video/x-raw ! nvvidconv ! video/x-raw(memory:NVMM), format=(string)RGBA ! nvegltransform ! nveglglessink

I cannot pipe straight from nvvidconv to an xvimagesink as is suggested in the Multimedia User Guide in multiple places. I noticed that in the Guide there is always an openmax plugin before the nvvidconv plugin in those examples, which makes me wonder if that is not coincidental.

In the end that means that the below command doesn’t work even with using video/x-raw between nvjpegdec and nvvidconv.

gst-launch-1.0 -vvv -e v4l2src device=/dev/video0 do-time-stamp=true ! image/jpeg, width=1280, height=720, framerate=60/1 ! nvjpegdec ! video/x-raw ! nvvidconv ! video/x-raw, format=(string)I420 ! xvimagesink sync=false

I have also tried not specifying the colorspace output to xvimagesink and using a videoconvert between nvvidconv and xvimagesink just to see if something would show up.

My end goal would be to have njpegdec and nvvidonv use NVMM to pass images and have nvvidconv pass out in the video/x-raw, format=RGBA format that can then be used in the ROS/OpenCV application.

The gstreamer videoconvert plugin can be used in the place of the nvvidconv plugin but it results in high cpu usage.

All help is greatly appreciated.

2

hi LBurner,
nvvidconv supports
video/x-raw(memory:NVMM) ! nvvidconv ! video/x-raw(memory:NVMM)
video/x-raw(memory:NVMM) ! nvvidconv ! video/x-raw
video/x-raw ! nvvidconv ! video/x-raw(memory:NVMM)

,but does not support
video/x-raw ! nvvidconv ! video/x-raw

The reason is that HW engine only accepts NVMM memory inputs/outputs, so the following cases perform memcpy():
video/x-raw(memory:NVMM) ! nvvidconv ! memcpy(video/x-raw(memory:NVMM)->video/x-raw)
memcpy(video/x-raw->video/x-raw(memory:NVMM)) ! nvvidconv ! video/x-raw(memory:NVMM)

For video/x-raw ! nvvidconv ! video/x-raw, it has to perform two memcpy for each conversion. It is probably worse than videoconvert.

1 Like
3

Thank you DaneLLL,

That explains the nvvidconv usage issues thank you. Is “memcpy” a gstreamer plugin? It does not appear to be in the Jetson’s installation thus I was not able to attempt the pipelines you listed with gst-launch-1.0.

What memory configurations does nvjpegdec support? gst-inspect-1.0 says it can source and sink x-raw and x-raw(memory:NVMM) however the below line does not work despite being compatible with the nvvidconv usage indicated above.

image/jpeg, width=1280, height=720, framerate=60/1 ! nvjpegdec ! video/x-raw(memory:NVMM) ! nvvidconv ! video/x-raw, format=(string)RGBA

I suppose that nvjpegdec has similar limitations as nvvidconv but the documentation does not explain them.

Currently videoconvert is using 82% of a core to perform the colorspace conversion at 30fps instead of the 60fps we would like to use. We would like to test to see if memcpy results in similar usage.

1 Like
4

Hi LBurner,
memory copy is internally done in nvvidconv, not a plugin.

Can cvtColor of openCV do I420 to RGBA conversion? If yes, we would suggest run the pipeline:

gst-launch-1.0 v4l2src ! image/jpeg, width=1280, height=720, framerate=60/1 ! nvjpegdec ! video/x-raw, format=(string)I420 ! appsink

And do conversion via cvtColor()

nvjpegdec can output video/x-raw(memory:NVMM) in single jpeg decoding, but cannot in continuous jpeg decoding.

5

It appears that opencv may support the conversion on the CPU however not on the GPU through CUDA which would make the speed difference. More investigation into that option is ongoing.

In theory the plugin jpegparse should be able to split continuous jpeg frames into single jpegs according to: jpegparse

However the pipeline:

gst-launch-1.0 v4l2src ! image/jpeg, width=1280, height=720, framerate=60/1 ! jpegparse ! nvjpegdec ! video/x-raw(memory:NVMM), format=(string)I420 ! nvvidconv ! video/x-raw, format=(string)RGBA

Does not appear to work. Is this because nvjpegdec needs to output video/x-raw(memory:NVMM) and that counts as a continuous stream even though the input should be single images? Or is there another reason?

6

Hi LBurner, it works in single JPEG case like

gst-launch-1.0 filesrc location= output_.jpg ! nvjpegdec ! 'video/x-raw(memory:NVMM), format=(string)I420' ! nvvidconv ! 'video/x-raw,format=I420' ! imagefreeze ! xvimagesink
7

I see thank you.

Connecting an nvvidconv into another nvvidconv to get video/x-raw to video/x-raw support seems to be the best option so far. The first nvvidconv does the conversion and dumps the result into NVMM memory. The next nvvidconv performs no conversion and copies the image to standard x-raw memory.

gst-launch-1.0 v4l2src ! image/jpeg, width=1280, height=720, framerate=30/1 ! jpegparse ! nvjpegdec ! video/x-raw, format=(string)I420 ! nvvidconv ! video/x-raw, format=(string)RGBA ! nvvidconv ! video/x-raw, format=(string)RGBA ! appsink

This allows us to not use the CPU based videoconvert plugin for the conversion and drops our total CPU usage (not just what videoconvert was using) from 95% to 20% when processing 720p at 30fps. This is almost a 5x speedup. The extra memcpy is significantly cheaper than doing the video conversion on the cpu.

Increasing the framerate to 720p at 60fps only uses 15% more cpu for a total of 35% of a single core.

8

It’s surprising that two nvvidconv perform better than videoconvert. Many thanks for the sharing.