Gstreamer and nvvidconv

I am working on an application where I am working with video and am trying to use Nvidia’s accelerated Gstreamer elments. I have a pipeline that is recording a camera’s video at 3840x2160. I am working on an Nvidia Xavier (16GB) module with MIPI cameras. The pipeline that I am using is:

gst-launch-1.0 v4l2src device=/dev/video3 ! “video/x-raw, format=UYVY, width=3840” ! nvvidconv interpolation-method=0 ! “video/x-raw(memory:NVMM)” ! omxvp9enc bitrate=20000000 ! matroskamux ! filesink location=test.mkv

When I am looking at what is the bottleneck in this pipeline, I see that it is nvvidconv. Gst-shark is showing me that the processing time for this element is alternating between about 10 and 60 ms. Approximately 2/3 of the time it is taking 60 ms. This is dropping my frame rate to approximately 25 fps. Everything else in the pipeline is capable of running at a full 30fps. I’d like to hit 30fps and maintain that with no dropped frames since I will ultimately be recording multiple cameras that will be used to reconstruct the depth of the things that they are seeing. I need an image in each time slot from both cameras or else I can’t estimate depths.

I can’t change the input or output format and am not scaling or cropping. I’ve tried changing the scaling method to nearest neighbor just to see if that did anything but it doesn’t . I suspect that this has something to do with copying from main RAM to GPU RAM but don’t know what I would do to speed that up. Does anyone here have any idea what I could do to eliminate this bottleneck?

Hi @jacksmfht,

Have you tried using a queue between elements? For example:

gst-launch-1.0 v4l2src device=/dev/video3 ! “video/x-raw, format=UYVY, width=3840” ! nvvidconv interpolation-method=0 ! “video/x-raw(memory:NVMM)” ! queue ! omxvp9enc bitrate=20000000 ! matroskamux ! filesink location=test.mkv

I don’t have such a high resolution camera for trying, but with my Xavier R32.3.1, this works fine at 30 fps:

gst-launch-1.0 -v videotestsrc ! video/x-raw, format=UYVY, width=3840, height=2160, framerate=30/1 ! nvvidconv interpolation-method=0 ! 'video/x-raw(memory:NVMM)' ! tee name=t ! queue ! omxvp9enc bitrate=20000000 ! matroskamux ! filesink location=test.mkv    t. ! queue ! fpsdisplaysink video-sink=fakesink text-overlay=false

I tried queues. They didn’t make any difference.

Interesting. It seems like you have hit on something here. I stripped everything away to just:

gst-launch-1.0 -v videotestsrc ! video/x-raw, format=UYVY, width=3840, height=2160, framerate=30/1 ! nvvidconv interpolation-method=0 ! 'video/x-raw(memory:NVMM)' ! queue ! fpsdisplaysink video-sink=fakesink text-overlay=false

This gets a solid 30.00 fps.

If I swap in my camera:

gst-launch-1.0 -v v4l2src device=/dev/video3 ! video/x-raw, format=UYVY, width=3840, height=2160, framerate=30/1 ! nvvidconv interpolation-method=0 ! 'video/x-raw(memory:NVMM)' ! queue ! fpsdisplaysink video-sink=fakesink text-overlay=false

I now get 16.60 fps.

Stripping it back further by removing the nvvidconv, I jump up to 29.21 fps.

gst-launch-1.0 -v v4l2src device=/dev/video3 ! video/x-raw, format=UYVY, width=3840, height=2160, framerate=30/1 ! queue ! fpsdisplaysink video-sink=fakesink text-overlay=false

That still isn’t 30 fps but it is a lot closer. I’m going to get in touch with the camera manufacturer since they advertise this as being able to ‘stream’ in 4K resolution. There seems to be some sort of delay caused by the data coming out of the camera that causes nvvidconv to slow down.

I cannot say much more… You may try:

  • using various io-mode of v4l2src and see if one improves
  • removing interpolation-method=0. Although it self describes with gst-inspect as default being smart, it uses nearest in fact. In one of my trials, this seemed to make a slight difference. Not sure however.

See if gst-shark can tell you more about cpuusage and proctime.