High CPU usage for video capturing

Hello,
We are using Jetson NX Xavier, capturing video (v4l2) like this:
gst-launch-1.0 v4l2src device=/dev/video0 ! ‘video/x-raw, format=UYVY, width=4192, height=3120, framerate=10/1’ ! nvvideoconvert ! ‘video/x-raw, format=NV12, width=4192, height=3120, framerate=10/1’ ! identity drop-allocation=1 ! v4l2sink device=/dev/video100 sync=false

The Stream looks fine besides it uses 100% CPU.

Is this expected?
I have verified omx plugin is good.

Any clues?

Thanks

Hi,
You may try to use nvv4l2camerasrc. Please check
Jetson Nano FAQ
[Q: I have a USB camera. How can I launch it on Jetson Nano?]

This may reduce partial CPU loading. The optimal solution is to pass NVMM buffers from source to sink, but it only accepts CPU buffers in v4l2sink. Certain CPU loading is expected in this usecase.

I tried this before (taken from your reference):
gst-launch-1.0 nvv4l2camerasrc device=/dev/video0 ! ‘video/x-raw(memory:NVMM),format=UYVY,width=4192,height=3120,framerate=10/1’ ! nvvideoconvert ! ‘video/x-raw(memory:NVMM),format=NV12’ ! nvoverlaysink -e
It gives me this:
Setting pipeline to PAUSED …
Pipeline is live and does not need PREROLL …
Setting pipeline to PLAYING …
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstNvV4l2CameraSrc:nvv4l2camerasrc0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstNvV4l2CameraSrc:nvv4l2camerasrc0:
streaming stopped, reason error (-5)
EOS on shutdown enabled – waiting for EOS after Error
Waiting for EOS…

What I would like to do is:
gst-launch-1.0 nvv4l2camerasrc device=/dev/video0 ! ‘video/x-raw,format=UYVY,width=4192,height=3120,framerate=10/1’ ! nvvideoconvert ! ‘video/x-raw(memory:NVMM),format=NV12,width=4192,height=3120,framerate=10/1’ ! v4l2sink device=/dev/video100 sync=false
WARNING: erroneous pipeline: could not link nvv4l2camerasrc0 to nvvideoconvert0, nvv4l2camerasrc0 can’t handle caps video/x-raw, format=(string)UYVY, width=(int)4192, height=(int)3120, framerate=(fraction)10/1

But it doesnt work either…
Please note I changed the example you gave: nvvidconv → nvvideoconvert because nvvidconv is not known.

Thanks

Hi,
For linking nvv4l2camerasrc and nvvideoconvert, please set the property to nvv4l2camerasrc:

  bufapi-version      : set to use new Buffer API
                        flags: readable, writable
                        Boolean. Default: false
1 Like

Thanks, with this I am able to run the first command and use nvoverlaysink.

However, still not able to sink to a v4l2loopback device:
gst-launch-1.0 nvv4l2camerasrc bufapi-version=TRUE device=/dev/video0 ! ‘video/x-raw(memory:NVMM),format=UYVY,width=4192,height=3120,framerate=10/1’ ! nvvideoconvert ! ‘video/x-raw(memory:NVMM),format=NV12’ ! v4l2sink device=/dev/video100 sync=false
WARNING: erroneous pipeline: could not link nvvideoconvert0 to v4l2sink0, v4l2sink0 can’t handle caps video/x-raw(memory:NVMM), format=(string)NV12

Actually I now got this to work by adding identity drop allocation before the sink.
Not sure how this solved the issue, would appreciate if you can explain

Hi I have the same problem, but I’m not familiar with Gsstreamer, could you please share the command that worked? Maybe works for me too
thanks

The root cause is probably in v4l2loopback. See this.

For CPU usage, it is probably v4l2loopback as well causing high CPU load. Try with fakesink instead of v4l2sink for checking.

Hi,
Now having issue with the data itself…
The pipeline seems to work (at least says its working), but I get a scrambled image on screen (no matter which sink I will use).
FOr example:
gst-launch-1.0 nvv4l2camerasrc bufapi-version=TRUE device=/dev/video0 ! ‘video/x-raw(memory:NVMM),format=UYVY,width=4192,height=3120,framerate=10/1’ ! nvvideoconvert ! nvegltransform ! nveglglessink

Same thing if I do this (stream to v4l2loopback device and consume from it):
gst-launch-1.0 nvv4l2camerasrc bufapi-version=TRUE device=/dev/video0 ! ‘video/x-raw(memory:NVMM),format=UYVY,width=4192,height=3120,framerate=10/1’ ! nvvideoconvert ! ‘video/x-raw,format=NV12’ ! identity drop-allocation=1 ! v4l2sink device=/dev/video100 sync=false

Camera is 13MP, needed resolution is 41923120.
It works with lower resolution (1920
1080).

Please advice.

This works for me for 1920,1080 and lower resolution:
gst-launch-1.0 nvv4l2camerasrc bufapi-version=TRUE device=/dev/video0 ! ‘video/x-raw(memory:NVMM),format=UYVY,width=1920,height=1080,framerate=30/1’ ! nvvideoconvert ! ‘video/x-raw,format=NV12’ ! identity drop-allocation=1 ! v4l2sink device=/dev/video100 sync=false

It doesn’t work for 4192,3120…

Please advice.
I am able to capture video till 4096x2160 resolution
Using 4192x3120 the video goes up scrambled, like lines…
Same command being used for both resolutions except for resolution itself and fps:

This one works:
gst-launch-1.0 nvv4l2camerasrc bufapi-version=TRUE device=/dev/video0 ! ‘video/x-raw(memory:NVMM),format=UYVY,width=4096,height=2160,framerate=15/1’ ! nvvideoconvert ! ‘video/x-raw,format=NV12’ ! identity drop-allocation=1 ! v4l2sink device=/dev/video100 sync=false

This one doesnt:
gst-launch-1.0 nvv4l2camerasrc bufapi-version=TRUE device=/dev/video0 ! ‘video/x-raw(memory:NVMM),format=UYVY,width=4192,height=3120,framerate=10/1’ ! nvvideoconvert ! ‘video/x-raw,format=NV12’ ! identity drop-allocation=1 ! v4l2sink device=/dev/video100 sync=false

In both cases pipelines are not complaining during execution.

This sounds like memory or capability. I looked at the nvv4l2camerasrc source code but didnt see any limitation with regards to resolution.

There is no problem with the camera since it works with the v4l2 plugin but consumes too much CPU due to not being able to use the NVMM memory.

Thanks

I think there may be a 4096 colums HW limit for VIC (and maybe other HW blocks).
Someone from NVIDIA may further clarifiy these.

Hi,
In nvv4l2camerasrc plugin, the frames are captured and stored into NVMM buffer directly. NVMM buffer is hardware DMA buffer and for certain resolution, the pitch and width may not be identical and the captured frames look broken. So for 4192x3120, you may not be able to use nvv4l2camersrc plugin and have to use v4l2src plugin.

Thanks.
Is there a list of resolutions that are supported by this plugin?

Going back to v4l2src is horrible. Using this plugin results in a very high CPU usage for the q first 10 minutes (100%) and video is quite bad during this time. After 10 minutes CPU usage goes down to 40% which is much higher than using the nvv4l2camerasrc and the video becomes ok.

Any other suggestions?
This will be very disappointing if we are left like this especially for a production system…

Thanks

Hi,
The data alignment is a hardware contraint, so cannot be changed. One possible solution is to check if your source can output in different pitch and width. You can call NvBufferGetParams to get value of pitch, width, height. See if your source can output data to match the buffer.

I already checked this and it doesn’t…
I mean it can go down to 8mp, 4096 width, but nothing in between.

Using v4l2src is there a way I can use dma/nvmm?
Does this pipeline looks good:
gst-launch-1.0 v4l2src device=/dev/video0 ! ‘video/x-raw, format=UYVY, width=4192, height=3120, framerate=10/1’ ! nvvideoconvert ! ‘video/x-raw, format=NV12, width=4192, height=3120, framerate=10/1’ ! identity drop-allocation=1 ! v4l2sink device=/dev/video100 sync=false

This one works but with high CPU usage.
Maybe pipeline can be optimized, need your feedback on this…

Hi,
A user has shared a solution for GREY format:
Gstnvv4l2camerasrc with GRAY8 support - #17 by rbi

Please check if you can apply it to your usecase. If not, the only method is to match the source data with hardware DMA buffer.