How to hardware accelerate mjpeg decoding of frames from webcam using gst

Hi,

I am using Logitech C920 webcam on Jetson Nano and trying to capture a MJPEG video stream and to convert its frames to BGR (or BGRx) format while doing the decoding on the Jetson hardware (same issue reproduce on Jetson TX2).

  1. When using:
    gst-launch-1.0 v4l2src device=/dev/video0 ! "image/jpeg,framerate=30/1,width=1280,height=720" ! jpegparse ! nvv4l2decoder mjpeg=1 ! nvvidconv ! "video/x-raw,width=1280,height=720,format=BGRx,framerate=30/1" ! ximagesink
    

    I can see the video stream but the frames look grayed and not colored.

  2. When using the nvv4l2decoder without the mjpeg=1:
    gst-launch-1.0 v4l2src device=/dev/video0 ! "image/jpeg,framerate=30/1,width=1280,height=720" ! jpegparse ! nvv4l2decoder ! nvvidconv ! "video/x-raw,width=1280,height=720,format=BGRx,framerate=30/1" ! ximagesink
    

    I got one colored image and the video stream was stuck.

  3. When using nvjpegdec instead of the nvv4l2decoder:
    gst-launch-1.0 v4l2src device=/dev/video0 ! "image/jpeg,framerate=30/1,width=1280,height=720" ! jpegparse ! nvjpegdec ! nvvidconv ! "video/x-raw,width=1280,height=720,format=BGRx,framerate=30/1" ! ximagesink
    

    I got the following error:

    Setting pipeline to PAUSED ...
    Pipeline is live and does not need PREROLL ...
    Setting pipeline to PLAYING ...
    New clock: GstSystemClock
    ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src0: Internal data stream error.
    Additional debug info:
    gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstV4l2Src:v4l2src0:
    streaming stopped, reason error (-5)
    Execution ended after 0:00:00.784941670
    Setting pipeline to PAUSED ...
    Setting pipeline to READY ...
    Setting pipeline to NULL ...
    Freeing pipeline ...
    
  4. When using regular jpegdec it was working fine visually, BUT it was not hardware accelerated
    gst-launch-1.0 v4l2src device=/dev/video0 ! "image/jpeg,framerate=30/1,width=1280,height=720" ! jpegparse ! jpegdec ! videoconvert ! "video/x-raw,width=1280,height=720,format=BGRx,framerate=30/1" ! ximagesink
    

BTW, capturing h264 format is working fine but I need to use a camera that supports only MJPEG:

gst-launch-1.0 v4l2src device=/dev/video0 ! "video/x-h264,framerate=30/1,width=1280,height=720" ! h264parse ! nvv4l2decoder ! nvvidconv ! "video/x-raw,width=1280,height=720,format=BGRx,framerate=30/1" ! ximagesink

Can anyone suggest a way to capture MJPEG frames and to convert it to BGRx while the decoding is done on the jetson’s accelerated hardware?

Hi,
Please try

gst-launch-1.0 v4l2src device=/dev/video0 ! "image/jpeg,framerate=30/1,width=1280,height=720" ! jpegparse ! nvjpegdec ! video/x-raw ! videoconvert ! "video/x-raw,width=1280,height=720,format=BGRx,framerate=30/1" ! ximagesink

Hi DaneLLL,

I tried your command line.
I got colored image but only a singe frame and it being stuck on it.

the output was:

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
WARNING: from element /GstPipeline:pipeline0/GstXImageSink:ximagesink0: A lot of buffers are being dropped.
Additional debug info:
gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstXImageSink:ximagesink0:
There may be a timestamping problem, or this computer is too slow.
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:39.748317142
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

Hi,
You may try nvoverlaysink

gst-launch-1.0 v4l2src device=/dev/video0 ! "image/jpeg,framerate=30/1,width=1280,height=720" ! jpegparse ! nvjpegdec ! video/x-raw ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! nvoverlaysink

Hi DaneLLL,

I tried your command line.
I got colored image but only a singe frame and it being stuck on it. I canceled it after 10 seconds…

The output was:

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:10.368161244
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

Hi,
Please execute ‘sudo jetson_clocks’ and run v4l2src in ‘io-mode=2’

$ gst-launch-1.0 v4l2src device=/dev/video1 io-mode=2  ! "image/jpeg,framerate=30/1,width=1280,height=720" ! jpegparse ! nvjpegdec ! video/x-raw ! nvvidconv ! 'video/x-raw(memory:NVMM)' ! nvoverlaysink

Hi DaneLLL,

This one worked fine (i.e. the hardware accelerated decoding) but, I can see a video stream only for few seconds, after it it shows green display and I stopped it.

The output was:

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
NvRmChannelSubmit: NvError_IoctlFailed with error code 22
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 22, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 22, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 22, SyncPointValue = 0)
..... and so on many times ....
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 22, SyncPointValue = 0)
NvRmPrivFlush: NvRmChannelSubmit failed (err = 196623, SyncPointIdx = 22, SyncPointValue = 0)
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:18.156703620
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

Hi,
Please check
https://devtalk.nvidia.com/default/topic/1048640/jetson-nano/power-supply-considerations-for-jetson-nano-developer-kit/
We can run Logitech c930e. Suspect your issue is due to insufficient power supply.

Hi DaneLLL,

The same issue (green screen after few seconds) happens also on TX2 with the original power supply.

My bad. Forgot there is a known issue.

Please check
https://elinux.org/L4T_Jetson/r32.2.1_patch

Hi DaneLLL,

Thank you for referring me to the r32.2.1 patch. It is working fine with your command line and I manage to convert if to BGRx afterwards.

Can you please tell why nvv4l2decoder with the mjpeg=1 is not working properly and it it something you are planning to fix?

Hi,
nvv4l2decoder can decode compressed YUV420 JPEGs. Logitech c920 outputs compressed YUV422(YUYV) JPEGs and you should use nvjpegdec.

Got it.
Thank you very much.