How to use deepstream-app with MJPEG format stream?

[936694123]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) Jetson Xavier NX
• DeepStream Version 6.0
**• JetPack Version (valid for Jetson only)**4.6.4
• TensorRT Version8.2
• NVIDIA GPU Driver Version (valid for GPU only)
**• Issue Type( questions, new requirements, bugs)**usb camera format
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing) commend line: v4l2-ctl -d /dev/video0 --list-formats-ext,then it will display:
Index : 0
Type : Video Capture
Pixel Format: ‘MJPEG’ (compressed)
Name : Motion-JPEG
Size: Discrete 3840x2160
Interval: Discrete 0.017s (60.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 2560x1440
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.200s (5.000 fps)
Size: Discrete 1920x1080
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.040s (25.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.200s (5.000 fps) There’s a part of it I didn’t let out，the formats are NV12,YV12…
And I want to use USB camera in MJPEG format, what should I do?
The command I used was：sudo deepstream-app -c config_yoloV5.txt.
I guess the format I’m using now is NV12。But i want to use MJPEG format with 4K60fps!!!

• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

[yuweiw]

You can try to modify the source code: opt\nvidia\deepstream\deepstream\sources\apps\apps-common\src\deepstream_source_bin.c

      caps1 = gst_caps_new_simple ("video/x-raw", "format", G_TYPE_STRING, "MJPEG",
          "width", G_TYPE_INT, config->source_width, "height", G_TYPE_INT,
          config->source_height, "framerate", GST_TYPE_FRACTION,
          config->source_fps_n, config->source_fps_d, NULL);

[936694123]

OK! Let me try and thanks to your reply!

[936694123]

ubuntu:/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/dsAppN_3$ sudo dsAppN_shi_3 -c deepstream_app_config3.txt
[Invalid UTF-8] here is config file '\xa0\xd8\x0b\x9cU’here is config number file ‘1’/opt/nvidia/deepstream/deepstream/sources/DeepStream-Yolo/deepstream_app_config3.txt

Please wait: Disabling —

** ERROR: <create_camera_source_bin:173>: Failed to link ‘nvvidconv2’ (video/x-raw, width=(int)3840, height=(int)2160, framerate=(fraction)60/1, format=(string){ GRAY8, YUY2, YVYU, UYVY, Y42B, NV12, I420, BGRx }; video/x-raw, width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)60/1, format=(string){ I420, NV12, P010_10LE, BGRx, RGBA, GRAY8, YUY2, UYVY, YVYU, Y42B }; video/x-raw(memory:NVMM), width=(int)[ 1, 2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)60/1, format=(string){ I420, NV12, P010_10LE, BGRx, RGBA, GRAY8, YUY2, UYVY, YVYU, Y42B }) and ‘src_cap_filter’ (video/x-raw(memory:NVMM), format=(string)MJPG, width=(int)3840, height=(int)2160, framerate=(fraction)60/1)
** ERROR: <create_camera_source_bin:218>: create_camera_source_bin failed
** ERROR: <create_pipeline:1326>: create_pipeline failed
** ERROR: main:932: Failed to create pipeline
Cleaning up…
App run failed
Cleanup complete

How can i reslove this problem?Does it means my NX can not use deepstream-app with camera format “MJPG”?

[936694123]

Can you tell me where change the code with the exact number of rows.
I try to change the code but it does not work!
and when i use the command:
gst-launch-1.0 v4l2src device=/dev/video0 ! image/jpeg,width=3840,height=2160,framerate=60/1 ! jpegdec ! videoconvert ! xvimagesink. It works!!!

[yuweiw]

static gboolean
create_camera_source_bin (NvDsSourceConfig * config, NvDsSrcBin * bin)

      caps1 = gst_caps_new_simple ("video/x-raw",
          "width", G_TYPE_INT, config->source_width, "height", G_TYPE_INT,
          config->source_height, "framerate", GST_TYPE_FRACTION,
          config->source_fps_n, config->source_fps_d, NULL);

Please configure this caps1 according to your own actual parameters.

[936694123]

When I tested it like you said something wrong happened：
[Invalid UTF-8] here is config file '\xa0\xe8\xd2\xa5U’here is config number file ‘1’/opt/nvidia/deepstream/deepstream/sources/DeepStream-Yolo/deepstream_app_config3.txt
Please wait: Disabling —
** ERROR: <create_camera_source_bin:162>: Failed to link ‘src_elem’ (image/jpeg; video/mpeg, mpegversion=(int)4, systemstream=(boolean)false; video/mpeg, mpegversion=(int)2; video/mpegts, systemstream=(boolean)true; video/x-bayer, format=(string){ bggr, gbrg, grbg, rggb }, width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-dv, systemstream=(boolean)true; video/x-h263, variant=(string)itu; video/x-h264, stream-format=(string){ byte-stream, avc }, alignment=(string)au; video/x-pwc1, width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-pwc2, width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-raw, format=(string){ RGB16, BGR, RGB, GRAY8, GRAY16_LE, GRAY16_BE, YVU9, YV12, YUY2, YVYU, UYVY, Y42B, Y41B, YUV9, NV12_64Z32, NV24, NV61, NV16, NV21, NV12, I420, BGRA, BGRx, ARGB, xRGB, BGR15, RGB15 }, width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-sonix, width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-vp8; video/x-vp9; video/x-wmv, wmvversion=(int)3, format=(string)WVC1) and ‘src_cap_filter1’ (video/x-raw, format=(string)mjpg, width=(int)3840, height=(int)2160, framerate=(fraction)60/1)
** ERROR: <create_camera_source_bin:217>: create_camera_source_bin failed
** ERROR: <create_pipeline:1326>: create_pipeline failed
** ERROR: main:932: Failed to create pipeline
Cleaning up…
App run failed
Cleanup complete
And what should i do with the error?

[936694123]

How to change source pixel format from YUYV to MJPG

Hi, I’m using deepstream_app of recent version 4.0, and using with yolo detector through logitech webcam(c930e). When I run deepstream app with yolo config file, it runs with YUYV pixel format, but I would like to use with MJPG format, as its fps is faster than YUYV’s.(As shown in bottom) :~$ v4l2-ctl --list-formats-ext ioctl: VIDIOC_ENUM_FMT Index : 0 Type : Video Capture Pixel Format: 'YUYV' Name : YUYV 4:2:2 Size: Discrete 640x480 Interval: Discrete 0.033s (30.…

Should i try this with this problem?

[yuweiw]

Could you attach what parameters are you using for gst_caps_new_simple ? Is it like below?

caps1 = gst_caps_new_simple ("image/jpeg",
      "width", G_TYPE_INT, 3840, "height", G_TYPE_INT,
     2160, "framerate", GST_TYPE_FRACTION,
      60, 1, NULL);

[936694123]

It likes this:
caps1 = gst_caps_new_simple (“video/x-raw”,“format”, G_TYPE_STRING, “mjpg”, dth", G_TYPE_INT, config->source_width, “height”, G_TYPE_INT,
config->source_height, “framerate”, GST_TYPE_FRACTION,
config->source_fps_n, config->source_fps_d, NULL);

I will change the code likes yours and have a try!

[936694123]

[Invalid UTF-8] here is config file '\xa0(\xcfjU’here is config number file ‘1’/opt/nvidia/deepstream/deepstream/sources/DeepStream-Yolo/deepstream_app_config3.txt

Please wait: Disabling —

** ERROR: <create_camera_source_bin:163>: Failed to link ‘src_elem’ (image/jpeg; video/mpeg, mpegversion=(int)4, systemstream=(boolean)false; video/mpeg, mpegversion=(int)2; video/mpegts, systemstream=(boolean)true; video/x-bayer, format=(string){ bggr, gbrg, grbg, rggb }, width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-dv, systemstream=(boolean)true; video/x-h263, variant=(string)itu; video/x-h264, stream-format=(string){ byte-stream, avc }, alignment=(string)au; video/x-pwc1, width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-pwc2, width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-raw, format=(string){ RGB16, BGR, RGB, GRAY8, GRAY16_LE, GRAY16_BE, YVU9, YV12, YUY2, YVYU, UYVY, Y42B, Y41B, YUV9, NV12_64Z32, NV24, NV61, NV16, NV21, NV12, I420, BGRA, BGRx, ARGB, xRGB, BGR15, RGB15 }, width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-sonix, width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-vp8; video/x-vp9; video/x-wmv, wmvversion=(int)3, format=(string)WVC1) and ‘src_cap_filter1’ (images/jpeg, width=(int)1280, height=(int)1024, framerate=(fraction)30/1)
** ERROR: <create_camera_source_bin:218>: create_camera_source_bin failed
** ERROR: <create_pipeline:1326>: create_pipeline failed
** ERROR: main:932: Failed to create pipeline
Cleaning up…
App run failed
Cleanup complete

[936694123]

Through How to use deepstream-app with MJPEG format stream? - #9 by 936694123 can use the YSB camera with JPEG format,but the frameate is too low,below is the new problem!
XXXX@ubuntu:/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/dsAppN_3$ sudo dsAppN_shi_3 -c deepstream_app_config3.txt
[sudo] password for sfhs:
[Invalid UTF-8] here is config file '\xa08W\x8eU’here is config number file ‘1’/opt/nvidia/deepstream/deepstream/sources/DeepStream-Yolo/deepstream_app_config3.txt

Please wait: Disabling —

Using winsys: x11

Runtime commands:
h: Print this help
q: Quit

p: Pause
r: Resume

s: enable

d: disable 

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
To go back to the tiled display, right-click anywhere on the window.

** INFO: <bus_callback:194>: Pipeline ready

** INFO: <bus_callback:180>: Pipeline running

**PERF: FPS 0 (Avg)
**PERF: 15.58 (14.62)
WARNING from sink_sub_bin_sink1: A lot of buffers are being dropped.
Debug info: gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin1/GstEglGlesSink:sink_sub_bin_sink1:
There may be a timestamping problem, or this computer is too slow.
WARNING from sink_sub_bin_sink1: A lot of buffers are being dropped.
Debug info: gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin1/GstEglGlesSink:sink_sub_bin_sink1:
There may be a timestamping problem, or this computer is too slow.
**PERF: 17.09 (16.84)
WARNING from sink_sub_bin_sink1: A lot of buffers are being dropped.
Debug info: gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin1/GstEglGlesSink:sink_sub_bin_sink1:
There may be a timestamping problem, or this computer is too slow.
WARNING from sink_sub_bin_sink1: A lot of buffers are being dropped.
Debug info: gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin1/GstEglGlesSink:sink_sub_bin_sink1:
There may be a timestamping problem, or this computer is too slow.
**PERF: 16.94 (16.91)
WARNING from sink_sub_bin_sink1: A lot of buffers are being dropped.
Debug info: gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin1/GstEglGlesSink:sink_sub_bin_sink1:
There may be a timestamping problem, or this computer is too slow.
WARNING from sink_sub_bin_sink1: A lot of buffers are being dropped.
Debug info: gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin1/GstEglGlesSink:sink_sub_bin_sink1:
There may be a timestamping problem, or this computer is too slow.
WARNING from sink_sub_bin_sink1: A lot of buffers are being dropped.
Debug info: gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin1/GstEglGlesSink:sink_sub_bin_sink1:
There may be a timestamping problem, or this computer is too slow.
WARNING from sink_sub_bin_sink1: A lot of buffers are being dropped.
Debug info: gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin1/GstEglGlesSink:sink_sub_bin_sink1:
There may be a timestamping problem, or this computer is too slow.
**PERF: 16.85 (16.87)
WARNING from sink_sub_bin_sink1: A lot of buffers are being dropped.
Debug info: gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin1/GstEglGlesSink:sink_sub_bin_sink1:
There may be a timestamping problem, or this computer is too slow.
WARNING from sink_sub_bin_sink1: A lot of buffers are being dropped.
Debug info: gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin1/GstEglGlesSink:sink_sub_bin_sink1:
There may be a timestamping problem, or this computer is too slow.
**PERF: 16.91 (16.86)
WARNING from sink_sub_bin_sink1: A lot of buffers are being dropped.
Debug info: gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin1/GstEglGlesSink:sink_sub_bin_sink1:
There may be a timestamping problem, or this computer is too slow.
WARNING from sink_sub_bin_sink1: A lot of buffers are being dropped.
Debug info: gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin1/GstEglGlesSink:sink_sub_bin_sink1:
There may be a timestamping problem, or this computer is too slow.
WARNING from sink_sub_bin_sink1: A lot of buffers are being dropped.
Debug info: gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin1/GstEglGlesSink:sink_sub_bin_sink1:
There may be a timestamping problem, or this computer is too slow.
WARNING from sink_sub_bin_sink1: A lot of buffers are being dropped.
Debug info: gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin1/GstEglGlesSink:sink_sub_bin_sink1:
There may be a timestamping problem, or this computer is too slow.
**PERF: 17.09 (16.92)
WARNING from sink_sub_bin_sink1: A lot of buffers are being dropped.
Debug info: gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline/GstBin:processing_bin_0/GstBin:sink_bin/GstBin:sink_sub_bin1/GstEglGlesSink:sink_sub_bin_sink1:
There may be a timestamping problem, or this computer is too

[yuweiw]

Your video is 4K, 60fps. There may be some performance issues for such video. You can try to change the sink plugin to fakesin and lower the frame rate and resolution of camera.