V4L2 MJPG low framerate

aron.h · July 16, 2024, 6:46pm

Environment

• Hardware Platform (Jetson / GPU)
Jetson Orin NX
• DeepStream Version
DeepStream 6.1
• JetPack Version (valid for Jetson only)
Host: 5.1.1-b56
Docker: 5.0.2-b231
• TensorRT Version
Host: 8.4.1.5
Docker: 8.5.2.2
• Issue Type( questions, new requirements, bugs)
Question

I have been testing with two different usb cameras and I find that MJPG framerate is lower than what they claim during negotiation.
Each is connected to their own Jetson Orin NX via USB2.0 with identical setup.

Camera A:
OV2643

v4l2-ctl --list-formats-ext
[0]: 'YUYV' (YUYV 4:2:2)
        Size: Discrete 1280x720
                Interval: Discrete 0.033s (30.000 fps)
                Interval: Discrete 0.040s (25.000 fps)
                Interval: Discrete 0.050s (20.000 fps)
                Interval: Discrete 0.067s (15.000 fps)
                Interval: Discrete 0.100s (10.000 fps)
                Interval: Discrete 0.200s (5.000 fps)
        Size: Discrete 640x480
                Interval: Discrete 0.033s (30.000 fps)
                Interval: Discrete 0.040s (25.000 fps)
                Interval: Discrete 0.050s (20.000 fps)
                Interval: Discrete 0.067s (15.000 fps)
                Interval: Discrete 0.100s (10.000 fps)
                Interval: Discrete 0.200s (5.000 fps)
        Size: Discrete 320x240
                Interval: Discrete 0.067s (15.000 fps)
                Interval: Discrete 0.100s (10.000 fps)
                Interval: Discrete 0.200s (5.000 fps)
        Size: Discrete 352x288
                Interval: Discrete 0.067s (15.000 fps)
                Interval: Discrete 0.100s (10.000 fps)
                Interval: Discrete 0.200s (5.000 fps)
        Size: Discrete 800x600
                Interval: Discrete 0.033s (30.000 fps)
                Interval: Discrete 0.040s (25.000 fps)
                Interval: Discrete 0.050s (20.000 fps)
                Interval: Discrete 0.067s (15.000 fps)
                Interval: Discrete 0.100s (10.000 fps)
                Interval: Discrete 0.200s (5.000 fps)
        Size: Discrete 1280x1024
                Interval: Discrete 0.033s (30.000 fps)
                Interval: Discrete 0.040s (25.000 fps)
                Interval: Discrete 0.050s (20.000 fps)
                Interval: Discrete 0.067s (15.000 fps)
                Interval: Discrete 0.100s (10.000 fps)
                Interval: Discrete 0.200s (5.000 fps)
        Size: Discrete 1600x1200
                Interval: Discrete 0.033s (30.000 fps)
                Interval: Discrete 0.040s (25.000 fps)
                Interval: Discrete 0.050s (20.000 fps)
                Interval: Discrete 0.067s (15.000 fps)
                Interval: Discrete 0.100s (10.000 fps)
                Interval: Discrete 0.200s (5.000 fps)
[1]: 'MJPG' (Motion-JPEG, compressed)
        Size: Discrete 1280x720
                Interval: Discrete 0.033s (30.000 fps)
                Interval: Discrete 0.040s (25.000 fps)
                Interval: Discrete 0.050s (20.000 fps)
                Interval: Discrete 0.067s (15.000 fps)
                Interval: Discrete 0.100s (10.000 fps)
                Interval: Discrete 0.200s (5.000 fps)
        Size: Discrete 640x480
                Interval: Discrete 0.033s (30.000 fps)
                Interval: Discrete 0.040s (25.000 fps)
                Interval: Discrete 0.050s (20.000 fps)
                Interval: Discrete 0.067s (15.000 fps)
                Interval: Discrete 0.100s (10.000 fps)
                Interval: Discrete 0.200s (5.000 fps)
        Size: Discrete 320x240
                Interval: Discrete 0.033s (30.000 fps)
                Interval: Discrete 0.040s (25.000 fps)
                Interval: Discrete 0.050s (20.000 fps)
                Interval: Discrete 0.067s (15.000 fps)
                Interval: Discrete 0.100s (10.000 fps)
                Interval: Discrete 0.200s (5.000 fps)
        Size: Discrete 352x288
                Interval: Discrete 0.033s (30.000 fps)
                Interval: Discrete 0.040s (25.000 fps)
                Interval: Discrete 0.050s (20.000 fps)
                Interval: Discrete 0.067s (15.000 fps)
                Interval: Discrete 0.100s (10.000 fps)
                Interval: Discrete 0.200s (5.000 fps)
        Size: Discrete 800x600
                Interval: Discrete 0.033s (30.000 fps)
                Interval: Discrete 0.040s (25.000 fps)
                Interval: Discrete 0.050s (20.000 fps)
                Interval: Discrete 0.067s (15.000 fps)
                Interval: Discrete 0.100s (10.000 fps)
                Interval: Discrete 0.200s (5.000 fps)
        Size: Discrete 1280x1024
                Interval: Discrete 0.033s (30.000 fps)
                Interval: Discrete 0.040s (25.000 fps)
                Interval: Discrete 0.050s (20.000 fps)
                Interval: Discrete 0.067s (15.000 fps)
                Interval: Discrete 0.100s (10.000 fps)
                Interval: Discrete 0.200s (5.000 fps)

Camera B :
See3CAM_24CUG

v4l2-ctl --list-formats-ext
[0]: 'UYVY' (UYVY 4:2:2)
        Size: Discrete 1280x720
                Interval: Discrete 0.067s (15.000 fps)
        Size: Discrete 1920x1080
                Interval: Discrete 0.125s (8.000 fps)
        Size: Discrete 1920x1200
                Interval: Discrete 0.167s (6.000 fps)
[1]: 'MJPG' (Motion-JPEG, compressed)
        Size: Discrete 1280x720
                Interval: Discrete 0.008s (120.000 fps)
                Interval: Discrete 0.017s (60.000 fps)
        Size: Discrete 1920x1080
                Interval: Discrete 0.008s (120.000 fps)
                Interval: Discrete 0.017s (60.000 fps)
                Interval: Discrete 0.033s (30.000 fps)
        Size: Discrete 1920x1200
                Interval: Discrete 0.009s (114.000 fps)
                Interval: Discrete 0.017s (60.000 fps)

Camera A using YUYV:

Is cheaper and seems to be promising unrealistic YUYV capabilites.

gst-launch-1.0 -e v4l2src device=/dev/video0 ! video/x-raw,width=1280,height=1024,framerate=30/1 ! queue ! fpsdisplaysink video-sink=fakesink --verbose

gst-launch-1.0 -e nvv4l2camerasrc device=/dev/video0 ! "video/x-raw(memory:NVMM)",width=1280,height=1024,framerate=30/1 ! queue ! fpsdispla ysink video-sink=fakesink --verbose

Both these pipes yield an fps of only 7.5 out of 30.
The usbtop tool shows that 20Mb/s of data is passing through the USB2.0.
I think it is safe to assume that 30 fps would not be attainable here anyway and I’m fine with that.

Camera B using UYVY:

This camera is more expensive and is promising plausible capabilities.

gst-launch-1.0 -e v4l2src device=/dev/video0 ! video/x-raw,width=1920,height=1080,framerate=8/1 ! queue ! fpsdisplaysink video-sink=fakesink --verbose

gst-launch-1.0 -e nvv4l2camerasrc device=/dev/video0 ! "video/x-raw(memory:NVMM)",width=1920,height=1080,framerate=8/1 ! queue ! fpsdisplaysink video-sink=fakesink --verbose

Both these pipelines show the expected fps of 8. I’m happy about that.
usbtop shows that 40Mb/s of data is passing through the USB2.0.
This is probably approaching the max of what one can consistently expect from a video stream through USB2.

Camera A using MJPG:

gst-launch-1.0 -e v4l2src device=/dev/video0 io-mode=2 ! image/jpeg,width=1280,height=1024,framerate=30/1 ! queue ! fpsdisplaysink video-sink=fakesink --verbose

This pipe yields an fps of 15 out of the promised 30.
usbtop shows only 1.5Mb/s of data flowing, so there should be plenty to spare to get the full fps.

Camera B using MJPG:

gst-launch-1.0 -e v4l2src device=/dev/video0 io-mode=2 ! image/jpeg,width=1920,height=1080,framerate=120/1 ! queue ! fpsdisplaysink video-sink=fakesink --verbose

This pipe yields an fps of 94 out of the promised 120.
usbtop shows 18Mb/s of data flow, so there should be some to spare for the remaining fps.

Does anybody have any thoughts on the low fps when using MJPG?

aron.h · July 16, 2024, 6:54pm

Update:

I tested again during evening time…

Camera A using YUYV:
unchanged

Camera B using YUYV:
unchanged

Camera A using MJPG:
unchanged

Camera B using MJPG:
Now this pipe yields an fps of 15 out of the promised 120.
usbtop shows 3Mb/s of data flow.
This shows that the fps and data flow is significantly reduced.
Is this related to the exposure time of the camera?

JerryChang · July 17, 2024, 2:55am

hello aron.h,

could you please try moving to the latest Jetpack release version for confirmation.

anyways,
let’s test with v4l2 IOCTL to check the sensor basic functionality.
for instance,
$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1200,pixelformat=UYVY --set-ctrl bypass_mode=0 --stream-mmap --stream-count=300

aron.h · July 17, 2024, 10:47am

Hi Jerry,

Thank you for your reply.
The above test results have looked the same on both host and docker so far.
Are you saying that the host Jetson needs upgrading the Jetpack version, or the docker I’m working in?

At the moment, camera B and it’s Jetson are unavailable to me. I expect to re-gain access in a couple of days.

Camera A YUYV:

v4l2-ctl -d /dev/video0 --set-fmt-video=width=1280,height=1024,pixelformat=YUYV --set-ctrl bypass_mode=0 --stream-mmap --stream-count=300
unknown control 'bypass_mode'

bypass mode doesn’t seem to be on the list of options here.

sudo v4l2-ctl --list-ctrls
                     brightness 0x00980900 (int)    : min=-255 max=255 step=1 default=0 value=0
                       contrast 0x00980901 (int)    : min=0 max=30 step=1 default=16 value=16
                     saturation 0x00980902 (int)    : min=0 max=127 step=1 default=32 value=32
                            hue 0x00980903 (int)    : min=-16000 max=16000 step=1 default=0 value=0
 white_balance_temperature_auto 0x0098090c (bool)   : default=1 value=1
                          gamma 0x00980910 (int)    : min=20 max=250 step=10 default=110 value=110
           power_line_frequency 0x00980918 (menu)   : min=0 max=2 default=1 value=1
      white_balance_temperature 0x0098091a (int)    : min=2500 max=7000 step=1 default=5000 value=5000 flags=inactive
                      sharpness 0x0098091b (int)    : min=0 max=4 step=1 default=3 value=3
         backlight_compensation 0x0098091c (int)    : min=0 max=2 step=1 default=1 value=1

Omitting the bypass argument, I get the same 7.5 fps as before.

v4l2-ctl -d /dev/video0 --set-fmt-video=width=1280,height=1024,pixelformat=YUYV --stream-mmap --stream-count=300

<<<<<<<<< 7.52 fps
<<<<<<<< 7.52 fps
<<<<<<< 7.52 fps
<<<<<<<< 7.52 fps
<<<<<<< 7.52 fps
<<<<<<<< 7.51 fps

JerryChang · July 18, 2024, 2:34am

hello aron.h,

V4L2 IOCTL is standard utility to verify basic camera functionality, it means your sensor only outputting frames at 7.5-fps.
it’s the issue on sensor side, please review the sensor configuration.

aron.h · July 18, 2024, 3:31pm

There are some inconsistencies in how much Camera A is advertised to handle.

                    |               fps               |
-------------------------------------------------------
Format | Resolution | list-formats | website | actual |
-------------------------------------------------------
 YUY2  |   320x240  |      15      |   30    |   15   |
 YUV2  |   352x288  |      15      |  N/A    |   25   |
 YUY2  |   640x480  |      30      |   30    |   15   |
 YUY2  |   800x600  |      30      |   15    |  7.5   |
 YUY2  |  1280x720  |      30      |    8    |  7.5   |
 YUY2  | 1280x1024  |      30      |    8    |  7.5   |
 YUY2  | 1600x1200  |      30      |    4    |  3.8   |

The list-formats column shows:
Some of the capabilities listed by v4l2-ctl --list-formats-ext look like they are beyond what USB2.0 can handle.

The website column shows:
The capabilities listed on the website are much more plausible.
It is interesting that the website doesn’t even mention the 352x288 resolution.

The actual column shows:
v4l2-ctl -d /dev/video0 --set-fmt-video=width=1280,height=1024,pixelformat=YUYV --stream-mmap --stream-count=300
What you are saying is that this is the true capability of the device and that the other columns are not to be taken seriously?

JerryChang · July 19, 2024, 2:36am

hello aron.h,

v4l2-ctl --list-formats-ext, only check the sensor’s kernel driver to report its capability.
it’s V4L2 IOCTL (a stream command-line) to enable the camera, processing the buffer, and running the camera stream.

aron.h · July 23, 2024, 2:35pm

Now I have access to the Jetson connected to Camera B again.

gst-launch-1.0 -e v4l2src device=/dev/video0 io-mode=2 ! image/jpeg,width=1920,height=1080,framerate=120/1 ! queue ! fpsdisplaysink video-sink=fakesink --verbose

v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=MJPG --stream-mmap --stream-count=300

Both of these commands now show the full expected 120fps.
USB 2.0 data flow is around 20Mbps.

I’m not sure why I was getting lower fps previoulsy on Camera B.
The advertised capabilites of Camera A were clearly overstated.

system · August 13, 2024, 7:50am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Jetpack 4.2.2 USB webcam very low framerate Jetson TX2 camera	6	871	October 18, 2021
V4l2 -capture 4k raw data Jetson AGX Orin camera , mmapi	8	1339	October 12, 2023
Issue in connecting camera Jetson AGX Xavier	15	2969	October 18, 2021
Capture Video from a AV.io DVI-USB3 Device not working on TX2 Jetson TX2	14	833	October 18, 2021
Jetson TX2 + Logitech Brio video capture size + detectNet Jetson TX2	12	1486	October 18, 2021
"v4l2-ctl -d /dev/video0 --stream-mmap --stream-to-host 192.168.1.111" cannot produce a picture Jetson Orin NX camera	13	758	April 10, 2024
Deepstream processing dat from Logitech 930e with only 5FPS DeepStream SDK nvbugs	32	2531	October 12, 2021
Jetson is struggling with high resolution camera Jetson AGX Orin camera , opencv	11	1592	September 14, 2022
high cpu0 usage with high resolution usb 3 camera Jetson TX2	9	1141	October 18, 2021
How many webcam can Nvidia nano accept? Jetson Nano usb	9	1114	July 27, 2022

V4L2 MJPG low framerate

Environment

Related topics