Framerate issues with devkit camera and v4l2-ctl

Autonomous Machines Jetson & Embedded Systems Jetson TX2
1

Hi,

I’m trying to capture with the devkit camera with 1280x720@120fps mode using v4l2-ctl, however, the framerate reaches only 60fps. Checking the notify traces, I see that between consecutive frames information, there is one frame id missing consistently over all the log as shown below, where after frame 673 information comes frame 675 and then frame 677 and so on:

kworker/5:1-1313  [005] ....   164.410037: rtcpu_vinotify_event: tstamp:5298204736 tag:CHANSEL_PXL_SOF channel:0x00 frame:673 vi_tstamp:5298203929 data:0x00000001
     kworker/5:1-1313  [005] ....   164.410041: rtcpu_vinotify_event: tstamp:5298205381 tag:ATOMP_FS channel:0x00 frame:673 vi_tstamp:5298203936 data:0x00000000
     kworker/5:1-1313  [005] ....   164.410044: rtcpu_vinotify_event: tstamp:5298450855 tag:CHANSEL_PXL_EOF channel:0x00 frame:673 vi_tstamp:5298450201 data:0x02cf0002
     kworker/5:1-1313  [005] ....   164.410047: rtcpu_vinotify_event: tstamp:5298450968 tag:ATOMP_FE channel:0x00 frame:673 vi_tstamp:5298450242 data:0x00000000
     kworker/5:1-1313  [005] ....   164.410051: rtcpu_vinotify_event: tstamp:5298725481 tag:CHANSEL_PXL_SOF channel:0x00 frame:675 vi_tstamp:5298724740 data:0x00000001
     kworker/5:1-1313  [005] ....   164.410054: rtcpu_vinotify_event: tstamp:5298726157 tag:ATOMP_FS channel:0x00 frame:675 vi_tstamp:5298724746 data:0x00000000
     kworker/5:1-1313  [005] ....   164.466031: rtcpu_vinotify_event: tstamp:5298971906 tag:CHANSEL_PXL_EOF channel:0x00 frame:675 vi_tstamp:5298971011 data:0x02cf0002
     kworker/5:1-1313  [005] ....   164.466036: rtcpu_vinotify_event: tstamp:5298972471 tag:ATOMP_FE channel:0x00 frame:675 vi_tstamp:5298971052 data:0x00000000
     kworker/5:1-1313  [005] ....   164.466038: rtcpu_vinotify_event: tstamp:5299246049 tag:CHANSEL_PXL_SOF channel:0x00 frame:677 vi_tstamp:5299245550 data:0x00000001
     kworker/5:1-1313  [005] ....   164.466040: rtcpu_vinotify_event: tstamp:5299246206 tag:ATOMP_FS channel:0x00 frame:677 vi_tstamp:5299245556 data:0x00000000
     kworker/5:1-1313  [005] ....   164.466042: rtcpu_vinotify_event: tstamp:5299492513 tag:CHANSEL_PXL_EOF channel:0x00 frame:677 vi_tstamp:5299491821 data:0x02cf0002
     kworker/5:1-1313  [005] ....   164.466044: rtcpu_vinotify_event: tstamp:5299492623 tag:ATOMP_FE channel:0x00 frame:677 vi_tstamp:5299491862 data:0x00000000
     kworker/5:1-1313  [005] ....   164.466046: rtcpu_vinotify_event: tstamp:5299767095 tag:CHANSEL_PXL_SOF channel:0x00 frame:679 vi_tstamp:5299766360 data:0x00000001
     kworker/5:1-1313  [005] ....   164.466048: rtcpu_vinotify_event: tstamp:5299767779 tag:ATOMP_FS channel:0x00 frame:679 vi_tstamp:5299766366 data:0x00000000
     kworker/5:1-1313  [005] ....   164.466050: rtcpu_vinotify_event: tstamp:5300013282 tag:CHANSEL_PXL_EOF channel:0x00 frame:679 vi_tstamp:5300012631 data:0x02cf0002
     kworker/5:1-1313  [005] ....   164.466052: rtcpu_vinotify_event: tstamp:5300013397 tag:ATOMP_FE channel:0x00 frame:679 vi_tstamp:5300012672 data:0x00000000

On the other hand, with an argus app, the traces show consecutive frames data with one missing more sporadically.

kworker/4:2-3146  [004] ....   319.596737: rtcpu_vinotify_event: tstamp:10147049745 tag:ISPBUF_FS channel:0x00 frame:206 vi_tstamp:10147049360 data:0x00000000
     kworker/4:2-3146  [004] ....   319.596739: rtcpu_vinotify_event: tstamp:10147054418 tag:CHANSEL_PXL_SOF channel:0x00 frame:206 vi_tstamp:10147054051 data:0x00000001
     kworker/4:2-3146  [004] ....   319.596740: rtcpu_vinotify_event: tstamp:10147300984 tag:CHANSEL_PXL_EOF channel:0x00 frame:206 vi_tstamp:10147300321 data:0x02cf0002
     kworker/4:2-3146  [004] ....   319.596740: rtcpu_vinotify_event: tstamp:10147301103 tag:ISPBUF_FE channel:0x00 frame:206 vi_tstamp:10147300360 data:0x00000000
     kworker/4:2-3146  [004] ....   319.596741: rtcpu_vinotify_event: tstamp:10147310127 tag:ISPBUF_FS channel:0x00 frame:207 vi_tstamp:10147309765 data:0x00000000
     kworker/4:2-3146  [004] ....   319.596741: rtcpu_vinotify_event: tstamp:10147314814 tag:CHANSEL_PXL_SOF channel:0x00 frame:207 vi_tstamp:10147314455 data:0x00000001
     kworker/4:2-3146  [004] ....   319.596742: rtcpu_vinotify_event: tstamp:10147561398 tag:CHANSEL_PXL_EOF channel:0x00 frame:207 vi_tstamp:10147560726 data:0x02cf0002
     kworker/4:2-3146  [004] ....   319.596742: rtcpu_vinotify_event: tstamp:10147561516 tag:ISPBUF_FE channel:0x00 frame:207 vi_tstamp:10147560767 data:0x00000000
     kworker/4:2-3146  [004] ....   319.596743: rtcpu_vinotify_event: tstamp:10147831029 tag:ISPBUF_FS channel:0x00 frame:209 vi_tstamp:10147830575 data:0x00000000
     kworker/4:2-3146  [004] ....   319.596743: rtcpu_vinotify_event: tstamp:10147835638 tag:CHANSEL_PXL_SOF channel:0x00 frame:209 vi_tstamp:10147835266 data:0x00000001
     kworker/4:2-3146  [004] ....   319.596744: rtcpu_vinotify_event: tstamp:10148082192 tag:CHANSEL_PXL_EOF channel:0x00 frame:209 vi_tstamp:10148081537 data:0x02cf0002
     kworker/4:2-3146  [004] ....   319.596744: rtcpu_vinotify_event: tstamp:10148082305 tag:ISPBUF_FE channel:0x00 frame:209 vi_tstamp:10148081576 data:0x00000000
     kworker/4:2-3146  [004] ....   319.596745: rtcpu_vinotify_event: tstamp:10148091341 tag:ISPBUF_FS channel:0x00 frame:210 vi_tstamp:10148090980 data:0x00000000
     kworker/4:2-3146  [004] ....   319.654797: rtcpu_vinotify_event: tstamp:10148096039 tag:CHANSEL_PXL_SOF channel:0x00 frame:210 vi_tstamp:10148095671 data:0x00000001
     kworker/4:2-3146  [004] ....   319.654801: rtcpu_vinotify_event: tstamp:10148342602 tag:CHANSEL_PXL_EOF channel:0x00 frame:210 vi_tstamp:10148341942 data:0x02cf0002
     kworker/4:2-3146  [004] ....   319.654802: rtcpu_vinotify_event: tstamp:10148342720 tag:ISPBUF_FE channel:0x00 frame:210 vi_tstamp:10148341982 data:0x00000000
     kworker/4:2-3146  [004] ....   319.654803: rtcpu_vinotify_event: tstamp:10148351747 tag:ISPBUF_FS channel:0x00 frame:211 vi_tstamp:10148351385 data:0x00000000
     kworker/4:2-3146  [004] ....   319.654803: rtcpu_vinotify_event: tstamp:10148356435 tag:CHANSEL_PXL_SOF channel:0x00 frame:211 vi_tstamp:10148356076 data:0x00000001
     kworker/4:2-3146  [004] ....   319.654804: rtcpu_vinotify_event: tstamp:10148603022 tag:CHANSEL_PXL_EOF channel:0x00 frame:211 vi_tstamp:10148602346 data:0x02cf0002
     kworker/4:2-3146  [004] ....   319.654805: rtcpu_vinotify_event: tstamp:10148603141 tag:ISPBUF_FE channel:0x00 frame:211 vi_tstamp:10148602387 data:0x00000000
     kworker/4:2-3146  [004] ....   319.654806: rtcpu_vinotify_event: tstamp:10148612186 tag:ISPBUF_FS channel:0x00 frame:212 vi_tstamp:10148611790 data:0x00000000
     kworker/4:2-3146  [004] ....   319.654806: rtcpu_vinotify_event: tstamp:10148616839 tag:CHANSEL_PXL_SOF channel:0x00 frame:212 vi_tstamp:10148616481 data:0x00000001
     kworker/4:2-3146  [004] ....   319.654807: rtcpu_vinotify_event: tstamp:10148863426 tag:CHANSEL_PXL_EOF channel:0x00 frame:212 vi_tstamp:10148862753 data:0x02cf0002
     kworker/4:2-3146  [004] ....   319.654807: rtcpu_vinotify_event: tstamp:10148863544 tag:ISPBUF_FE channel:0x00 frame:212 vi_tstamp:10148862792 data:0x00000000
     kworker/4:2-3146  [004] ....   319.654808: rtcpu_vinotify_event: tstamp:10149132970 tag:ISPBUF_FS channel:0x00 frame:214 vi_tstamp:10149132600 data:0x00000000

I’ve tried increasing vi clock as in https://devtalk.nvidia.com/default/topic/1031413/raw-capture-over-csi-works-half-speed-only/ and setting nvp model to 0, but still, I get only 60fps.

I’m using the command below to launch the capture. Also, I verified that the driver is actually in the 120fps mode and that is setting the frame length value correctly to get 120fps.

Do you have any ideas of what might be happening on the v4l2 path?

v4l2-ctl  -d /dev/video0 --set-fmt-video=width=1280,height=720,pixelformat=BG10 --set-ctrl=frame_rate=120000000 --stream-mmap  --stream-count=1000 --set-ctrl=bypass_mode=0

Thanks

2

Does boost the VI/CSI clock help for the v4l2-ctl?
The argus may limit by the display monitor.

3

Setting vi/rate to vi/max_rate did not increase the v4l2-ctl framerate, I get 60fps consistently instead of the expected 120fps.

Thanks

4

Hi ShaneCCC,

Argus app is showing 120fps despite displaying to monitor (probably the sporadic framerate drops are due to the displaying as you mentioned), however, with v4l2-ctl command there is no displaying and yet, I get 60fps only, despite setting vi rate to max.

Is there any known limitation in v4l2 path causing this?

Thanks

5

@jcaballero
For your case I believe the sensor not output as 120FPS as expect.

6

Hi JCaballero,
The fps print from argus_camera application corresponds to the display frame rate. If you want to validate the frame rate coming from the ISP, I would suggest you to check using gstreamer plugin fpsdisplaysink over nvarguscamerasrc.

gst-launch-1.0 nvarguscamerasrc ! "video/x-raw(memory:NVMM), format=NV12, width=1280, height=720, framerate=120/1" ! nvvidconv ! fpsdisplaysink video-sink=fakesink sync=false -v

This will show the current and average fps output from the camera setup.

/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstTextOverlay:fps-display-text-overlay: text = rendered: 243, dropped: 0, <b>current: 120.08</b>, <b>average: 120.51</b>
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 243, dropped: 0, <b>current: 120.08</b>, <b>average: 120.51</b>
7

Hi,

Thank you, I do get 120fps with your pipeline. Currently, I’m trying to bypass ISP by capturing with v4l2-ctl.

Given that the capture with argus-app/GStreamer-nvarguscamerasrc (ISP path) delivers 120fps, my questions is, Why v4l2-ctl capture with the same setup reaches 60fps max?

Also, Why in the VI NOTIFY events for the v4l4-ctl capture there is a frame id being skipped? Is the VI dropping frames?

Thanks