Jetson AGX Xavier MIPI CSI-2 Half FPS Problem

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Xavier
1

Hello there. I’ve been modifying L4T 35.3.1 kernel for accept MIPI CSI-2 data without using I2C. With the help of these threads, I’m successfully getting video data;
https://forums.developer.nvidia.com/t/jetson-agx-xavier-4-csi-raw-input-from-fpga/275628
https://forums.developer.nvidia.com/t/jetson-agx-xavier-mipi-csi-2-without-i2c-from-fpga/165969

But the output is half the FPS that I’m sending. For example, I’m sending 1920x1080@60p video data to the Jetson, but I’m seeing 30 fps video on the Jetson. I’ve tried boosting the clocks. It didn’t worked. How should I debug this problem? Here is the trace outputs:

# tracer: nop
#
# entries-in-buffer/entries-written: 1041/1041   #P:4
#
#                                _-----=> irqs-off
#                               / _----=> need-resched
#                              | / _---=> hardirq/softirq
#                              || / _--=> preempt-depth
#                              ||| /     delay
#           TASK-PID     CPU#  ||||   TIMESTAMP  FUNCTION
#              | |         |   ||||      |         |
     kworker/0:0-2213    [000] ....   486.415111: rtcpu_string: tstamp:15853938338 id:0x04010000 str:"VM0 deactivating."
  gst-launch-1.0-2239    [001] ....   490.578318: tegra_channel_open: vi-output, ov5693 2-0036
  gst-launch-1.0-2239    [001] ....   490.578524: tegra_channel_close: vi-output, ov5693 2-0036
  gst-launch-1.0-2239    [001] ....   490.578542: tegra_channel_open: vi-output, ov5693 2-0037
  gst-launch-1.0-2239    [001] ....   490.578743: tegra_channel_close: vi-output, ov5693 2-0037
  gst-launch-1.0-2239    [001] ....   490.578760: tegra_channel_open: vi-output, ov5693 2-0038
  gst-launch-1.0-2239    [001] ....   490.579029: tegra_channel_close: vi-output, ov5693 2-0038
  gst-launch-1.0-2239    [001] ....   490.579047: tegra_channel_open: vi-output, ov5693 2-0039
  gst-launch-1.0-2239    [001] ....   490.579088: tegra_channel_close: vi-output, ov5693 2-0039
  gst-launch-1.0-2239    [001] ....   490.810808: tegra_channel_open: vi-output, ov5693 2-0036
    v4l2src0:src-2246    [002] ....   490.830303: tegra_channel_set_power: ov5693 2-0036 : 0x1
    v4l2src0:src-2246    [002] ....   490.830315: camera_common_s_power: status : 0x1
    v4l2src0:src-2246    [002] ....   490.834772: tegra_channel_set_power: 13e10000.host1x:nvcsi@15a00000- : 0x1
    v4l2src0:src-2246    [002] ....   490.834775: csi_s_power: enable : 0x1
    v4l2src0:src-2246    [002] ....   490.835275: tegra_channel_capture_setup: vnc_id 0 W 1920 H 1080 fmt 9
    v4l2src0:src-2246    [002] ....   490.842476: tegra_channel_set_stream: enable : 0x1
    v4l2src0:src-2246    [002] ....   490.843650: tegra_channel_set_stream: 13e10000.host1x:nvcsi@15a00000- : 0x1
    v4l2src0:src-2246    [002] ....   490.843652: csi_s_stream: enable : 0x1
    v4l2src0:src-2246    [002] ....   490.844035: tegra_channel_set_stream: ov5693 2-0036 : 0x1
     kworker/0:0-2213    [000] ....   490.846986: rtcpu_string: tstamp:15993893739 id:0x04010000 str:"VM0 activating."
     kworker/0:0-2213    [000] ....   490.846990: rtcpu_vinotify_error: tstamp:15994078863 cch:-1 vi:0 tag:CSIMUX_STREAM channel:0x00 frame:0 vi_tstamp:511810500224 data:0x0000000000000001
 vi-output, ov56-2249    [003] ....   490.881720: tegra_channel_capture_frame: sof:511.816421568
 vi-output, ov56-2249    [003] ....   490.881722: tegra_channel_capture_frame: eof:511.848074784
     kworker/0:0-2213    [000] ....   490.906994: rtcpu_vinotify_event: tstamp:15994330445 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:511808973184 data:0xcd9ce50010000000
     kworker/0:0-2213    [000] ....   490.906997: rtcpu_vinotify_event: tstamp:15994330581 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:511808978688 data:0x0000000031000001
     kworker/0:0-2213    [000] ....   490.906998: rtcpu_vinotify_event: tstamp:15994330734 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:511809033344 data:0xcd9ce20010000000
     kworker/0:0-2213    [000] ....   490.906999: rtcpu_vinotify_event: tstamp:15994330868 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:511809038912 data:0x0000000031000002
     kworker/0:0-2213    [000] ....   490.907030: rtcpu_vinotify_event: tstamp:15994331025 cch:-1 vi:0 tag:CSIMUX_STREAM channel:0x00 frame:0 vi_tstamp:511810500224 data:0x0000000000000001
     kworker/0:0-2213    [000] ....   490.907030: rtcpu_vinotify_event: tstamp:15994331158 cch:0 vi:0 tag:FS channel:0x00 frame:0 vi_tstamp:511816420992 data:0x0000000000000010
     kworker/0:0-2213    [000] ....   490.907034: rtcpu_vinotify_event: tstamp:15994331313 cch:0 vi:0 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:511816420992 data:0x0000000800000000
     kworker/0:0-2213    [000] ....   490.907035: rtcpu_vinotify_event: tstamp:15994331446 cch:0 vi:0 tag:CHANSEL_PXL_SOF channel:0x23 frame:0 vi_tstamp:511816421568 data:0x0000000000000001
     kworker/0:0-2213    [000] ....   490.907036: rtcpu_vinotify_event: tstamp:15994331595 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:0 vi_tstamp:511816425888 data:0x0000000008020001
     kworker/0:0-2213    [000] ....   490.907038: rtcpu_vinotify_event: tstamp:15995255851 cch:0 vi:0 tag:CHANSEL_PXL_EOF channel:0x23 frame:0 vi_tstamp:511848074272 data:0x0000000004370002
     kworker/0:0-2213    [000] ....   490.907039: rtcpu_vinotify_event: tstamp:15995256008 cch:0 vi:0 tag:ATOMP_FRAME_DONE channel:0x23 frame:0 vi_tstamp:511848074720 data:0x0000000000000000
     kworker/0:0-2213    [000] ....   490.907040: rtcpu_vinotify_event: tstamp:15995256141 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:0 vi_tstamp:511848077664 data:0x0000000002020001
     kworker/0:0-2213    [000] ....   490.907040: rtcpu_vinotify_event: tstamp:15995256291 cch:0 vi:0 tag:FE channel:0x00 frame:0 vi_tstamp:511848074784 data:0x0000000000000020
     kworker/0:0-2213    [000] ....   490.907041: rtcpu_vinotify_event: tstamp:15995256429 cch:0 vi:0 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:511848074784 data:0x0000000800000000
     kworker/0:0-2213    [000] ....   490.907042: rtcpu_vinotify_event: tstamp:15995256581 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:0 vi_tstamp:511848082912 data:0x0000000000020001
     kworker/0:0-2213    [000] ....   490.907042: rtcpu_vinotify_event: tstamp:15995256712 cch:0 vi:0 tag:FS channel:0x00 frame:0 vi_tstamp:511848075296 data:0x0000000000000010
     kworker/0:0-2213    [000] ....   490.907043: rtcpu_vinotify_event: tstamp:15995551153 cch:0 vi:0 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:511848075296 data:0x0000000800000000
     kworker/0:0-2213    [000] ....   490.907062: rtcpu_vinotify_event: tstamp:15995551287 cch:0 vi:0 tag:CHANSEL_PXL_SOF channel:0x23 frame:0 vi_tstamp:511848075872 data:0x0000000000000001
     kworker/0:0-2213    [000] ....   490.907063: rtcpu_vinotify_event: tstamp:15995551438 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:0 vi_tstamp:511848094048 data:0x0000000008020002
     kworker/0:0-2213    [000] ....   490.907063: rtcpu_vinotify_event: tstamp:15995551572 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:511848186656 data:0xcd9ce50010000000
     kworker/0:0-2213    [000] ....   490.907064: rtcpu_vinotify_event: tstamp:15995551721 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:511848192192 data:0x0000000031000003
 vi-output, ov56-2249    [002] ....   490.913352: tegra_channel_capture_frame: sof:511.848075872
 vi-output, ov56-2249    [002] ....   490.913355: tegra_channel_capture_frame: eof:511.879729088
 vi-output, ov56-2249    [002] ....   490.944997: tegra_channel_capture_frame: sof:511.879730144
 vi-output, ov56-2249    [002] ....   490.945019: tegra_channel_capture_frame: eof:511.911383424
     kworker/0:0-2213    [000] ....   490.963042: rtcpu_vinotify_event: tstamp:15996490143 cch:0 vi:0 tag:CHANSEL_PXL_EOF channel:0x23 frame:0 vi_tstamp:511879728576 data:0x0000000004370002
     kworker/0:0-2213    [000] ....   490.963045: rtcpu_vinotify_event: tstamp:15996490299 cch:0 vi:0 tag:ATOMP_FRAME_DONE channel:0x23 frame:0 vi_tstamp:511879729024 data:0x0000000000000000
     kworker/0:0-2213    [000] ....   490.963046: rtcpu_vinotify_event: tstamp:15996490436 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:0 vi_tstamp:511879732000 data:0x0000000002020002
     kworker/0:0-2213    [000] ....   490.963046: rtcpu_vinotify_event: tstamp:15996490585 cch:0 vi:0 tag:FE channel:0x00 frame:0 vi_tstamp:511879729088 data:0x0000000000000020
     kworker/0:0-2213    [000] ....   490.963047: rtcpu_vinotify_event: tstamp:15996490721 cch:0 vi:0 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:511879729088 data:0x0000000800000000
     kworker/0:0-2213    [000] ....   490.963047: rtcpu_vinotify_event: tstamp:15996490872 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:0 vi_tstamp:511879737216 data:0x0000000000020002
     kworker/0:0-2213    [000] ....   490.963048: rtcpu_vinotify_event: tstamp:15996491004 cch:0 vi:0 tag:FS channel:0x00 frame:0 vi_tstamp:511879729600 data:0x0000000000000010
     kworker/0:0-2213    [000] ....   490.963049: rtcpu_vinotify_event: tstamp:15996491158 cch:0 vi:0 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:511879729600 data:0x0000000800000000
     kworker/0:0-2213    [000] ....   490.963049: rtcpu_vinotify_event: tstamp:15996491292 cch:0 vi:0 tag:CHANSEL_PXL_SOF channel:0x23 frame:0 vi_tstamp:511879730144 data:0x0000000000000001
     kworker/0:0-2213    [000] ....   490.963050: rtcpu_vinotify_event: tstamp:15996491442 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:0 vi_tstamp:511879746368 data:0x0000000008020003
     kworker/0:0-2213    [000] ....   490.963050: rtcpu_vinotify_event: tstamp:15996634641 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:511892266848 data:0xcd9ce20010000000
     kworker/0:0-2213    [000] ....   490.963051: rtcpu_vinotify_event: tstamp:15996928363 cch:0 vi:0 tag:VIFALC_TDSTATE channel:0x23 frame:0 vi_tstamp:511892283168 data:0x0000000031000004
     kworker/0:0-2213    [000] ....   490.963052: rtcpu_vinotify_event: tstamp:15997241446 cch:0 vi:0 tag:CHANSEL_PXL_EOF channel:0x23 frame:0 vi_tstamp:511911382880 data:0x0000000004370002
     kworker/0:0-2213    [000] ....   490.963053: rtcpu_vinotify_event: tstamp:15997241603 cch:0 vi:0 tag:ATOMP_FRAME_DONE channel:0x23 frame:0 vi_tstamp:511911383296 data:0x0000000000000000
     kworker/0:0-2213    [000] ....   490.963054: rtcpu_vinotify_event: tstamp:15997241739 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:0 vi_tstamp:511911386272 data:0x0000000002020003
     kworker/0:0-2213    [000] ....   490.963054: rtcpu_vinotify_event: tstamp:15997241888 cch:0 vi:0 tag:FE channel:0x00 frame:0 vi_tstamp:511911383392 data:0x0000000000000020
     kworker/0:0-2213    [000] ....   490.963055: rtcpu_vinotify_event: tstamp:15997242027 cch:0 vi:0 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:511911383424 data:0x0000000800000000
     kworker/0:0-2213    [000] ....   490.963056: rtcpu_vinotify_event: tstamp:15997242177 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:0 vi_tstamp:511911391520 data:0x0000000000020003
     kworker/0:0-2213    [000] ....   490.963056: rtcpu_vinotify_event: tstamp:15997242308 cch:0 vi:0 tag:FS channel:0x00 frame:0 vi_tstamp:511911383872 data:0x0000000000000010
     kworker/0:0-2213    [000] ....   490.963057: rtcpu_vinotify_event: tstamp:15997242463 cch:0 vi:0 tag:ATOMP_FS channel:0x00 frame:0 vi_tstamp:511911383904 data:0x0000000800000000
     kworker/0:0-2213    [000] ....   490.963057: rtcpu_vinotify_event: tstamp:15997242596 cch:0 vi:0 tag:CHANSEL_PXL_SOF channel:0x23 frame:0 vi_tstamp:511911384480 data:0x0000000000000001
     kworker/0:0-2213    [000] ....   490.963058: rtcpu_vinotify_event: tstamp:15997242745 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:0 vi_tstamp:511911400672 data:0x0000000008020004
 vi-output, ov56-2249    [002] ....   490.976718: tegra_channel_capture_frame: sof:511.911384480
 vi-output, ov56-2249    [002] ....   490.976721: tegra_channel_capture_frame: eof:511.943037696
     kworker/0:0-2213    [000] ....   491.019100: rtcpu_vinotify_error: tstamp:15998222168 cch:0 vi:0 tag:CHANSEL_NOMATCH channel:0x01 frame:0 vi_tstamp:511943038784 data:0x0000000000000489
     kworker/0:0-2213    [000] ....   491.019104: rtcpu_vinotify_event: tstamp:15998223197 cch:0 vi:0 tag:CHANSEL_PXL_EOF channel:0x23 frame:0 vi_tstamp:511943037184 data:0x0000000004370002
     kworker/0:0-2213    [000] ....   491.019104: rtcpu_vinotify_event: tstamp:15998223351 cch:0 vi:0 tag:ATOMP_FRAME_DONE channel:0x23 frame:0 vi_tstamp:511943037632 data:0x0000000000000000
     kworker/0:0-2213    [000] ....   491.019105: rtcpu_vinotify_event: tstamp:15998223484 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:0 vi_tstamp:511943040416 data:0x0000000002020004
     kworker/0:0-2213    [000] ....   491.019105: rtcpu_vinotify_event: tstamp:15998223635 cch:0 vi:0 tag:FE channel:0x00 frame:0 vi_tstamp:511943037696 data:0x0000000000000020
     kworker/0:0-2213    [000] ....   491.019106: rtcpu_vinotify_event: tstamp:15998223772 cch:0 vi:0 tag:ATOMP_FE channel:0x00 frame:0 vi_tstamp:511943037696 data:0x0000000800000000
     kworker/0:0-2213    [000] ....   491.019107: rtcpu_vinotify_event: tstamp:15998524685 cch:0 vi:0 tag:VIFALC_ACTIONLST channel:0x23 frame:0 vi_tstamp:511943046688 data:0x0000000000020004
     kworker/0:0-2213    [000] ....   491.019107: rtcpu_vinotify_event: tstamp:15998524829 cch:0 vi:0 tag:FS channel:0x00 frame:0 vi_tstamp:511943038208 data:0x0000000000000010
     kworker/0:0-2213    [000] ....   491.019108: rtcpu_vinotify_event: tstamp:15998524983 cch:0 vi:0 tag:CHANSEL_NOMATCH channel:0x01 frame:0 vi_tstamp:511943038784 data:0x0000000000000489
     kworker/0:0-2213    [000] ....   491.019108: rtcpu_vinotify_error: tstamp:15999209948 cch:0 vi:0 tag:CHANSEL_NOMATCH channel:0x01 frame:0 vi_tstamp:511974693088 data:0x0000000000000489
     kworker/0:0-2213    [000] ....   491.019109: rtcpu_vinotify_event: tstamp:15999463715 cch:0 vi:0 tag:FE channel:0x00 frame:0 vi_tstamp:511974692000 data:0x0000000000000020
     kworker/0:0-2213    [000] ....   491.019109: rtcpu_vinotify_event: tstamp:15999463872 cch:0 vi:0 tag:FS channel:0x00 frame:0 vi_tstamp:511974692512 data:0x0000000000000010
     kworker/0:0-2213    [000] ....   491.019110: rtcpu_vinotify_event: tstamp:15999464008 cch:0 vi:0 tag:CHANSEL_NOMATCH channel:0x01 frame:0 vi_tstamp:511974693088 data:0x0000000000000489
     kworker/0:0-2213    [000] ....   491.079025: rtcpu_vinotify_error: tstamp:16000199147 cch:0 vi:0 tag:CHANSEL_NOMATCH channel:0x01 frame:0 vi_tstamp:512006347360 data:0x0000000000000489
     kworker/0:0-2213    [000] ....   491.079029: rtcpu_vinotify_event: tstamp:16000402766 cch:0 vi:0 tag:FE channel:0x00 frame:0 vi_tstamp:512006346304 data:0x0000000000000020
     kworker/0:0-2213    [000] ....   491.079030: rtcpu_vinotify_event: tstamp:16000402902 cch:0 vi:0 tag:FS channel:0x00 frame:0 vi_tstamp:512006346784 data:0x0000000000000010
     kworker/0:0-2213    [000] ....   491.079030: rtcpu_vinotify_event: tstamp:16000403054 cch:0 vi:0 tag:CHANSEL_NOMATCH channel:0x01 frame:0 vi_tstamp:512006347360 data:0x0000000000000489
     kworker/0:0-2213    [000] ....   491.079031: rtcpu_vinotify_error: tstamp:16001188343 cch:0 vi:0 tag:CHANSEL_NOMATCH channel:0x01 frame:0 vi_tstamp:512038001696 data:0x0000000000000489
     kworker/0:0-2213    [000] ....   491.079032: rtcpu_vinotify_event: tstamp:16001341687 cch:0 vi:0 tag:FE channel:0x00 frame:0 vi_tstamp:512038000608 data:0x0000000000000020
     kworker/0:0-2213    [000] ....   491.079032: rtcpu_vinotify_event: tstamp:16001341840 cch:0 vi:0 tag:FS channel:0x00 frame:0 vi_tstamp:512038001088 data:0x0000000000000010
     kworker/0:0-2213    [000] ....   491.079033: rtcpu_vinotify_event: tstamp:16001341974 cch:0 vi:0 tag:CHANSEL_NOMATCH channel:0x01 frame:0 vi_tstamp:512038001696 data:0x0000000000000489
 vi-output, ov56-2249    [001] ....   491.134912: tegra_channel_capture_frame: sof:512.69656000
 vi-output, ov56-2249    [001] ....   491.134915: tegra_channel_capture_frame: eof:512.101309216

Kind regards,
Ibrahim

3

hello varolaibrahim,

may I know which sensor driver you’re working with?
did you also evaluate with v4l2 IOCTL, for instance, $ v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=300

4

Hello there,

I’m using “non-i2c” modified version of OV5693 sensor. And I also checked with the command that you sent. Still I’m getting half of the FPS.

5

Hi again.

I restarted Jetson again. Now, I can confirm with using v4l2-ctl -d /dev/video0 --set-fmt-video=width=1280,height=720,pixelformat=AR24 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=300command, I can see the real output is 60fps.(Actually 67.06 but it’s not gonna be a problem)

By the way, I’m using RGB888 input on Jetson side.

6

so… is the issue resolved?

7

v4l2-ctl works but GStreamer not. I want to use GStreamer on my project. I’m using the command below:

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=BGRA,width=1280,height=720,framerate=60/1 ! fpsdisplaysink text-overlay=true video-sink=glimagesink
8

hello varolaibrahim,

may I know what’s the error reported, is it erroneous pipeline with not support formats?
you may also running… $ gst-inspect-1.0 v4l2src to check v4l2src support formats.

10

Hello there.

I’ve been doing research for a while. I think there’s an issue on the GStreamer side. I can achieve 60 fps capturing the image with the following command. However, when I run it via GStreamer, the fps drops by half, affecting V4L2 as well. Even when I run it through V4L2 after running it through GStreamer, I still only get 30 fps.

Commands:

v4l2-ctl --set-fmt-video=width=1280,height=720,pixelformat=AR24 --stream-mmap --set-ctrl bypass_mode=0 --stream-count=500 -d /dev/video0

gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw,format=BGRA,width=1280,height=720,framerate=60/1 ! fpsdisplaysink text-overlay=true video-sink=glimagesink

Through my research on the GStreamer side, I realized that it would be more reasonable to use the “nvv4l2camerasrc” plugin. However, at that point, I noticed that RGB888 support was not available. Therefore, I modified the plugin to add “BGRA” support. Still, my fps problem persists. Additionally, I’m unable to stream the image to the screen with “nvv4l2camerasrc”. The command I’m using is as follows:

gst-launch-1.0 nvv4l2camerasrc device=/dev/video0 num-buffers=300 ! 'video/x-raw(memory:NVMM),format=(string)BGRA, width=(int)1280, height=(int)720,framerate=(fraction)60/1' ! fpsdisplaysink text-overlay=true glimagesink
11

hello varolaibrahim,

you may have color convert before sending to fpsdisplaysink.
please refer to MMAPI sample, which is available via… $ sudo apt install nvidia-l4t-jetson-multimedia-api
for instance,
/usr/src/jetson_multimedia_api/samples/unittest_samples/transform_unit_sample/

12

Thanks for your reply.

Actually, I don’t need a conversion on video data. I’m sending RGB888 data from MIPI CSI-2 input. But, I’ll try.

13

BTW, VI did not support RGB888 memory formats, please see-also Topic 108194.

14

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.