tegra-vi4 constraints

Hi all,

I need help to understand better the limits of the vi4, so here my questions:

vi4 limits
I read in the forum that vi4 could achieve 4k,60fps but I can’t with my testing sensor.

Link:
https://devtalk.nvidia.com/default/topic/998322/jetson-tx2/tegra-x2-4k-x-2k-60hz-video-input/post/5107388/#5107388

So, I tried to test our sensor against tegra camera platform and these are the results for the tegra-vi4:

  • Maximum active resolution: 3968x2998
  • Width constraint: multiple of 128
  • Height constratint: multiple of 2
  • Line Length (width): 8700
  • Frame length (height): 3200
  • Pixel rate (pixel/s): 5904000000 (Hz)
  • mclock: 18 (MHz)
  • What happens if values are higher?
    Our sensor is capable of larger images but, if we increase the (active) height to 3000, the tegra vi4 sends these errors:

    [   74.158742] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
    

    And this:

    kworker/3:1-53    [003] ...1    80.558101: rtcpu_vinotify_handle_msg: tstamp:2872256131 tag:ATOMP_FS channel:0x00 frame:157 vi_tstamp:2872255721 data:0x00000000
    kworker/3:1-53    [003] ...1    80.610098: rtcpu_vinotify_handle_msg: tstamp:2873729833 tag:CHANSEL_SHORT_FRAME channel:0x01 frame:157 vi_tstamp:2873729259 data:0x00000001
    kworker/3:1-53    [003] ...1    80.610100: rtcpu_vinotify_handle_msg: tstamp:2873729963 tag:ATOMP_FE channel:0x00 frame:157 vi_tstamp:2873729259 data:0x00000000
    

    So, even if we increase the line and the frame lenght, the error is still the same. But if we continue increasing the active height pixels, the vi4 ends up complaining about the following:

    [  281.014822] tegra-vi4 15700000.vi: Status:  7 channel:00 frame:0002
    [  281.021093] tegra-vi4 15700000.vi:          timestamp sof 292351296384 eof 292397949120 data 0x00000001
    [  281.030483] tegra-vi4 15700000.vi:          capture_id 49 stream  0 vchan  0
    [  281.964916] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
    [  282.969054] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
    [  283.972984] tegra-vi4 15700000.vi: ATOMP_FE syncpt timeout!
    

    And this:

    kworker/3:1-53    [003] ...1   324.356841: rtcpu_vinotify_handle_msg: tstamp:10491389719 tag:CSIMUX_FRAME channel:0x00 frame:3 vi_tstamp:1901454590 data:0x00000220
    kworker/3:1-53    [003] ...1   324.356842: rtcpu_vinotify_handle_msg: tstamp:10491389848 tag:ATOMP_FS channel:0x00 frame:2 vi_tstamp:1901454623 data:0x00000000
    kworker/3:1-53    [003] ...1   324.356843: rtcpu_vinotify_handle_msg: tstamp:10491405258 tag:CHANSEL_PXL_SOF channel:0x00 frame:2 vi_tstamp:1901470264 data:0x00000001
    kworker/3:1-53    [003] ...1   324.356843: rtcpu_vinotify_handle_msg: tstamp:10491407968 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:2 vi_tstamp:1901472972 data:0x08000000
    kworker/3:1-53    [003] ...1   324.408839: rtcpu_vinotify_handle_msg: tstamp:10492863328 tag:CHANSEL_SHORT_FRAME channel:0x01 frame:2 vi_tstamp:1902928162 data:0x00000001
    kworker/3:1-53    [003] ...1   324.408840: rtcpu_vinotify_handle_msg: tstamp:10492863511 tag:ATOMP_FE channel:0x00 frame:2 vi_tstamp:1902928163 data:0x00000000
    

    According to the CHANSEL notifications:

    SHORT_FRAME (many-channel) is emitted when a Frame End appears from NVCSI before the normal number of pixels
    has appeared (or when zero pixels have appeared, and therefore no channel could be selected). An additional payload
    field indicates that too few embedded lines have been received.

    LOAD_FRAMED (many-channel) is emitted when a LOAD command is received for a channel while that channel is
    currently in a frame.

    So, what do you think what’s going on with the vi4?

    More information about vi4
    Also, we would like to know more about the limits of the tegra-vi4+nvcsi and where to find more documentation about them. We have the TRM document but still we think it is not enough.

    Other details

  • L4T: 28.2.1
  • Platform: Jetson TX2
  • Linux kernel version: 4.4
  • Of course, I’m using camera brinup wiki to fix some errors:
    https://elinux.org/Jetson_TX2_Camera_BringUp

    But also the camera Software Development Solution:
    https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%2520Linux%2520Driver%2520Package%2520Development%2520Guide%2Fjetson_xavier_camera_soft_archi.html%23wwconnect_header

    Thanks,

    right, the “tag:CHANSEL_SHORT_FRAME” tell the output line less than 3000

    I have updated the system to the latest 32.1 release together with the driver. Now, the configuration I had working in the previous version doesn’t work anymore. But the tegra-vi4 is now saying the next:

    [  334.021378] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
    [  334.027816] tegra-vi4 15700000.vi: tegra_channel_error_recovery: attempting to reset the capture channel
    [  334.038647] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) ERROR_STATUS2VI_VC0 = 0x0000000a
    [  334.047492] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) ERROR_STATUS2VI_VC1 = 0x00000002
    [  334.056251] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) ERROR_STATUS2VI_VC2 = 0x0000000a
    [  334.065059] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) INTR_STATUS 0x00010a0a
    [  334.073032] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) ERR_INTR_STATUS 0x00010a0a
    [  334.120825] tegra-vi4 15700000.vi: Status:  2 channel:00 frame:FF5F
    [  334.127235] tegra-vi4 15700000.vi:      timestamp sof 346544537312 eof 346544540800 data 0x00000400
    [  334.136398] tegra-vi4 15700000.vi:      capture_id 427 stream  0 vchan  0
    [  334.321408] tegra-vi4 15700000.vi: PXL_SOF syncpt timeout! err = -11
    [  334.327892] tegra-vi4 15700000.vi: tegra_channel_error_recovery: attempting to reset the capture channel
    [  334.339044] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) ERROR_STATUS2VI_VC0 = 0x0000000a
    [  334.348110] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) ERROR_STATUS2VI_VC1 = 0x00000002
    [  334.356968] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) ERROR_STATUS2VI_VC2 = 0x0000000a
    [  334.365915] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) INTR_STATUS 0x00010a0a
    [  334.373919] nvcsi 150c0000.nvcsi: csi4_stream_check_status (0) ERR_INTR_STATUS 0x00010a0a
    
    kworker/0:1-1313  [000] ....   334.105381: rtos_queue_send_failed: tstamp:10828301379 queue:0x0b4a7258
    kworker/0:1-1313  [000] ....   334.105387: rtcpu_vinotify_event: tstamp:10828302897 tag:CSIMUX_STREAM channel:0xff frame:0 vi_tstamp:10828301078 data:0x00000001
    kworker/0:1-1313  [000] ....   334.105390: rtcpu_vinotify_event: tstamp:10828727933 tag:CSIMUX_STREAM channel:0xff frame:2 vi_tstamp:10828727506 data:0x00000001
    kworker/0:1-1313  [000] ....   334.161242: rtcpu_vinotify_event: tstamp:10829501488 tag:ATOMP_FS channel:0x00 frame:65375 vi_tstamp:10829501062 data:0x00000000
    kworker/0:1-1313  [000] ....   334.161247: rtcpu_vinotify_event: tstamp:10829517526 tag:CHANSEL_PXL_SOF channel:0x00 frame:65375 vi_tstamp:10829516791 data:0x00000001
    kworker/0:1-1313  [000] ....   334.161248: rtcpu_vinotify_event: tstamp:10829517817 tag:CSIMUX_FRAME channel:0xac frame:65375 vi_tstamp:10829516900 data:0x00000400
    kworker/0:1-1313  [000] ....   334.161249: rtcpu_vinotify_event: tstamp:10829518297 tag:CHANSEL_FAULT channel:0x00 frame:65375 vi_tstamp:10829516900 data:0x00000200
    kworker/0:1-1313  [000] ....   334.161250: rtcpu_vinotify_event: tstamp:10829518490 tag:CHANSEL_LOAD_FRAMED channel:0x01 frame:65375 vi_tstamp:10829517999 data:0x08000000
    kworker/0:1-1313  [000] ....   334.161251: rtcpu_vinotify_event: tstamp:10829519200 tag:CHANSEL_FAULT_FE channel:0x01 frame:65375 vi_tstamp:10829518620 data:0x00000001
    kworker/0:1-1313  [000] ....   334.161252: rtcpu_vinotify_event: tstamp:10829519412 tag:ATOMP_FE channel:0x00 frame:65375 vi_tstamp:10829518623 data:0x00000000
    

    This is the configuration I’m using for the above output:

    • Active resolution: 1920x1080
    • Line Length (width): 8700 (sensor & DT)
    • Frame length (height): 1200 (I've also tested incrementing it up to 3200 but the output is the same).
    • Pixel rate (pixel/s): 5904000000 (Hz) (sensor & DT)
    • mclock: 18 (MHz) (sensor & DT)
    • mclk multiplier: 328 (DT)

    L4T:

  • 32.1.0
  • Linux kernel 4.9
  • Any suggestion?

    Hi!

    I could get a picture with the maximum resolution by using the latest L4T release (32.1.0 and kernel 4.9). My problem was related to the resolution parameters on the sensor. Now, I can get max sensor resolution but still, the output is weird. Here you can find a screenshot of the image.

    https://photos.app.goo.gl/5vqpvDKfkHYkv9T98

    Do you know why this is happening? image is kind of distorted.

    Thanks

    If there’s no error output to get the frame, it’s could be the pixel format is not correct.

    Hi ShaneCCC,

    Thanks for your answer. I don’t think there is any problem with the pixel format. I’ve been doing more tests on the camera and the tegra-vi4 and it seems like the width resolutions supported by the vi4 need to be multiple of 32. Is that correct?

    My test confirm correct and maximum resolutions: 4032x3040, 4000x3040, 3968x3040. But if I try resolutions like 4008x3040 or 4056x3040 then the output is like the above image shared in the previous comment.

    Does it make sense? the sensor can achieve up to 4056x3040.

    All the driver’s configurations I found in kernel 4.9 supported in tegra186 platform are multiple of 32 except the imx219.

    Hi again,

    I could achieve the maximum resolution of the sensor but the image doesn’t look great. Actually, with lower resolutions the quality much better. Do you know what could it be?

    This is the command I’m using:

    gst-launch-1.0 nvarguscamerasrc \
    wbmode=9 \
    ! 'video/x-raw(memory:NVMM), width=(int)4056, height=(int)3040, framerate=(fraction)30/1' \
    ! nvvidconv flip-method=2 \
    ! queue ! xvimagesink
    

    Pictures:

    • 4056x3040 (bad result): https://photos.app.goo.gl/fjnSeYhTxMD16Ntt9
    • 1920x1080 (good quality): https://photos.app.goo.gl/BRp9vKHEng4bNNe27

    Besides that, is there any limit on the vi4 for v4l2 path? I mean, I can’t get more than 23 fps for 4056x3040 and more than 100fps for 1920x1080. However, I can get better performance by using nvarguscamerasrc plugin with gstreamer instead of v4l2-ctl or yavta.

    @danielellxz
    For getting better image quality you may need camera partner to help to tuning process for it.
    For the performance, the v4l2 did not involve the ISP pipeline and display.

    Hi ShaneCCC,

    nvarguscamerasrc:
    I’m sorry for the confusion but what I’m really asking is why I can get images with good quality using 1920x1080 but not when using the highest resolution.

    v4l2:
    How should I use the ISP and the display in v4l2 pipeline (yavta, v4l2-ctl, qv4l2 or any standard Linux v4l2 application) to improve and get the same perfomance as with nvarguscamerasrc?

    Example:
    With v4l2 I can get images up to 100fps for 1920x1080 but with nvarguscamerasrc I can get up to 120fps.

    The camera driver documentation points to use the next command:

    v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --stream-mmap --stream-count=1 -d /dev/video0 --stream-to=ov5693.raw
    

    Link:
    https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%2520Linux%2520Driver%2520Package%2520Development%2520Guide%2Fjetson_xavier_camera_soft_archi.html%23wwpID0EZHA

    1. I can’t tell why you can get good quality for 1080p. But for the image quality must need tuning process.
    2. Sorry for my typo. I mean v4l2-ctl pipeline did not go through the ISP process that why can get better performance.

    Hi again,

    Thanks ShaneCCC for your previous answer. Now, I’ve understood if we want to have a better performance like getting higher framerates we should go with argus and not through v4l2 since it is not possible to involve the ISP in the v4l2/open-source pipeline.

    Regarding the quality issue, we have found how to replicate it and get ‘proper’ images by covering the sensor and make the image completely black when we use nvarguscamera.

    Steps:

    • Start gst-launch-1.0: Image quality is poor and it has spread dots around.
    • Cover sensor: We put something in front of the sensor to cover it so, we get a partial black image.
    • Wait ~2 seconds: After this time nvarguscamerasrc starts synching/getting/showing full quality of the image.

    Here 2 videos showing up the issue:

    Command line:

    $ gst-launch-1.0 nvarguscamerasrc wbmode=9 ! 'video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, framerate=(fraction)30/1' ! nvvidconv flip-method=2 ! queue ! xvimagesink
    

    So, do you have any idea of what is happening?

    duplicated

    duplicated

    Any update on the issue?

    Here the videos:

    So, still with same argus behavior: every time you make black the image, argus starts working properly.