Image sequence numbers incrementing even if no image captured

goad · July 19, 2022, 12:12pm

Hello,
We successfully use a camera connected to the Jetson Nano using a minimal device tree without a camera mode section. We use an external pixel clock and control the sensor directly using a user-space library. Although this works well in low-latency mode we have noticed that the sequence number occasionally jumps by several values, even though the timestamps show (and our observations confirm) that images are not really being lost.

Looking at the source code in vi2_fops.c in the function tegra_channel_capture_frame_multi_thread it appears that a buffer is always de-queued but then re-queued whenever the FIFO depth of the syncpt events is already 1. After re-queueing, there is a short wait of 1 ms. Unfortunately, re-queueing buffers in this way causes the sequence number to be incremented every time (in release_buffer) and appears to be the reason for the inconsistent numbering.

    /* The fifo depth of PP_FRAME_START and MW_ACK_DONE is 2 */
    down_read(&chan->reset_lock);
    /* The fifo depth of syncpt event PP_FRAME_START and MW_ACK_DONE is 2 */
    if (atomic_read(&chan->syncpt_depth) < SYNCPT_FIFO_DEPTH) {
        atomic_inc(&chan->syncpt_depth);
    } else {
        up_read(&chan->reset_lock);
        /* requeue this vb2buf */
        buf->state = VB2_BUF_STATE_REQUEUEING;
        release_buffer(chan, buf);
        /* sleep, waiting for the programmed syncpt being handled */
        usleep_range(1000, 1010);
        return 0;
    }

In the JetPack 4.6 version the FIFO full condition is only occasionally met so that the sequence number is often, but not always correct. But since JetPack 4.6.1 this re-queueing happens many times between real images, causing the sequence number to jump significantly between frames. Our sensor runs at approx. 30 frames per second in full-resolution, so we see the sequence number jumping by around 30 between each frame, since it is incremented nearly every millisecond.

My questions are:

Would it not be easier to check the FIFO depth of the syncpt events before de-queuing e.g. in the function tegra_channel_kthread_capture_start? If the FIFO is already full then there would be no need to call the function tegra_channel_capture_frame yet and no need to re-queue or increment the sequence number.
Why is tegra_channel_capture_frame_multi_thread being called so often (unnecessarily?) in the JetPack 4.6.1 version? (I have another question relating to the changes between JetPack 4.6 and 4.6.1 which I will post separately).

As a proof-of-concept we have already tried to implement the FIFO depth check in tegra_channel_kthread_capture_start and it appears to work well. It looks something like this:

        /* 
         * Check the depth of the FIFO before dequeuing a buffer.
         * This saves having to re-queue it immediately if the depth counter
         * has already reached its limit and makes sure that the frame's
         * sequence number is not incremented for re-queued frames.
         */
        if (chan->low_latency) {
            /* The fifo depth of syncpt event PP_FRAME_START and MW_ACK_DONE is 2 */
            if (atomic_read(&chan->syncpt_depth) < SYNCPT_FIFO_DEPTH) {
                buf = dequeue_buffer(chan, !chan->low_latency);
                if (!buf)
                    continue;

                err = tegra_channel_capture_frame(chan, buf);
            } else {
                /*
                 * We now wait for a very short time here instead of in 'tegra_channel_capture_frame'.
                 */
                /* sleep, waiting for the programmed syncpt being handled */
                usleep_range(1000, 1010);
            }
        } else {
            /* not low latency */
            buf = dequeue_buffer(chan, !chan->low_latency);
            if (!buf)
                continue;

            err = tegra_channel_capture_frame(chan, buf);
        }

Thank you in advance.

Goad.

JerryChang · July 20, 2022, 5:54am

hello goad,

this VI-2 driver, vi2_fops is used on TX1 and Nano series for many releases. it also has fixes check-in to refactor low-latency mode recently.
may I know what’s your sensor types, what’s your test steps and how you check the image sequence. we’ll also review the code snippet you’ve shared.

goad · July 21, 2022, 3:35pm

Hello JerryChang,

We are using a Sony IMX567. We set up a V4L subdevice that gives us access to the sensor registers via custom V4L ioctls. The complete register initialisation is done in user space so that we can set up different resolutions and frame rates. We then use the standard video device e.g. /dev/video0 to access the image data, just like a normal V4L application.

However, since this is a special case using hardware and software that you do not have, I tried to reproduce the problem using a RaspberryPi HQ camera (IMX477) and the standard Device Tree and kernel supplied in JetPack 4.6.2. It looks like it is indeed possible to reproduce using v4l2-ctl:

v4l2-ctl -d /dev/video0 --set-ctrl=low_latency_mode=1 --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=20 --verbose

This produces an output like this (note: the resolution seems to have been ignored because 3840*2160 is being used):

IDIOC_QUERYCAP: ok
VIDIOC_S_EXT_CTRLS: ok
VIDIOC_G_FMT: ok
VIDIOC_S_FMT: ok
Format Video Capture:
        Width/Height      : 3840/2160
        Pixel Format      : 'RG10'
        Field             : None
        Bytes per Line    : 7680
        Size Image        : 16588800
        Colorspace        : sRGB
        Transfer Function : Default (maps to sRGB)
        YCbCr/HSV Encoding: Default (maps to ITU-R 601)
        Quantization      : Default (maps to Full Range)
        Flags             : 
VIDIOC_REQBUFS: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_STREAMON: ok
        Index    : 0
        Type     : Video Capture
        Flags    : mapped
        Field    : None
        Sequence : 41
        Length   : 16588800
        Bytesused: 16588800
        Timestamp: 648.232609s (Monotonic, End-of-Frame)

        Index    : 1
        Type     : Video Capture
        Flags    : mapped
        Field    : None
        Sequence : 74
        Length   : 16588800
        Bytesused: 16588800
        Timestamp: 648.232618s (Monotonic, End-of-Frame)

etc......

        Index    : 1
        Type     : Video Capture
        Flags    : mapped
        Field    : None
        Sequence : 568
        Length   : 16588800
        Bytesused: 16588800
        Timestamp: 648.774360s (Monotonic, End-of-Frame)

        Index    : 0
        Type     : Video Capture
        Flags    : mapped
        Field    : None
        Sequence : 599
        Length   : 16588800
        Bytesused: 16588800
        Timestamp: 648.807805s (Monotonic, End-of-Frame)

        Index    : 2
        Type     : Video Capture
        Flags    : mapped
        Field    : None
        Sequence : 630
        Length   : 16588800
        Bytesused: 16588800
        Timestamp: 648.841503s (Monotonic, End-of-Frame)

VIDIOC_STREAMOFF: ok

As you can see, the sequence number jumps significantly between frames. Turning on kernel debugging for the files channel.c and vi2_fops.c reveals that between real frames the buffer is being dequeued and requeued (and being counted!) approx. every millisecond i.e. the syncpt FIFO is already full when the function tegra_channel_capture_frame_multi_thread is being called.

[ 1257.841936] video4linux video0: tegra_channel_release_frame: vi2 got EOF syncpt buf[ffffffc0c76de000]
[ 1257.841941] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[597] to user-space
[ 1257.842308] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[598] to user-space
[ 1257.843335] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[599] to user-space
[ 1257.844363] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[600] to user-space
[ 1257.845392] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[601] to user-space
[ 1257.846418] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[602] to user-space
[ 1257.847445] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[603] to user-space
[ 1257.848520] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[604] to user-space
[ 1257.849587] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[605] to user-space
[ 1257.850655] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[606] to user-space
[ 1257.851724] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[607] to user-space
[ 1257.852763] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[608] to user-space
[ 1257.853794] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[609] to user-space
[ 1257.854823] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[610] to user-space
[ 1257.855851] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[611] to user-space
[ 1257.856879] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[612] to user-space
[ 1257.858006] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[613] to user-space
[ 1257.860577] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[614] to user-space
[ 1257.861608] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[615] to user-space
[ 1257.862644] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[616] to user-space
[ 1257.863669] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[617] to user-space
[ 1257.864752] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[618] to user-space
[ 1257.865783] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[619] to user-space
[ 1257.866809] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[620] to user-space
[ 1257.867836] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[621] to user-space
[ 1257.868867] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[622] to user-space
[ 1257.869897] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[623] to user-space
[ 1257.870924] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[624] to user-space
[ 1257.871951] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[625] to user-space
[ 1257.872983] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[626] to user-space
[ 1257.874013] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[627] to user-space
[ 1257.875038] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[628] to user-space
[ 1257.875224] video4linux video0: tegra_channel_release_frame: vi2 got EOF syncpt buf[ffffffc0c76dc000]
...... etc.....

In non low-latency mode everything is working fine:

v4l2-ctl -d /dev/video0 --set-ctrl=low_latency_mode=0 --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=20 --verbose

The result looks good:

VIDIOC_QUERYCAP: ok
VIDIOC_S_EXT_CTRLS: ok
VIDIOC_G_FMT: ok
VIDIOC_S_FMT: ok
Format Video Capture:
        Width/Height      : 3840/2160
        Pixel Format      : 'RG10'
        Field             : None
        Bytes per Line    : 7680
        Size Image        : 16588800
        Colorspace        : sRGB
        Transfer Function : Default (maps to sRGB)
        YCbCr/HSV Encoding: Default (maps to ITU-R 601)
        Quantization      : Default (maps to Full Range)
        Flags             : 
VIDIOC_REQBUFS: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_STREAMON: ok
        Index    : 0
        Type     : Video Capture
        Flags    : mapped
        Field    : None
        Sequence : 0
        Length   : 16588800
        Bytesused: 16588800
        Timestamp: 866.225197s (Monotonic, End-of-Frame)

        Index    : 1
        Type     : Video Capture
        Flags    : mapped
        Field    : None
        Sequence : 1
        Length   : 16588800
        Bytesused: 16588800
        Timestamp: 866.257029s (Monotonic, End-of-Frame)

        Index    : 2
        Type     : Video Capture
        Flags    : mapped
        Field    : None
        Sequence : 2
        Length   : 16588800
        Bytesused: 16588800
        Timestamp: 866.291356s (Monotonic, End-of-Frame)

etc......

        Index    : 1
        Type     : Video Capture
        Flags    : mapped
        Field    : None
        Sequence : 17
        Length   : 16588800
        Bytesused: 16588800
        Timestamp: 866.791423s (Monotonic, End-of-Frame)

        Index    : 2
        Type     : Video Capture
        Flags    : mapped
        Field    : None
        Sequence : 18
        Length   : 16588800
        Bytesused: 16588800
        Timestamp: 866.824411s (Monotonic, End-of-Frame)

        Index    : 3
        Type     : Video Capture
        Flags    : mapped
        Field    : None
        Sequence : 19
        Length   : 16588800
        Bytesused: 16588800
        Timestamp: 866.858118s (Monotonic, End-of-Frame)

VIDIOC_STREAMOFF: ok

The same test performed using JetPack 4.6 gives proper sequence numbers in low-latency mode, so it would appear that something is not quite right with the refactored low-latency handling introduced in JetPack 4.6.1.

Perhaps you are able to reproduce the problem yourself like this?

Regards,

Goad

goad · July 22, 2022, 9:45am

Hello JerryChang,

A possible solution, better than my original idea, might be to expand the condition check in the call wait_event_interruptible in the function tegra_channel_kthread_capture_start to include a check on the depth of the syncpt FIFO fill-level. Then you could presumably eliminate the 1ms wait in the function tegra_channel_capture_frame_multi_thread. The two threads for capture and release would synchronise with each other instead of running around the while-loop in the function tegra_channel_kthread_capture_start 30 times between frames.

I have attached a patch for review. This seems to work well with the RaspberryPi camera and also with our camera, although it does not solve the issue mentioned in my other post (Image timestamp: differences between JetPack 4.6 and JetPack 4.6.1?) with our camera (and I am not able to reproduce that problem with a RasPi camera yet).

Kind regards,

Goad
001_Sequence_Number.patch (1.4 KB)

JerryChang · July 25, 2022, 1:59am

hello goad,

thanks for sharing, I’ll arrange resources to review the patch.

goad · August 12, 2022, 7:00am

Hello JerryChang,

Do you have any new feedback on this? Are you able to reproduce the problem and is there any chance of a fix in any future 4.x JetPacks, even though Jetson Nano is no longer supported by JetPack 5? It makes a difference to us whether our customers will need to patch the kernel or can rely on a JetPack out-of-the-box.

Thanks.

Goad

JerryChang · August 12, 2022, 7:06am

hello goad,

we had reproduce the issue and confirmed this kernel patch could resolve the failure.
the review process is on-going, it’s may integrate to future JetPack-4 release, unfortunately, code-review take times, I cannot claim when will complete.

goad · August 12, 2022, 8:47am

Hello JerryChang,

Thank you. Please let me know if I can help to test anything.

Regards,

Goad

JerryChang · October 19, 2022, 2:59am

hello goad,

sorry for late response. there’re some internal v4l test failures for this patch,[001_Sequence_Number.patch].

we’ve tested and examined the code in your 1st post.
upon that, we integrate the fixes into rel-32 code-line, you may expect this changes in the next JetPack-4 public release.
thanks

goad · October 19, 2022, 8:12am

Hello JerryChang,
Thanks for the info. Yes, unfortunately I don’t have the facilities here to test all the different sensors and modes, so I’m not surprised the patch didn’t solve everything.
It’s also good to know that there will be a further Jetpack-4 release. Do you have any information about a release date?
Regards,

Goad

kayccc · November 2, 2022, 2:41am

Please wait for our announcement at coming weeks.

system · November 23, 2022, 4:45am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Image timestamp: differences between JetPack 4.6 and JetPack 4.6.1? Jetson Nano camera , kernel	5	738	July 22, 2022
Mipi-driver : frame start syncpt timeout!0 Jetson Nano camera , kernel , gstreamer	50	3683	October 15, 2021
One frame latency/delay in TX1 V4L stack Jetson TX1	44	9562	October 18, 2021
Potential issues with Falcon leading to issues in vi5_fops Jetson Orin NX camera , rtc , csi	29	53	February 7, 2025
tag:CSIMUX_FRAME channel:0x00 frame:1 vi_tstamp:3607277994400 data:0x00000000000000a0 Jetson Xavier NX camera	40	140	September 11, 2024
Gstreamer lag increases with frame rate recording from Raspberry Pi camera (nvarguscamerasrc and nvivafilter) Jetson Nano camera , cuda , gstreamer	26	2384	January 17, 2023
Multi_camera "No cameras available" Jetson Nano camera	58	4343	October 15, 2021
Possible multimedia api regression with decode interlace source Jetson TX2 mmapi , nvbugs	28	2671	February 9, 2022
Gstreamer reports Raspberry Pi camera streaming at 120fps when in reality it is only 60fps Jetson Nano camera , gstreamer	53	3799	July 19, 2022
record and run inference at the same time, split video Jetson Nano	26	2783	October 14, 2021

Image sequence numbers incrementing even if no image captured

Related topics