Losing first frame after stop/start streaming VI4

Hi,
I am capturing frames on a Jetson TX2 from an external FPGA through MIPI CSI-2 lane and managed to modify the vi4_fops.c to capture the 2 first frames. In my application, I am capturing frames which is working fine, I stop the streaming correctly, then apply a new video format (640x480 to another format), start the streaming again, then I lose the first frame (not receiving any PXL_SOF syncpt incrementation from hardware) and then the second frame arrives (incrementing the PXL_SOF syncpt) and the video capture works fine after. I am wondering why I lose the first frame after starting the stream again. I do not lose the first frame if I stop the application and start it again even though it is going through the same functions/process. Do you have any idea why after a stop/start streaming the first frame isn’t captured ? And why after a stop/start streaming by stopping/starting the application it is working ?
Thank you for your help

I think it could be the timing cause it. Maybe the sensor initial flow is different?

It isn’t the timing because even if we use an external trigger and wait a long time after the the VIDIOC_STREAMOFF/VIDIOC_STREAMON, the first trigger send the frame from FPGA on MIPI CSI-2 but the PXL_SOF syncpt doesn’t increment.

Don’t understand your experiment clearly.

The experiment is to capture frames using an external trigger and to modify the frame format whenever we want. In order to this I modified the vi4_fops to have an infinite timeout without blocking the user from changing the frame format. I’ve also done some modifications to put the frame in the ring buffer at the moment it arrives (and not at frame N+2). For the moment, everything is working fine appart from a frame being lost.

Between two frame formats we do the following : Stop the streaming (VIDIOC_STREAMOFF) → Apply new format (VIDIOC_S_FMT) → Start the streaming (VIDIOC_STREAMON).

After this sequence, we do not receive the first frame (from our external trigger), we only receive frames starting from the second one.
I have noticed it is after the following sequence in the same linux process that we have this problem : Stop the streaming (VIDIOC_STREAMOFF) → Start the streaming (VIDIOC_STREAMON) = first frame lost.
The strange thing is that when I kill the linux process and run it again, we go through the same sequence but here the first frame is captured : kill process → Stop the streaming (VIDIOC_STREAMOFF) → start process → Start the streaming (VIDIOC_STREAMON) = first frame captured.

Have a try below patch.

diff --git a/drivers/media/platform/tegra/camera/vi/vi4_fops.c b/drivers/media/platform/tegra/camera/vi/vi4_fops.c
index 9bcae78..044f6b5 100644
--- a/drivers/media/platform/tegra/camera/vi/vi4_fops.c
+++ b/drivers/media/platform/tegra/camera/vi/vi4_fops.c
@@ -36,6 +36,11 @@


 static void tegra_channel_error_recovery(struct tegra_channel *chan);
 static void tegra_channel_stop_kthreads(struct tegra_channel *chan);
 static int tegra_channel_stop_increments(struct tegra_channel *chan);
@@ -206,7 +211,7 @@ static bool vi_notify_wait(struct tegra_channel *chan,
         */
        for (i = 0; i < chan->valid_ports; i++) {
                err = nvhost_syncpt_wait_timeout_ext(chan->vi->ndev,
-                               chan->syncpt[i][SOF_SYNCPT_IDX], thresh[i],
+                               chan->syncpt[i][FE_SYNCPT_IDX], thresh[i],
                                chan->timeout, NULL, NULL);
                if (unlikely(err)) {
                        dev_err(chan->vi->dev,

I have the same problem when I apply the patch above.
Being able to capture the first frame is important for us.
Our application which is running on Jetson TX1 is able to do it so we want to do the same on Jetson TX2.

What’s the problem when apply the patch?

The patch doesn’t change the behaviour.
I just found out where the problem is coming from.

With a framerate of 0.2 Hz when I run “yavta /dev/video0 -c5”, then wait 600ms after the end of yavta and run “yavta /dev/video0 -c5” again, I get all the frames I wanted.
Now if I run “yavta /dev/video0 -c5 && yavta /dev/video0 -c5”, I don’t have the first frame of the yavta running just after the end of the first yavta (there is 10 seconds beetwen the last frame of the first yavta and the first frame of the second yavta, but there should be 5 seconds).

So my conclusion is that there must be 600ms between stop streaming and start streaming.
Can you do the experiment and let me know why some time is needed between the stop streaming and start streaming
please ?

Thank you for your help

Even stop streaming and start streaming take 600ms that shouldn’t be a problem.
The device should get start stream to enable the stream so I think stop stream to start stream should be the problem.

I am saying there is no problem with the following : Stop the streaming (VIDIOC_STREAMOFF) → usleep(600000) → Start the streaming (VIDIOC_STREAMON) = 1st frame received.
But there is a lost first frame with the following : Stop the streaming (VIDIOC_STREAMOFF) → Start the streaming (VIDIOC_STREAMON) = 1st frame lost.

Now using usleep is working but it is not a correct solution, so I need some information for solving this problem without usleep.

I think the key point should be the timing of the sensor stream_on.

Can you be more precise please ? All the functions of the stream OFF are done before the functions of the stream ON as you can see in the logs I have added

Logs

Have try remove below code from the csi4_fops.c

csi4_phy_write(chan, phy_num, NVCSI_CIL_A_SW_RESET,
SW_RESET1_EN | SW_RESET0_EN);

csi4_phy_write(chan, phy_num, NVCSI_CIL_B_SW_RESET,
SW_RESET1_EN | SW_RESET0_EN);

Thank you very much, it is working now. Can you tell me what these line of codes are doing usually ?

This code reset the CSI logic.

Thank you for your answers. I now have a problem concerning the syncpoints.
I have applied the patch you mentionned which is correcting the camera shake issue from (tx2)the camera delay N+2 frame :

  •                           chan->syncpt[i][SOF_SYNCPT_IDX], thresh[i],
    
  •                           chan->syncpt[i][FE_SYNCPT_IDX], thresh[i],
    

It is working well but when I stop and start the streaming the ATOMP_FE threshold and the syncpoint are not synchronized anymore as you can see from log below :

this is causing to lose all the frames after. Do you know what is causing the ATOMP_FE threshold to be reseted to 0 ? I would need it to follow the same threshold as PXL_SOF.

You may try to modify the time of syncpt wait function to try it.

The problem is not the timeout, the problem is that we are waiting for an ATOMP_FE threshold value of 1 but the syncpt here is at 257 and incrementing. Do you know what is causing the ATOMP_FE threshold to be reseted to 0 ?

The value will roll back when it reach to max. But I don’t know what’s the max value.