Hello JerryChang,
We are using a Sony IMX567. We set up a V4L subdevice that gives us access to the sensor registers via custom V4L ioctls. The complete register initialisation is done in user space so that we can set up different resolutions and frame rates. We then use the standard video device e.g. /dev/video0 to access the image data, just like a normal V4L application.
However, since this is a special case using hardware and software that you do not have, I tried to reproduce the problem using a RaspberryPi HQ camera (IMX477) and the standard Device Tree and kernel supplied in JetPack 4.6.2. It looks like it is indeed possible to reproduce using v4l2-ctl:
v4l2-ctl -d /dev/video0 --set-ctrl=low_latency_mode=1 --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=20 --verbose
This produces an output like this (note: the resolution seems to have been ignored because 3840*2160 is being used):
IDIOC_QUERYCAP: ok
VIDIOC_S_EXT_CTRLS: ok
VIDIOC_G_FMT: ok
VIDIOC_S_FMT: ok
Format Video Capture:
Width/Height : 3840/2160
Pixel Format : 'RG10'
Field : None
Bytes per Line : 7680
Size Image : 16588800
Colorspace : sRGB
Transfer Function : Default (maps to sRGB)
YCbCr/HSV Encoding: Default (maps to ITU-R 601)
Quantization : Default (maps to Full Range)
Flags :
VIDIOC_REQBUFS: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_STREAMON: ok
Index : 0
Type : Video Capture
Flags : mapped
Field : None
Sequence : 41
Length : 16588800
Bytesused: 16588800
Timestamp: 648.232609s (Monotonic, End-of-Frame)
Index : 1
Type : Video Capture
Flags : mapped
Field : None
Sequence : 74
Length : 16588800
Bytesused: 16588800
Timestamp: 648.232618s (Monotonic, End-of-Frame)
etc......
Index : 1
Type : Video Capture
Flags : mapped
Field : None
Sequence : 568
Length : 16588800
Bytesused: 16588800
Timestamp: 648.774360s (Monotonic, End-of-Frame)
Index : 0
Type : Video Capture
Flags : mapped
Field : None
Sequence : 599
Length : 16588800
Bytesused: 16588800
Timestamp: 648.807805s (Monotonic, End-of-Frame)
Index : 2
Type : Video Capture
Flags : mapped
Field : None
Sequence : 630
Length : 16588800
Bytesused: 16588800
Timestamp: 648.841503s (Monotonic, End-of-Frame)
VIDIOC_STREAMOFF: ok
As you can see, the sequence number jumps significantly between frames. Turning on kernel debugging for the files channel.c and vi2_fops.c reveals that between real frames the buffer is being dequeued and requeued (and being counted!) approx. every millisecond i.e. the syncpt FIFO is already full when the function tegra_channel_capture_frame_multi_thread is being called.
[ 1257.841936] video4linux video0: tegra_channel_release_frame: vi2 got EOF syncpt buf[ffffffc0c76de000]
[ 1257.841941] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[597] to user-space
[ 1257.842308] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[598] to user-space
[ 1257.843335] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[599] to user-space
[ 1257.844363] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[600] to user-space
[ 1257.845392] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[601] to user-space
[ 1257.846418] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[602] to user-space
[ 1257.847445] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[603] to user-space
[ 1257.848520] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[604] to user-space
[ 1257.849587] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[605] to user-space
[ 1257.850655] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[606] to user-space
[ 1257.851724] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[607] to user-space
[ 1257.852763] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[608] to user-space
[ 1257.853794] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[609] to user-space
[ 1257.854823] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[610] to user-space
[ 1257.855851] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[611] to user-space
[ 1257.856879] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[612] to user-space
[ 1257.858006] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[613] to user-space
[ 1257.860577] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[614] to user-space
[ 1257.861608] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[615] to user-space
[ 1257.862644] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[616] to user-space
[ 1257.863669] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[617] to user-space
[ 1257.864752] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[618] to user-space
[ 1257.865783] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[619] to user-space
[ 1257.866809] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[620] to user-space
[ 1257.867836] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[621] to user-space
[ 1257.868867] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[622] to user-space
[ 1257.869897] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[623] to user-space
[ 1257.870924] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[624] to user-space
[ 1257.871951] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[625] to user-space
[ 1257.872983] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[626] to user-space
[ 1257.874013] video4linux video0: release_buffer: release buf[ffffffc0c76de000] frame[627] to user-space
[ 1257.875038] video4linux video0: release_buffer: release buf[ffffffc0c76df800] frame[628] to user-space
[ 1257.875224] video4linux video0: tegra_channel_release_frame: vi2 got EOF syncpt buf[ffffffc0c76dc000]
...... etc.....
In non low-latency mode everything is working fine:
v4l2-ctl -d /dev/video0 --set-ctrl=low_latency_mode=0 --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=20 --verbose
The result looks good:
VIDIOC_QUERYCAP: ok
VIDIOC_S_EXT_CTRLS: ok
VIDIOC_G_FMT: ok
VIDIOC_S_FMT: ok
Format Video Capture:
Width/Height : 3840/2160
Pixel Format : 'RG10'
Field : None
Bytes per Line : 7680
Size Image : 16588800
Colorspace : sRGB
Transfer Function : Default (maps to sRGB)
YCbCr/HSV Encoding: Default (maps to ITU-R 601)
Quantization : Default (maps to Full Range)
Flags :
VIDIOC_REQBUFS: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_QUERYBUF: ok
VIDIOC_QBUF: ok
VIDIOC_STREAMON: ok
Index : 0
Type : Video Capture
Flags : mapped
Field : None
Sequence : 0
Length : 16588800
Bytesused: 16588800
Timestamp: 866.225197s (Monotonic, End-of-Frame)
Index : 1
Type : Video Capture
Flags : mapped
Field : None
Sequence : 1
Length : 16588800
Bytesused: 16588800
Timestamp: 866.257029s (Monotonic, End-of-Frame)
Index : 2
Type : Video Capture
Flags : mapped
Field : None
Sequence : 2
Length : 16588800
Bytesused: 16588800
Timestamp: 866.291356s (Monotonic, End-of-Frame)
etc......
Index : 1
Type : Video Capture
Flags : mapped
Field : None
Sequence : 17
Length : 16588800
Bytesused: 16588800
Timestamp: 866.791423s (Monotonic, End-of-Frame)
Index : 2
Type : Video Capture
Flags : mapped
Field : None
Sequence : 18
Length : 16588800
Bytesused: 16588800
Timestamp: 866.824411s (Monotonic, End-of-Frame)
Index : 3
Type : Video Capture
Flags : mapped
Field : None
Sequence : 19
Length : 16588800
Bytesused: 16588800
Timestamp: 866.858118s (Monotonic, End-of-Frame)
VIDIOC_STREAMOFF: ok
The same test performed using JetPack 4.6 gives proper sequence numbers in low-latency mode, so it would appear that something is not quite right with the refactored low-latency handling introduced in JetPack 4.6.1.
Perhaps you are able to reproduce the problem yourself like this?
Regards,
Goad