AGX Orin 64GB platform, How to modify vi5 to drop one frame every two frames?

Now I 'd like to implement drop frame in the driver. What should I do to modify the driver code? Thank you.

hello Dingdongkk,

may I know what’s the real use-case?

it’s VI-5 driver to handle camera buffer operations.
for instance,
$public_sources/kernel_src/kernel/nvidia/drivers/media/platform/tegra/camera/vi/vi5_fops.c

static void vi5_capture_dequeue(...)
{
...
rel_buf:
        vi5_release_buffer(chan, buf);

and…
here’s the code snippet for sending frames to use-space.
by default it’s sending each of capture frames in orders, you may add some code here for verificaiton.

static void vi5_release_buffer(...)
{
...
        vbuf->sequence = chan->sequence++;
        vb2_buffer_done(&vbuf->vb2_buf, buf->vb2_state);

Yeah, the default camera frame rate for the car is 30fps, but we need 15fps at the user-space to perform some deep learning-related tasks.
However, I tried the following code in vi5_release_buffer, and it seems like it doesn’t work.

static void vi5_release_buffer(struct tegra_channel *chan,
	struct tegra_channel_buffer *buf)
{
	struct vb2_v4l2_buffer *vbuf = &buf->buf;
	static bool drop_frame = true;
	if (drop_frame) {
		vbuf->sequence = chan->sequence++;
		vbuf->field = V4L2_FIELD_NONE;
		vb2_set_plane_payload(&vbuf->vb2_buf, 0, chan->format.sizeimage);
	
		vb2_buffer_done(&vbuf->vb2_buf, buf->vb2_state);
	}
	drop_frame = !drop_frame;
}

what did you meant “not works”, for instance, is there any failure logs or symptoms for reference?

This is my test command.

v4l2-ctl -d /dev/video3 --stream-mmap=4 --set-fmt-video=width=2880,height=1860,pixelformat=UYVY  --verbose

The log is as follows, but the dpbuf seems to be stuck after a few frames.

VIDIOC_QUERYCAP: ok
VIDIOC_G_FMT: ok
VIDIOC_S_FMT: ok
Format Video Capture:
	Width/Height      : 2880/1860
	Pixel Format      : 'UYVY' (UYVY 4:2:2)
	Field             : None
	Bytes per Line    : 5760
	Size Image        : 10713600
	Colorspace        : sRGB
	Transfer Function : Default (maps to sRGB)
	YCbCr/HSV Encoding: Default (maps to ITU-R 601)
	Quantization      : Default (maps to Limited Range)
	Flags             : 
		VIDIOC_REQBUFS returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_STREAMON returned 0 (Success)
cap dqbuf: 0 seq:      0 bytesused: 10713600 ts: 1710145739.466480 (ts-monotonic, ts-src-eof)
cap dqbuf: 2 seq:      1 bytesused: 10713600 ts: 1710145739.533268 delta: 66.788 ms (ts-monotonic, ts-src-eof)
cap dqbuf: 0 seq:      2 bytesused: 10713600 ts: 1710145739.683340 delta: 150.072 ms (ts-monotonic, ts-src-eof)
cap dqbuf: 0 seq:      3 bytesused: 10713600 ts: 1710145739.783332 delta: 99.992 ms (ts-monotonic, ts-src-eof)

And dmesg commend only shows the following two lines.

[   95.986782] bwmgr API not supported
[   96.099996] --------cntt=  1-----

When setting --stream-mmap=10, the log is as follows. It seems like there might be an issue with dqbuf number.

nvidia@nvidia-desktop:~$ v4l2-ctl -d /dev/video3 --stream-mmap=10 --set-fmt-video=width=2880,height=1860,pixelformat=UYVY --stream-skip=1 --verbose
VIDIOC_QUERYCAP: ok
VIDIOC_G_FMT: ok
VIDIOC_S_FMT: ok
Format Video Capture:
	Width/Height      : 2880/1860
	Pixel Format      : 'UYVY' (UYVY 4:2:2)
	Field             : None
	Bytes per Line    : 5760
	Size Image        : 10713600
	Colorspace        : sRGB
	Transfer Function : Default (maps to sRGB)
	YCbCr/HSV Encoding: Default (maps to ITU-R 601)
	Quantization      : Default (maps to Limited Range)
	Flags             : 
		VIDIOC_REQBUFS returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QUERYBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_QBUF returned 0 (Success)
		VIDIOC_STREAMON returned 0 (Success)
cap dqbuf: 0 seq:      0 bytesused: 10713600 ts: 1710146439.233835 (ts-monotonic, ts-src-eof)
cap dqbuf: 2 seq:      1 bytesused: 10713600 ts: 1710146439.333178 delta: 99.343 ms (ts-monotonic, ts-src-eof)
cap dqbuf: 4 seq:      2 bytesused: 10713600 ts: 1710146439.433170 delta: 99.992 ms (ts-monotonic, ts-src-eof)
cap dqbuf: 6 seq:      3 bytesused: 10713600 ts: 1710146439.533485 delta: 100.315 ms (ts-monotonic, ts-src-eof)
cap dqbuf: 8 seq:      4 bytesused: 10713600 ts: 1710146439.633325 delta: 99.840 ms fps: 10.01 (ts-monotonic, ts-src-eof)
cap dqbuf: 0 seq:      5 bytesused: 10713600 ts: 1710146439.733490 delta: 100.165 ms fps: 10.01 (ts-monotonic, ts-src-eof)
cap dqbuf: 4 seq:      6 bytesused: 10713600 ts: 1710146439.833483 delta: 99.993 ms fps: 10.01 (ts-monotonic, ts-src-eof)
cap dqbuf: 8 seq:      7 bytesused: 10713600 ts: 1710146439.933184 delta: 99.701 ms fps: 10.01 (ts-monotonic, ts-src-eof)
cap dqbuf: 4 seq:      8 bytesused: 10713600 ts: 1710146440.033528 delta: 100.344 ms fps: 10.00 (ts-monotonic, ts-src-eof)
cap dqbuf: 4 seq:      9 bytesused: 10713600 ts: 1710146440.133527 delta: 99.999 ms fps: 10.00 (ts-monotonic, ts-src-eof)


hello Dingdongkk,

instead of dropping frames, could you please try below settings, i.e. --set-ctrl frame_rate=xxx,override_enable=1 to configure frame-rate for your use-case,
for example, here’s my camera sensor capability, 2592x1944@30-fps.
$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=2592,height=1944,pixelformat=BG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=100
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 30.00 fps

by adding the property settings,
it could adjust the frame-rate (I’ve configure it as 15-fps) and sending 15-fps to user-space.
for exmaple,
$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=2592,height=1944,pixelformat=BG10 --set-ctrl bypass_mode=0 --set-ctrl frame_rate=15000000,override_enable=1 --stream-mmap --stream-count=100
<<<<<<<<<<<<<<<<< 15.00 fps

Thank you for your quick response. I tested the frame rate, and it’s still 30fps. Here are the test logs.

nvidia@nvidia-desktop:~$ v4l2-ctl -d /dev/video3 --set-fmt-video=width=2880,height=1860,pixelformat=UYVY --set-ctrl 
bypass_mode=0 --set-ctrl frame_rate=20000000,override_enable=1 --stream-mmap --stream-count=100
VIDIOC_S_EXT_CTRLS: failed: Remote I/O error
frame_rate: Remote I/O error
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 29.13 fps

The dmesg error is as follows.

camera_set_frame_rate: FRAME_RATE control error

The possible reason is that our camera doesn’t support setting the frame rate. Therefore, it is necessary to handle frame dropping settings in the driver.

hello Dingdongkk,

it’s basic vl4 control to call sensor driver for changing frame_length, is it possible to implement this in the sensor driver side?

Sorry, we consulted the camera manufacturer and they said that the frame rate output is fixed and cannot be adjusted.
For now, we are handling frame dropping at the application layer. Thank you.

hello Dingdongkk,

thanks for sharing, are we able to close this discussion thread now?

Sure, of course. thank you.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.