Tegra Multimedia APIs could get raw image via v4l2 only at 15fps

forever3000 · November 7, 2017, 6:42am

Dear all,

When I tried to run 12_camera_v4l2_cuda in tegra_multimedia_api examples, I could get only 15 FPS as below

----------- Element = renderer0 -----------
Total Profiling time = 6.85006
Average FPS = 15.0364
Total units processed = 104
Num. of late units = 103

I also tried to measure polling time then result was same.

Polling Time : 0.066571 
Polling Time : 0.066521 
Polling Time : 0.066563 
Polling Time : 0.066589 
Polling Time : 0.066581 
Polling Time : 0.066580 
Polling Time : 0.066573 
Polling Time : 0.066569 
Polling Time : 0.066564 
Polling Time : 0.066583

But when I stream camera with gstreamer, it worked look like 30 fps

gst-launch-1.0 v4l2src device=/dev/video0 ! 'video/x-raw,width=1280,height=720,framerate=30/1,format=(string)UYVY' ! nvvidconv ! 'video/x-raw(memory:NVMM), width=(int)128, height=(int)72, format=(string)N
V12' ! nvoverlaysink -e

I’m wondering this is camera driver issue or I need to change any configuration of tegra_multimedia_api.

Thanks
Vu Nguyen

waynezhu · November 15, 2017, 7:37am

Vu Nguyen,
Sorry for late, are u use 1080p for test?

I don’t think it is camera dirver issue, but it may be caused by buffer copy in APP side.

It is better if you can do some tirage to know which module is the bottleneck.

forever3000 · November 16, 2017, 6:24am

Hi waynezhu,

I used only 720p for testing.

I don’t think it is caused by buffer copy in APP side because I did test with below command :

v4l2-ctl -d /dev/video1 --set-fmt-video=width=1920,height=1080, --set-ctrl bypass_mode=0 --stream-mmap --stream-count=600

then result still was 15fps.

I also checked my driver setting to confirm that v4l2 setting in driver was set to 30fps. I still didn’t figure out the reason.

Here is MIPI output

Thanks,
Vu Nguyen

waynezhu · November 16, 2017, 8:28am

Vu Nguyen,
Are you connecting to FPGA or sensor?

Can you change fps to 20, does it get 10 fps in commandline?

forever3000 · November 16, 2017, 8:57am

Hi waynezhu,

I’m connecting to sensor. I tried to test on TX1 and TX2, camera driver same for both but the result when I run this command is different

v4l2-ctl -d /dev/video1 --set-fmt-video=width=1920,height=1080, --set-ctrl bypass_mode=0 --stream-mmap --stream-count=600

TX1 return 30fps
TX2 return 15fps

So I think there is difference related v4l2 between TX1 and TX2. Please help me confirm v4l2 on TX2.

Thanks

DaneLLL · November 17, 2017, 2:02am

Hi Vu Nguyen,
Which sensor do you use?
So you see different framerate by using v4l2-ctl on TX1 and TX2?

forever3000 · November 17, 2017, 7:08am

Hi DaneLLL,

I’m using OV10635 image sensor (via Serializer/Deserializer). There is only one baseboard and I just replace TX1 and TX2 modules each other so I could confirm that it was not hardware issue.

Camera driver for TX1 and TX2 also almost same so I think the most different is kernel (3.10 vs 4.x).

The result return from this command

v4l2-ctl -d /dev/video1 --set-fmt-video=width=1920,height=1080, --set-ctrl bypass_mode=0 --stream-mmap --stream-count=600

as above post
TX1 return 30fps
TX2 return 15fps

Thanks
Vu Nguyen

forever3000 · November 20, 2017, 2:54am

Update

The result still was 15fps even though below command return 30fps

nvidia@tegra-ubuntu:~$ v4l2-ctl --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'UYVY'
	Name        : UYVY 4:2:2
		Size: Discrete 1280x720
			Interval: Discrete 0.033s (30.000 fps)

	Index       : 1
	Type        : Video Capture
	Pixel Format: 'NV16'
	Name        : Y/CbCr 4:2:2

	Index       : 2
	Type        : Video Capture
	Pixel Format: 'UYVY'
	Name        : UYVY 4:2:2
		Size: Discrete 1280x720
			Interval: Discrete 0.033s (30.000 fps)

JerryChang · November 20, 2017, 3:07am

hello forever3000,

please try below commands to boost the system performance (VI clock and ISP clock)

sudo ./jetson_clocks.sh
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate
echo max_rate > /sys/kernel/debug/bpmp/debug/clk/vi/rate
echo max_rate > /sys/kernel/debug/bpmp/debug/clk/isp/rate

forever3000 · November 20, 2017, 7:19am

JerryChang:

hello forever3000,

please try below commands to boost the system performance (VI clock and ISP clock)

sudo ./jetson_clocks.sh
echo 1 > /sys/kernel/debug/bpmp/debug/clk/vi/mrq_rate_locked
echo 1 > /sys/kernel/debug/bpmp/debug/clk/isp/mrq_rate_locked
cat /sys/kernel/debug/bpmp/debug/clk/vi/max_rate
cat /sys/kernel/debug/bpmp/debug/clk/isp/max_rate
echo max_rate > /sys/kernel/debug/bpmp/debug/clk/vi/rate
echo max_rate > /sys/kernel/debug/bpmp/debug/clk/isp/rate

Hi JerryChang,

Thanks. I did as you prompt but result still was same.

DaneLLL · November 23, 2017, 2:10am

Hi Vu,
Looks like 1920x1080p30 is not shown in ‘v4l2-ctl --list-formats-ext’. It shows 1280x720p30, is this mode working?

And for TX1, do you run r28.1?

forever3000 · November 23, 2017, 8:07am

Hi DaneLLL,

Yes my image sensor is 1280x720p30.
For TX1, I’m running r24.2. I will try r28.1 for TX1.

Thanks
Vu Nguyen

forever3000 · November 24, 2017, 5:49am

Hi DaneLLL,

I tried to test TX1 with 28.1 kernel. Driver was same because I just copied from TX2 and pasted to TX1, hardware baseboard also same just replace TX2 by TX1 module. And the result was:

TX1: return 30 fps
TX2: return 15 fps

Thanks

forever3000 · November 28, 2017, 5:39am

Hi DaneLLL,

Is there anyone at your side tried to test with YUV image sensor? I still couldn’t figure out why TX2 can run only at 15fps.

Thanks

WayneWWW · November 28, 2017, 6:48am

Hi forever3000,
Could you try command “v4l2-ctl --list-formats-ext -d /dev/video0” and paste the result?
We can achieve 72 fps when using 1920x1080 mode with our sensor.

nvidia@tegra-ubuntu:~$ v4l2-ctl --list-formats-ext -d /dev/video0 
ioctl: VIDIOC_ENUM_FMT
Index       : 0
Type        : Video Capture
Pixel Format: 'UYVY'
Name        : UYVY 4:2:2
Size: Discrete 4224x3156
Interval: Discrete 0.050s (20.000 fps)
Size: Discrete 3840x2160
Interval: Discrete 0.033s (30.000 fps)
Size: Discrete 1920x1080
Interval: Discrete 0.014s (72.000 fps)
Size: Discrete 1280x720
Interval: Discrete 0.014s (72.000 fps)
Size: Discrete 640x480
Interval: Discrete 0.017s (60.000 fps)

nvidia@tegra-ubuntu:~$ v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat="UYVY" --device=/dev/video0  --stream-count=300 --stream-mmap
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 72.27 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 72.13 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 72.09 fps
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< 72.06 fps
<<<<<<<<<<

forever3000 · November 28, 2017, 6:54am

Hi WayneWWW,

This is my result

nvidia@tegra-ubuntu:~$ v4l2-ctl --list-formats-ext -d /dev/video0
ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'UYVY'
	Name        : UYVY 4:2:2
		Size: Discrete 1280x720
			Interval: Discrete 0.033s (30.000 fps)

	Index       : 1
	Type        : Video Capture
	Pixel Format: 'NV16'
	Name        : Y/CbCr 4:2:2

nvidia@tegra-ubuntu:~$ v4l2-ctl --set-fmt-video=width=1280,height=720,pixelformat="UYVY" --device=/dev/video0 --stream-count=300 --stream-mmap
<<<<<<<<<<<<<<<<< 15.09 fps
<<<<<<<<<<<<<<< 15.04 fps
<<<<<<<<<<<<<<< 15.03 fps
<<<<<<<<<<<<<<< 15.02 fps
<<<<<<<<<<<<<<< 15.01 fps
<<<<<<<<<<<<<<< 15.01 fps
<<<<<<<<<<<<<<< 15.01 fps
<<<<<<<<<<<<<<< 15.01 fps
<<<<<<<<<<<<<<< 15.01 fps
<<<<<<<<<<<<<<< 15.00 fps
<<<<<<<<<<<<<<< 15.00 fps
<<<<<<<<<<<<<<< 15.00 fps
<<<<<<<<<<<<<<< 15.00 fps
<<<<<<<<<<<<<<< 15.00 fps
<<<<<<<<<<<<<<< 15.00 fps
<<<<<<<<<<<<<<< 15.00 fps
<<<<<<<<<<<<<<< 15.00 fps
<<<<<<<<<<<<<<< 15.00 fps
<<<<<<<<<<<<<<< 15.00 fps
<<<<<<<<<<<<<

Which camera was you running test?

Thanks
Vu Nguyen

DaneLLL · November 29, 2017, 2:27am

Hi Vu,
Please share your device tree for reference.

DaneLLL · November 29, 2017, 3:36am

Hi Vu,
Please also check if you have following change to support UYVY:

diff --git a/drivers/media/platform/tegra/camera/camera_common.c b/drivers/media/platform/tegra/camera/camera_common.c
index c0fabbd..cd43d34 100644
--- a/drivers/media/platform/tegra/camera/camera_common.c
+++ b/drivers/media/platform/tegra/camera/camera_common.c
@@ -34,6 +34,11 @@
 
 static const struct camera_common_colorfmt camera_common_color_fmts[] = {
 	{
+		MEDIA_BUS_FMT_UYVY8_1X16,
+		V4L2_COLORSPACE_SRGB,
+		V4L2_PIX_FMT_UYVY,
+	},
+	{
 		MEDIA_BUS_FMT_SRGGB12_1X12,
 		V4L2_COLORSPACE_SRGB,
 		V4L2_PIX_FMT_SRGGB12,
diff --git a/drivers/media/platform/tegra/camera/sensor_common.c b/drivers/media/platform/tegra/camera/sensor_common.c
index 62941cf..7bd9fbe 100644
--- a/drivers/media/platform/tegra/camera/sensor_common.c
+++ b/drivers/media/platform/tegra/camera/sensor_common.c
@@ -111,6 +111,8 @@ static int extract_pixel_format(
 		*format = V4L2_PIX_FMT_XBGGR10P;
 	else if (strncmp(pixel_t, "bayer_xrggb10p", size) == 0)
 		*format = V4L2_PIX_FMT_XRGGB10P;
+	else if (strncmp(pixel_t, "uyvy", size) == 0)
+		*format = V4L2_PIX_FMT_UYVY;
 	else {
 		pr_err("%s: Need to extend format%s\n", __func__, pixel_t);
 		return -EINVAL;
diff --git a/drivers/media/platform/tegra/camera/vi/vi4_formats.h b/drivers/media/platform/tegra/camera/vi/vi4_formats.h
index bbdbbab..c40854c 100644
--- a/drivers/media/platform/tegra/camera/vi/vi4_formats.h
+++ b/drivers/media/platform/tegra/camera/vi/vi4_formats.h
@@ -125,8 +125,6 @@ static const struct tegra_video_format vi4_video_formats[] = {
 	/* YUV422 */
 	TEGRA_VIDEO_FORMAT(YUV422, 16, UYVY8_1X16, 2, 1, T_U8_Y8__V8_Y8,
 				YUV422_8, UYVY, "YUV 4:2:2"),
-	TEGRA_VIDEO_FORMAT(YUV422, 16, UYVY8_1X16, 1, 1, T_Y8__V8U8_N422,
-				YUV422_8, NV16, "NV16"),
 	TEGRA_VIDEO_FORMAT(YUV422, 16, UYVY8_2X8, 2, 1, T_U8_Y8__V8_Y8,
 				YUV422_8, UYVY, "YUV 4:2:2 UYVY"),
 	TEGRA_VIDEO_FORMAT(YUV422, 16, VYUY8_2X8, 2, 1, T_T_V8_Y8__U8_Y8,

forever3000 · November 29, 2017, 7:16am

Hi DaneLLL,

Yes I already applied but result was still same.

DaneLLL · November 29, 2017, 8:04am

Hi Vu,
Please share your device tree for reference.

Topic		Replies	Views
V4L2 Drivers for Tegra X2 - v4l2 and nvcamerasrc Jetson TX2	16	9479	June 23, 2017
tx2 mipi virtual channel error. Jetson TX2	2	1746	October 18, 2021
4-lane CSI Camera Problems Jetson AGX Xavier	9	2660	October 18, 2021
Camera outputs to 1 buffer only Jetson TX1	59	5615	October 18, 2021
Camera streaming not working Jetson Orin Nano camera	10	137	July 11, 2024
Rtcpu_nvcsi_intr Jetson AGX Orin - ISL79987 MIPI CSI 2 Jetson AGX Orin device-tree , driver	14	1312	January 11, 2023
PXL_SOF syncpt timeout! err = -11 on TX2-4GB L4T 32.4.4 Jetson TX2 camera , fpga	7	1299	October 18, 2021
V4l2-ctl can't capture video data Jetson AGX Orin camera , board-design	44	3255	May 17, 2023
Timeout on /dev/video3 capture Jetson AGX Orin camera	9	464	February 1, 2024
SOF T/O errors from FPGA generated dual VC stream in JP4.2.1 on TX2 Jetson TX2	4	842	October 18, 2021

Tegra Multimedia APIs could get raw image via v4l2 only at 15fps

Related topics