12_camera_v4l2_cuda works, but only after "warming up" with gstreamer

Hello!

I’m testing the 12_camera_v4l2_cuda sample. After compiling and running, I get a few seconds of a black screen and then the application quits with no frames displayed:

./12_camera_v4l2_cuda -d /dev/video0 -s 1920x1080 -f UYVY -v
INFO: camera_initialize(): (line:303) Camera ouput format: (1920 x 1080)  stride: 3840, imagesize: 4147200, frate: 0 / 0
[INFO] (NvEglRenderer.cpp:110) <renderer0> Setting Screen width 1920 height 1080
INFO: init_components(): (line:342) Initialize v4l2 components successfully
WARN: request_camera_buff(): (line:380) Camera v4l2 buf length is not expected
WARN: request_camera_buff(): (line:380) Camera v4l2 buf length is not expected
WARN: request_camera_buff(): (line:380) Camera v4l2 buf length is not expected
WARN: request_camera_buff(): (line:380) Camera v4l2 buf length is not expected
INFO: prepare_buffers(): (line:527) Succeed in preparing stream buffers
INFO: start_stream(): (line:544) Camera video streaming on ...
----------- Element = renderer0 -----------
Total Profiling time = 0
Average FPS = 0
Total units processed = 0
Num. of late units = 0
-------------------------------------
INFO: stop_stream(): (line:710) Camera video streaming off ...
App run was successful

Running this gstreamer pipeline successfully shows video output:

gst-launch-1.0 -vvvvv \
  v4l2src device=/dev/video0 ! 'video/x-raw, format=UYVY, width=1920, height=1080' ! \
  nvvidconv flip-method=1 ! 'video/x-raw(memory:NVMM), format=I420, width=480, height=640' ! \
  nvoverlaysink sync=false

Strangely, if I run 12_camera_v4l2_cuda again after running gstreamer, the sample app works as expected and I can view video:

./12_camera_v4l2_cuda -d /dev/video0 -s 1920x1080 -f UYVY -v
INFO: camera_initialize(): (line:303) Camera ouput format: (1920 x 1080)  stride: 3840, imagesize: 4147200, frate: 0 / 0
[INFO] (NvEglRenderer.cpp:110) <renderer0> Setting Screen width 1920 height 1080
INFO: init_components(): (line:342) Initialize v4l2 components successfully
WARN: request_camera_buff(): (line:380) Camera v4l2 buf length is not expected
WARN: request_camera_buff(): (line:380) Camera v4l2 buf length is not expected
WARN: request_camera_buff(): (line:380) Camera v4l2 buf length is not expected
WARN: request_camera_buff(): (line:380) Camera v4l2 buf length is not expected
INFO: prepare_buffers(): (line:527) Succeed in preparing stream buffers
INFO: start_stream(): (line:544) Camera video streaming on ...
^CQuit due to exit command from user!
----------- Element = renderer0 -----------
Total Profiling time = 10.0577
Average FPS = 30.0269
Total units processed = 303
Num. of late units = 0
-------------------------------------
INFO: stop_stream(): (line:710) Camera video streaming off ...
App run was successful

At this point, 12_camera_v4l2_cuda will continue working fine until I restart the device, which brings it back to the original no-frames behavior and requires another “warm up” with gstreamer.

Any ideas what might be happening here?

thank you!

For context, this is using a TC358743 HDMI-to-CSI2 bridge configured using a slightly modified driver (ref A, ref B)

Hi,
For information, please provide your release version( $ head -1 /etc/nv_tegra_release ).

head -1 /etc/nv_tegra_release 
# R32 (release), REVISION: 5.1, GCID: 27362550, BOARD: t210ref, EABI: aarch64, DATE: Wed May 19 18:07:59 UTC 2021

thank you!

Hi,
It looks like the sensor mode is not set correctly. In default sample it does not set framerate. Please add the code for a try:

@@ -293,7 +293,12 @@ camera_initialize(context_t * ctx)
     memset (&streamparm, 0x00, sizeof (struct v4l2_streamparm));
     streamparm.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
     ioctl (ctx->cam_fd, VIDIOC_G_PARM, &streamparm);
-
+{
+    streamparm.parm.capture.timeperframe.denominator = 30;
+    streamparm.parm.capture.timeperframe.numerator = 1;
+    ioctl (ctx->cam_fd, VIDIOC_S_PARM, &streamparm);
+    ioctl (ctx->cam_fd, VIDIOC_G_PARM, &streamparm);
+}
     INFO("Camera ouput format: (%d x %d)  stride: %d, imagesize: %d, frate: %u / %u",
             fmt.fmt.pix.width,
             fmt.fmt.pix.height,

Hi, thank you for the idea.

I tried that and while it now prints the frame rate correctly (frate: 30 / 1), it still has the same behavior otherwise:

./12_camera_v4l2_cuda -d /dev/video0 -s 1920x1080 -f UYVY -v
INFO: camera_initialize(): (line:286) Camera ouput format: (1920 x 1080)  stride: 3840, imagesize: 4147200, frate: 30 / 1
[INFO] (NvEglRenderer.cpp:110) <renderer0> Setting Screen width 1920 height 1080
INFO: init_components(): (line:318) Initialize v4l2 components successfully
WARN: request_camera_buff(): (line:351) Camera v4l2 buf length is not expected
WARN: request_camera_buff(): (line:351) Camera v4l2 buf length is not expected
WARN: request_camera_buff(): (line:351) Camera v4l2 buf length is not expected
WARN: request_camera_buff(): (line:351) Camera v4l2 buf length is not expected
INFO: prepare_buffers(): (line:488) Succeed in preparing stream buffers
INFO: start_stream(): (line:502) Camera video streaming on ...
----------- Element = renderer0 -----------
Total Profiling time = 0
Average FPS = 0
Total units processed = 0
Num. of late units = 0
-------------------------------------
INFO: stop_stream(): (line:653) Camera video streaming off ...
App run was successful

Any other ideas? Thank you!

Hi,
Please run

$ v4l2-ctl --list-formats-ext

and share information for reference. This generally happens when wrong sensor mode is set. Probably format, width, height, or framerate is not set correctly.

Hi, that sounds plausible. The kernel driver for the TC358743 chip was hacked together and hard-coded with some format information including resolution, framerate, EDID info, etc. It appears this information is what gets reported by v4l2-ctl --list-formats-ext:

v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'AR24'
	Name        : 32-bit BGRA 8-8-8-8

	Index       : 1
	Type        : Video Capture
	Pixel Format: 'UYVY'
	Name        : UYVY 4:2:2
		Size: Discrete 1920x1080
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete infs (0.000 fps)
			Interval: Discrete 0.000s (892560244.000 fps)

	Index       : 2
	Type        : Video Capture
	Pixel Format: 'NV16'
	Name        : Y/CbCr 4:2:2
		Size: Discrete 1920x1080
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete infs (0.000 fps)
			Interval: Discrete 0.000s (892560244.000 fps)

On the other hand, running a basic (working) gstreamer pipeline shows:

gst-launch-1.0 -vvvvv v4l2src device=/dev/video0 ! videoconvert !  xvimagesink          
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
/GstPipeline:pipeline0/GstV4l2Src:v4l2src0.GstPad:src: caps = video/x-raw, format=(string)UYVY, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, colorimetry=(string)2:3:5:4, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:src: caps = video/x-raw, format=(string)UYVY, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, colorimetry=(string)2:3:5:4, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstXvImageSink:xvimagesink0.GstPad:sink: caps = video/x-raw, format=(string)UYVY, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, colorimetry=(string)2:3:5:4, interlace-mode=(string)progressive
/GstPipeline:pipeline0/GstVideoConvert:videoconvert0.GstPad:sink: caps = video/x-raw, format=(string)UYVY, width=(int)1920, height=(int)1080, framerate=(fraction)30/1, colorimetry=(string)2:3:5:4, interlace-mode=(string)progressive
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:09.322889202
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

The main discrepancy appears to be the colorimetry string: 2:3:5:4 negotiated and 4:2:2 reported in the listed format names.

Do you think that incorrect colorimetry info from the kernel module could be causing this issue, and any guidance on how to fix that either in the driver code or the example code? (I’m fairly new to all of this, so still learning the basics).

Thank you!

Hi,
It looks like certain parameters do not have default value and would need to be set in first time. But not sure which parameter is. We have tried with USB cameras but do not observe the issue. Probably you may run the gstreamer pipeline as a quick fix. Please try to capture only one frame and then run 12_camera_v4l2_cuda:

gst-launch-1.0  v4l2src device=/dev/video0 num-buffers=1 ! 'video/x-raw, format=UYVY, width=1920, height=1080' ! fakesink

Appreciate the guidance. I’ll continue exploring this next week.