Manage performance to capture videos

TimoK · June 14, 2018, 12:09pm

Hi devs,

i’ve a little question and maybe someone of you has the answer!

My challenge:
I want to process and save videos with my webcam (1920x1080 30fps). First of all, i Just want to save them on my SSD. I am using the ROS System with some nodes to solve the challenge.

My Setup:

Jetson TX2 developer kit
JetPack-L4T-3.2 (28.2)
USB Cam: Logitech BRIO https://www.logitech.com/de-de/product/brio
Ubuntu 16.04 with ROS kinetic
Intenso 512GB SSD disk on SATA connector

Read out the frames is done with OpenCV3.4.1 with gstreamer support in a ROS node.

Writing the video is done with a cv::VideoWriter() in a ROS node.

My Problem:
I am able to read the frames from the camera with 30 fps. I am able to write 30 fps to file. But if i’am trying to read and write, the performance is going down to ~23fps (read) and ~27fps(write).

When i am using

sudo ~ubuntu/jetson_clocks.sh

the rates are going up to ~25fps (read) and ~30fps(write).

My questions:

Is Jetson able to read and write FHD videos 30 fps?
How can i teach Jetson to work with maximum availlable performance automatically?

Thanks for all answers!

EDIT:
One addition: If i am compressing the video to mp4 (maybe with codec h264), how can i enable GPU support?

sudo ~ubuntu/jetson_clocks.sh --show

says that the GPU is working with minimum performance.

Honey_Patouceul · June 14, 2018, 7:38pm

You may try to see resources usage when you run your app with this command in another shell:

sudo /home/nvidia/tegrastats

You should see where is the bootleneck if any.
You may also ensure you are using all cores in performance mode with:

sudo nvpmodel -m0

TimoK · June 14, 2018, 8:31pm

Thanks!

From now i am using

sudo nvpmodel -m0

Here the output from tegrastats:

nvidia@tegra-ubuntu:~/ros_catkin_ws/src$ sudo /home/nvidia/tegrastats
RAM 1545/7846MB (lfb 762x4MB) CPU [76%@1984,40%@1728,56%@1727,69%@1983,67%@1985,70%@1988] EMC_FREQ 10%@1866 GR3D_FREQ 0%@140 APE 150 MTS fg 0% bg 0% GPU@43C MCPU@45C Tdiode@43.25C PLL@45C PMIC@100C BCPU@45C Tboard@40C thermal@44.5C VDD_IN 7386/7280 VDD_CPU 3291/3195 VDD_GPU 153/153 VDD_SOC 1071/1071 VDD_WIFI 115/137 VDD_DDR 1852/1844
RAM 1545/7846MB (lfb 762x4MB) CPU [79%@2030,47%@806,52%@806,68%@2031,68%@2030,64%@2031] EMC_FREQ 10%@1866 GR3D_FREQ 0%@140 APE 150 MTS fg 0% bg 0% GPU@43C MCPU@45.5C Tdiode@43.25C PLL@45.5C PMIC@100C BCPU@45.5C Tboard@40C thermal@44.5C VDD_IN 7386/7292 VDD_CPU 3366/3214 VDD_GPU 153/153 VDD_SOC 1071/1071 VDD_WIFI 115/134 VDD_DDR 1852/1845
RAM 1560/7846MB (lfb 762x4MB) CPU [80%@2028,43%@2036,44%@2035,71%@2030,72%@2030,72%@2029] EMC_FREQ 10%@1866 GR3D_FREQ 0%@140 APE 150 MTS fg 0% bg 2% GPU@43C MCPU@45.5C Tdiode@43.25C PLL@45.5C PMIC@100C BCPU@45.5C Tboard@40C thermal@44.5C VDD_IN 7195/7282 VDD_CPU 3138/3206 VDD_GPU 153/153 VDD_SOC 1071/1071 VDD_WIFI 115/132 VDD_DDR 1833/1844
RAM 1553/7846MB (lfb 762x4MB) CPU [80%@2032,50%@806,37%@806,74%@2031,66%@2030,70%@2030] EMC_FREQ 10%@1866 GR3D_FREQ 0%@140 APE 150 MTS fg 0% bg 0% GPU@43C MCPU@45.5C Tdiode@43.25C PLL@45.5C PMIC@100C BCPU@45.5C Tboard@40C thermal@44.5C VDD_IN 7156/7271 VDD_CPU 3138/3200 VDD_GPU 153/153 VDD_SOC 1071/1071 VDD_WIFI 115/131 VDD_DDR 1852/1845
RAM 1520/7846MB (lfb 762x4MB) CPU [77%@2029,53%@1420,46%@1421,70%@2028,67%@2030,70%@2028] EMC_FREQ 10%@1866 GR3D_FREQ 0%@140 APE 150 MTS fg 0% bg 1% GPU@43.5C MCPU@46C Tdiode@43.5C PLL@46C PMIC@100C BCPU@46C Tboard@40C thermal@45C VDD_IN 7424/7283 VDD_CPU 3366/3214 VDD_GPU 153/153 VDD_SOC 1071/1071 VDD_WIFI 115/129 VDD_DDR 1871/1847

In my opinion the CPU is not on top performance…
Can i see if the GPU is working?

Do you know how i can enable GPU support for OpenCV VideoWriter?

Honey_Patouceul · June 14, 2018, 10:10pm

I don’t see any resource outage from this log.
TX2 is able to capture and record 30 fps:

<b>gst-launch-1.0 -e nvcamerasrc num-buffers=300 ! 'video/x-raw(memory:NVMM),framerate=30/1' ! omxh264enc ! h264parse ! matroskamux ! filesink location=test.mkv</b>
Setting pipeline to PAUSED ...

Available Sensor modes : 
2592 x 1944 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
2592 x 1458 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1280 x 720 FR=120.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
Pipeline is live and does not need PREROLL ...

NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 1 WxH = 2592x1458 FrameRate = 30.000000 ...

Setting pipeline to PLAYING ...
New clock: GstSystemClock
Framerate set to : 30 at NvxVideoEncoderSetParameterNvMMLiteOpen : Block : BlockType = 4 
===== MSENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
===== MSENC blits (mode: 1) into tiled surfaces =====
Got EOS from element "pipeline0".
Execution ended after 0:00:10.222439458
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

can be played back with:

gst-launch-1.0 filesrc location=./test.mkv ! matroskademux ! h264parse ! omxh264dec ! nvvidconv ! xvimagesinkSetting pipeline to PAUSED ...
Pipeline is PREROLLING ...
NvMMLiteOpen : Block : BlockType = 261 
TVMR: NvMMLiteTVMRDecBlockOpen: 7650: NvMMLiteBlockOpen 
NvMMLiteBlockCreate : Block : BlockType = 261 
TVMR: cbBeginSequence: 1179: BeginSequence  640x480, bVPR = 0
TVMR: LowCorner Frequency = 100000 
TVMR: cbBeginSequence: 1529: DecodeBuffers = 17, pnvsi->eCodec = 4, codec = 0 
TVMR: cbBeginSequence: 1600: Display Resolution : (640x480) 
TVMR: cbBeginSequence: 1601: Display Aspect Ratio : (640x480) 
TVMR: cbBeginSequence: 1669: ColorFormat : 5 
TVMR: cbBeginSequence:1683 ColorSpace = NvColorSpace_YCbCr601
TVMR: cbBeginSequence: 1809: SurfaceLayout = 3
TVMR: cbBeginSequence: 1902: NumOfSurfaces = 24, InteraceStream = 0, InterlaceEnabled = 0, bSecure = 0, MVC = 0 Semiplanar = 1, bReinit = 1, BitDepthForSurface = 8 LumaBitDepth = 8, ChromaBitDepth = 8, ChromaFormat = 5
TVMR: cbBeginSequence: 1904: BeginSequence  ColorPrimaries = 2, TransferCharacteristics = 2, MatrixCoefficients = 2
Allocating new output: 640x480 (x 24), ThumbnailMode = 0
OPENMAX: HandleNewStreamFormat: 3464: Send OMX_EventPortSettingsChanged : nFrameWidth = 640, nFrameHeight = 480 
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
TVMR: FrameRate = 90 
TVMR: NVDEC LowCorner Freq = (300000 * 1024) 
---> TVMR: Video-conferencing detected !!!!!!!!!
TVMR: FrameRate = 30.173558 
TVMR: FrameRate = 30.000030 
TVMR: NvMMLiteTVMRDecDoWork: 6534: NVMMLITE_TVMR: EOS detected
TVMR: TVMRBufferProcessing: 5489: Processing of EOS 
TVMR: TVMRBufferProcessing: 5566: Processing of EOS Done
Got EOS from element "pipeline0".
Execution ended after 0:00:09.976340504
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
TVMR: TVMRFrameStatusReporting: 6135: Closing TVMR Frame Status Thread -------------
TVMR: TVMRVPRFloorSizeSettingThread: 5945: Closing TVMRVPRFloorSizeSettingThread -------------
TVMR: TVMRFrameDelivery: 5985: Closing TVMR Frame Delivery Thread -------------
TVMR: NvMMLiteTVMRDecBlockClose: 7818: Done 
Setting pipeline to NULL ...
Freeing pipeline ...

If this doesn’t work, you check what bandwidth is available on your SATA SSD…For example I have:

sudo hdparm -tT /dev/sda1 
/dev/sda1:
 Timing cached reads:   6772 MB in  2.00 seconds = 3393.14 MB/sec
 Timing buffered disk reads: 754 MB in  3.00 seconds = 251.14 MB/sec

Otherwise, and more likely, problem may be with your gstreamer pipeline for capture or videoWriter. You may share more details about these.

DaneLLL · June 15, 2018, 1:56am

Starting from OpenCV 3.3, it supports NV12/I420 in appsink. If you sends BGR to appsink, you may try to run NV12/I420 and check if it helps.

We have seen better performance in nvcamerasrc:
[url]https://devtalk.nvidia.com/default/topic/1024245/jetson-tx2/opencv-3-3-and-integrated-camera-problems-/post/5210735/#5210735[/url]

But don’t know if it can benefit v4l2src.

TimoK · June 15, 2018, 7:53am

I tried NV12:

VIDEOIO ERROR: V4L: device v4l2src device=/dev/video1 ! video/x-raw, format=NV12, width=1920, height=1080 ! appsink: Unable to query number of channels
OpenCV Error: Unspecified error (GStreamer: unable to start pipeline) in cvCaptureFromCAM_GStreamer

I dont know why but the pipeline could not be started

TimoK · June 15, 2018, 7:58am

I don’t see any resource outage from this log.
TX2 is able to capture and record 30 fps:
<b>gst-launch-1.0 -e nvcamerasrc num-buffers=300 ! 'video/x-raw(memory:NVMM),framerate=30/1' ! omxh264enc ! h264parse ! matroskamux ! filesink location=test.mkv</b>

I got this caps in OpenCV with GStreamer support for my Logitech Webcam

v4l2src device=/dev/video1 ! video/x-raw,format=YUY2,width=640,height=480 ! appsink

Can you show me how to write a caps for using this camera for writing into file like in your reply?

The SATA stats:

nvidia@tegra-ubuntu:/media/vol0/videos$ sudo hdparm -tT /dev/sda1 

/dev/sda1:
 Timing cached reads:   4950 MB in  2.00 seconds = 2478.75 MB/sec
 Timing buffered disk reads: 592 MB in  3.00 seconds = 197.30 MB/sec

TimoK · June 15, 2018, 8:44am

Okay i will describe all relevant issues of my program(s):

Video input:
I’m using a Logitech Brio with this formats:

v4l2-ctl -d /dev/video1 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'YUYV'
	Name        : YUYV 4:2:2
		Size: Discrete 640x480
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 160x120
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 176x144
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 320x180
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 320x240
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 352x288
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 340x340
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 424x240
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 440x440
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 480x270
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 640x360
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 800x448
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 800x600
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 848x480
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 960x540
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 1024x576
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 1600x896
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)

	Index       : 1
	Type        : Video Capture
	Pixel Format: 'MJPG' (compressed)
	Name        : Motion-JPEG
		Size: Discrete 640x480
			Interval: Discrete 0.008s (120.000 fps)
			Interval: Discrete 0.011s (90.000 fps)
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 160x120
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 176x144
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 320x180
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 320x240
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 352x288
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 424x240
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 480x270
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 640x360
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 800x448
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 800x600
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 848x480
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 960x540
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 1024x576
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.011s (90.000 fps)
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 1600x896
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.017s (60.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 2560x1440
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 3840x2160
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 4096x2160
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)

	Index       : 2
	Type        : Video Capture
	Pixel Format: 'NV12'
	Name        : Y/CbCr 4:2:0
		Size: Discrete 640x480
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 640x360
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.033s (30.000 fps)
			Interval: Discrete 0.042s (24.000 fps)
			Interval: Discrete 0.050s (20.000 fps)
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.100s (10.000 fps)
			Interval: Discrete 0.133s (7.500 fps)
			Interval: Discrete 0.200s (5.000 fps)

To grab the video frames i pogrammed a ROS node. In it a OpenCV VideoCapture with GStreamer support is used. The code is

std::string caps = "v4l2src device=/dev/video1 ! video/x-raw,format=YUY2,width=640,height=480 ! appsink"
cv::VideoCapture *vidCap_ = new cv::VideoCapture()
vidCap_->open(caps);

Then in a loop (triggered by a ROS timer) the capture is red out

/* Read out frame */
  cv::Mat frame, frame_conv;
  cv_bridge::CvImage cv_image;
  *vidCap_ >> frame; // Read frame from stream
  /* Convert to RGB */
  cv::cvtColor(frame,
               frame_conv,
               cv::COLOR_YUV2BGR_YUY2);
  cv_image = cv_bridge::CvImage(header, "bgr8", frame_conv);
 
  /* Publish image */
  image_pub_.publish(cv_image.toImageMsg());

I think a problem could be the cv::cvtColor() function. This is needed to use cv::VideoWriter, because OpenCV is only able to write 3 channel frames. I think the image_pub_.publish(cv_image.toImageMsg()) is not killing performance, because cv_bridge just sends a shared_ptr into the ROS System.

Write the video:

The writing is realized in an other ROS node. The writer is a standart OpenCV VideoWriter

writer = new cv::VideoWriter();
writer->open(filename, // Filename
             "MJPG", // Codec
             fps_, // FPS
             size); // Framesize

Every incoming frame is handled by a routine like this:

cv_ptr_color_ = cv_bridge::toCvCopy(image, sensor_msgs::image_encodings::BGR8);
writer->write(this->cv_ptr_color_->image);

I hope my programming is not killing the whole performance…

DaneLLL · June 15, 2018, 8:49am

Hi,
nvcamerasrc is for onboard camera ov5693. for USB camera, it uses v4l2src.

Your case requires high CPU loading. As @Honey_Patoucuel suggests, you can try to configure nvpmodel and run jetson_clocks.sh. If it still does not bring you enough performance, it probably is the system limitation.

We don’t work on ROS so not sure what performance it can achieve. Other users may share their experience on ROS.

Honey_Patouceul · June 15, 2018, 11:31am

The following code uses gstreamer pipelines for capture and saving to mkv in h264.
It is not optimal, but should show that capturing and encoding and writing works on Jetson.
It works with ZED camera at 2560x720@60fps without jetson_clocks.sh.
You may just adjust size and framerate for your brio cam.

#include <iostream>

#include <opencv2/opencv.hpp>
#include <opencv2/videoio.hpp>
#include <opencv2/highgui.hpp>
[s]#include <opencv2/cudaimgproc.hpp>
[/s]
int main()
{
    //printf("%s\n", cv::getBuildInformation().c_str()); 
	
 
    const char* gst_capture = "v4l2src device=/dev/video1 ! video/x-raw, width=2560, height=720, format=YUY2, framerate=60/1 ! "
			      "videoconvert ! video/x-raw, width=2560, height=720, format=BGR ! "
			      "appsink";

    const char*  gst_writer = "appsrc ! video/x-raw, format=(string)BGR ! \
			      videoconvert ! video/x-raw, format=(string)I420 ! \
 		              nvvidconv ! video/x-raw(memory:NVMM) ! \
			      omxh264enc ! h264parse ! matroskamux ! filesink location=test.mkv"; 

    //setenv("GST_DEBUG", "*:3", 0); 

    cv::VideoCapture cap(gst_capture);
    if(!cap.isOpened()) {
	std::cout<<"Failed to open camera."<<std::endl;
	return (-1);
    }
    
    unsigned int width  = cap.get(CV_CAP_PROP_FRAME_WIDTH); 
    unsigned int height = cap.get(CV_CAP_PROP_FRAME_HEIGHT); 
    unsigned int fps    = cap.get(CV_CAP_PROP_FPS);
    unsigned int pixels = width*height;
    std::cout <<" Frame size : "<<width<<" x "<<height<<", "<<pixels<<" Pixels "<<fps<<" FPS"<<std::endl;

    cv::VideoWriter writer(gst_writer, cap.get(CV_CAP_PROP_FOURCC), fps, cv::Size(width, height));
    if (!writer.isOpened()) {
		std::cout<<"Failed to open writer."<<std::endl;
		return (-2);
    }
    
    cv::namedWindow("MyCameraPreview", CV_WINDOW_AUTOSIZE);
    cv::Mat frame_in, frame_out;
    
    while(1)
    {
	if (!cap.read(frame_in)) {
		std::cout<<"Capture read error"<<std::endl;
		break;
	}
	else  {
		frame_in.copyTo(frame_out);
		cv::imshow("MyCameraPreview",frame_in);
		writer.write(frame_out);
		cv::waitKey(1); // let imshow draw 
	}	
    }

    cap.release();
    writer.release();

    return 0;
}

TimoK · June 15, 2018, 11:38am

The following code uses gstreamer pipelines for capture and saving to mkv in h264.
It is not optimal, but should show that capturing and encoding and writing works on Jetson.
It works with ZED camera at 2560x720@60fps without jetson_clocks.sh.
You may just adjust size and framerate for your brio cam.

#include <iostream>

#include <opencv2/opencv.hpp>
#include <opencv2/videoio.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/cudaimgproc.hpp>




int main()
{
    //printf("%s\n", cv::getBuildInformation().c_str()); 
	
 
    const char* capture = "v4l2src device=/dev/video1 ! video/x-raw, width=2560, height=720, format=YUY2, framerate=60/1 ! "
			"videoconvert ! video/x-raw, width=2560, height=720, format=BGR ! "
			"appsink";

    const char*  gst_writer = "appsrc ! video/x-raw, format=(string)BGR ! \
			videoconvert ! video/x-raw, format=(string)I420 ! \
 		        nvvidconv ! video/x-raw(memory:NVMM) ! \
			omxh264enc ! h264parse ! matroskamux ! filesink location=test.mkv"; 

    //setenv("GST_DEBUG", "*:3", 0); 

    cv::VideoCapture cap(capture);
    if(!cap.isOpened()) {
	std::cout<<"Failed to open camera."<<std::endl;
	return (-1);
    }
    
    unsigned int width  = cap.get(CV_CAP_PROP_FRAME_WIDTH); 
    unsigned int height = cap.get(CV_CAP_PROP_FRAME_HEIGHT); 
    unsigned int fps    = cap.get(CV_CAP_PROP_FPS);
    unsigned int pixels = width*height;
    std::cout <<" Frame size : "<<width<<" x "<<height<<", "<<pixels<<" Pixels "<<fps<<" FPS"<<std::endl;

    cv::VideoWriter writer(gst_writer, cap.get(CV_CAP_PROP_FOURCC), fps, cv::Size(width, height));
    if (!writer.isOpened()) {
		std::cout<<"Failed to open writer."<<std::endl;
		return (-2);
    }
    
    cv::namedWindow("MyCameraPreview", CV_WINDOW_AUTOSIZE);
    cv::Mat frame_in, frame_out;
    
    while(1)
    {
	if (!cap.read(frame_in)) {
		std::cout<<"Capture read error"<<std::endl;
		break;
	}
	else  {
			frame_in.copyTo(frame_out);
			cv::imshow("MyCameraPreview",frame_in);
			writer.write(frame_out);
			cv::waitKey(1); // let imshow draw 
		}	
    }

    cap.release();
    writer.release();

    return 0;

}

Thank you very much! I will try it as fast as possible. But one question before: is there a special reason to import “opencv2/cudaimgproc.hpp” ? Im asking because i have an other problem with including OpenCV with cuda enabled to ROS compiler, but thats an other issue…

TimoK · June 15, 2018, 12:26pm

I implemented your code snippet to my ROS implementation:
Using GStreamer in OpenCV works quite well.

I think i can close this thread!

Thank you!

Honey_Patouceul · June 15, 2018, 2:28pm

No, it should not be there…it is just because I made this example from another project using gpu conversion, and I forgot to remove it. Sorry for confusion.

Topic		Replies	Views
The web camera does not work on jetson nano Jetson Nano camera , opencv , gstreamer	9	1899	October 15, 2021
Jetson TX2 + Logitech Brio video capture size + detectNet Jetson TX2	12	1482	October 18, 2021
Video Broadcasting - Latency/Lag reduction Jetson Nano gstreamer	16	1962	October 18, 2021
Logitech Brio USB cam 'select timeout' while capturing Jetson TX2	18	9742	October 18, 2021
Jetson nano Python Deepstream test-3 application for multi-USB camera inferance pipeline Low FPS Issue DeepStream SDK	13	848	October 12, 2021
Python Deepstream 2 USB camera inferance pipeline DeepStream SDK camera , gstreamer	15	678	October 12, 2021
connected more than two usb cameras problem on deepstream-app (Jetson Nano Dev Kit) DeepStream SDK	18	7461	October 12, 2021
How to force Deepstream on Xavier to achieve USB Camera Resolution at 1080P DeepStream SDK	5	309	November 9, 2021
Jetson nano rtsp stream failed with intel realsense 415 usb camera DeepStream SDK	35	5244	October 12, 2021
[JETPACK 4.2] Jetson TX2 with Logitec C920 Camera - ERROR Jetson TX2	4	629	October 18, 2021

Manage performance to capture videos

Related topics