camera => CUDA => h264 encoding

Hello, I can’t find any simple example out there to get frames from a camera, apply some CUDA treatment and encode the result in h264 before sending it to a file or the network, could someone give me one ?

You may use gstreamer for this with a pipeline like this: nvcamerasrc -> nvivafilter -> omxh264enc -> …

You need to get the source code of library for nvivafilter, build your own version where you’ll implement your CUDA processing, and use it with gstreamer plugin nvivafilter. You may also check: https://devtalk.nvidia.com/default/topic/1022543/jetson-tx2/gstreamer-nvmm-lt-gt-opencv-gpumat/post/5208232/#5208232

Pipeline for saving to file may be something like :

gst-launch-1.0 -ev nvcamerasrc ! 'video/x-raw(memory:NVMM), format=I420, framerate=30/1' ! nvivafilter cuda-process=true ! 'video/x-raw(memory:NVMM), format=RGBA'! nvvidconv ! omxh264enc ! h264parse ! mp4mux ! filesink location=testout.mp4

I don’t like gstreame because you can’t optimize the h264 encoder part. I need a low latency encoder with the possibility to change the bitrate from one frame to the other, and based on some data coming from a C library.

Why can’t we do that with a low level library like libargus ?

Argus is probably the way you’ll go…but I am not able to give good advices for this. Someone else would have to help further.

ok thanks anyway.

I will finally use gstreamer as it could be programmatically setup and in real time configured.
The real advantage of gstreamer is that it has been optimized for the GPU and all the hard part of buffer management and sharing has already been done.

Hello pcrochat42,

can you give more information how did you do it as I am doing something similar? How do you change bitrate/profile?

Best regards,
Matevz

hello mp12, I’m not arrived that far, but it should be as easy as changing the omxh264enc parameters
profile and bitrate

see my gst-inspect-1.0 omxh264enc output below :

please confirm on this post if that works for you

ubuntu@tegra-ubuntu:~$ gst-inspect-1.0 omxh264enc
Factory Details:
Rank primary + 10 (266)
Long-name OpenMAX H.264 Video Encoder
Klass Codec/Encoder/Video
Description Encode H.264 video streams
Author Sebastian Dröge sebastian.droege@collabora.co.uk

Plugin Details:
Name omx
Description GStreamer OpenMAX Plug-ins
Filename /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstomx.so
Version 1.0.0.1
License LGPL
Source module gst-omx
Source release date 2014-02-08
Binary package GStreamer source release
Origin URL Unknown package origin

GObject
±—GInitiallyUnowned
±—GstObject
±—GstElement
±—GstVideoEncoder
±—GstOMXVideoEnc
±—GstOMXH264Enc
±—GstOMXH264Enc-omxh264enc

Implemented Interfaces:
GstPreset

Pad Templates:
SRC template: ‘src’
Availability: Always
Capabilities:
video/x-h264
width: [ 16, 4096 ]
height: [ 16, 4096 ]
stream-format: { byte-stream, avc }
alignment: au

SINK template: ‘sink’
Availability: Always
Capabilities:
video/x-raw(memory:NVMM)
format: { I420, NV12 }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]
video/x-raw
format: { I420, NV12 }
width: [ 1, 2147483647 ]
height: [ 1, 2147483647 ]
framerate: [ 0/1, 2147483647/1 ]

Element Flags:
no flags set

Element Implementation:
Has change_state() function: gst_omx_video_enc_change_state

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
SINK: ‘sink’
Pad Template: ‘sink’
SRC: ‘src’
Pad Template: ‘src’

Element Properties:
name : The name of the object
flags: readable, writable
String. Default: “omxh264enc-omxh264enc0”
parent : The parent of the object
flags: readable, writable
Object of type “GstObject”
control-rate : Bitrate control method
flags: readable, writable, changeable only in NULL or READY state
Enum “GstOMXVideoEncControlRate” Default: 1, “variable”
(0): disable - Disable
(1): variable - Variable
(2): constant - Constant
(3): variable-skip-frames - Variable Skip Frames
(4): constant-skip-frames - Constant Skip Frames
bitrate : Target bitrate
flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state
Unsigned Integer. Range: 0 - 4294967295 Default: 4000000
quant-i-frames : Quantization parameter for I-frames (0xffffffff=component default)
flags: readable, writable, changeable only in NULL or READY state
Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295
quant-p-frames : Quantization parameter for P-frames (0xffffffff=component default)
flags: readable, writable, changeable only in NULL or READY state
Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295
quant-b-frames : Quantization parameter for B-frames (0xffffffff=component default)
flags: readable, writable, changeable only in NULL or READY state
Unsigned Integer. Range: 0 - 4294967295 Default: 4294967295
iframeinterval : Encoding Intra Frame occurance frequency
flags: readable, writable, changeable only in NULL or READY state
Unsigned Integer. Range: 0 - 4294967295 Default: 0
quality-level : Encoding quality-level
flags: readable, writable, changeable only in NULL or READY state
Unsigned Integer. Range: 0 - 2 Default: 0
SliceIntraRefreshEnable: Enable Slice Intra Refresh while encoding
flags: readable, writable, changeable only in NULL or READY state
Boolean. Default: false
SliceIntraRefreshInterval: Set SliceIntraRefreshInterval
flags: readable, writable, changeable only in NULL or READY state
Unsigned Integer. Range: 0 - 4294967295 Default: 60
bit-packetization : Whether or not Packet size is based upon Number Of bits
flags: readable, writable, changeable only in NULL or READY state
Boolean. Default: false
vbv-size : virtual buffer size = vbv-size * (bitrate/fps)
flags: readable, writable, changeable only in NULL or READY state
Unsigned Integer. Range: 0 - 30 Default: 10
temporal-tradeoff : Temporal Tradeoff value for encoder
flags: readable, writable, changeable only in NULL or READY state
Enum “GstOmxVideoEncTemporalTradeoffType” Default: 0, “Do not drop frames”
(0): Do not drop frames - GST_OMX_VIDENC_DROP_NO_FRAMES
(1): Drop 1 in 5 frames - GST_OMX_VIDENC_DROP_1_IN_5_FRAMES
(2): Drop 1 in 3 frames - GST_OMX_VIDENC_DROP_1_IN_3_FRAMES
(3): Drop 1 in 2 frames - GST_OMX_VIDENC_DROP_1_IN_2_FRAMES
(4): Drop 2 in 3 frames - GST_OMX_VIDENC_DROP_2_IN_3_FRAMES
EnableMVBufferMeta : Enable Motion Vector Meta data for encoding
flags: readable, writable, changeable only in NULL or READY state
Boolean. Default: false
qp-range : Qunatization range for P and I frame,
Use string with values of Qunatization Range
in MinQpP-MaxQpP:MinQpI-MaxQpP:MinQpB-MaxQpB order, to set the property.
flags: readable, writable
String. Default: “-1,-1:-1,-1:-1,-1”
EnableTwopassCBR : Enable two pass CBR while encoding
flags: readable, writable, changeable only in NULL or READY state
Boolean. Default: false
preset-level : HW preset level for encoder
flags: readable, writable, changeable only in NULL or READY state
Enum “GstOMXVideoEncHwPreset” Default: 0, “UltraFastPreset”
(0): UltraFastPreset - UltraFastPreset for high perf
(1): FastPreset - FastPreset
(2): MediumPreset - MediumPreset
(3): SlowPreset - SlowPreset
insert-sps-pps : Insert H.264 SPS, PPS at every IDR frame
flags: readable, writable
Boolean. Default: false
num-B-Frames : Number of B Frames between two reference frames (not recommended)
flags: readable, writable, changeable only in NULL or READY state
Unsigned Integer. Range: 0 - 2 Default: 0
slice-header-spacing: Slice Header Spacing number of macroblocks/bits in one packet
flags: readable, writable, changeable only in NULL or READY state
Unsigned Long. Range: 0 - 18446744073709551615 Default: 0
profile : Set profile for encode
flags: readable, writable, changeable only in NULL or READY state
Enum “GstOmxVideoEncProfileType” Default: 1, “baseline”
(1): baseline - GST_OMX_VIDENC_BASELINE_PROFILE
(2): main - GST_OMX_VIDENC_MAIN_PROFILE
(8): high - GST_OMX_VIDENC_HIGH_PROFILE

Element Actions:
“force-IDR” : void user_function (GstElement* object);

Hi pcrochat42,

You may found interesting the following information about the GstCUDA framework, I think that is exactly what you are looking for. Below you will find a more detailed description, but in summary, it consists of a framework that allows to easily and optimally interface GStreamer with CUDA, guaranteeing zero memory copies. It also supports several inputs.

GstCUDA is a RidgeRun developed GStreamer plug-in enabling easy CUDA algorithm integration into GStreamer pipelines. GstCUDA offers a framework that allows users to develop custom GStreamer elements that execute any CUDA algorithm. The GstCUDA framework is a series of base classes abstracting the complexity of both CUDA and GStreamer. With GstCUDA, developers avoid writing elements from scratch, allowing the developer to focus on the algorithm logic, thus accelerating time to market.

GstCUDA offers a GStreamer plugin that contains a set of elements, that are ideal for GStreamer/CUDA quick prototyping. Those elements consist in a set of filters with different input/output pads combinations, that are run-time loadable with an external custom CUDA library that contains the algorithm to be executed on the GPU on each video frame that passes through the pipeline. GstCUDA plugin allows users to develop their own CUDA processing library, pass the library into the GstCUDA filter element that best adapts to the algorithm requirements, executes the library on the GPU, passing upstream frames from the GStreamer pipeline to the GPU and passing the modified frames downstream to the next element in the GStreamer pipeline. Those elements were created with the CUDA algorithm developer in mind - supporting quick prototyping and abstracting all GStreamer concepts. The elements are fully adaptable to different project needs, making GstCUDA a powerful tool that is essential for CUDA/GStreamer project development.

One remarkable feature of GstCUDA is that it provides a zero memory copy interface between CUDA and GStreamer on Jetson TX1/TX2 platforms. This enables heavy algorithms and large amounts of data (up to 2x 4K 60fps streams) to be processed on CUDA without the performance caused by copies or memory conversions. GstCUDA provides the necessary APIs to directly handle NVMM buffers to achieve the best possible performance on Jetson TX1/TX2 platforms. It provides a series of base classes and utilities that abstract the complexity of handle memory interface between GStreamer and CUDA, so the developer can focus on what actually gives value to the end product. GstCuda ensures an optimal performance for GStreamer/CUDA applications on Jetson platforms.

You can find detailed information and examples about GstCUDA on the following link:
http://developer.ridgerun.com/wiki/index.php?title=GstCUDA

I hope this information can be useful to you.

Best regards,
-Daniel