Building a GStreamer Pipeline

[Honey_Patouceul]

Probably the thing you would have to know is that most of the doc for a plugin is embedded into it.
A gstreamer command named gst-inspect allow to list all available plugins:

gst-inspect-1.0 --version   # Would display gstreamer version
gst-inspect-1.0             # Would list all available plugins and more

You can use it to inspect a plugin:

gst-inspect-1.0 nvarguscamerasrc

It would display the library providing it, then its SRC and SINK capabilities, that are different types of multimedia data, formats,… So you would only be able to use a plugin giving one of its listed available types/formats in SINK as input. These CAPS should also match the previous plugin SRC (output) CAPS.
So basically a pipeline goes like this:

SrcPlugin -> CAPS1 -> ProcessPlugin1 -> CAPS2 -> ProcessPlugin2 -> CAPS3 -> SinkPlugin

Src Plugin has no SINK capabilities, for example a camera. SinkPlugin has no SRC capabilities, so no output (ex a display).
CAPS1 would the belong to the intersection of SrcPlugin SRC caps and ProcessPlugin1 SINK caps, …etc.

Since we are lazy, and for some conciseness, instead of writing the caps between each pair of plugins, you may let gstreamer find for you one that matches both ends. Using gst-launch with -v (verbose) option, it will display the caps used for each plugin input and output.

For your case, I assume you’re using a RPi V2 cam with IMX219 sensor.

@icornejo.a

If I select a sensor-mode = 3 , which means a width = 1280 and a height = 720 pixels, do I still then need to deal with specifying a width and height in the video/x raw(memory:NVMM) part of the configuration? If so how come? What would be the application of using sensor-mode then? I’ve seen it used on some examples online, but then they start specifying a height, width, frame rate anyway. This same type of questions also applies to frame rate as well. I thought by picking a sensor mode, you automatically define a frame rate for that mode as well.

If you want to use a native mode of your sensor, you don’t need to specify any caps if next plugin SINK caps are ok.
You might however specify caps for resizing input into a different resolution from source or dividing framerate.

@icornejo.a

Are there any other options for NVMM , other than NVMM=NV12? Can I please get a reference on what this means? Or a link to a reference where I can read up more? Or is this more of a “It just is” type of thing? I’m asking because in the NVIDIA Input and Output Formats section of the Developers Guide, it talks about different formats for nvvidconv .

Usually for video in raw format the caps would specify type video/x-raw. This refers to standard memory and can be be used by a CPU plugin. For HW accelerators or GPU, contiguous memory is preferred. So with Jetson you have the video/x-raw(memory:NVMM) caps for using dedicated HW such as nvarguscamerasrc.
Note that NVMM=NV12 is not correct. You would have to use caps: video/x-raw(memory:NVMM),format=NV12
Plugin nvvidconv allows to copy between both types. It can also convert format and resize, but expects at least one among its input or output to be in NVMM memory.

@icornejo.a

What’s not clear is that when we’re building the pipeline, do we always expect NV12 as an input or can you select another input format? This question is an attempt to minimize transformations and conversions that I see in almost every example of OpenCV online. If I can immediatley get a stream that is JPEG, and then stream that to an iPywidget, wouldn’t that be more effeciant than importing an image to BGR, converting to JPEG and then displaying?

Where do I find information on video/x-raw ? It looks like I have to specify it a height, width, but am I stuck using BGR? Or can I create a stream that outputs right to JPEG? (I get that this question may be answered already above, but in case I have #2 completely wrong, I’m asking another question).

I have no knowledge of iPywidget. Is there a way it can be connected to gstreamer ? If it can be linked to gstreamer, maybe it would use appsink for iPywidget application and you would try:

gst-launch-1.0 nvarguscamerasrc ! nvvidconv ! jpegenc ! appsink

# Or specifying some caps
gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM)' ! nvvidconv ! video/x-raw ! jpegenc ! image/jpeg ! appsink

Note that in a shell command, in order to prevent it from interpreting the parenthesis, the caps have to be quoted.

If it works, you would use as pipeline from your application :

pipeline_string="nvarguscamerasrc ! video/x-raw(memory:NVMM) ! nvvidconv ! video/x-raw ! jpegenc ! image/jpeg ! appsink"

@icornejo.a

Where do I find more information on video sinks? My goal is to display an image/video stream using an iPywidget on a remote machine running Jupyter Lab (Not unlike the DLI courses, except using ethernet instead of USB). I’m already able to ssh into the nano and run juypter lab, I have even successfully diplayed an image using an iPywidget running jet-cam. However, the more I learn about GStreamer pipelines, the more questions I have on the examples I see online. They don’t seem to make sense in the context of building the right pipeline.

gst-inspect-1.0 | grep sink | grep video

would give most of them. They use different backends.

nvhdmioverlaysink or other EGL sinks would expect a local monitor and for EGL a X session running for the user.
xvimagesink may be used remotely with ssh -X or ssh -Y but you wouldn’t set DISPLAY for remote display.
cacasink would work on any terminal.