revising streaming with Xavier

Autonomous Machines Jetson & Embedded Systems Jetson AGX Xavier
1

My concern is to implement a running demo that will allow with high-resolution stream from Xavier CSI camera via network nearly realtime

I can think of:

  1. WebRTC or web browser-based streaming
  2. gstreamer rtsp streaming
  3. deepstream implementation of some sort

for the first scenario, I tried a web browser implementation to that via v4l2loopback redirected a stream from CSI camera

for the second scenario I can run rtsp stream but I can not play it:

./test-launch "nvarguscamerasrc ! video/x-raw(memory:NVMM), format=NV12, width=1920, height=1080, framerate=30/1 ! nvvidconv ! video/x-raw, width=640, height=480, format=NV12, framerate=30/1 ! omxh265enc ! rtph265pay name=pay0 pt=96 config-interval=1"
stream ready at rtsp://127.0.0.1:8554/test

gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! queue ! decodebin ! videoconvert ! xvimagesink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Progress: (open) Opening Stream
Progress: (connect) Connecting to rtsp://127.0.0.1:8554/test
Progress: (open) Retrieving server options
Progress: (open) Retrieving media info
Progress: (request) SETUP stream 0
Progress: (open) Opened Stream
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Progress: (request) Sending PLAY request
Progress: (request) Sending PLAY request
Progress: (request) Sent PLAY request
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 279 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 279 
WARNING: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0: not negotiated
Additional debug info:
gstbasetransform.c(1415): gst_base_transform_reconfigure (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0:
not negotiated
WARNING: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0: not negotiated
Additional debug info:
gstbasetransform.c(1415): gst_base_transform_reconfigure (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0:
not negotiated
WARNING: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0: not negotiated
Additional debug info:
gstbasetransform.c(1415): gst_base_transform_reconfigure (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0:
not negotiated
WARNING: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0: not negotiated
Additional debug info:
gstbasetransform.c(1415): gst_base_transform_reconfigure (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0:
not negotiated
WARNING: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0: not negotiated
Additional debug info:
gstbasetransform.c(1415): gst_base_transform_reconfigure (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0:
not negotiated
WARNING: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0: not negotiated
Additional debug info:
gstbasetransform.c(1415): gst_base_transform_reconfigure (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0:
not negotiated
WARNING: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0: not negotiated
Additional debug info:
gstbasetransform.c(1415): gst_base_transform_reconfigure (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0:
not negotiated
WARNING: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0: not negotiated
Additional debug info:
gstbasetransform.c(1415): gst_base_transform_reconfigure (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0:
not negotiated
WARNING: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0: not negotiated
Additional debug info:
gstbasetransform.c(1415): gst_base_transform_reconfigure (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0:
not negotiated
WARNING: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0: not negotiated
Additional debug info:
gstbasetransform.c(1415): gst_base_transform_reconfigure (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0:
not negotiated
WARNING: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0: not negotiated
Additional debug info:
2

Hi,
Please try

$ gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! queue ! decodebin ! nvvidconv ! videoconvert ! xvimagesink

Utilizing nvoverlaysink can give better performance:

$ gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! queue ! decodebin ! nvoverlaysink
3

Hi,
Also notice that there is a redundant memcpy in test-launc command. Please run

./test-launch "nvarguscamerasrc ! video/x-raw(memory:NVMM), format=NV12, width=1920, height=1080, framerate=30/1 ! nvvidconv ! video/x-raw(memory:NVMM), width=640, height=480, format=NV12, framerate=30/1 ! omxh265enc ! rtph265pay name=pay0 pt=96 config-interval=1"

‘video/x-raw(memory:NVMM) ! nvvidconv ! video/x-raw’ executes a redundant NVMM → CPU buffer copy.

4

Hi DaneLLL,
Thank you for your response.
And thank you for sharing the code.
However, when executing

./test-launch "nvarguscamerasrc ! video/x-raw(memory:NVMM), format=NV12, width=1920, height=1080, framerate=30/1 ! nvvidconv ! video/x-raw(memory:NVMM), width=640, height=480, format=NV12, framerate=30/1 ! omxh265enc ! rtph265pay name=pay0 pt=96 config-interval=1"

and

gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! queue ! decodebin ! nvoverlaysink

it wont pop ip a window with video output. I am just wondering if some arguments need to be added in order to display the video stream.

gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! queue ! decodebin ! nvoverlaysink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Progress: (open) Opening Stream
Progress: (connect) Connecting to rtsp://127.0.0.1:8554/test
Progress: (open) Retrieving server options
Progress: (open) Retrieving media info
Progress: (request) SETUP stream 0
Progress: (open) Opened Stream
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Progress: (request) Sending PLAY request
Progress: (request) Sending PLAY request
Progress: (request) Sent PLAY request
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 279 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 279

On the other hand,

gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! queue ! decodebin ! nvvidconv ! videoconvert ! xvimagesink

works and displays the video feed

5

Hi,
We can run the pipeline on r32.2.3. Probably you use other version. You may link the plugins directly instead of using decodebin:

gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! queue ! rtph265depay ! h265parse ! nvvl4l2decoder ! nvoverlaysink
6

Hi, Thank you for your response.
I just flashed with Jetpack 4.3.
And I am using usb-c display;
When I run the code above it will throw error

nvidia@nvidia-desktop:~/gst-rtsp-server-1.14.1/examples$ gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! queue ! rtph265depay ! h265parse ! nvvl4l2decoder ! nvoverlaysink
WARNING: erroneous pipeline: no element "nvvl4l2decoder"

gst-inspect-1.0 nvv4l2decoder
Factory Details:
  Rank                     primary + 11 (267)
  Long-name                NVIDIA v4l2 video decoder
  Klass                    Codec/Decoder/Video
  Description              Decode video streams via V4L2 API
  Author                   Nicolas Dufresne <nicolas.dufresne@collabora.com>, Viranjan Pagar <vpagar@nvidia.com>

Plugin Details:
  Name                     nvvideo4linux2
  Description              Nvidia elements for Video 4 Linux
  Filename                 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvideo4linux2.so
  Version                  1.14.0
  License                  LGPL
  Source module            nvvideo4linux2
  Binary package           nvvideo4linux2
  Origin URL               http://nvidia.com/

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstVideoDecoder
                         +----GstNvV4l2VideoDec
                               +----nvv4l2decoder

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw(memory:NVMM)
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
  
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      image/jpeg
      video/x-h264
          stream-format: { (string)byte-stream }
              alignment: { (string)au }
      video/x-h265
          stream-format: { (string)byte-stream }
              alignment: { (string)au }
      video/mpeg
            mpegversion: 4
           systemstream: false
                 parsed: true
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
      video/mpeg
            mpegversion: [ 1, 2 ]
           systemstream: false
                 parsed: true
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
      video/x-divx
            divxversion: [ 4, 5 ]
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
      video/x-vp8
      video/x-vp9
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SINK: 'sink'
    Pad Template: 'sink'
  SRC: 'src'
    Pad Template: 'src'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "nvv4l2decoder0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  device              : Device location
                        flags: readable
                        String. Default: "/dev/nvhost-nvdec"
  device-name         : Name of the device
                        flags: Opening in BLOCKING MODE 
readable
                        String. Default: ""
  device-fd           : File descriptor of the device
                        flags: readable
                        Integer. Range: -1 - 2147483647 Default: -1 
  output-io-mode      : Output side I/O mode (matches sink pad)
                        flags: readable, writable
                        Enum "GstNvV4l2DecOutputIOMode" Default: 0, "auto"
                           (0): auto             - GST_V4L2_IO_AUTO
                           (2): mmap             - GST_V4L2_IO_MMAP
                           (3): userptr          - GST_V4L2_IO_USERPTR
  capture-io-mode     : Capture I/O mode (matches src pad)
                        flags: readable, writable
                        Enum "GstNvV4l2DecCaptureIOMode" Default: 0, "auto"
                           (0): auto             - GST_V4L2_IO_AUTO
                           (2): mmap             - GST_V4L2_IO_MMAP
  extra-controls      : Extra v4l2 controls (CIDs) for the device
                        flags: readable, writable
                        Boxed pointer of type "GstStructure"
  skip-frames         : Type of frames to skip during decoding
                        flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state
                        Enum "SkipFrame" Default: 0, "decode_all"
                           (0): decode_all       - Decode all frames
                           (1): decode_non_ref   - Decode non-ref frames
                           (2): decode_key       - decode key frames
  drop-frame-interval : Interval to drop the frames,ex: value of 5 means every 5th frame will be given by decoder, rest all dropped
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 30 Default: 0 
  num-extra-surfaces  : Additional number of surfaces in addition to min decode surfaces given by the v4l2 driver
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 24 Default: 1 
  disable-dpb         : Set to disable DPB buffer for low latency
                        flags: readable, writable
                        Boolean. Default: false
  enable-full-frame   : Whether or not the data is full framed
                        flags: readable, writable
                        Boolean. Default: false
  enable-frame-type-reporting: Set to enable frame type reporting
                        flags: readable, writable
                        Boolean. Default: false
  enable-error-check  : Set to enable error check
                        flags: readable, writable
                        Boolean. Default: false
  enable-max-performance: Set to enable max performance
                        flags: readable, writable
                        Boolean. Default: false
  mjpeg               : Set to open MJPEG block
                        flags: readable, writable
                        Boolean. Default: false
  bufapi-version      : Set to use new buf API
                        flags: readable, writable
                        Boolean. Default: false
7

Hi,
I made a typo. It is nvv4l2decoder, not nvvl4l2decoder. My bad.

8

Hi DaneLLL,
Thank you for the update.

gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! queue ! rtph265depay ! h265parse ! nvv4l2decoder ! nvoverlaysink

The code above works, however, it won’t opp up a window with displayed video stream.

gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! queue ! rtph265depay ! h265parse ! nvv4l2decoder ! nvoverlaysink
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE 
Pipeline is live and does not need PREROLL ...
Progress: (open) Opening Stream
Progress: (connect) Connecting to rtsp://127.0.0.1:8554/test
Progress: (open) Retrieving server options
Progress: (open) Retrieving media info
Progress: (request) SETUP stream 0
Progress: (open) Opened Stream
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Progress: (request) Sending PLAY request
Progress: (request) Sending PLAY request
Progress: (request) Sent PLAY request
NvMMLiteOpen : Block : BlockType = 279 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 279

May be you could add on how to use WebRTC on system flashed by Jetpack 4.3 in a way it will start a CSI server so that from a network it would be possible connect to it?

9

Could you confirm that the line below will execute on L4T 32.3.1 with video output? without video output? with hdmi monitor? with usb-c monitor? anyhow?

1. gst-launch-1.0 videotestsrc ! nvvidconv ! nvoverlaysink
2. gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! queue ! decodebin ! nvoverlaysink
10

trying to change the situation with

root@nvidia-desktop:/sys/class/graphics/fb1#  echo 0x0 > device/win_mask ; cd /sys/class/graphics/fb2 ; echo 0x0 > device/win_mask ; cd /sys/class/graphics/fb0 ; echo 4 > blank ; echo 0x0 > device/win_mask ; echo 0x3f > device/win_mask ; echo 0  > blank
bash: echo: write error: Invalid argument
bash: echo: write error: Invalid argument
root@nvidia-desktop:/sys/class/graphics/fb0#

gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test latency=30 ! queue ! rtph265depay ! h265parse ! nvv4l2decoder ! nvoverlaysink
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE 
Pipeline is live and does not need PREROLL ...
Progress: (open) Opening Stream
Progress: (connect) Connecting to rtsp://127.0.0.1:8554/test
Progress: (open) Retrieving server options
Progress: (open) Retrieving media info
Progress: (request) SETUP stream 0
Progress: (open) Opened Stream
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Progress: (request) Sending PLAY request
Progress: (request) Sending PLAY request
Progress: (request) Sent PLAY request
NvMMLiteOpen : Block : BlockType = 279 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 279 
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552 
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552

gst-launch-1.0 videotestsrc ! nvvidconv ! nvoverlaysink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552 
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0:
streaming stopped, reason error (-5)
Execution ended after 0:00:00.000257257
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552 
Setting pipeline to NULL ...

it was said that the pipeline works with Xavier device flashed with the latter Jetpack; I assume it is the issue of USB-C nature and nvoverlaysink

11

Hi,
The case we verify is running a server:

./test-launch "nvarguscamerasrc ! video/x-raw(memory:NVMM), format=NV12, width=1920, height=1080, framerate=30/1 ! nvvidconv ! video/x-raw(memory:NVMM), width=640, height=480, format=NV12, framerate=30/1 ! omxh265enc ! rtph265pay name=pay0 pt=96 config-interval=1"

And client(HDMI video out):

gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! queue ! rtph265depay ! h265parse ! nvvl4l2decoder ! nvoverlaysink
12

Thank you.
For USB-c the sequence below worked:

gst-launch-1.0 videotestsrc ! nvvidconv ! nvoverlaysink display-id=2