revising streaming with Xavier

My concern is to implement a running demo that will allow with high-resolution stream from Xavier CSI camera via network nearly realtime

I can think of:

  1. WebRTC or web browser-based streaming
  2. gstreamer rtsp streaming
  3. deepstream implementation of some sort

for the first scenario, I tried a web browser implementation to that via v4l2loopback redirected a stream from CSI camera

for the second scenario I can run rtsp stream but I can not play it:

./test-launch "nvarguscamerasrc ! video/x-raw(memory:NVMM), format=NV12, width=1920, height=1080, framerate=30/1 ! nvvidconv ! video/x-raw, width=640, height=480, format=NV12, framerate=30/1 ! omxh265enc ! rtph265pay name=pay0 pt=96 config-interval=1"
stream ready at rtsp://127.0.0.1:8554/test
gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! queue ! decodebin ! videoconvert ! xvimagesink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Progress: (open) Opening Stream
Progress: (connect) Connecting to rtsp://127.0.0.1:8554/test
Progress: (open) Retrieving server options
Progress: (open) Retrieving media info
Progress: (request) SETUP stream 0
Progress: (open) Opened Stream
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Progress: (request) Sending PLAY request
Progress: (request) Sending PLAY request
Progress: (request) Sent PLAY request
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 279 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 279 
WARNING: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0: not negotiated
Additional debug info:
gstbasetransform.c(1415): gst_base_transform_reconfigure (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0:
not negotiated
WARNING: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0: not negotiated
Additional debug info:
gstbasetransform.c(1415): gst_base_transform_reconfigure (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0:
not negotiated
WARNING: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0: not negotiated
Additional debug info:
gstbasetransform.c(1415): gst_base_transform_reconfigure (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0:
not negotiated
WARNING: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0: not negotiated
Additional debug info:
gstbasetransform.c(1415): gst_base_transform_reconfigure (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0:
not negotiated
WARNING: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0: not negotiated
Additional debug info:
gstbasetransform.c(1415): gst_base_transform_reconfigure (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0:
not negotiated
WARNING: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0: not negotiated
Additional debug info:
gstbasetransform.c(1415): gst_base_transform_reconfigure (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0:
not negotiated
WARNING: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0: not negotiated
Additional debug info:
gstbasetransform.c(1415): gst_base_transform_reconfigure (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0:
not negotiated
WARNING: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0: not negotiated
Additional debug info:
gstbasetransform.c(1415): gst_base_transform_reconfigure (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0:
not negotiated
WARNING: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0: not negotiated
Additional debug info:
gstbasetransform.c(1415): gst_base_transform_reconfigure (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0:
not negotiated
WARNING: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0: not negotiated
Additional debug info:
gstbasetransform.c(1415): gst_base_transform_reconfigure (): /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0:
not negotiated
WARNING: from element /GstPipeline:pipeline0/GstDecodeBin:decodebin0/GstCapsFilter:capsfilter0: not negotiated
Additional debug info:

Hi,
Please try

$ gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! queue ! decodebin ! nvvidconv ! videoconvert ! xvimagesink

Utilizing nvoverlaysink can give better performance:

$ gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! queue ! decodebin ! nvoverlaysink

Hi,
Also notice that there is a redundant memcpy in test-launc command. Please run

./test-launch "nvarguscamerasrc ! video/x-raw(memory:NVMM), format=NV12, width=1920, height=1080, framerate=30/1 ! nvvidconv ! video/x-raw(memory:NVMM), width=640, height=480, format=NV12, framerate=30/1 ! omxh265enc ! rtph265pay name=pay0 pt=96 config-interval=1"

‘video/x-raw(memory:NVMM) ! nvvidconv ! video/x-raw’ executes a redundant NVMM -> CPU buffer copy.

Hi DaneLLL,
Thank you for your response.
And thank you for sharing the code.
However, when executing

./test-launch "nvarguscamerasrc ! video/x-raw(memory:NVMM), format=NV12, width=1920, height=1080, framerate=30/1 ! nvvidconv ! video/x-raw(memory:NVMM), width=640, height=480, format=NV12, framerate=30/1 ! omxh265enc ! rtph265pay name=pay0 pt=96 config-interval=1"

and

gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! queue ! decodebin ! nvoverlaysink

it wont pop ip a window with video output. I am just wondering if some arguments need to be added in order to display the video stream.

gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! queue ! decodebin ! nvoverlaysink
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Progress: (open) Opening Stream
Progress: (connect) Connecting to rtsp://127.0.0.1:8554/test
Progress: (open) Retrieving server options
Progress: (open) Retrieving media info
Progress: (request) SETUP stream 0
Progress: (open) Opened Stream
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Progress: (request) Sending PLAY request
Progress: (request) Sending PLAY request
Progress: (request) Sent PLAY request
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 279 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 279

On the other hand,

gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! queue ! decodebin ! nvvidconv ! videoconvert ! xvimagesink

works and displays the video feed

Hi,
We can run the pipeline on r32.2.3. Probably you use other version. You may link the plugins directly instead of using decodebin:

gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! queue ! rtph265depay ! h265parse ! nvvl4l2decoder ! nvoverlaysink

Hi, Thank you for your response.
I just flashed with Jetpack 4.3.
And I am using usb-c display;
When I run the code above it will throw error

nvidia@nvidia-desktop:~/gst-rtsp-server-1.14.1/examples$ gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! queue ! rtph265depay ! h265parse ! nvvl4l2decoder ! nvoverlaysink
WARNING: erroneous pipeline: no element "nvvl4l2decoder"
gst-inspect-1.0 nvv4l2decoder
Factory Details:
  Rank                     primary + 11 (267)
  Long-name                NVIDIA v4l2 video decoder
  Klass                    Codec/Decoder/Video
  Description              Decode video streams via V4L2 API
  Author                   Nicolas Dufresne <nicolas.dufresne@collabora.com>, Viranjan Pagar <vpagar@nvidia.com>

Plugin Details:
  Name                     nvvideo4linux2
  Description              Nvidia elements for Video 4 Linux
  Filename                 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvvideo4linux2.so
  Version                  1.14.0
  License                  LGPL
  Source module            nvvideo4linux2
  Binary package           nvvideo4linux2
  Origin URL               http://nvidia.com/

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstVideoDecoder
                         +----GstNvV4l2VideoDec
                               +----nvv4l2decoder

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw(memory:NVMM)
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
  
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      image/jpeg
      video/x-h264
          stream-format: { (string)byte-stream }
              alignment: { (string)au }
      video/x-h265
          stream-format: { (string)byte-stream }
              alignment: { (string)au }
      video/mpeg
            mpegversion: 4
           systemstream: false
                 parsed: true
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
      video/mpeg
            mpegversion: [ 1, 2 ]
           systemstream: false
                 parsed: true
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
      video/x-divx
            divxversion: [ 4, 5 ]
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
      video/x-vp8
      video/x-vp9
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SINK: 'sink'
    Pad Template: 'sink'
  SRC: 'src'
    Pad Template: 'src'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "nvv4l2decoder0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  device              : Device location
                        flags: readable
                        String. Default: "/dev/nvhost-nvdec"
  device-name         : Name of the device
                        flags: Opening in BLOCKING MODE 
readable
                        String. Default: ""
  device-fd           : File descriptor of the device
                        flags: readable
                        Integer. Range: -1 - 2147483647 Default: -1 
  output-io-mode      : Output side I/O mode (matches sink pad)
                        flags: readable, writable
                        Enum "GstNvV4l2DecOutputIOMode" Default: 0, "auto"
                           (0): auto             - GST_V4L2_IO_AUTO
                           (2): mmap             - GST_V4L2_IO_MMAP
                           (3): userptr          - GST_V4L2_IO_USERPTR
  capture-io-mode     : Capture I/O mode (matches src pad)
                        flags: readable, writable
                        Enum "GstNvV4l2DecCaptureIOMode" Default: 0, "auto"
                           (0): auto             - GST_V4L2_IO_AUTO
                           (2): mmap             - GST_V4L2_IO_MMAP
  extra-controls      : Extra v4l2 controls (CIDs) for the device
                        flags: readable, writable
                        Boxed pointer of type "GstStructure"
  skip-frames         : Type of frames to skip during decoding
                        flags: readable, writable, changeable in NULL, READY, PAUSED or PLAYING state
                        Enum "SkipFrame" Default: 0, "decode_all"
                           (0): decode_all       - Decode all frames
                           (1): decode_non_ref   - Decode non-ref frames
                           (2): decode_key       - decode key frames
  drop-frame-interval : Interval to drop the frames,ex: value of 5 means every 5th frame will be given by decoder, rest all dropped
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 30 Default: 0 
  num-extra-surfaces  : Additional number of surfaces in addition to min decode surfaces given by the v4l2 driver
                        flags: readable, writable, changeable only in NULL or READY state
                        Unsigned Integer. Range: 0 - 24 Default: 1 
  disable-dpb         : Set to disable DPB buffer for low latency
                        flags: readable, writable
                        Boolean. Default: false
  enable-full-frame   : Whether or not the data is full framed
                        flags: readable, writable
                        Boolean. Default: false
  enable-frame-type-reporting: Set to enable frame type reporting
                        flags: readable, writable
                        Boolean. Default: false
  enable-error-check  : Set to enable error check
                        flags: readable, writable
                        Boolean. Default: false
  enable-max-performance: Set to enable max performance
                        flags: readable, writable
                        Boolean. Default: false
  mjpeg               : Set to open MJPEG block
                        flags: readable, writable
                        Boolean. Default: false
  bufapi-version      : Set to use new buf API
                        flags: readable, writable
                        Boolean. Default: false

Hi,
I made a typo. It is nvv4l2decoder, not nvvl4l2decoder. My bad.

Hi DaneLLL,
Thank you for the update.

gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! queue ! rtph265depay ! h265parse ! nvv4l2decoder ! nvoverlaysink

The code above works, however, it won’t opp up a window with displayed video stream.

gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! queue ! rtph265depay ! h265parse ! nvv4l2decoder ! nvoverlaysink
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE 
Pipeline is live and does not need PREROLL ...
Progress: (open) Opening Stream
Progress: (connect) Connecting to rtsp://127.0.0.1:8554/test
Progress: (open) Retrieving server options
Progress: (open) Retrieving media info
Progress: (request) SETUP stream 0
Progress: (open) Opened Stream
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Progress: (request) Sending PLAY request
Progress: (request) Sending PLAY request
Progress: (request) Sent PLAY request
NvMMLiteOpen : Block : BlockType = 279 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 279

May be you could add on how to use WebRTC on system flashed by Jetpack 4.3 in a way it will start a CSI server so that from a network it would be possible connect to it?

Could you confirm that the line below will execute on L4T 32.3.1 with video output? without video output? with hdmi monitor? with usb-c monitor? anyhow?

1. gst-launch-1.0 videotestsrc ! nvvidconv ! nvoverlaysink
2. gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! queue ! decodebin ! nvoverlaysink

trying to change the situation with

root@nvidia-desktop:/sys/class/graphics/fb1#  echo 0x0 > device/win_mask ; cd /sys/class/graphics/fb2 ; echo 0x0 > device/win_mask ; cd /sys/class/graphics/fb0 ; echo 4 > blank ; echo 0x0 > device/win_mask ; echo 0x3f > device/win_mask ; echo 0  > blank
bash: echo: write error: Invalid argument
bash: echo: write error: Invalid argument
root@nvidia-desktop:/sys/class/graphics/fb0#
gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test latency=30 ! queue ! rtph265depay ! h265parse ! nvv4l2decoder ! nvoverlaysink
Setting pipeline to PAUSED ...
Opening in BLOCKING MODE 
Pipeline is live and does not need PREROLL ...
Progress: (open) Opening Stream
Progress: (connect) Connecting to rtsp://127.0.0.1:8554/test
Progress: (open) Retrieving server options
Progress: (open) Retrieving media info
Progress: (request) SETUP stream 0
Progress: (open) Opened Stream
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Progress: (request) Sending PLAY request
Progress: (request) Sending PLAY request
Progress: (request) Sent PLAY request
NvMMLiteOpen : Block : BlockType = 279 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 279 
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552 
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552
gst-launch-1.0 videotestsrc ! nvvidconv ! nvoverlaysink
Setting pipeline to PAUSED ...
Pipeline is PREROLLING ...
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552 
Pipeline is PREROLLED ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
ERROR: from element /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0: Internal data stream error.
Additional debug info:
gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline0/GstVideoTestSrc:videotestsrc0:
streaming stopped, reason error (-5)
Execution ended after 0:00:00.000257257
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
NvxBaseWorkerFunction[2575] comp OMX.Nvidia.std.iv_renderer.overlay.yuv420 Error -2147479552 
Setting pipeline to NULL ...

it was said that the pipeline works with Xavier device flashed with the latter Jetpack; I assume it is the issue of USB-C nature and nvoverlaysink

Hi,
The case we verify is running a server:

./test-launch "nvarguscamerasrc ! video/x-raw(memory:NVMM), format=NV12, width=1920, height=1080, framerate=30/1 ! nvvidconv ! video/x-raw(memory:NVMM), width=640, height=480, format=NV12, framerate=30/1 ! omxh265enc ! rtph265pay name=pay0 pt=96 config-interval=1"

And client(HDMI video out):

gst-launch-1.0 rtspsrc location=rtsp://127.0.0.1:8554/test ! queue ! rtph265depay ! h265parse ! nvvl4l2decoder ! nvoverlaysink

Thank you.
For USB-c the sequence below worked:

gst-launch-1.0 videotestsrc ! nvvidconv ! nvoverlaysink display-id=2