Jetson Nano - unable to decode frames from IP camera streams with GPU (nvv412decoder)

Robotics & Edge Computing Jetson & Embedded Systems Jetson Nano
1

We’ve been experimenting with Deepstream and GStreamer pipelines on our Jetson Xavier and Nano devices, and are running into some technical issues/questions about how to properly leverage GPU-frame decoding/conversion. We’re attempting to decode from an RTSP stream coming from an IP (PoE) camera.

These GStreamer gst-launch pipelines definitely work with the IP camera on Jetson hardware, but only use CPU decoding – but they have let us experiment with basic pipelines, to validate that we can get data from an IP camera.

  • gst-launch-1.0 rtspsrc location=rtsp://192.168.0.103:554/s1 latency=300 ! rtph264depay ! avdec_h264 ! xvimagesink -e
  • gst-launch-1.0 rtspsrc location=rtsp://192.168.0.103:554/s1 latency=300 ! rtph264depay ! avdec_h264 ! autovideosink
  • gst-launch-1.0 rtspsrc location=rtsp://192.168.0.103:554/s1 latency=1000 ! rtph264depay ! queue ! h264parse ! omxh264dec ! nvvidconv ! nv3dsink
  • gst-launch-1.0 rtspsrc location=rtsp://192.168.0.103:554/s1 latency=1000 ! rtph264depay ! queue ! h264parse ! omxh264dec ! nveglglessink -e
  • gst-launch-1.0 rtspsrc location=rtsp://192.168.0.103:554/s1 latency=1000 ! rtph264depay ! queue ! h264parse ! omxh264dec ! autovideosink

Sidenote – this next pipeline – which may use hardware acceleration (I’m still a bit fuzzy on what omxh264dec actually does), seems to error out every few frames due to a lower latency value (300), compared to the above 1000 – we’d love to better understand why this is, if there is any insight you could provide.

  • gst-launch-1.0 rtspsrc location=rtsp://192.168.0.103:554/s1 latency=300 ! rtph264depay ! queue ! h264parse ! omxh264dec ! nveglglessink -e 
WARNING: from element /GstPipeline:pipeline0/GstEglGlesSink:eglglessink0: A lot of buffers are being dropped.
    Additional debug info:
    gstbasesink.c(2902): gst_base_sink_is_too_late (): /GstPipeline:pipeline0/GstEglGlesSink:eglglessink0:
    There may be a timestamping problem, or this computer is too slow.

Finally, these gst-launch pipelines, which should be leveraging NVIDIA hardware to do GPU-accelerated frame decoding/conversation (following guidance in the Accelerated GStreamer User Guide), unfortunately do not work on our Jetson hardware – and we’re trying to understand why/what we may be doing incorrectly here:

  • gst-launch-1.0 rtspsrc location=rtsp://192.168.0.103:554/s1 latency=1000 ! rtph264depay ! qtdemux ! queue ! h264parse ! nvv4l2decoder ! nv3dsink -e
  • Results in: WARNING: erroneous pipeline: could not link rtph264depay0 to qtdemux0
  • gst-launch-1.0 rtspsrc location=rtsp://192.168.0.103:554/s1 latency=1000 ! rtph264depay ! queue ! h264parse ! nvv4l2decoder ! nv3dsink -e
  • Results in: (gst-launch-1.0:16180): GStreamer-CRITICAL **: 17:08:19.337: gst_mini_object_unref: assertion 'mini_object != NULL' failed
  • gst-launch-1.0 rtspsrc location=rtsp://192.168.0.103:554/s1 latency=1000 ! rtph264depay ! queue ! h264parse ! nvv4l2decoder ! nvvidconv ! nv3dsink
  • Results in: (gst-launch-1.0:16180): GStreamer-CRITICAL **: 17:08:19.337: gst_mini_object_unref: assertion 'mini_object != NULL' failed
  • gst-launch-1.0 rtspsrc location=rtsp://192.168.0.103:554/s1 latency=1000 ! rtph264depay ! queue ! h264parse ! nvv4l2decoder ! autovideosink
  • Results in: (gst-launch-1.0:16180): GStreamer-CRITICAL **: 17:08:19.337: gst_mini_object_unref: assertion 'mini_object != NULL' failed
  • gst-launch-1.0 rtspsrc location=rtsp://192.168.0.103:554/s1 latency=1000 ! rtph264depay ! queue ! h264parse ! nvv4l2decoder ! nveglglessink -e
  • Results in: "WARNING: erroneous pipeline: could not link nvv4l2decoder0 to eglglessink0"

Is there anything obvious we’re doing wrong, particularly with nvv412decoder?

2

Hi,
On Jetson Nano, it would be better to use nvoverlaysink:

gst-launch-1.0 rtspsrc location=rtsp://192.168.0.103:554/s1 latency=1000 ! rtph264depay ! queue ! h264parse ! nvv4l2decoder ! nvoverlaysink

Since nveglglessink take certain GPU resources. It may reduce performance in runnug deepstream usecases. Please also execute ‘sudo jetson_clocks’ to run in max performance.

The reference config fie is source8_1080p_dec_infer-resnet_tracker_tiled_display_fp16_nano.txt. You may enter ‘deepstream_sdk_v4.0.1_jetson\samples\configs\deepstream-app’ and run

deepstream-app -c source8_1080p_dec_infer-resnet_tracker_tiled_display_fp16_nano.txt

It is ‘type=5’(nvoverlaysink) by default. Please refer to
https://docs.nvidia.com/metropolis/deepstream/dev-guide/index.html#page/DeepStream_Development_Guide%2Fdeepstream_app_config.3.2.html%23wwpID0ENHA

3

Hi - unfortunately, this does not work either on my Jetson Nano or Jetson Xavier when ingesting from my IP camera. For example, this is from the Xavier, but this is what I’m getting on both devices:

$ gst-launch-1.0 rtspsrc location=rtsp://192.168.0.103:554/s1 latency=1000 ! rtph264depay ! queue ! h264parse ! nvv4l2decoder ! nvoverlaysink

Setting pipeline to PAUSED ...
Opening in BLOCKING MODE 
Pipeline is live and does not need PREROLL ...
Progress: (open) Opening Stream
Progress: (connect) Connecting to rtsp://192.168.0.102:554/s1
Progress: (open) Retrieving server options
Progress: (open) Retrieving media info
Progress: (request) SETUP stream 0
Progress: (request) SETUP stream 1
Progress: (open) Opened Stream
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Progress: (request) Sending PLAY request
Progress: (request) Sending PLAY request
Progress: (request) Sent PLAY request
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 

(gst-launch-1.0:10879): GStreamer-CRITICAL **: 16:01:22.036: gst_mini_object_unref: assertion 'mini_object != NULL' failed
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261

Long version (-v) attached.
output_long.txt (42.6 KB)

4

Hi,
Please check if you are able to run this case:
https://devtalk.nvidia.com/default/topic/1058086/deepstream-sdk/how-to-run-rtp-camera-in-deepstream-on-nano/post/5366807/#5366807

Also please clean cache and try again.

If the application encounters errors and cannot create Gst elements, remove the GStreamer cache, then try again. To remove the GStreamer cache, enter this command:
$ rm ${HOME}/.cache/gstreamer-1.0/registry.aarch64.bin

For informatio, do you install the whole system/packages through sdkmanager? Which release do you use?