Live streaming with inference using deepstream sdk via RTMP

[royn]

Please provide complete information as applicable to your setup.

**• Hardware Platform (Jetson / GPU) - Nano
• DeepStream Version - 6.0
• JetPack Version (valid for Jetson only) - 4.6.2

I’m using usb camera to stream live object detection inference using rtmp sink from deepstream pipeline. Below is the pipeline used:

gst-launch-1.0 v4l2src device=/dev/video0 ! videoconvert ! “video/x-raw,format=(string)UYVY” ! nvvideoconvert ! “video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1” ! nvvideoconvert ! “video/x-raw,format=(string)NV12,width=1280,height=720,framerate=30/1” ! nvvideoconvert ! mux.sink_0 nvstreammux live-source=1 name=mux batch-size=1 width=1280 height=720 ! nvinfer config-file-path=dstest1_pgie_config.txt batch-size=1 ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! queue ! flvmux name=muxer alsasrc device=hw:1 ! audioresample ! “audio/x-raw,rate=48000” ! queue ! voaacenc bitrate=32000 ! aacparse ! queue ! muxer. muxer. ! rtmpsink location= rtmp://a.rtmp.youtube.com/live2/xxxxxxxx sync=false -v

It is streaming only the video NOT the inference output with bounding box and its classes.

When i tried to display locally it works fine with the below command :

gst-launch-1.0 v4l2src device=/dev/video0 ! videoconvert ! “video/x-raw,format=(string)UYVY” ! nvvidconv ! “video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1” ! nvvidconv ! “video/x-raw,format=(string)NV12,width=1280,height=720,framerate=30/1” ! nvvidconv ! mux.sink_0 nvstreammux live-source=1 name=mux batch-size=1 width=1280 height=720 ! nvinfer config-file-path=dstest1_pgie_config.txt ! nvvidconv ! nvv4l2h264enc ! h264parse ! queue ! flvmux name=muxer alsasrc device=hw:1 ! audioresample ! “audio/x-raw,rate=48000” ! queue ! voaacenc bitrate=32000 ! aacparse ! queue ! muxer. muxer. ! nveglglesink -e

How to stream the object detection output via rtmp sink?

Any help would be appreciated.

Thanks.

[Fiona.Chen]

Your pipeline looks fine. It should work.

[Fiona.Chen]

@royn

gst-launch-1.0 v4l2src device=/dev/video0 ! videoconvert ! “video/x-raw,format=(string)UYVY” ! nvvidconv ! “video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1” ! nvvidconv ! “video/x-raw,format=(string)NV12,width=1280,height=720,framerate=30/1” ! nvvidconv ! mux.sink_0 nvstreammux live-source=1 name=mux batch-size=1 width=1280 height=720 ! nvinfer config-file-path=dstest1_pgie_config.txt ! nvvidconv ! nvv4l2h264enc ! h264parse ! queue ! flvmux name=muxer alsasrc device=hw:1 ! audioresample ! “audio/x-raw,rate=48000” ! queue ! voaacenc bitrate=32000 ! aacparse ! queue ! muxer. muxer. ! nveglglesink -e

I don’t think nveglglessink can be connected to flvmux src pad. This is a wrong pipeline.

[royn]

gst-launch-1.0 v4l2src device=/dev/video0 ! videoconvert ! “video/x-raw,format=(string)UYVY” ! nvvideoconvert ! “video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1” ! nvvideoconvert ! “video/x-raw,format=(string)NV12,width=1280,height=720,framerate=30/1” ! nvvideoconvert ! mux.sink_0 nvstreammux live-source=1 name=mux batch-size=1 width=1280 height=720 ! nvinfer config-file-path=dstest1_pgie_config.txt batch-size=1 ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! queue ! flvmux name=muxer alsasrc device=hw:1 ! audioresample ! “audio/x-raw,rate=48000” ! queue ! voaacenc bitrate=32000 ! aacparse ! queue ! muxer. muxer. ! rtmpsink location= rtmp://a.rtmp.youtube.com/live2/xxxxxxxx sync=false -v

With the above command - it is not streaming the inference output - instead it is streaming only the camera input.

Any suggestions?

[Fiona.Chen]

Please dump pipeline graph to check. DeepStream SDK FAQ - Intelligent Video Analytics / DeepStream SDK - NVIDIA Developer Forums

[royn]

Attached the output image of the pipeline graph here.
Let me know anything is missing.

[Fiona.Chen]

@royn

gst-launch-1.0 v4l2src device=/dev/video0 ! videoconvert ! “video/x-raw,format=(string)UYVY” ! nvvideoconvert ! “video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1” ! nvvideoconvert ! “video/x-raw,format=(string)NV12,width=1280,height=720,framerate=30/1” ! nvvideoconvert ! mux.sink_0 nvstreammux live-source=1 name=mux batch-size=1 width=1280 height=720 ! nvinfer config-file-path=dstest1_pgie_config.txt batch-size=1 ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! queue ! flvmux name=muxer alsasrc device=hw:1 ! audioresample ! “audio/x-raw,rate=48000” ! queue ! voaacenc bitrate=32000 ! aacparse ! queue ! muxer. muxer. ! rtmpsink location= rtmp://a.rtmp.youtube.com/live2/xxxxxxxx sync=false -v

Please use nvdsosd to draw the bbox and print text. Gst-nvdsosd — DeepStream 6.1 Release documentation

gst-launch-1.0 v4l2src device=/dev/video0 ! videoconvert ! “video/x-raw,format=(string)UYVY” ! nvvideoconvert ! “video/x-raw(memory:NVMM),width=1280,height=720,framerate=30/1” ! nvvideoconvert ! “video/x-raw,format=(string)NV12,width=1280,height=720,framerate=30/1” ! nvvideoconvert ! mux.sink_0 nvstreammux live-source=1 name=mux batch-size=1 width=1280 height=720 ! nvinfer config-file-path=dstest1_pgie_config.txt batch-size=1 ! nvvideoconvert ! nvdsosd ! nvvideoconvert ! nvv4l2h264enc ! h264parse ! queue ! flvmux name=muxer alsasrc device=hw:1 ! audioresample ! “audio/x-raw,rate=48000” ! queue ! voaacenc bitrate=32000 ! aacparse ! queue ! muxer. muxer. ! rtmpsink location= rtmp://a.rtmp.youtube.com/live2/xxxxxxxx sync=false -v

[royn]

Thanks for you suggestion.
It works now - However there is a latency of 30 sec to 1 minute in the live streaming.

Any recommendations to reduce the latency in this pipeline?

[Fiona.Chen]

Please create new topic for the latency issue.