Synchronizing audio and video after inference of RTSP streams

Accelerated Computing Intelligent Video Analytics DeepStream SDK
, ,
1

Hi,

Is there a way to add support for RTSP streams with audio on DeppStream?
I have an RTSP live streaming input that contains a video channel and an audio channel. I need to decode the video frames, infer the video frames with a segmentation network, get the segmentation mask and encode it, sync it back with audio, and sent it to an output RTMP.

Does Deepstream have a way of doing that? I am especially concerned with the segmentation model latency for latter synchronization with the audio track.

Thanks

3

Hi,
We have sample for doing inference to audio data. Please take a look at:
Search — DeepStream 6.1.1 Release documentation

If you need to bypass audio data, there is no existing implementation by default. You would need to add audio path to deepstream-app.

4

Hi, @DaneLLL

I saw this audio inference example but it does not provide a way to synchronize audio and video after running inference on the video stream.

Do you have references or sample codes showing how I can implement that?

Thanks

5

Hi,
Please check this python sample:

#!/usr/bin/env python3

import gi
import time
gi.require_version('Gst', '1.0')
from gi.repository import Gst, GObject, GLib

pipeline = None
bus = None
message = None

# initialize GStreamer
Gst.init(None)

# build the pipeline
pipeline = Gst.parse_launch(
    "rtspsrc location=rtsp://10.19.107.227:8554/test name=src ! rtph264depay ! h264parse ! nvv4l2decoder ! mx.sink_0 nvstreammux width=1920 height=1080 batch-size=1 name=mx ! nvinfer name=myinfer ! nvstreamdemux name=demx demx.src_0 ! nvvideoconvert ! nvdsosd ! nvoverlaysink "
    "src. ! rtpmp4adepay ! avdec_aac ! audioconvert ! alsasink device=hw:0,8 "
)
infer = pipeline.get_by_name("myinfer")
infer.set_property("config-file-path", "/opt/nvidia/deepstream/deepstream-5.1/samples/configs/deepstream-app/config_infer_primary_nano.txt")

# start playing
print("Switch to PLAYING state")
pipeline.set_state(Gst.State.PLAYING)

time.sleep(15)
print("Send EoS")
Gst.Element.send_event(pipeline, Gst.Event.new_eos())
# wait until EOS/Error
bus = pipeline.get_bus()
msg = bus.timed_pop_filtered(
    Gst.CLOCK_TIME_NONE, Gst.MessageType.ERROR | Gst.MessageType.EOS)

# free resources
print("Switch to NULL state")
pipeline.set_state(Gst.State.NULL)
time.sleep(1)

hw:0,8 is card 0 and device 8 since we have it in our setup:

nvidia@dhcp-10-19-115244:~$ aplay -l
**** List of PLAYBACK Hardware Devices ****
(skip...)
card 0: tegrahdagalent1 [tegra-hda-galen-t194], device 8: HDMI 0 [HDMI 0]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
(...skip)

You may try the sample and refer to it for further customization.