Hi,
Is there a way to add support for RTSP streams with audio on DeppStream?
I have an RTSP live streaming input that contains a video channel and an audio channel. I need to decode the video frames, infer the video frames with a segmentation network, get the segmentation mask and encode it, sync it back with audio, and sent it to an output RTMP.
Does Deepstream have a way of doing that? I am especially concerned with the segmentation model latency for latter synchronization with the audio track.
Thanks
Hi,
We have sample for doing inference to audio data. Please take a look at:
Search — DeepStream 6.1.1 Release documentation
If you need to bypass audio data, there is no existing implementation by default. You would need to add audio path to deepstream-app.
Hi, @DaneLLL
I saw this audio inference example but it does not provide a way to synchronize audio and video after running inference on the video stream.
Do you have references or sample codes showing how I can implement that?
Thanks
Hi,
Please check this python sample:
#!/usr/bin/env python3
import gi
import time
gi.require_version('Gst', '1.0')
from gi.repository import Gst, GObject, GLib
pipeline = None
bus = None
message = None
# initialize GStreamer
Gst.init(None)
# build the pipeline
pipeline = Gst.parse_launch(
"rtspsrc location=rtsp://10.19.107.227:8554/test name=src ! rtph264depay ! h264parse ! nvv4l2decoder ! mx.sink_0 nvstreammux width=1920 height=1080 batch-size=1 name=mx ! nvinfer name=myinfer ! nvstreamdemux name=demx demx.src_0 ! nvvideoconvert ! nvdsosd ! nvoverlaysink "
"src. ! rtpmp4adepay ! avdec_aac ! audioconvert ! alsasink device=hw:0,8 "
)
infer = pipeline.get_by_name("myinfer")
infer.set_property("config-file-path", "/opt/nvidia/deepstream/deepstream-5.1/samples/configs/deepstream-app/config_infer_primary_nano.txt")
# start playing
print("Switch to PLAYING state")
pipeline.set_state(Gst.State.PLAYING)
time.sleep(15)
print("Send EoS")
Gst.Element.send_event(pipeline, Gst.Event.new_eos())
# wait until EOS/Error
bus = pipeline.get_bus()
msg = bus.timed_pop_filtered(
Gst.CLOCK_TIME_NONE, Gst.MessageType.ERROR | Gst.MessageType.EOS)
# free resources
print("Switch to NULL state")
pipeline.set_state(Gst.State.NULL)
time.sleep(1)
hw:0,8
is card 0 and device 8 since we have it in our setup:
nvidia@dhcp-10-19-115244:~$ aplay -l
**** List of PLAYBACK Hardware Devices ****
(skip...)
card 0: tegrahdagalent1 [tegra-hda-galen-t194], device 8: HDMI 0 [HDMI 0]
Subdevices: 1/1
Subdevice #0: subdevice #0
(...skip)
You may try the sample and refer to it for further customization.