Nvinfer + nvinfer audio in same pipeline

[user64451]

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
• DeepStream Version: 6.1.1 - Docker ENV
• NVIDIA GPU Driver Version (valid for GPU only) 525
• Issue Type( questions)

Hi there, I have some questions to ask and need confirmation from your side. Thank in advance.

Firstly, I already read some posts about this topic:
https://forums.developer.nvidia.com/t/how-to-build-a-deepstream-pipeline-using-both-nvinfer-and-nvinferaudio/263196
https://forums.developer.nvidia.com/t/about-running-both-nvinfer-and-nvinferaudio-in-parallel-in-jetson-boards/270095

Secondly, I have 2 questions:
Q1: Can I implement a deepstream pipeline like the image below. If cannot, could you explain why?

Q2: In case, I wanna run both nvinfer and nvinfer-audio, I have to follow deepstream-6.1/sources/apps/sample_apps/deepstream-avsync, right? or do you have any other method to do without external service like riva?

[Fiona.Chen]

@user64451

Q1: Can I implement a deepstream pipeline like the image below. If cannot, could you explain why?

Seems the nvstreamdemux has some issue with audio batch. We will investigate the root cause.

@user64451

Q2: In case, I wanna run both nvinfer and nvinfer-audio, I have to follow deepstream-6.1/sources/apps/sample_apps/deepstream-avsync , right? or do you have any other method to do without external service like riva?

The following pipeline is for your reference.

USE_NEW_NVSTREAMMUX=yes gst-launch-1.0 filesrc location=test.mov ! qtdemux name=m m.audio_0 ! tee name=t1 t1.src_0 ! avdec_aac ! audioconvert ! audioresample ! mux.sink_0 nvstreammux name=mux batch-size=1 ! nvinferaudio config-file-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-audio/configs/config_infer_audio_sonyc.txt batch-size=1 audio-transform="melsdb,fft_length=2560,hop_size=692,dsp_window=hann,num_mels=128,sample_rate=44100,p2db_ref=(float)1.0,p2db_min_power=(float)0.0,p2db_top_db=(float)80.0" audio-framesize=441000 audio-hopsize=110250 ! queue ! fakesink sync=0 enable-last-sample=false async=false m.video_0 ! h264parse ! nvv4l2decoder ! mux2.sink_0 nvstreammux name=mux2 batch-size=1 ! nvinfer config-file-path=/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt ! queue ! nvv4l2h264enc ! h264parse ! qm.video_0  qtmux name=qm ! filesink location=out.mp4 sync=0 enable-last-sample=false async=false t1.src_1 ! queue ! qm.audio_0

[user64451]

Thank you for reference

Because nvstreamdemux has some issue with audio batch, my custom-plugin-2 after nvinfer-audio has to run frame by frame, cannot parallel.
So if I run the pipeline like the image blow, how can I sync frame and audio for further processing? and should I change my pipeline using riva service like example deepstream-avsync?

Thank you so much.

[Fiona.Chen]

@user64451

So if I run the pipeline like the image blow, how can I sync frame and audio for further processing?

I’ve given you the sample, please investigate the pipeline I gave you. No RIVA service is needed.

Post it again

USE_NEW_NVSTREAMMUX=yes gst-launch-1.0 filesrc location=test.mov ! qtdemux name=m m.audio_0 ! tee name=t1 t1.src_0 ! avdec_aac ! audioconvert ! audioresample ! mux.sink_0 nvstreammux name=mux batch-size=1 ! nvinferaudio config-file-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-audio/configs/config_infer_audio_sonyc.txt batch-size=1 audio-transform="melsdb,fft_length=2560,hop_size=692,dsp_window=hann,num_mels=128,sample_rate=44100,p2db_ref=(float)1.0,p2db_min_power=(float)0.0,p2db_top_db=(float)80.0" audio-framesize=441000 audio-hopsize=110250 ! queue ! fakesink sync=0 enable-last-sample=false async=false m.video_0 ! h264parse ! nvv4l2decoder ! mux2.sink_0 nvstreammux name=mux2 batch-size=1 ! nvinfer config-file-path=/opt/nvidia/deepstream/deepstream/samples/configs/deepstream-app/config_infer_primary.txt ! queue ! nvv4l2h264enc ! h264parse ! qm.video_0  qtmux name=qm ! filesink location=out.mp4 sync=0 enable-last-sample=false async=false t1.src_1 ! queue ! qm.audio_0

[user64451]

yup, thank you so much

[user64451]

@Fiona.Chen
I wanna save the raw audio for further processing so
Could you mind showing me the source code or any example to get the raw audio of the pipeline at tiler_sink_pad_buffer_probe (for example)? Thank you so much.

[Fiona.Chen]

You can save the wav data after “avdec_aac” with “wavenc”. This is basic audio decoding and GStreamer skills. GStreamer: open source multimedia framework. Please google by yourself.