Nvstreammux cannot handle audio

[samfarfar]

Hardware Platform : GPU
DeepStream Version 6.2
TensorRT Version 530.41.03
NVIDIA GPU Driver Version 535.54.03
Issue Type:
Currently I am trying to run the 3rd usecase in the documentation for the gst_nvmetautilselement
I am coming to an issue in that the nvstreammux element only has caps for video/x-raw(memory:NVMM) for its sink.

In the usecase the caps of the element before is “audio/x-raw, rate=(int)44100”. How would this pipeline possibly work. The nvstreammux element needs to have the ability to accept audio because as of right now the nvstreammux does not have any way to deal with the audio. Is there another way this can be done?

[yuweiw]

You can try to use Gst-nvstreammux New:
https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvstreammux2.html

[samfarfar]

I believe that by using the USE_NEW_NVSTREAMMUX environment variable and setting it to yes should allow you to use the new mux however when I use the gst-inspect-1.0 command on nvstreammux i get the following:

When I run the example:

gst-launch-1.0 export USE_NEW_NVSTREAMMUX=yes filesrc location=/opt/nvidia/deepstream/deepstream-6.2/samples/streams/sonyc_mixed_audio.wav ! audioconvert ! audioresample ! “audio/x-raw, rate=(int)44100” ! queue ! m.sink_0 nvstreammux name=m batch-size=2 ! queue ! nvinferaudio audio-framesize=44100 audio-hopsize=11025 batch-size=2 config-file-path= /opt/nvidia/deepstream/deepstream-6.2/sources/SONYCAudioClassifier/config_infer_audio_sonyc.txt ! fakesink sync=true

This pipeline causes a failure in the audio transform creation which does not make sense as I am using the config file that was provided.

[yuweiw]

Could you try the steps below and see if it runs well?

   $ cd /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-audio/configs/
   $ deepstream-audio -c ds_audio_sonyc_test_config.txt

[samfarfar]

After running the above commands I am hit with a segmentation fault after a little bit of waiting

[yuweiw]

OK. I have run this demo on my T4 with DeepsSream 6.2. It runs well. Could you use gdb to locate the address of the crash?

$ gdb deepstream-audio
$ r -c ds_audio_sonyc_test_config.txt
$ bt

If you want to use gst-inspect-1.0 check the pad capabilities, you can clean the cache of Gstreamer after set the new nvstreammux macro.

rm -rf ${HOME}/.cache/gstreamer-1.0

[samfarfar]

This was the output after inputting bt:

#0 0x00007fffddedc133 in () at /lib64/gstreamer-1.0/deepstream/libnvdsgst_inferaudio.so
#1 0x00007fffddebe7eb in () at ///opt/nvidia/deepstream/deepstream-6.2/lib/libnvdsgst_inferbase.so
#2 0x00007ffff769e9d0 in gst_base_transform_chain
(pad=pad@entry=0x555555ef8b40 [GstPad|sink], parent=parent@entry=0x555555ef8500 [GstObject|audio_classifier], buffer=buffer@entry=0x7fff40001100 [None])
at …/libs/gst/base/gstbasetransform.c:2348
#3 0x00007ffff7b43a1c in gst_pad_chain_data_unchecked (pad=pad@entry=0x555555ef8b40 [GstPad|sink], type=type@entry=4112, data=data@entry=0x7fff40001100)
at …/gst/gstpad.c:4463
#4 0x00007ffff7b447ae in gst_pad_push_data (pad=pad@entry=0x555555b7e2b0 [GstPad|src], type=type@entry=4112, data=data@entry=0x7fff40001100) at …/gst/gstpad.c:4739
#5 0x00007ffff7b44de4 in gst_pad_push (pad=0x555555b7e2b0 [GstPad|src], buffer=buffer@entry=0x7fff40001100 [None]) at …/gst/gstpad.c:4858
#6 0x00007fffddf33ea9 in gst_queue_push_one (queue=0x555555b7d900 [GstQueue|classifier_queue]) at …/plugins/elements/gstqueue.c:1388
#7 gst_queue_loop (pad=) at …/plugins/elements/gstqueue.c:1541
#8 0x00007ffff7b7dee4 in gst_task_func (task=0x55558e6ca820 [GstTask|classifier_queue:src]) at …/gst/gsttask.c:384
#9 0x00007ffff79f3fb2 in g_thread_pool_thread_proxy (data=) at …/glib/gthreadpool.c:350
#10 0x00007ffff79f1893 in g_thread_proxy (data=0x7fffbc0010d0) at …/glib/gthread.c:831
#11 0x00007ffff7815907 in start_thread (arg=) at pthread_create.c:444
#12 0x00007ffff789b870 in clone3 () at …/sysdeps/unix/sysv/linux/x86_64/clone3.S:81

[yuweiw]

It’s weird that it works well on my T4 with DeepStream 6.2. Could you compare the version of the dependent software on your platform with our official recommended version? dGPU model Platform and OS Compatibility.
Can you provide a detailed description of the steps you took to run our demo?

[samfarfar]

running the following command:

USE_NEW_NVSTREAMMUX=yes gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deeps
tream-6.2/sources/apps/audio_apps/deepstream_asr_tts_app/streams/1272-135031-0000.wav ! wavparse ! audioconvert ! audioresample
! “audio/x-raw, rate=(int)44100” ! queue ! m.sink_0 nvstreammux name=m batch-size=2 ! queue ! nvinferaudio audio-framesize=44100
audio-hopsize=11025 batch-size=2 config-file-path= config_infer_audio_sonyc.txt audio-transform=“melsdb,fft_length=2560,hop_siz
e=692,dsp_window=hann,num_mels=128,sample_rate=44100,p2db_ref=(float)1.0,p2db_min_power=(float)0.0,p2db_top_db=(float)80.0” ! nv
dsmetainsert serialize-lib = “libnvds_audio_metadata_serialization.so” ! nvdsmetaextract deserialize-lib = “libnvds_audio_metada
ta_serialization.so” ! fakesink filesrc location=/opt/nvidia/deepstream/deepstream-6.2/sources/apps/audio_apps/deepstream_asr_t
ts_app/streams/1272-135031-0000.wav ! wavparse ! audioconvert ! audioresample ! “audio/x-raw, rate=(int)44100” ! m.sink_1

which is the equivalent of the demo with the only difference being the source is an actual source provided in the deepstream-6.2 installation. /opt/nvidia/deepstream/deepstream6.2/sources/apps/audio_apps/deepstream_asr_tts_app/streams/1272-135031-0000.wav

The following occurs:

[yuweiw]

I use your gst-lunch-1.0 cli and the demo app below:

   $ cd /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-audio/configs/
   $ deepstream-audio -c ds_audio_sonyc_test_config.txt

They both work normally. Are there any personal changes or library replacements in your environment? You can try to use our docker to test that: nvcr.io/nvidia/deepstream:6.2-triton

[samfarfar]

The link provided does not lead to anywhere and times out when clicked on is there any other links to the same thing?

[yuweiw]

This is not a link to a webpage, this is a docker link. You can refer to the link: Docker Containers to learn how to use docker. It is more convenient to recovery the status.
You can also reconfirm some of your software versions， like CUDA, RensorRT… It needs to be completely consistent with the version we provide. dGPU model Platform and OS Compatibility: Ubuntu 20.04, CUDA 11.8, TRT 8.5.2.2, Driver R525.85.12.

[samfarfar]

I have another question as well Why is there 2 inputs of the same source in this example do you need to have 2 versions of the source for audio?

[yuweiw]

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

This is just a demonstration of how to configure multiple sources. You can configure that according to your own needs.