Gstreamer pipeline error

Running an a 4GB Jetson Nano with Deepstrem 6.0,
I want to figure out how nvinferaudio works. So I wanted to setup a small gstreamer pipeline to
read a soundfile ( provided with the deepstream framework)
write the found info in a file
play the soundfile on the jetson nano

the pipeline is as follows :
gst-launch-1.0 --gst-debug=3 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sonyc_mixed_audio.wav ! wavparse ! nvinferaudio config-file-path= /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-audio/configs/config_infer_audio_sonyc.txt ! autoaudiosink
i get the followingt output

Setting pipeline to PAUSED …
ERROR: Deserialize engine failed because file path: /opt/nvidia/deepstream/deepstream-6.0/sources/apps/sample_apps/deepstream-audio/configs/…/…/…/…/…/samples/models/SONYC_Audio_Classifier/sonyc_audio_classify.onnx_b2_gpu0_fp32.engine open error
0:00:03.684046053 14831 0x557d3192a0 WARN nvinferbase gstnvinferbase.cpp:424:gst_nvinfer_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1889> [UID = 1]: deserialize engine from file :/opt/nvidia/deepstream/deepstream-6.0/sources/apps/sample_apps/deepstream-audio/configs/…/…/…/…/…/samples/models/SONYC_Audio_Classifier/sonyc_audio_classify.onnx_b2_gpu0_fp32.engine failed
0:00:03.703437430 14831 0x557d3192a0 WARN nvinferbase gstnvinferbase.cpp:424:gst_nvinfer_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:1996> [UID = 1]: deserialize backend context from engine from file :/opt/nvidia/deepstream/deepstream-6.0/sources/apps/sample_apps/deepstream-audio/configs/…/…/…/…/…/samples/models/SONYC_Audio_Classifier/sonyc_audio_classify.onnx_b2_gpu0_fp32.engine failed, try rebuild
WARNING: [TRT]: onnx2trt_utils.cpp:364: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
WARNING: [TRT]: DLA requests all profiles have same min, max, and opt value. All dla layers are falling back to GPU
WARNING: [TRT]: Detected invalid timing cache, setup a local cache instead
ERROR: Serialize engine failed because of file path: /opt/nvidia/deepstream/deepstream-6.0/samples/models/SONYC_Audio_Classifier/sonyc_audio_classify.onnx_b2_gpu0_fp32.engine opened error
0:03:46.292193925 14831 0x557d3192a0 WARN nvinferbase gstnvinferbase.cpp:424:gst_nvinfer_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::buildModel() <nvdsinfer_context_impl.cpp:1942> [UID = 1]: failed to serialize cude engine to file: /opt/nvidia/deepstream/deepstream-6.0/samples/models/SONYC_Audio_Classifier/sonyc_audio_classify.onnx_b2_gpu0_fp32.engine
INFO: [FullDims Engine Info]: layers num: 2
0 INPUT kFLOAT input.1 1x635x128 min: 1x1x635x128 opt: 2x1x635x128 Max: 2x1x635x128
1 OUTPUT kFLOAT output1 31 min: 0 opt: 0 Max: 0

0:03:46.477738145 14831 0x557d3192a0 WARN nvinferaudio gstnvinferaudio.cpp:283:gst_nvinferaudio_start: error: Failed to create audio transform
0:03:46.486117768 14831 0x557d3192a0 WARN basesrc gstbasesrc.c:3583:gst_base_src_start_complete: pad not activated yet
Pipeline is PREROLLING …
ERROR: from element /GstPipeline:pipeline0/GstNvInferAudio:nvinferaudio0: Failed to create audio transform
Additional debug info:
/dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinferaudio/gstnvinferaudio.cpp(283): gst_nvinferaudio_start (): /GstPipeline:pipeline0/GstNvInferAudio:nvinferaudio0
0:03:46.488809845 14831 0x557d2ba190 FIXME default gstutils.c:3981:gst_pad_create_stream_id_internal:wavparse0:src Creating random stream-id, consider implementing a deterministic way of creating a stream-id
ERROR: pipeline doesn’t want to preroll.
Setting pipeline to NULL …

(gst-launch-1.0:14831): GStreamer-CRITICAL **: 17:06:46.909: gst_object_unref: assertion ‘object != NULL’ failed
0:03:46.514756852 14831 0x557d2ba190 WARN wavparse gstwavparse.c:1789:gst_wavparse_stream_headers: error: Couldn’t read in header -2 (flushing)
0:03:46.514961804 14831 0x557d2ba190 WARN wavparse gstwavparse.c:2293:gst_wavparse_loop: error: Internal data stream error.
0:03:46.515022690 14831 0x557d2ba190 WARN wavparse gstwavparse.c:2293:gst_wavparse_loop: error: streaming stopped, reason error (-5)
Freeing pipeline …

couls someone please help me in getting this to work ?
Fo exemple where is the file sonyc_audio_classify.onnx_b2_gpu0_fp32.engine ? i only see this one sonyc_audio_classify.onnx

when trying to add the raw-output-file-write element to get the following pipeline

gst-launch-1.0 --gst-debug=3 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sonyc_mixed_audio.wav ! wavparse ! nvinferaudio config-file-path= /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-audio/configs/config_infer_audio_sonyc.txt raw-output-file-write=inferaudioresults.txt! autoaudiosink

i get this error
0:00:01.441027322 14701 0x556c7f1580 ERROR GST_PIPELINE grammar.y:455:gst_parse_element_set: could not set property “raw-output-file-write” in element “nvinferaudio0” to “~/Documents/inferaudioresults.txt!”

The nvstreammux (New) must be used to generate batched data for inference. Gst-nvstreammux New (Beta) — DeepStream 6.0 Release documentation

To enable nvstreammux(New), should run “export USE_NEW_NVSTREAMMUX=yes”

And the following pipeline can work

gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sonyc_mixed_audio.wav ! wavparse ! audioresample ! m.sink_0 nvstreammux name=m batch-size=1 ! nvinferaudio config-file-path= /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-audio/configs/config_infer_audio_sonyc.txt ! autoaudiosink
or

gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sonyc_mixed_audio.wav ! wavparse ! m.sink_0 nvstreammux name=m batch-size=1 ! nvinferaudio config-file-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-audio/configs/config_infer_audio_sonyc.txt audio-transform=melsdb,fft_length=2560,hop_size=692,dsp_window=hann,num_mels=128,sample_rate=44100 ! autoaudiosink

Hi Fiona, thanks for the reply,
Unfortunately I get errors,
for the 1 st command :
gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sonyc_mixed_audio.wav ! wavparse ! audioresample ! m.sink_0 nvstreammux name=m batch-size=1 ! nvinferaudio config-file-path= /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-audio/configs/config_infer_audio_sonyc.txt ! autoaudiosink
i get the following result max_fps_dur 8.33333e+06 min_fps_dur 2e+08
Setting pipeline to PAUSED …
INFO: [FullDims Engine Info]: layers num: 2
0 INPUT kFLOAT input.1 1x635x128 min: 1x1x635x128 opt: 2x1x635x128 Max: 2x1x635x128
1 OUTPUT kFLOAT output1 31 min: 0 opt: 0 Max: 0

0:00:31.376290577 31162 0x559ebe8270 WARN nvinferaudio gstnvinferaudio.cpp:283:gst_nvinferaudio_start: error: Failed to create audio transform
Pipeline is PREROLLING …
ERROR: from element /GstPipeline:pipeline0/GstNvInferAudio:nvinferaudio0: Failed to create audio transform
Additional debug info:
/dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinferaudio/gstnvinferaudio.cpp(283): gst_nvinferaudio_start (): /GstPipeline:pipeline0/GstNvInferAudio:nvinferaudio0
ERROR: pipeline doesn’t want to preroll.
Setting pipeline to NULL …

(gst-launch-1.0:31162): GStreamer-WARNING **: 16:51:43.431: Failed to load plugin ‘/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstlibav.so’: /usr/lib/aarch64-linux-gnu/libgomp.so.1: cannot allocate memory in static TLS block

several more same lines and then pipeline gets freed up

and the second command :

gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sonyc_mixed_audio.wav ! wavparse ! m.sink_0 nvstreammux name=m batch-size=1 ! nvinferaudio config-file-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-audio/configs/config_infer_audio_sonyc.txt audio-transform=melsdb,fft_length=2560,hop_size=692,dsp_window=hann,num_mels=128,sample_rate=44100 ! autoaudiosink
I get the following output ( an a 4GB Jetson Nano :
max_fps_dur 8.33333e+06 min_fps_dur 2e+08
Setting pipeline to PAUSED …
INFO: [FullDims Engine Info]: layers num: 2
0 INPUT kFLOAT input.1 1x635x128 min: 1x1x635x128 opt: 2x1x635x128 Max: 2x1x635x128
1 OUTPUT kFLOAT output1 31 min: 0 opt: 0 Max: 0

0:01:20.610181502 31011 0x558ad8c4d0 WARN nvinferaudio gstnvinferaudio.cpp:283:gst_nvinferaudio_start: error: Failed to create audio transform
Pipeline is PREROLLING …
ERROR: from element /GstPipeline:pipeline0/GstNvInferAudio:nvinferaudio0: Failed to create audio transform
Additional debug info:
/dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinferaudio/gstnvinferaudio.cpp(283): gst_nvinferaudio_start (): /GstPipeline:pipeline0/GstNvInferAudio:nvinferaudio0
ERROR: pipeline doesn’t want to preroll.
Setting pipeline to NULL …

(gst-launch-1.0:31011): GStreamer-WARNING **: 16:48:09.520: Failed to load plugin ‘/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstlibav.so’: /usr/lib/aarch64-linux-gnu/libgomp.so.1: cannot allocate memory in static TLS block

(gst-launch-1.0:31011): GStreamer-WARNING **: 16:48:10.377: Failed to load plugin ‘/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstlibav.so’: /usr/lib/aarch64-linux-gnu/libgomp.so.1: cannot allocate memory in static TLS block
the last line repeating about 20 times followed by : freeing pipeline

i

Seems your deepstream is not installed correctly. Please reinstall your board. Quickstart Guide — DeepStream 6.0 Release documentation

i reinstalled everything, still not working.
this pipeline works well :
USE_NEW_NVSTREAMMUX=yes gst-launch-1.0
uridecodebin uri=file:///opt/nvidia/deepstream/deepstream/samples/streams/sonyc_mixed_audio.wav name=source_0 ! queue ! audioconvert ! audioresample ! mux.sink_0
nvstreammux name=mux batch-size=2 !
nvstreamdemux name=demux
demux.src_0 ! audioconvert ! audioresample ! alsasink async=false

I now added the following :
export LD_PRELOAD=/usr/lib/aarch64-linux-gnu/libgomp.so.1
added a
source ~/.bashrc

and now I get
max_fps_dur 8.33333e+06 min_fps_dur 2e+08
Setting pipeline to PAUSED …
INFO: [FullDims Engine Info]: layers num: 2
0 INPUT kFLOAT input.1 1x635x128 min: 1x1x635x128 opt: 2x1x635x128 Max: 2x1x635x128
1 OUTPUT kFLOAT output1 31 min: 0 opt: 0 Max: 0

0:00:04.658258691 27111 0x55c4f75070 WARN nvinferaudio gstnvinferaudio.cpp:283:gst_nvinferaudio_start: error: Failed to create audio transform
Pipeline is PREROLLING …
ERROR: from element /GstPipeline:pipeline0/GstNvInferAudio:nvinferaudio0: Failed to create audio transform
Additional debug info:
/dvs/git/dirty/git-master_linux/deepstream/sdk/src/gst-plugins/gst-nvinferaudio/gstnvinferaudio.cpp(283): gst_nvinferaudio_start (): /GstPipeline:pipeline0/GstNvInferAudio:nvinferaudio0
ERROR: pipeline doesn’t want to preroll.
Setting pipeline to NULL …

(gst-launch-1.0:27111): GStreamer-CRITICAL **: 11:39:05.330: gst_object_unref: assertion ‘object != NULL’ failed
Freeing pipeline …

when launching
gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sonyc_mixed_audio.wav ! wavparse ! audioresample ! m.sink_0 nvstreammux name=m batch-size=1 ! nvinferaudio config-file-path= /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-audio/configs/config_infer_audio_sonyc.txt ! autoaudiosink

or when launching :
gst-launch-1.0 filesrc location=/opt/nvidia/deepstream/deepstream/samples/streams/sonyc_mixed_audio.wav ! wavparse ! m.sink_0 nvstreammux name=m batch-size=1 ! nvinferaudio config-file-path=/opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-audio/configs/config_infer_audio_sonyc.txt audio-transform=melsdb,fft_length=2560,hop_size=692,dsp_window=hann,num_mels=128,sample_rate=44100 ! autoaudiosin

In order to use inferaudio, I need an a source with the CAP audio/x-raw (memory:NVMM)

Or for video streams there is a plugin nvvv4l2decoder. It has a source of video/x-raw(memory:NVMM).
Is there any corresponding audio decoder available, allowing to have a wav file as input and regenerate audio/x- (memory:NVMM)

any ideas about that

We are getting the same issue as jetson1 (‘Failed to create audio transform’), when we are trying to run both pipelines. Could you please suggest, how it can be possible to solve it (i tried to make some modifications in pipeline, but it didn’t help).
And also one other small question regarding the mel spectrograms, that are generated with nvinferaudio plugin for inference… is there a possibility to visualize them? Probably they will be saved in the metadata? We want to compare them with the spectrograms from Python librosa library, that we now using for our model.
Thanks in advance!

Hi user49080, it is interesting that some other user has now the same problem, for months i have not found any other use case on the web. I even do not understand quit well what the inferaudio is really intended to do. At the beginning i thought it was using an audio recognition cnn, but now i really think that the audio is transformed into some kind of image and the network is comparing aginst that image. Really hard to find any other example except the deepstream-audio example in C, but hard to understand for me. If you get any other results, please let me know.

Hi, jetson1, ok:) i’ll write if we’ll be able to get any results. Our goal now is to run the pipeline and to try to compare the mel spectograms from nvinferaudio with the mel spectrograms from librosa.