Nvstreammux for audio data

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU) GPU
• DeepStream Version 7.1
• JetPack Version (valid for Jetson only)
• TensorRT Version10.3.0.26
• NVIDIA GPU Driver Version (valid for GPU only) 575.51.03
**• Issue Type( questions, new requirements, bugs)**question
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

Dear all,
I am trying to do inference on my own audio classifier based on MobilenetV3. However the inference never gets done because (i suspect) streammux is not correctly passing data downstream. I provide my .py here and my config file for nvinferaudio.
config.txt (666 Bytes).

I am trying to perform inference over a 5 seconds .wav file with characteristics as per parameters in capsfilter. The output i am getting looks like this:

image
image1821×782 56.4 KB

As you can see by bInferDone=0 the inference is not being done, besides, there is a sudden drop on the buffer’ size right after caps module. Am i making a mistake assuming data is being lost in streammux?

3

The “bInferDone” is only used by the nvtracker. For audio case, there is no object, so we don’t touch it.

How do you measure the buffer size?

4

Thanks for the answer @Fiona.Chen .

To get the buffer info i am using the following probe function:


def generic_buffer_probe_fn(pad, info, user_tag):
    gst_buffer = info.get_buffer()
    if not gst_buffer:
        print(f"PROBE ({user_tag}): Unable to get GstBuffer")
        return Gst.PadProbeReturn.OK

    duration_str = "unknown"
    if gst_buffer.duration != Gst.CLOCK_TIME_NONE:
        duration_str = f"{gst_buffer.duration / Gst.SECOND:.6f} s ({gst_buffer.duration} ns)"
    else:
        duration_str = f"unknown s ({gst_buffer.duration} ns)"


    print(f"PROBE ({user_tag}): Buffer received. Duration: {duration_str}, Size: {gst_buffer.get_size()} bytes")
    return Gst.PadProbeReturn.OK

I then probe on the different pipeline elements, for example:

    srcpad = capsfilter.get_static_pad("src")
    srcpad.add_probe(Gst.PadProbeType.BUFFER, generic_buffer_probe_fn, "CAPSFILTER_SRC")
5

Update.

After more (a lot) of debugging i have come to understand that there was not problem with streammux. The 24 bytes the streammux has been sending correspond to the pointer (correct me if i’m wrong) of the 4096 bytes of actual data. However during my debugging proccess i found out that

USE_NEW_NVSTREAMMUX=yes gst-launch-1.0 -v filesrc location=my_audio.wav ! wavparse ! audioconvert ! audioresample ! "audio/x-raw,format=S16LE,layout=interleaved,rate=22050,channels=1" ! nvstreammux name=mux batch-size=1 batched-push-timeout=5000000 num-surfaces-per-frame=1 ! fakesink dump=true

gave also an error since the mux was not accepting the caps. Following the deepstream-audio-app i got my pipeline to work, solving this issue.

USE_NEW_NVSTREAMMUX=yes gst-launch-1.0 -v     uridecodebin uri=file:///mypath/N+T_clip_12723.wav name=source_0 ! queue ! audioconvert ! audioresample ! mux.sink_0     nvstreammux name=mux batch-size=1 batched-push-timeout=5000000 !     nvinferaudio name=infer config-file-path=/mypath/config.txt         audio-framesize=110250         audio-hopsize=110250         audio-transform='melsdb,fft_length=512,hop_size=256,dsp_window=hann,num_mels=90,sample_rate=22050,p2db_ref=(float)1.0,p2db_min_power=(float)0.0,p2db_top_db=(float)80.0' !     fakesink

The main difference is the use of uridecodebin instead of filesrc. I let the python code here in case is useful to someone.

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.