How to integrate custom audio model into DeepStream app?

I want to integrate a custom audio model into DeepStream app, I run successfully sample audio app in sources/apps/sample_apps/deepstream-audio/. Now I want to change the audio model to other customed audio model. But I cannot find any document or tutorial to guide me for implementing.
Please give me some documents or other example using audio model in DeepStream app.
Thank you very much.

Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU)
• DeepStream Version
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

1 Like

Thank you for your reply. I want to share my setup with you as below:
• Hardware Platform (Jetson / GPU): Jetson
• DeepStream Version: 6.2
• JetPack Version (valid for Jetson only): 5.1.1
• TensorRT Version: TensorRT 8.5.2
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs): questions
• Detail: I want to change the audio classify model (“sonyc_audio_classify.onnx” in deepstream-audio sample app that I run successfully) to other audio classify model. Can you give me some docs or source to refer?

please read the readme \opt\nvidia\deepstream\deepstream-6.2\sources\apps\sample_apps\deepstream-audio\README, the configuration file is ds_audio_sonyc_test_config.txt. you need to modify the [audio-classifier] part in ds_audio_sonyc_test_config.txt.
deepstream uses nvinferaudio plugin to do audio inference. please refer to this doc for parameters 's explanation.

1 Like

Is this still an DeepStream issue to support?

1 Like

Thank you for your documentations. I read all docs you sent, I understand parameters 's explanation and config them; I can run deepstream-audio sample app successfully.
Now, I want to know flow how to integrate other audio classify model such as BEATs, etc (in Audio Classification | Papers With Code) into DeepStream App to replace for “sonyc_audio_classify.onnx” in deepstream-audio sample app. I used to custom a YOLOv7 object detection (getting output of model, parsing bounding box, …) model following DS_using_custom_model and it works correctly, but for audio I don’t see any dev guide to custom a audio model.
Thank you very much.

There is no update from you for a period, assuming this is not an issue any more. Hence we are closing this topic. If need further support, please open a new one. Thanks.
first please understand the preprocess of audio model 's input, then replace the custom paramters in ds_audio_sonyc_test_config.txt and config_infer_audio_sonyc.txt.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.