How to integrate custom audio model into DeepStream app?

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

I want to integrate a custom audio model into DeepStream app, I run successfully sample audio app in sources/apps/sample_apps/deepstream-audio/. Now I want to change the audio model to other customed audio model. But I cannot find any document or tutorial to guide me for implementing.
Please give me some documents or other example using audio model in DeepStream app.
Thank you very much.

3

Please provide complete information as applicable to your setup.
• Hardware Platform (Jetson / GPU)
• DeepStream Version
• JetPack Version (valid for Jetson only)
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

1 Like
4

Thank you for your reply. I want to share my setup with you as below:
• Hardware Platform (Jetson / GPU): Jetson
• DeepStream Version: 6.2
• JetPack Version (valid for Jetson only): 5.1.1
• TensorRT Version: TensorRT 8.5.2
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs): questions
• Detail: I want to change the audio classify model (“sonyc_audio_classify.onnx” in deepstream-audio sample app that I run successfully) to other audio classify model. Can you give me some docs or source to refer?

5

please read the readme \opt\nvidia\deepstream\deepstream-6.2\sources\apps\sample_apps\deepstream-audio\README, the configuration file is ds_audio_sonyc_test_config.txt. you need to modify the [audio-classifier] part in ds_audio_sonyc_test_config.txt.
deepstream uses nvinferaudio plugin to do audio inference. please refer to this doc for parameters 's explanation.

1 Like
6

Is this still an DeepStream issue to support?

1 Like
7

Thank you for your documentations. I read all docs you sent, I understand parameters 's explanation and config them; I can run deepstream-audio sample app successfully.
Now, I want to know flow how to integrate other audio classify model such as BEATs, etc (in Audio Classification | Papers With Code) into DeepStream App to replace for “sonyc_audio_classify.onnx” in deepstream-audio sample app. I used to custom a YOLOv7 object detection (getting output of model, parsing bounding box, …) model following DS_using_custom_model and it works correctly, but for audio I don’t see any dev guide to custom a audio model.
Thank you very much.

8

There is no update from you for a period, assuming this is not an issue any more. Hence we are closing this topic. If need further support, please open a new one. Thanks.
first please understand the preprocess of audio model 's input, then replace the custom paramters in ds_audio_sonyc_test_config.txt and config_infer_audio_sonyc.txt.

9

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.