# How audio-transform works? **URL:** \ **Category:** DeepStream SDK\ **Created:** [January 3, 2024, 5:17am UTC](https://forums.developer.nvidia.com/t/how-audio-transform-works/277720 "2024-01-03T05:17:14Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![jmren](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@jmren](https://forums.developer.nvidia.com/u/jmren)\ **Post date:** [January 3, 2024, 5:17am UTC](https://forums.developer.nvidia.com/t/how-audio-transform-works/277720/1 "2024-01-03T05:17:14Z")
From the nvinferaudio guide ([Gst-nvinferaudio — DeepStream documentation 6.4 documentation](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinferaudio.html)), I can find parameters of deepstream audio. However, it’s not clear how audio-transform works. 1. When I set hop\_size to 1024 (the original one is 692), in my experience, the input size of model should be changed accordingly. But the current size (batch\_size, 1, 635, 128) still works. So, if possible, could anyone give more explaination of audio-transform work flow? 2. To compare with other open source (mel-scale spectrogram), is it possible to print out the computation result of audio-transform or save the result to another file? Many thanks. ---
**Author:** ![Fiona.Chen](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/fiona.chen/32/15508_2.png) [@Fiona.Chen](https://forums.developer.nvidia.com/u/Fiona.Chen)\ **Post date:** [January 3, 2024, 9:55am UTC](https://forums.developer.nvidia.com/t/how-audio-transform-works/277720/3 "2024-01-03T09:55:30Z")
> [@jmren](#): > > When I set hop\_size to 1024 (the original one is 692), in my experience, the input size of model should be changed accordingly. The transform parameters are decided by your own model. Do you know which kind of preprocessing is needed for your model? > [@jmren](#): > > 1. To compare with other open source (mel-scale spectrogram), is it possible to print out the computation result of audio-transform or save the result to another file? No. There is no data dump function. The parameters are explained : [Gst-nvinferaudio — DeepStream 6.3 Release documentation](https://docs.nvidia.com/metropolis/deepstream/6.3/dev-guide/text/DS_plugin_gst-nvinferaudio.html#id5) ---
**Author:** ![jmren](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@jmren](https://forums.developer.nvidia.com/u/jmren)\ **Post date:** [January 3, 2024, 4:40pm UTC](https://forums.developer.nvidia.com/t/how-audio-transform-works/277720/4 "2024-01-03T16:40:25Z")
> [@Fiona.Chen](#): > > The transform parameters are decided by your own model. Do you know which kind of preprocessing is needed for your model? We’re trying to deploy “[Efficient Pre-Trained CNNs for Audio Pattern Recognition](https://github.com/fschmid56/EfficientAT)” on deepstream. The preprocess is to compute mel-scale spectrogram of a given waveform ([EfficientAT/models/preprocess.py at main · fschmid56/EfficientAT · GitHub](https://github.com/fschmid56/EfficientAT/blob/main/models/preprocess.py)). As far as I know, we need to **make sure the computation of mel-scale spectrogram match audio-transform process in deepstream when we train the model**. Thus, the pre-trained model can be converted to an onnx format and we don’t need to compute mel-scale spectrogram by ourself in inference time (becuase audio-transform does). That’s why we need to know how audio-transform works. For the current input size (batch\_size, 1, 635, 128) of the model , what does 635 mean? the number of windows? In mel-scale spectrogram computation, increasing hop\_size will result in decreasing the number of windows (or say frames) so as to get a smaller input size of model. But it does not in audio-transform computation. Besides, how the window length (or say framesize _K_ in the following diagram) is setting in audio-transform? the same as fft\_length? (Q means hop\_size) ([c - Hop\_Size meaning in aubio - Stack Overflow](https://stackoverflow.com/questions/51276464/hop-size-meaning-in-aubio)) ![](https://global.discourse-cdn.com/nvidia/original/4X/f/7/0/f706d6a6c0062bf7ba1ee521a77f087f2ae772ad.png) > [@Fiona.Chen](#): > > No. There is no data dump function. Could you please tell us where can we find the source code of audio-transfrom? Is it released? We think we may need to trace the code. Thanks again. ---
**Author:** ![Fiona.Chen](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/fiona.chen/32/15508_2.png) [@Fiona.Chen](https://forums.developer.nvidia.com/u/Fiona.Chen)\ **Post date:** [January 4, 2024, 7:40am UTC](https://forums.developer.nvidia.com/t/how-audio-transform-works/277720/5 "2024-01-04T07:40:30Z")
> [@jmren](#): > > Could you please tell us where can we find the source code of audio-transfrom? Is it released? We think we may need to trace the code. No. It is not open source. ---
**Author:** ![jmren](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@jmren](https://forums.developer.nvidia.com/u/jmren)\ **Post date:** [January 4, 2024, 8:43am UTC](https://forums.developer.nvidia.com/t/how-audio-transform-works/277720/6 "2024-01-04T08:43:29Z")
Thanks for the reply. But my question is “ **why the change of hop\_size does not affect the input size of the default SONYC audio model?** ” Is there any exception processing? Or do I misunderstand sth? Thanks again. ---
**Author:** ![jmren](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@jmren](https://forums.developer.nvidia.com/u/jmren)\ **Post date:** [January 5, 2024, 12:46am UTC](https://forums.developer.nvidia.com/t/how-audio-transform-works/277720/7 "2024-01-05T00:46:47Z")
Hi @Fiona.Chen Or I would say, if the input size of my model is (batch\_size, 1, 625, 128), how could I set hop\_size? As far as I know, it’s not clear how to set hop\_size in documentation. Thanks. ---
**Author:** ![jmren](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@jmren](https://forums.developer.nvidia.com/u/jmren)\ **Post date:** [January 9, 2024, 12:45am UTC](https://forums.developer.nvidia.com/t/how-audio-transform-works/277720/8 "2024-01-09T00:45:39Z")
Hi @Fiona.Chen From this link [How to feed raw audio into the model by nvinferaudio?](https://forums.developer.nvidia.com/t/how-to-feed-raw-audio-into-the-model-by-nvinferaudio/241922), I found that you said “For the model used in /opt/nvidia/deepstream/deepstream/sources/apps/sample\_apps/deepstream-audio, **the shape is batch x channel x frame\_samples x sample\_size** ” If so, my question is **why I changed the hop\_size, the frame\_samples were not changed accordingly**? It’s weird. Beside, you said “the nvinferaudio plugin does not support feeding raw audio input to the model. **The input audio is first converted to log mel spectogram and then passed on to the model**. **We will add this feature in our roadmap**.” Does deepstream 6.4 support the feeding of raw audio to the model? Thanks for your kindly replay. ---
**Author:** ![Fiona.Chen](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/fiona.chen/32/15508_2.png) [@Fiona.Chen](https://forums.developer.nvidia.com/u/Fiona.Chen)\ **Post date:** [January 17, 2024, 5:58am UTC](https://forums.developer.nvidia.com/t/how-audio-transform-works/277720/9 "2024-01-17T05:58:49Z")
Sorry for late response. In the model input dimensions (batch\_size, 1, 635, 128), 635 is the number of windows or number of time frames, 128 is the number of mel frequency bands (num\_mels). The audio transform used in this case is log mel spectogram, which calculates the number of time frames as: num\_time\_frames = 1 + (audio-framesize - fft\_length) / hop\_size; for e.g. 634 = 1 + ( 441000 - 2560) / 692. ‘fft\_length’ is same as the window length K in the above diagram. The output of the audio transform is stored in a tensor, whose size is allocated as per the model input dimensions. The tensor memory is memset to 0. So in the above example, 634 time frames will be filled in the tensor and the last 128 values corresponding to the last time frame will be zero. If the hop\_size is increased, the audio\_framesize should be adjusted as per above formula to get the required number of time frames or the model should be updated as per the resulting dimensions. Feeding raw audio to the nvinferaudio plugin is not supported as of now. ---
**Author:** ![jmren](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@jmren](https://forums.developer.nvidia.com/u/jmren)\ **Post date:** [January 18, 2024, 3:08am UTC](https://forums.developer.nvidia.com/t/how-audio-transform-works/277720/10 "2024-01-18T03:08:50Z")
@Fiona.Chen Thanks for the replay To make sure our computed log-mel spectrogram is the same as what audio\_transform did, could you please help to check weather audio\_transform performs the following steps? If missing, please also update it. We appreciate your help. (we use [torchaudio.compliance.kaldi.fbank — Torchaudio 2.1.2 documentation](https://pytorch.org/audio/stable/generated/torchaudio.compliance.kaldi.fbank.html#torchaudio.compliance.kaldi.fbank) to compute log-mel-spectrogram) step 1. subtract each frame by its mean? step 2. preemphasize input audio via coefficient 0.97 ? step 3. compute spectrum of the result of step 2 ? step 4. compute the power spectrum of the result of step 3? step 5. compute mel banks using num\_mels? If so, what’s low\_frequency and high\_frequency do you set for cutting freuqnecy for mel bins? low\_freq is 20 and high\_freq is 8000 for sample rate 16000 (high\_freq is set as sample\_rate /2) step 6. sum with mel filterbanks over the power spectrum? step 7. transform the result of step 6 to log-scale ? ---
**Author:** ![Fiona.Chen](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/fiona.chen/32/15508_2.png) [@Fiona.Chen](https://forums.developer.nvidia.com/u/Fiona.Chen)\ **Post date:** [January 18, 2024, 10:33am UTC](https://forums.developer.nvidia.com/t/how-audio-transform-works/277720/11 "2024-01-18T10:33:28Z")
The log mel spectogram audio transform is implemented as follows: 1. Mel spectogram with the specified fft\_length, window type etc 2. power to db conversion Pre-emphasis or mean subtraction is not performed. ---
**Author:** ![jmren](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@jmren](https://forums.developer.nvidia.com/u/jmren)\ **Post date:** [January 19, 2024, 5:59am UTC](https://forums.developer.nvidia.com/t/how-audio-transform-works/277720/12 "2024-01-19T05:59:26Z")
Thanks for the replay, and it works. ---
**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\ **Post date:** [February 2, 2024, 5:59am UTC](https://forums.developer.nvidia.com/t/how-audio-transform-works/277720/13 "2024-02-02T05:59:48Z")
Closed on February 2, 2024, 5:59am UTC This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.