However, it’s not clear how audio-transform works.
When I set hop_size to 1024 (the original one is 692), in my experience, the input size of model should be changed accordingly. But the current size (batch_size, 1, 635, 128) still works. So, if possible, could anyone give more explaination of audio-transform work flow?
To compare with other open source (mel-scale spectrogram), is it possible to print out the computation result of audio-transform or save the result to another file?
As far as I know, we need to make sure the computation of mel-scale spectrogram match audio-transform process in deepstream when we train the model. Thus, the pre-trained model can be converted to an onnx format and we don’t need to compute mel-scale spectrogram by ourself in inference time (becuase audio-transform does). That’s why we need to know how audio-transform works.
For the current input size (batch_size, 1, 635, 128) of the model , what does 635 mean? the number of windows? In mel-scale spectrogram computation, increasing hop_size will result in decreasing the number of windows (or say frames) so as to get a smaller input size of model. But it does not in audio-transform computation.
Besides, how the window length (or say framesize K in the following diagram) is setting in audio-transform? the same as fft_length? (Q means hop_size) (c - Hop_Size meaning in aubio - Stack Overflow)
But my question is “why the change of hop_size does not affect the input size of the default SONYC audio model?” Is there any exception processing? Or do I misunderstand sth? Thanks again.
Or I would say, if the input size of my model is (batch_size, 1, 625, 128), how could I set hop_size? As far as I know, it’s not clear how to set hop_size in documentation. Thanks.
From this link How to feed raw audio into the model by nvinferaudio?, I found that you said “For the model used in /opt/nvidia/deepstream/deepstream/sources/apps/sample_apps/deepstream-audio, the shape is batch x channel x frame_samples x sample_size”
If so, my question is why I changed the hop_size, the frame_samples were not changed accordingly? It’s weird.
Beside, you said “the nvinferaudio plugin does not support feeding raw audio input to the model. The input audio is first converted to log mel spectogram and then passed on to the model. We will add this feature in our roadmap.”
Does deepstream 6.4 support the feeding of raw audio to the model?
In the model input dimensions (batch_size, 1, 635, 128), 635 is the number of windows or number of time frames, 128 is the number of mel frequency bands (num_mels).
The audio transform used in this case is log mel spectogram, which calculates the number of time frames as:
num_time_frames = 1 + (audio-framesize - fft_length) / hop_size;
for e.g. 634 = 1 + ( 441000 - 2560) / 692.
‘fft_length’ is same as the window length K in the above diagram.
The output of the audio transform is stored in a tensor, whose size is allocated as per the model input dimensions.
The tensor memory is memset to 0. So in the above example, 634 time frames will be filled in the tensor and the last 128 values corresponding to the last time frame will be zero.
If the hop_size is increased, the audio_framesize should be adjusted as per above formula to get the required number of time frames or the model should be updated as per the resulting dimensions.
Feeding raw audio to the nvinferaudio plugin is not supported as of now.
To make sure our computed log-mel spectrogram is the same as what audio_transform did, could you please help to check weather audio_transform performs the following steps? If missing, please also update it. We appreciate your help.
step 2. preemphasize input audio via coefficient 0.97 ?
step 3. compute spectrum of the result of step 2 ?
step 4. compute the power spectrum of the result of step 3?
step 5. compute mel banks using num_mels? If so, what’s low_frequency and
high_frequency do you set for cutting freuqnecy for mel bins? low_freq is 20 and high_freq is 8000 for sample rate 16000 (high_freq is set as sample_rate /2)
step 6. sum with mel filterbanks over the power spectrum?
step 7. transform the result of step 6 to log-scale ?
