To generate the beams .tsv file, do I need to run [scripts/asr_language_modeling/ngram_lm/eval_beamsearch_ngram.py]link with another LM that has to be n-gram LM?
→ This script is needed to produce the beams but it is not necessary to use N-gram LM. You may just use the beam search decoder without any LM by passing --decoding_mode=beamsearch instead of beamsearch_ngram which is the default mode.But, it is strongly recommended to use N-gram to improve the accuracy. It is very easy and fast to train these models and would not increase the decoding time significantly. In some cases, just N-gram LM with beam search decoder can give you similar results as neural rescorer LM which is slower and need computation.
Can I train an N-gram LM by myself? Should I use the same tokenizer as in the Conformer?
→ Yes, you need to use the LM text data of librispeech downloadable from here. You may use the normalized version of this dataset. You may also concatenate the text transcriptions of train set of LS or any other dataset you have to this data. If you have enough text data, you can also train on your own data. For the tokenizer, no need to worry about it since our script extracts it automatically out of the NeMo file.
Is there another way to get the beams file for the neural rescorer? that doesn’t involve training a new LM?
→ Yes, You may do it with the two scripts mentioned here
Thank you for sharing the links.