Is there a way to convert the BioMegatron and Gatortron .nemo or checkpoint files to be used in Hugging Face? I created a script based on transformers/convert_megatron_bert_checkpoint.py at main · huggingface/transformers · GitHub
Unfortunately, after my attempt of conversion to hugging face format from BioMegatron .ckpt file to pytorch_model.bin I tried fine-tuning for a NER task but I noticed low F1 scores after 1 epoch of training unlike what I see in NeMo/Token_Classification-BioMegatron.ipynb at main · NVIDIA/NeMo · GitHub. I am assuming the weights were not set correctly even though I only saw initializing warnings related to [cls.predictions.decoder.weight’] and [classifier.weight, classifier.bias] using MegatronBertForTokenClassification.
Any help would be appreciated.