Were there any solutions to this @AakankshaS @SunilJB ? I am experiencing a similar issue currently following this tutorial: https://github.com/NVIDIA/NeMo/blob/stable/tutorials/nlp/Joint_Intent_and_Slot_Classification.ipynb
The model trains easily, but when loading the model I get the error Missing key(s) in state_dict: “bert_model.embeddings.position_ids”.
[NeMo W 2023-10-17 07:47:23 nemo_logging:349] /usr/local/lib/python3.8/dist-packages/pytorch_lightning/utilities/cloud_io.py:41: LightningDeprecationWarning: `pytorch_lightning.utilities.cloud_io.load` has been deprecated in v1.8.0 and will be removed in v2.0.0. This function is internal but you can copy over its implementation.
rank_zero_deprecation(
[NeMo I 2023-10-17 07:47:29 intent_slot_classification_descriptor:87] Stats calculating for train mode...
[NeMo I 2023-10-17 07:47:29 intent_slot_classification_descriptor:112] Three most popular intents in train mode:
[NeMo I 2023-10-17 07:47:29 data_preprocessing:194] label: 0, 175 out of 9960 (1.76%).
[NeMo I 2023-10-17 07:47:29 data_preprocessing:194] label: 2, 175 out of 9960 (1.76%).
[NeMo I 2023-10-17 07:47:29 data_preprocessing:194] label: 6, 175 out of 9960 (1.76%).
[NeMo I 2023-10-17 07:47:29 intent_slot_classification_descriptor:118] Three most popular slots in train mode:
[NeMo I 2023-10-17 07:47:29 data_preprocessing:194] label: 54, 53689 out of 65658 (81.77%).
[NeMo I 2023-10-17 07:47:29 data_preprocessing:194] label: 38, 1290 out of 65658 (1.96%).
[NeMo I 2023-10-17 07:47:29 data_preprocessing:194] label: 12, 1233 out of 65658 (1.88%).
[NeMo I 2023-10-17 07:47:29 intent_slot_classification_descriptor:121] Total Number of Intents: 9960
[NeMo I 2023-10-17 07:47:29 intent_slot_classification_descriptor:122] Intent Label Frequencies: {0: 175, 2: 175, 6: 175, 7: 175, 8: 175, 9: 175, 11: 175, 13: 175, 14: 175, 15: 175, 16: 175, 17: 175, 18: 175, 19: 175, 20: 175, 22: 175, 23: 175, 24: 175, 25: 175, 27: 175, 28: 175, 30: 175, 35: 175, 36: 175, 37: 175, 39: 175, 41: 175, 42: 175, 43: 175, 44: 175, 45: 175, 46: 175, 47: 175, 48: 175, 49: 175, 51: 175, 52: 175, 53: 175, 55: 175, 57: 175, 58: 175, 59: 175, 61: 175, 63: 175, 62: 171, 56: 165, 60: 163, 38: 162, 26: 151, 4: 142, 50: 134, 5: 126, 32: 126, 29: 114, 21: 110, 1: 106, 54: 97, 33: 89, 10: 79, 12: 79, 3: 72, 40: 70, 34: 69, 31: 35}
[NeMo I 2023-10-17 07:47:29 intent_slot_classification_descriptor:123] Total Number of Slots: 65658
[NeMo I 2023-10-17 07:47:29 intent_slot_classification_descriptor:124] Slots Label Frequencies: {54: 53689, 38: 1290, 12: 1233, 46: 1076, 14: 637, 36: 599, 5: 579, 18: 526, 19: 458, 11: 423, 52: 356, 28: 345, 23: 330, 26: 317, 43: 298, 6: 252, 4: 247, 20: 233, 13: 207, 35: 176, 37: 167, 48: 158, 9: 138, 44: 137, 41: 136, 2: 134, 7: 133, 34: 131, 45: 120, 53: 114, 16: 106, 47: 95, 33: 91, 39: 87, 42: 70, 25: 66, 49: 60, 22: 51, 40: 48, 8: 42, 27: 37, 1: 34, 32: 33, 0: 28, 24: 23, 3: 23, 50: 23, 29: 22, 30: 22, 51: 21, 17: 15, 15: 9, 10: 8, 31: 4, 21: 1}
[NeMo I 2023-10-17 07:47:29 intent_slot_classification_descriptor:128] Intent Weights: {0: 0.8892857142857142, 2: 0.8892857142857142, 6: 0.8892857142857142, 7: 0.8892857142857142, 8: 0.8892857142857142, 9: 0.8892857142857142, 11: 0.8892857142857142, 13: 0.8892857142857142, 14: 0.8892857142857142, 15: 0.8892857142857142, 16: 0.8892857142857142, 17: 0.8892857142857142, 18: 0.8892857142857142, 19: 0.8892857142857142, 20: 0.8892857142857142, 22: 0.8892857142857142, 23: 0.8892857142857142, 24: 0.8892857142857142, 25: 0.8892857142857142, 27: 0.8892857142857142, 28: 0.8892857142857142, 30: 0.8892857142857142, 35: 0.8892857142857142, 36: 0.8892857142857142, 37: 0.8892857142857142, 39: 0.8892857142857142, 41: 0.8892857142857142, 42: 0.8892857142857142, 43: 0.8892857142857142, 44: 0.8892857142857142, 45: 0.8892857142857142, 46: 0.8892857142857142, 47: 0.8892857142857142, 48: 0.8892857142857142, 49: 0.8892857142857142, 51: 0.8892857142857142, 52: 0.8892857142857142, 53: 0.8892857142857142, 55: 0.8892857142857142, 57: 0.8892857142857142, 58: 0.8892857142857142, 59: 0.8892857142857142, 61: 0.8892857142857142, 63: 0.8892857142857142, 62: 0.9100877192982456, 56: 0.9431818181818182, 60: 0.9547546012269938, 38: 0.9606481481481481, 26: 1.0306291390728477, 4: 1.095950704225352, 50: 1.1613805970149254, 5: 1.2351190476190477, 32: 1.2351190476190477, 29: 1.3651315789473684, 21: 1.4147727272727273, 1: 1.4681603773584906, 54: 1.6043814432989691, 33: 1.7485955056179776, 10: 1.9699367088607596, 12: 1.9699367088607596, 3: 2.1614583333333335, 40: 2.2232142857142856, 34: 2.255434782608696, 31: 4.446428571428571}
[NeMo I 2023-10-17 07:47:29 intent_slot_classification_descriptor:130] Slot Weights: {54: 0.02223512857720982, 38: 0.9254122621564482, 12: 0.9681928776819287, 46: 1.109462656302805, 14: 1.8740687883545026, 36: 1.9929579602367582, 5: 2.061799340555817, 18: 2.2695471828551677, 19: 2.606510520047638, 11: 2.8221792392005156, 52: 3.353319713993871, 28: 3.4602371541501977, 23: 3.617520661157025, 26: 3.765873243475767, 43: 4.005979255643685, 6: 4.737229437229437, 4: 4.833124769966875, 20: 5.1235271166601635, 13: 5.767061923583663, 35: 6.782851239669421, 37: 7.14839412084921, 48: 7.555581127733026, 9: 8.650592885375493, 44: 8.713735899137358, 41: 8.777807486631016, 2: 8.908819538670285, 7: 8.975803144224196, 34: 9.112838306731437, 45: 9.948181818181817, 53: 10.47177033492823, 16: 11.262092624356775, 47: 12.566124401913875, 33: 13.118481518481518, 39: 13.721630094043887, 42: 17.054025974025976, 25: 18.087603305785123, 49: 19.896363636363635, 22: 23.40748663101604, 40: 24.870454545454546, 8: 28.423376623376623, 27: 32.26437346437346, 1: 35.11122994652406, 32: 36.175206611570246, 0: 42.63506493506493, 24: 51.903557312252964, 3: 51.903557312252964, 50: 51.903557312252964, 29: 54.26280991735537, 30: 54.26280991735537, 51: 56.846753246753245, 17: 79.58545454545454, 15: 132.64242424242425, 10: 149.22272727272727, 31: 298.44545454545454, 21: 1193.7818181818182}
[NeMo I 2023-10-17 07:47:29 intent_slot_classification_descriptor:87] Stats calculating for test mode...
[NeMo I 2023-10-17 07:47:29 intent_slot_classification_descriptor:112] Three most popular intents in test mode:
[NeMo I 2023-10-17 07:47:29 data_preprocessing:194] label: 0, 19 out of 1076 (1.77%).
[NeMo I 2023-10-17 07:47:29 data_preprocessing:194] label: 2, 19 out of 1076 (1.77%).
[NeMo I 2023-10-17 07:47:29 data_preprocessing:194] label: 6, 19 out of 1076 (1.77%).
[NeMo I 2023-10-17 07:47:29 intent_slot_classification_descriptor:118] Three most popular slots in test mode:
[NeMo I 2023-10-17 07:47:29 data_preprocessing:194] label: 54, 5952 out of 7242 (82.19%).
[NeMo I 2023-10-17 07:47:29 data_preprocessing:194] label: 38, 148 out of 7242 (2.04%).
[NeMo I 2023-10-17 07:47:29 data_preprocessing:194] label: 46, 134 out of 7242 (1.85%).
[NeMo I 2023-10-17 07:47:29 intent_slot_classification_descriptor:121] Total Number of Intents: 1076
[NeMo I 2023-10-17 07:47:29 intent_slot_classification_descriptor:122] Intent Label Frequencies: {0: 19, 2: 19, 6: 19, 7: 19, 8: 19, 9: 19, 11: 19, 13: 19, 14: 19, 15: 19, 16: 19, 17: 19, 18: 19, 19: 19, 20: 19, 22: 19, 23: 19, 24: 19, 25: 19, 27: 19, 28: 19, 30: 19, 35: 19, 36: 19, 37: 19, 39: 19, 41: 19, 42: 19, 43: 19, 44: 19, 45: 19, 46: 19, 47: 19, 48: 19, 49: 19, 51: 19, 52: 19, 53: 19, 55: 19, 57: 19, 58: 19, 59: 19, 61: 19, 62: 19, 63: 19, 38: 18, 56: 18, 60: 18, 26: 16, 4: 15, 32: 14, 50: 14, 5: 13, 21: 12, 29: 12, 1: 11, 54: 10, 33: 9, 3: 8, 10: 8, 12: 8, 34: 7, 40: 7, 31: 3}
[NeMo I 2023-10-17 07:47:29 intent_slot_classification_descriptor:123] Total Number of Slots: 7242
[NeMo I 2023-10-17 07:47:29 intent_slot_classification_descriptor:124] Slots Label Frequencies: {54: 5952, 38: 148, 46: 134, 12: 112, 14: 80, 18: 69, 5: 55, 36: 52, 11: 50, 19: 43, 43: 38, 28: 37, 52: 35, 23: 33, 48: 28, 7: 25, 6: 24, 45: 22, 4: 22, 26: 21, 2: 21, 20: 21, 37: 20, 13: 20, 44: 17, 34: 17, 35: 17, 53: 17, 16: 14, 41: 13, 9: 12, 22: 9, 47: 9, 33: 9, 49: 9, 24: 6, 1: 6, 25: 4, 8: 4, 42: 4, 51: 4, 32: 3, 39: 2, 3: 2, 17: 1, 40: 1}
[NeMo I 2023-10-17 07:47:29 intent_slot_classification_model:116] Labels: {'alarm_type': 0, 'app_name': 1, 'artist_name': 2, 'audiobook_author': 3, 'audiobook_name': 4, 'business_name': 5, 'business_type': 6, 'change_amount': 7, 'coffee_type': 8, 'color_type': 9, 'cooking_type': 10, 'currency_name': 11, 'date': 12, 'definition_word': 13, 'device_type': 14, 'drink_type': 15, 'email_address': 16, 'email_folder': 17, 'event_name': 18, 'food_type': 19, 'game_name': 20, 'game_type': 21, 'general_frequency': 22, 'house_place': 23, 'ingredient': 24, 'joke_type': 25, 'list_name': 26, 'meal_type': 27, 'media_type': 28, 'movie_name': 29, 'movie_type': 30, 'music_album': 31, 'music_descriptor': 32, 'music_genre': 33, 'news_topic': 34, 'order_type': 35, 'person': 36, 'personal_info': 37, 'place_name': 38, 'player_setting': 39, 'playlist_name': 40, 'podcast_descriptor': 41, 'podcast_name': 42, 'radio_name': 43, 'relation': 44, 'song_name': 45, 'time': 46, 'time_zone': 47, 'timeofday': 48, 'transport_agency': 49, 'transport_descriptor': 50, 'transport_name': 51, 'transport_type': 52, 'weather_descriptor': 53, 'O': 54}
[NeMo I 2023-10-17 07:47:29 intent_slot_classification_model:117] Labels mapping saved to : ./nemo_format/slot_labels.csv
[NeMo I 2023-10-17 07:47:29 intent_slot_classification_model:116] Labels: {'alarm_query': 0, 'alarm_remove': 1, 'alarm_set': 2, 'audio_volume_down': 3, 'audio_volume_mute': 4, 'audio_volume_up': 5, 'calendar_query': 6, 'calendar_remove': 7, 'calendar_set': 8, 'cooking_recipe': 9, 'datetime_convert': 10, 'datetime_query': 11, 'email_addcontact': 12, 'email_query': 13, 'email_querycontact': 14, 'email_sendemail': 15, 'general_affirm': 16, 'general_commandstop': 17, 'general_confirm': 18, 'general_dontcare': 19, 'general_explain': 20, 'general_joke': 21, 'general_negate': 22, 'general_praise': 23, 'general_quirky': 24, 'general_repeat': 25, 'iot_cleaning': 26, 'iot_coffee': 27, 'iot_hue_lightchange': 28, 'iot_hue_lightdim': 29, 'iot_hue_lightoff': 30, 'iot_hue_lighton': 31, 'iot_hue_lightup': 32, 'iot_wemo_off': 33, 'iot_wemo_on': 34, 'lists_createoradd': 35, 'lists_query': 36, 'lists_remove': 37, 'music_likeness': 38, 'music_query': 39, 'music_settings': 40, 'news_query': 41, 'play_audiobook': 42, 'play_game': 43, 'play_music': 44, 'play_podcasts': 45, 'play_radio': 46, 'qa_currency': 47, 'qa_definition': 48, 'qa_factoid': 49, 'qa_maths': 50, 'qa_stock': 51, 'recommendation_events': 52, 'recommendation_locations': 53, 'recommendation_movies': 54, 'social_post': 55, 'social_query': 56, 'takeaway_order': 57, 'takeaway_query': 58, 'transport_query': 59, 'transport_taxi': 60, 'transport_ticket': 61, 'transport_traffic': 62, 'weather_query': 63}
[NeMo I 2023-10-17 07:47:29 intent_slot_classification_model:117] Labels mapping saved to : ./nemo_format/intent_labels.csv
[NeMo I 2023-10-17 07:47:29 tokenizer_utils:130] Getting HuggingFace AutoTokenizer with pretrained_model_name: bert-base-uncased, vocab_file: /root/.cache/huggingface/nemo_nlp_tmp/ae0d012864bdb2474ba67537c3f5e0fa/vocab.txt, merges_files: None, special_tokens_dict: {}, and use_fast: False
Using eos_token, but it is not set yet.
Using bos_token, but it is not set yet.
[NeMo W 2023-10-17 07:47:30 modelPT:244] You tried to register an artifact under config key=tokenizer.vocab_file but an artifact for it has already been registered.
[NeMo W 2023-10-17 07:47:30 modelPT:161] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
Train config :
prefix: train
batch_size: 32
shuffle: true
num_samples: -1
num_workers: 2
drop_last: false
pin_memory: false
[NeMo W 2023-10-17 07:47:30 modelPT:168] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s).
Validation config :
prefix: test
batch_size: 32
shuffle: false
num_samples: -1
num_workers: 2
drop_last: false
pin_memory: false
[NeMo W 2023-10-17 07:47:30 modelPT:174] Please call the ModelPT.setup_test_data() or ModelPT.setup_multiple_test_data() method and provide a valid configuration file to setup the test data loader(s).
Test config :
prefix: test
batch_size: 32
shuffle: false
num_samples: -1
num_workers: 2
drop_last: false
pin_memory: false
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertModel: ['cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight', 'cls.predictions.bias']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertEncoder: ['cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.seq_relationship.bias', 'cls.seq_relationship.weight', 'cls.predictions.bias']
- This IS expected if you are initializing BertEncoder from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertEncoder from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
Cell In[42], line 5
2 checkpoint_path = trainer.checkpoint_callback.best_model_path
4 # load the model from this checkpoint
----> 5 eval_model = nemo_nlp.models.IntentSlotClassificationModel.load_from_checkpoint(checkpoint_path=checkpoint_path)
File /usr/local/lib/python3.8/dist-packages/nemo/collections/nlp/models/nlp_model.py:355, in NLPModel.load_from_checkpoint(cls, checkpoint_path, map_location, hparams_file, strict, **kwargs)
353 model = ptl_load_state(cls, checkpoint, strict=strict, **kwargs)
354 else:
--> 355 model = ptl_load_state(cls, checkpoint, strict=strict, cfg=cfg, **kwargs)
356 # cfg = checkpoint[cls.CHECKPOINT_HYPER_PARAMS_KEY].cfg
357
358 # NMT models do not have a `tokenizer` attribute, they instead have an encoder_tokenizer and decoder_tokenizer attribute.
359 if hasattr(cfg, "tokenizer"):
File /usr/local/lib/python3.8/dist-packages/pytorch_lightning/core/saving.py:247, in _load_state(cls, checkpoint, strict, **cls_kwargs_new)
245 # load the state_dict on the model automatically
246 assert strict is not None
--> 247 keys = obj.load_state_dict(checkpoint["state_dict"], strict=strict)
249 if not strict:
250 if keys.missing_keys:
File /usr/local/lib/python3.8/dist-packages/nemo/collections/nlp/models/nlp_model.py:397, in NLPModel.load_state_dict(self, state_dict, strict)
391 if (
392 self.bert_model is not None
393 and "position_ids" not in self.bert_model.embeddings._modules
394 and "bert_model.embeddings.position_ids" in state_dict
395 ):
396 del state_dict["bert_model.embeddings.position_ids"]
--> 397 super(NLPModel, self).load_state_dict(state_dict, strict=strict)
File /usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py:2073, in Module.load_state_dict(self, state_dict, strict)
2068 error_msgs.insert(
2069 0, 'Missing key(s) in state_dict: {}. '.format(
2070 ', '.join('"{}"'.format(k) for k in missing_keys)))
2072 if len(error_msgs) > 0:
-> 2073 raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
2074 self.__class__.__name__, "\n\t".join(error_msgs)))
2075 return _IncompatibleKeys(missing_keys, unexpected_keys)
RuntimeError: Error(s) in loading state_dict for IntentSlotClassificationModel:
Missing key(s) in state_dict: "bert_model.embeddings.position_ids".