I’m trying to finetune FastPitch and HiFiGAN using Tao and mostly following the notebook from Text to Speech Notebook | NVIDIA NGC
When trying to finetune FastPitch, with the command below:
!tao spectro_gen finetune
-e $SPECS_DIR/spectro_gen/finetune.yaml
-g 1
-k tlt_encode
-r $RESULTS_DIR/spectro_gen/finetune
-m $pretrained_fastpitch_model
train_dataset=$DATA_DIR/$finetune_data_name/merged_train.json
validation_dataset=$DATA_DIR/$finetune_data_name/manifest_val.json
prior_folder=$RESULTS_DIR/spectro_gen/finetune/prior_folder
trainer.max_epochs=200
n_speakers=2
pitch_fmin=$pitch_fmin
pitch_fmax=$pitch_fmax
pitch_avg=$pitch_mean
pitch_std=$pitch_std
trainer.precision=16
it fails with this error:
[NeMo E 2022-12-20 21:25:37 common:503] Model instantiation failed!
Target class: nemo.collections.tts.models.fastpitch.FastPitchModel
Error(s): src path does not exist or it is not a path in nemo file. src value I got was: scripts/tts_dataset_files/cmudict-0.7b_nv22.08. Absolute: /opt/nvidia/tools/scripts/tts_dataset_files/cmudict-0.7b_nv22.08
Traceback (most recent call last):
File “”, line 764, in extract
File “/opt/conda/lib/python3.8/tarfile.py”, line 2060, in extract
tarinfo = self.getmember(member)
File “/opt/conda/lib/python3.8/tarfile.py”, line 1782, in getmember
raise KeyError(“filename %r not found” % name)
KeyError: “filename ‘manifest.yaml’ not found”
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “”, line 653, in restore_manifest
File “”, line 767, in extract
File “/opt/conda/lib/python3.8/tarfile.py”, line 2060, in extract
tarinfo = self.getmember(member)
File “/opt/conda/lib/python3.8/tarfile.py”, line 1782, in getmember
raise KeyError(“filename %r not found” % name)
KeyError: “filename ‘./manifest.yaml’ not found”
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “”, line 94, in restore_from
File “”, line 404, in restore_from
File “”, line 203, in validate_archive
File “”, line 657, in restore_manifest
TypeError: The indicated file ‘/data/tts_en_fastpitch_v1.4.0/tts_en_fastpitch_align.nemo’ is not an EFF archive
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “/opt/NeMo/nemo/core/classes/common.py”, line 482, in from_config_dict
instance = imported_cls(cfg=config, trainer=trainer)
File “/opt/NeMo/nemo/collections/tts/models/fastpitch.py”, line 105, in init
self._setup_tokenizer(tokenizer_conf)
File “/opt/NeMo/nemo/collections/tts/models/fastpitch.py”, line 188, in _setup_tokenizer
g2p_kwargs[“phoneme_dict”] = self.register_artifact(
File “/opt/NeMo/nemo/core/classes/modelPT.py”, line 222, in register_artifact
return self._save_restore_connector.register_artifact(self, config_path, src, verify_src_exists)
File “/opt/NeMo/nemo/core/connectors/save_restore_connector.py”, line 378, in register_artifact
raise FileNotFoundError(
FileNotFoundError: src path does not exist or it is not a path in nemo file. src value I got was: scripts/tts_dataset_files/cmudict-0.7b_nv22.08. Absolute: /opt/nvidia/tools/scripts/tts_dataset_files/cmudict-0.7b_nv22.08
Error executing job with overrides: [‘exp_manager.explicit_log_dir=/results/spectro_gen/finetune’, ‘trainer.gpus=1’, ‘restore_from=/data/tts_en_fastpitch_v1.4.0/tts_en_fastpitch_align.nemo’, ‘encryption_key=tlt_encode’, ‘train_dataset=/data/GLaDOS/merged_train.json’, ‘validation_dataset=/data/GLaDOS/manifest_val.json’, ‘prior_folder=/results/spectro_gen/finetune/prior_folder’, ‘trainer.max_epochs=200’, ‘n_speakers=2’, ‘pitch_fmin=80.0’, ‘pitch_fmax=2048.0’, ‘pitch_avg=165.458’, ‘pitch_std=40.1891’, ‘trainer.precision=16’]
An error occurred during Hydra’s exception formatting:
AssertionError()
Traceback (most recent call last):
File “/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py”, line 252, in run_and_report
assert mdl is not None
AssertionError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “</opt/conda/lib/python3.8/site-packages/nvidia_tao_pytorch/conv_ai/tts/spectro_gen/scripts/finetune.py>”, line 3, in
File “”, line 285, in
File “/opt/NeMo/nemo/core/config/hydra_runner.py”, line 104, in wrapper
_run_hydra(
File “/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py”, line 377, in _run_hydra
run_and_report(
File “/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py”, line 294, in run_and_report
raise ex
File “/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py”, line 211, in run_and_report
return func()
File “/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py”, line 378, in
lambda: hydra.run(
File “/opt/conda/lib/python3.8/site-packages/hydra/_internal/hydra.py”, line 111, in run
_ = ret.return_value
File “/opt/conda/lib/python3.8/site-packages/hydra/core/utils.py”, line 233, in return_value
raise self._return_value
File “/opt/conda/lib/python3.8/site-packages/hydra/core/utils.py”, line 160, in run_job
ret.return_value = task_function(task_cfg)
File “”, line 224, in main
File “/opt/NeMo/nemo/core/classes/modelPT.py”, line 311, in restore_from
instance = cls._save_restore_connector.restore_from(
File “”, line 94, in restore_from
File “/opt/NeMo/nemo/core/connectors/save_restore_connector.py”, line 235, in restore_from
loaded_params = self.load_config_and_state_dict(
File “/opt/NeMo/nemo/core/connectors/save_restore_connector.py”, line 158, in load_config_and_state_dict
instance = calling_cls.from_config_dict(config=conf, trainer=trainer)
File “/opt/NeMo/nemo/core/classes/common.py”, line 504, in from_config_dict
raise e
File “/opt/NeMo/nemo/core/classes/common.py”, line 496, in from_config_dict
instance = cls(cfg=config, trainer=trainer)
File “/opt/NeMo/nemo/collections/tts/models/fastpitch.py”, line 105, in init
self._setup_tokenizer(tokenizer_conf)
File “/opt/NeMo/nemo/collections/tts/models/fastpitch.py”, line 188, in _setup_tokenizer
g2p_kwargs[“phoneme_dict”] = self.register_artifact(
File “/opt/NeMo/nemo/core/classes/modelPT.py”, line 222, in register_artifact
return self._save_restore_connector.register_artifact(self, config_path, src, verify_src_exists)
File “/opt/NeMo/nemo/core/connectors/save_restore_connector.py”, line 378, in register_artifact
raise FileNotFoundError(
FileNotFoundError: src path does not exist or it is not a path in nemo file. src value I got was: scripts/tts_dataset_files/cmudict-0.7b_nv22.08. Absolute: /opt/nvidia/tools/scripts/tts_dataset_files/cmudict-0.7b_nv22.08
I’m using pretrained fastpitch 1.4.0 from ngc (TTS En FastPitch | NVIDIA NGC). I’ve tried the other 3 versions too, but get equally un awesome results
So many different errors in that output, I don’t even know what ones to start tracking down. The YAML is the default one downloaded from tao spectro_gen download_specs
• Hardware (3090ti, Ryzen 7 5800X, 32Gb)
• Network Type (TAO with FastPitch)
• TLT Version (4.0.0-pyt)