[NeMo W 2021-05-17 06:43:28 experimental:28] Module is experimental, not ready for production and is not fully supported. Use at your own risk. [NeMo W 2021-05-17 06:43:29 experimental:28] Module nemo.collections.asr.data.audio_to_text.AudioToCharDataset is experimental, not ready for production and is not fully supported. Use at your own risk. [NeMo W 2021-05-17 06:43:29 experimental:28] Module nemo.collections.asr.data.audio_to_text.AudioToBPEDataset is experimental, not ready for production and is not fully supported. Use at your own risk. [NeMo W 2021-05-17 06:43:29 experimental:28] Module nemo.collections.asr.data.audio_to_text.AudioLabelDataset is experimental, not ready for production and is not fully supported. Use at your own risk. [NeMo W 2021-05-17 06:43:29 experimental:28] Module nemo.collections.asr.data.audio_to_text.TarredAudioToTextDataset is experimental, not ready for production and is not fully supported. Use at your own risk. [NeMo W 2021-05-17 06:43:29 experimental:28] Module nemo.collections.asr.data.audio_to_text.TarredAudioToCharDataset is experimental, not ready for production and is not fully supported. Use at your own risk. [NeMo W 2021-05-17 06:43:29 experimental:28] Module nemo.collections.asr.data.audio_to_text.TarredAudioToBPEDataset is experimental, not ready for production and is not fully supported. Use at your own risk. [NeMo W 2021-05-17 06:43:29 experimental:28] Module is experimental, not ready for production and is not fully supported. Use at your own risk. [NeMo I 2021-05-17 06:43:29 tlt_logging:20] Experiment configuration: restore_from: /results/speechtotext_english_jasper_vtrainable_v1.2/speechtotext_english_jasper.tlt save_to: ??? exp_manager: explicit_log_dir: /results/model exp_dir: null name: finetuned-model version: null use_datetime_version: true resume_if_exists: true resume_past_end: false resume_ignore_no_checkpoint: true create_tensorboard_logger: false summary_writer_kwargs: null create_wandb_logger: false wandb_logger_kwargs: null create_checkpoint_callback: true checkpoint_callback_params: filepath: null monitor: val_loss verbose: true save_last: true save_top_k: 3 save_weights_only: false mode: auto period: 1 prefix: null postfix: .tlt save_best_model: false files_to_copy: null trainer: logger: false checkpoint_callback: false callbacks: null default_root_dir: null gradient_clip_val: 0.0 process_position: 0 num_nodes: 1 num_processes: 1 gpus: 1 auto_select_gpus: false tpu_cores: null log_gpu_memory: null progress_bar_refresh_rate: 1 overfit_batches: 0.0 track_grad_norm: -1 check_val_every_n_epoch: 1 fast_dev_run: false accumulate_grad_batches: 1 max_epochs: 3 min_epochs: 1 max_steps: null min_steps: null limit_train_batches: 1.0 limit_val_batches: 1.0 limit_test_batches: 1.0 val_check_interval: 1.0 flush_logs_every_n_steps: 100 log_every_n_steps: 50 accelerator: ddp sync_batchnorm: false precision: 32 weights_summary: full weights_save_path: null num_sanity_val_steps: 2 truncated_bptt_steps: null resume_from_checkpoint: null profiler: null benchmark: false deterministic: false reload_dataloaders_every_epoch: false auto_lr_find: false replace_sampler_ddp: true terminate_on_nan: false auto_scale_batch_size: false prepare_data_per_node: true amp_backend: native amp_level: O0 change_vocabulary: false finetuning_ds: manifest_filepath: /data/train_total.json batch_size: 32 sample_rate: 16000 labels: - ' ' - a - b - c - d - e - f - g - h - i - j - k - l - m - 'n' - o - p - q - r - s - t - u - v - w - x - 'y' - z - '''' num_workers: 20 trim_silence: true shuffle: true max_duration: 16.7 is_tarred: false tarred_audio_filepaths: null validation_ds: manifest_filepath: /data/dev/dev_com.json batch_size: 32 sample_rate: 16000 labels: - ' ' - a - b - c - d - e - f - g - h - i - j - k - l - m - 'n' - o - p - q - r - s - t - u - v - w - x - 'y' - z - '''' num_workers: 20 trim_silence: true shuffle: false max_duration: null is_tarred: false tarred_audio_filepaths: null optim: name: novograd lr: 0.001 encryption_key: '*********' [NeMo W 2021-05-17 06:43:30 exp_manager:380] Exp_manager is logging to /results/model, but it already exists. [NeMo W 2021-05-17 06:43:30 exp_manager:321] There were no checkpoints found in /results/model/checkpoints. Training from scratch. [NeMo I 2021-05-17 06:43:30 exp_manager:194] Experiments will be logged at /results/model [NeMo W 2021-05-17 06:43:30 nemo_logging:349] /opt/conda/lib/python3.6/site-packages/pytorch_lightning/utilities/distributed.py:49: UserWarning: Checkpoint directory /results/model/checkpoints exists and is not empty. warnings.warn(*args, **kwargs) [NeMo W 2021-05-17 06:43:38 modelPT:146] Please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader. Train config : manifest_filepath: /data/fisher_min5sec/small_manifest.json batch_size: 32 sample_rate: 16000 labels: - ' ' - a - b - c - d - e - f - g - h - i - j - k - l - m - 'n' - o - p - q - r - s - t - u - v - w - x - 'y' - z - '''' num_workers: null trim_silence: true shuffle: true max_duration: 16.7 is_tarred: false tarred_audio_filepaths: null [NeMo W 2021-05-17 06:43:38 modelPT:153] Please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s). Validation config : manifest_filepath: /data/fisher_min5sec/small_manifest.json batch_size: 32 sample_rate: 16000 labels: - ' ' - a - b - c - d - e - f - g - h - i - j - k - l - m - 'n' - o - p - q - r - s - t - u - v - w - x - 'y' - z - '''' num_workers: null trim_silence: true shuffle: false max_duration: null is_tarred: false tarred_audio_filepaths: null [NeMo I 2021-05-17 06:43:38 features:236] PADDING: 16 [NeMo I 2021-05-17 06:43:38 features:252] STFT using torch [NeMo I 2021-05-17 06:43:59 finetune:113] Model restored from '/results/speechtotext_english_jasper_vtrainable_v1.2/speechtotext_english_jasper.tlt' [NeMo I 2021-05-17 06:44:12 collections:173] Dataset loaded with 280815 files totalling 448.66 hours [NeMo I 2021-05-17 06:44:12 collections:174] 2569 files were filtered totalling 16.03 hours [NeMo I 2021-05-17 06:44:13 collections:173] Dataset loaded with 7044 files totalling 11.16 hours [NeMo I 2021-05-17 06:44:13 collections:174] 0 files were filtered totalling 0.00 hours [NeMo I 2021-05-17 06:44:13 modelPT:753] Optimizer config = Novograd ( Parameter Group 0 amsgrad: False betas: (0.95, 0.98) eps: 1e-08 grad_averaging: False lr: 0.001 weight_decay: 0 ) [NeMo I 2021-05-17 06:44:13 lr_scheduler:492] Scheduler not initialized as no `sched` config supplied to setup_optimizer() [NeMo I 2021-05-17 06:44:15 modelPT:753] Optimizer config = Novograd ( Parameter Group 0 amsgrad: False betas: (0.95, 0.98) eps: 1e-08 grad_averaging: False lr: 0.001 weight_decay: 0 ) [NeMo I 2021-05-17 06:44:15 lr_scheduler:492] Scheduler not initialized as no `sched` config supplied to setup_optimizer() [NeMo W 2021-05-17 06:44:20 patch_utils:50] torch.stft() signature has been updated for PyTorch 1.7+ Please update PyTorch to remain compatible with later versions of NeMo. [NeMo W 2021-05-17 06:44:24 nemo_logging:349] /opt/conda/lib/python3.6/site-packages/torch/functional.py:516: UserWarning: The function torch.rfft is deprecated and will be removed in a future PyTorch release. Use the new torch.fft module functions, instead, by importing torch.fft and calling torch.fft.fft or torch.fft.rfft. (Triggered internally at ../aten/src/ATen/native/SpectralOps.cpp:674.) normalized, onesided, return_complex) [NeMo W 2021-05-17 06:46:07 nemo_logging:349] /opt/conda/lib/python3.6/site-packages/pytorch_lightning/utilities/distributed.py:49: UserWarning: The validation_epoch_end should not return anything as of 9.1. To log, use self.log(...) or self.write(...) directly in the LightningModule warnings.warn(*args, **kwargs)