Speech_to_text infer: model_weights.ckpt not found

I’m trying to use Tao for generating some transcripts for audiofiles using the speech_to_text infer command.
It starts loading the spec file and the model etc. but after that it throws the following error:

FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp8iua4jmw/model_weights.ckpt'

What could be the reason for that?

Here my .tlt_mounts.json:

{
    "Mounts": [
        {
            "source": "/home/ubuntu/x/nvidia-riva-experiments/models",
            "destination": "/workspace/tao-experiments/models"
        },
        {
            "source": "/home/ubuntu/x/nvidia-riva-experiments/data/tao",
            "destination": "/workspace/tao-experiments/specs"
        },
        {
            "source": "/home/ubuntu/x/nvidia-riva-experiments/data/transcripts/tao",
            "destination": "/workspace/tao-experiments/transcripts"
        },
        {
            "source": "/tmp",
            "destination": "/tmp"
        }
    ]
}

Here the complete shell command & output.

> tao speech_to_text infer -e "/workspace/tao-experiments/specs/inference.spec" \
                                                -m "/workspace/tao-experiments/models/speechtotext_en_us_citrinet_vdeployable_v3.0/citrinet-1024-Jarvis-asrset-3_0-encrypted.riva" \
                                                 -g 1 \
                                                 -r "/workspace/tao-experiments/transcripts/test3"

~/.tao_mounts.json wasn’t found. Falling back to obtain mount points and docker configs from ~/.tlt_mounts.json.
Please note that this will be deprecated going forward.
2022-02-23 19:53:07,605 [INFO] root: Registry: [‘nvcr.io’]
2022-02-23 19:53:07,681 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-pyt:v3.21.11-py3
2022-02-23 19:53:07,711 [WARNING] tlt.components.docker_handler.docker_handler:
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the “user”:“UID:GID” in the
DockerOptions portion of the “/home/ubuntu/.tlt_mounts.json” file. You can obtain your
users UID and GID by using the “id -u” and “id -g” commands on the
terminal.
[NeMo W 2022-02-23 19:53:11 nemo_logging:349] /opt/conda/lib/python3.8/site-packages/torchaudio-0.7.0a0+42d447d-py3.8-linux-x86_64.egg/torchaudio/backend/utils.py:53: UserWarning: “sox” backend is being deprecated. The default backend will be changed to “sox_io” backend in 0.8.0 and “sox” backend will be removed in 0.9.0. Please migrate to “sox_io” backend. Please refer to [Announcement] Improving I/O for correct and consistent experience · Issue #903 · pytorch/audio · GitHub for the detail.
warnings.warn(

I0223 19:53:12.061039 139949109749568 font_manager.py:1443] generated new fontManager
[NeMo W 2022-02-23 19:53:12 experimental:27] Module <class ‘nemo.collections.asr.data.audio_to_text_dali._AudioTextDALIDataset’> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo W 2022-02-23 19:53:15 nemo_logging:349] /opt/conda/lib/python3.8/site-packages/torchaudio-0.7.0a0+42d447d-py3.8-linux-x86_64.egg/torchaudio/backend/utils.py:53: UserWarning: “sox” backend is being deprecated. The default backend will be changed to “sox_io” backend in 0.8.0 and “sox” backend will be removed in 0.9.0. Please migrate to “sox_io” backend. Please refer to [Announcement] Improving I/O for correct and consistent experience · Issue #903 · pytorch/audio · GitHub for the detail.
warnings.warn(

[NeMo W 2022-02-23 19:53:16 experimental:27] Module <class ‘nemo.collections.asr.data.audio_to_text_dali._AudioTextDALIDataset’> is experimental, not ready for production and is not fully supported. Use at your own risk.
[NeMo I 2022-02-23 19:53:16 tlt_logging:20] Experiment configuration:
restore_from: /workspace/tao-experiments/models/speechtotext_en_us_citrinet_vdeployable_v3.0/citrinet-1024-Jarvis-asrset-3_0-encrypted.riva
exp_manager:
task_name: infer
explicit_log_dir: /workspace/tao-experiments/transcripts/test3
file_paths: ???
encryption_key: ‘*****’

[NeMo W 2022-02-23 19:53:16 exp_manager:26] Exp_manager is logging to /workspace/tao-experiments/transcripts/test3``, but it already exists. Metadata doesn't have the required key nemo_archive_version`
[NeMo I 2022-02-23 19:53:19 mixins:147] Tokenizer SentencePieceTokenizer initialized with 1024 tokens
[NeMo W 2022-02-23 19:53:19 modelPT:130] If you intend to do training or fine-tuning, please call the ModelPT.setup_training_data() method and provide a valid configuration file to setup the train data loader.
Train config :
manifest_filepath: null
sample_rate: 16000
batch_size: 32
trim_silence: false
max_duration: 20.0
shuffle: true
is_tarred: false
tarred_audio_filepaths: null
use_start_end_token: false

[NeMo W 2022-02-23 19:53:19 modelPT:137] If you intend to do validation, please call the ModelPT.setup_validation_data() or ModelPT.setup_multiple_validation_data() method and provide a valid configuration file to setup the validation data loader(s).
Validation config :
manifest_filepath: null
sample_rate: 16000
batch_size: 32
shuffle: false
use_start_end_token: false

[NeMo W 2022-02-23 19:53:19 modelPT:143] Please call the ModelPT.setup_test_data() or ModelPT.setup_multiple_test_data() method and provide a valid configuration file to setup the test data loader(s).
Test config :
manifest_filepath: null
sample_rate: 16000
batch_size: 32
shuffle: false
use_start_end_token: false

[NeMo I 2022-02-23 19:53:19 features:252] PADDING: 16
[NeMo I 2022-02-23 19:53:19 features:269] STFT using torch
Error executing job with overrides: [‘exp_manager.explicit_log_dir=/workspace/tao-experiments/transcripts/test3’, ‘restore_from=/workspace/tao-experiments/models/speechtotext_en_us_citrinet_vdeployable_v3.0/citrinet-1024-Jarvis-asrset-3_0-encrypted.riva’]
Traceback (most recent call last):
File “/home/jenkins/agent/workspace/tlt-pytorch-main-nightly/tlt_utils/connectors/save_restore_connector.py”, line 77, in restore_from
File “/home/jenkins/agent/workspace/tlt-pytorch-main-nightly/tlt_utils/cookbooks/nemo_cookbook.py”, line 396, in restore_from
TypeError: Archive doesn’t have the required runtime, format, version or object class type

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py”, line 211, in run_and_report
return func()
File “/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py”, line 368, in
lambda: hydra.run(
File “/opt/conda/lib/python3.8/site-packages/hydra/_internal/hydra.py”, line 110, in run
_ = ret.return_value
File “/opt/conda/lib/python3.8/site-packages/hydra/core/utils.py”, line 233, in return_value
raise self._return_value
File “/opt/conda/lib/python3.8/site-packages/hydra/core/utils.py”, line 160, in run_job
ret.return_value = task_function(task_cfg)
File “/home/jenkins/agent/workspace/tlt-pytorch-main-nightly/asr/speech_to_text/scripts/infer.py”, line 64, in main
File “/opt/conda/lib/python3.8/site-packages/nemo/core/classes/modelPT.py”, line 270, in restore_from
instance = cls._save_restore_connector.restore_from(
File “/home/jenkins/agent/workspace/tlt-pytorch-main-nightly/tlt_utils/connectors/save_restore_connector.py”, line 87, in restore_from
File “/opt/conda/lib/python3.8/site-packages/nemo/core/connectors/save_restore_connector.py”, line 140, in restore_from
self._load_state_dict_from_disk(model_weights, map_location=map_location), strict=strict
File “/opt/conda/lib/python3.8/site-packages/nemo/core/connectors/save_restore_connector.py”, line 390, in _load_state_dict_from_disk
return torch.load(model_weights, map_location=map_location)
File “/opt/conda/lib/python3.8/site-packages/torch/serialization.py”, line 594, in load
with _open_file_like(f, ‘rb’) as opened_file:
File “/opt/conda/lib/python3.8/site-packages/torch/serialization.py”, line 230, in _open_file_like
return _open_file(name_or_buffer, mode)
File “/opt/conda/lib/python3.8/site-packages/torch/serialization.py”, line 211, in init
super(_open_file, self).init(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: ‘/tmp/tmp8iua4jmw/model_weights.ckpt’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/home/jenkins/agent/workspace/tlt-pytorch-main-nightly/asr/speech_to_text/scripts/infer.py”, line 80, in
File “/opt/conda/lib/python3.8/site-packages/nemo/core/config/hydra_runner.py”, line 101, in wrapper
_run_hydra(
File “/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py”, line 367, in _run_hydra
run_and_report(
File “/opt/conda/lib/python3.8/site-packages/hydra/_internal/utils.py”, line 251, in run_and_report
assert mdl is not None
AssertionError
2022-02-23 19:53:27,240 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

Hardware - GPU T4
Operating System: Ubuntu 18.04
Riva Version: ?
TLT Version (if relevant): RIVA Citrinet ASR English | NVIDIA NGC