Trying to install riva_quickstart_v2.15.1
has hung at this point___________________
05/24/2024-18:25:28] [TRT] [E] 1: Unexpected exception std::exception
[05/24/2024-18:25:28] [TRT] [E] 1: Unexpected exception std::exception
[05/24/2024-18:25:28] [TRT] [E] 1: Unexpected exception std::exception
[05/24/2024-18:25:29] [TRT] [E] 1: Unexpected exception std::exception
[05/24/2024-18:25:29] [TRT] [E] 1: Unexpected exception std::exception
[05/24/2024-18:29:11] [TRT] [W] TensorRT encountered issues when converting weights between types and that could affect accuracy.
[05/24/2024-18:29:11] [TRT] [W] If this is not the desired behavior, please modify the weights or retrain with regularization to adjust the magnitude of the weights.
[05/24/2024-18:29:11] [TRT] [W] Check verbose logs for the list of affected weights.
[05/24/2024-18:29:11] [TRT] [W] - 84 weights are affected by this issue: Detected subnormal FP16 values.
[05/24/2024-18:29:11] [TRT] [W] - 25 weights are affected by this issue: Detected values less than smallest positive FP16 subnormal value and converted them to the FP16 minimum subnormalized value.
2024-05-24 18:29:12,041 [INFO] Writing engine to model repository: /data/models/riva-trt-hifigan-English-US/1/model.plan
2024-05-24 18:29:12,524 [INFO] Extract_binaries for postprocessor → /data/models/tts_postprocessor-English-US/1
2024-05-24 18:29:12,524 [INFO] No binaries to extract. Creating empty file at /data/models/tts_postprocessor-English-US/1
2024-05-24 18:29:12,527 [INFO] Extract_binaries for self → /data/models/fastpitch_hifigan_ensemble-English-US/1
2024-05-24 18:29:12,528 [INFO] No binaries to extract. Creating empty file at /data/models/fastpitch_hifigan_ensemble-English-US/1
2024-05-24 18:29:12,528 [INFO] [{‘model_name’: ‘tts_preprocessor-English-US’, ‘model_version’: 1, ‘input_map’: {‘input_string’: ‘INPUT’, ‘speaker’: ‘SPEAKER’}, ‘output_map’: {‘output’: ‘input_encoder’, ‘is_last_sentence’: ‘chunker_is_last_sentence’, ‘output_string’: ‘PROCESSED_TEXT’, ‘sentence_num’: ‘SENTENCE_NUM’, ‘pitch’: ‘input_encoder_pitch’, ‘duration’: ‘input_encoder_dur’, ‘speaker’: ‘input_speaker’, ‘volume’: ‘input_volume’}}, {‘model_name’: ‘riva-onnx-fastpitch_encoder-English-US’, ‘model_version’: 1, ‘input_map’: {‘text’: ‘input_encoder’, ‘pitch’: ‘input_encoder_pitch’, ‘pace’: ‘input_encoder_dur’, ‘speaker’: ‘input_speaker’, ‘volume’: ‘input_volume’}, ‘output_map’: {‘spect’: ‘output_encoder’, ‘num_frames’: ‘num_valid_frames_encoder’, ‘durs_predicted’: ‘durs_predicted’, ‘volume_aligned’: ‘volume_out’}}, {‘model_name’: ‘spectrogram_chunker-English-US’, ‘model_version’: 1, ‘input_map’: {‘SPECTROGRAM’: ‘output_encoder’, ‘IS_LAST_SENTENCE’: ‘chunker_is_last_sentence’, ‘NUM_VALID_FRAMES_IN’: ‘num_valid_frames_encoder’, ‘SENTENCE_NUM’: ‘SENTENCE_NUM’, ‘DURATIONS’: ‘durs_predicted’, ‘PROCESSED_TEXT’: ‘PROCESSED_TEXT’, ‘VOLUME’: ‘volume_out’}, ‘output_map’: {‘SPECTROGRAM_CHUNK’: ‘spectrogram_chunk’, ‘END_FLAG’: ‘END_FLAG’, ‘NUM_VALID_SAMPLES_OUT’: ‘num_valid_samples’, ‘SENTENCE_NUM’: ‘OUT_SENTENCE_NUM’, ‘DURATIONS’: ‘OUT_DURATIONS’, ‘PROCESSED_TEXT’: ‘OUT_PROCESSED_TEXT’, ‘VOLUME’: ‘OUT_VOLUME’}}, {‘model_name’: ‘riva-trt-hifigan-English-US’, ‘model_version’: 1, ‘input_map’: {‘spec’: ‘spectrogram_chunk’}, ‘output_map’: {‘audio’: ‘audio_chunk’}}, {‘model_name’: ‘tts_postprocessor-English-US’, ‘model_version’: 1, ‘input_map’: {‘INPUT’: ‘audio_chunk’, ‘NUM_VALID_SAMPLES’: ‘num_valid_samples’, ‘Prosody_volume’: ‘OUT_VOLUME’}, ‘output_map’: {‘OUTPUT’: ‘OUTPUT’}}]
2024-05-24 18:29:40,081 [INFO] Using obey-precision pass with fp16 TRT
2024-05-24 18:29:40,082 [INFO] Extract_binaries for nn → /data/models/riva-trt-conformer-en-US-asr-offline-am-streaming-offline/1
2024-05-24 18:29:40,082 [INFO] extracting {‘onnx’: (‘nemo.collections.asr.models.ctc_bpe_models.EncDecCTCModelBPE’, ‘model_graph.onnx’)} → /data/models/riva-trt-conformer-en-US-asr-offline-am-streaming-offline/1
2024-05-24 18:29:41,034 [INFO] Printing copied artifacts:
2024-05-24 18:29:41,035 [INFO] {‘onnx’: ‘/data/models/riva-trt-conformer-en-US-asr-offline-am-streaming-offline/1/model_graph.onnx’}
2024-05-24 18:29:41,035 [INFO] Building TRT engine from ONNX file /data/models/riva-trt-conformer-en-US-asr-offline-am-streaming-offline/1/model_graph.onnx
[05/24/2024-19:49:44] [TRT] [W] onnx2trt_utils.cpp:374: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[05/24/2024-19:49:44] [TRT] [W] onnx2trt_utils.cpp:400: One or more weights outside the range of INT32 was clamped
[05/24/2024-19:49:45] [TRT] [E] ModelImporter.cpp:535: Parse was called with a non-empty network definition
2024-05-24 19:49:45,456 [INFO] Mixed-precision net: 5911 layers, 5911 tensors, 1 outputs…
2024-05-24 19:49:45,741 [INFO] Mixed-precision net: 0 layers / 0 outputs fixed
it is now 23.35
Any clues ?
$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
nvcr.io/nvidia/riva/riva-speech 2.15.1-servicemaker e795656de871 9 days ago 11GB
nvcr.io/nvidia/riva/riva-speech 2.15.1 3fb4bf2d6357 9 days ago 11.4GB