Running container
jetson-containers run --env HUGGINGFACE_TOKEN=???
$(autotag nano_llm)
python3 -m nano_llm.studio
dustynv/nano_llm r36.4.0 1f60691fe50b 7 days ago 30.6GB
whisperASR Produces the following error.
14:22:59 | INFO | loading Whisper model ‘base.en’ with TensorRT
[10/21/2024-14:23:00] [TRT] [E] IRuntime::deserializeCudaEngine: Error Code 1: Serialization (Serialization assertion stdVersionRead == kSERIALIZATION_VERSION failed.Version tag does not match. Note: Current Version: 239, Serialized Engine Version: 236)
14:23:00 | ERROR | Exception occurred handling websocket message:
{ ‘add_plugin’: { ‘language_code’: ‘en_US’,
‘layout_node’: {‘x’: 10, ‘y’: 10},
‘model’: ‘base’,
‘name’: ‘WhisperASR’,
‘partial_transcripts’: 0.25,
‘type’: ‘WhisperASR’,
‘use_cache’: ‘false’}}
Traceback (most recent call last):
File “/opt/NanoLLM/nano_llm/web/server.py”, line 193, in on_message
callback(payload, payload_size=payload_size, msg_type=msg_type, msg_id=msg_id,
File “/opt/NanoLLM/nano_llm/agents/dynamic_agent.py”, line 442, in on_websocket
on_message(self, message)
File “/opt/NanoLLM/nano_llm/agents/dynamic_agent.py”, line 432, in on_message
if invoke_handler(obj, key, msg):
File “/opt/NanoLLM/nano_llm/agents/dynamic_agent.py”, line 414, in invoke_handler
response = func(**msg)
File “/opt/NanoLLM/nano_llm/agents/dynamic_agent.py”, line 58, in add_plugin
threading.Thread(target=self.add_plugin, kwargs={‘type’: type, ‘wait’: True, ‘state_dict’: state_dict, ‘layout_node’: layout_node, **kwargs}).run()
File “/usr/lib/python3.10/threading.py”, line 953, in run
self._target(*self._args, **self._kwargs)
File “/opt/NanoLLM/nano_llm/agents/dynamic_agent.py”, line 65, in add_plugin
plugin = DynamicPlugin(type, **init_kwargs)
File “/opt/NanoLLM/nano_llm/plugins/dynamic_plugin.py”, line 35, in new
instance = plugin(*args, **kwargs)
File “/opt/NanoLLM/nano_llm/plugins/speech/whisper_asr.py”, line 71, in init
self.model = load_trt_model(self.model_name, verbose=True)
File “/opt/whisper_trt/whisper_trt/model.py”, line 424, in load_trt_model
return builder.load(path)
File “/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py”, line 116, in decorate_context
return func(*args, **kwargs)
File “/opt/whisper_trt/whisper_trt/model.py”, line 339, in load
audio_encoder_engine.load_state_dict(checkpoint[‘audio_encoder_engine’])
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 2201, in load_state_dict
load(self, state_dict)
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 2183, in load
module._load_from_state_dict(
File “/usr/local/lib/python3.10/dist-packages/torch2trt/trt_module.py”, line 80, in _load_from_state_dict
self.context = self.engine.create_execution_context()
AttributeError: ‘NoneType’ object has no attribute ‘create_execution_context’
rivaASR not working- location tool produces error. I am having trouble with number of presets that appear to work with previous nano_llm.
I will try and be more precise when I have more time to investigate.
Any help will be grateful