Whisper not working nano_llm

Running container
jetson-containers run --env HUGGINGFACE_TOKEN=???
$(autotag nano_llm)
python3 -m nano_llm.studio

dustynv/nano_llm r36.4.0 1f60691fe50b 7 days ago 30.6GB

whisperASR Produces the following error.

14:22:59 | INFO | loading Whisper model ‘base.en’ with TensorRT
[10/21/2024-14:23:00] [TRT] [E] IRuntime::deserializeCudaEngine: Error Code 1: Serialization (Serialization assertion stdVersionRead == kSERIALIZATION_VERSION failed.Version tag does not match. Note: Current Version: 239, Serialized Engine Version: 236)
14:23:00 | ERROR | Exception occurred handling websocket message:

{ ‘add_plugin’: { ‘language_code’: ‘en_US’,
‘layout_node’: {‘x’: 10, ‘y’: 10},
‘model’: ‘base’,
‘name’: ‘WhisperASR’,
‘partial_transcripts’: 0.25,
‘type’: ‘WhisperASR’,
‘use_cache’: ‘false’}}
Traceback (most recent call last):
File “/opt/NanoLLM/nano_llm/web/server.py”, line 193, in on_message
callback(payload, payload_size=payload_size, msg_type=msg_type, msg_id=msg_id,
File “/opt/NanoLLM/nano_llm/agents/dynamic_agent.py”, line 442, in on_websocket
on_message(self, message)
File “/opt/NanoLLM/nano_llm/agents/dynamic_agent.py”, line 432, in on_message
if invoke_handler(obj, key, msg):
File “/opt/NanoLLM/nano_llm/agents/dynamic_agent.py”, line 414, in invoke_handler
response = func(**msg)
File “/opt/NanoLLM/nano_llm/agents/dynamic_agent.py”, line 58, in add_plugin
threading.Thread(target=self.add_plugin, kwargs={‘type’: type, ‘wait’: True, ‘state_dict’: state_dict, ‘layout_node’: layout_node, **kwargs}).run()
File “/usr/lib/python3.10/threading.py”, line 953, in run
self._target(*self._args, **self._kwargs)
File “/opt/NanoLLM/nano_llm/agents/dynamic_agent.py”, line 65, in add_plugin
plugin = DynamicPlugin(type, **init_kwargs)
File “/opt/NanoLLM/nano_llm/plugins/dynamic_plugin.py”, line 35, in new
instance = plugin(*args, **kwargs)
File “/opt/NanoLLM/nano_llm/plugins/speech/whisper_asr.py”, line 71, in init
self.model = load_trt_model(self.model_name, verbose=True)
File “/opt/whisper_trt/whisper_trt/model.py”, line 424, in load_trt_model
return builder.load(path)
File “/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py”, line 116, in decorate_context
return func(*args, **kwargs)
File “/opt/whisper_trt/whisper_trt/model.py”, line 339, in load
audio_encoder_engine.load_state_dict(checkpoint[‘audio_encoder_engine’])
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 2201, in load_state_dict
load(self, state_dict)
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 2183, in load
module._load_from_state_dict(
File “/usr/local/lib/python3.10/dist-packages/torch2trt/trt_module.py”, line 80, in _load_from_state_dict
self.context = self.engine.create_execution_context()
AttributeError: ‘NoneType’ object has no attribute ‘create_execution_context’

rivaASR not working- location tool produces error. I am having trouble with number of presets that appear to work with previous nano_llm.
I will try and be more precise when I have more time to investigate.
Any help will be grateful

Hi,

Based on the below error:

[10/21/2024-14:23:00] [TRT] [E] IRuntime::deserializeCudaEngine: Error Code 1: Serialization (Serialization assertion stdVersionRead == kSERIALIZATION_VERSION failed.Version tag does not match. Note: Current Version: 239, Serialized Engine Version: 236)

You are using the TensorRT engine, which is built on other software environments.
The container you used is for r36.4.0/JetPack 6.1.

Do you use the same environment?
Have you manually upgraded the CUDA/cuDNN/TensorRT library?

Thanks.

Hi
Thanks for reply.
I am using Jetpack 6.1 [L4T 36.4.0]
I downloaded the latest nano-llm r36.4.0 yesterday and got the above result when attempting to use wisperASR in agent studio.
I have not manually upgraded anything. other than installing 6.1 when I upgraded from 1tb nvme to a 2tb nvme.
jtop tells me i’m using TensorRT 10.3.0.30
cuda 12.6.68

as an aside jetson-copilot would not work in 6.1 unless using
ollama r36.4.0 image and jetson-copilot r36.4.0 image this is now working fine.

I am now reinstalling image nano-llm r36.3.0 to see where this leads me

Cheers

Hi,

Do you get it to work after reinstalling nano-llm r36.3.0?
Thanks.

Hi
Yes
All I was doing was testing whether whisperASR would load in r36.4.0 (Agent Studio) which it did not giving the original error above.
The above works correctly using r36.3.0 WHY?
Thanks for replying

Hi,

[10/21/2024-14:23:00] [TRT] [E] IRuntime::deserializeCudaEngine: Error Code 1: Serialization (Serialization assertion stdVersionRead == kSERIALIZATION_VERSION failed.Version tag does not match. Note: Current Version: 239, Serialized Engine Version: 236)

The above error is triggered when running a TensorRT engine built on a different software environment.
As the model is built on TensorRT 8.6, it works on JetPack 6.0 but fails on JetPack 6.1.

Thanks.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.