Hello,
I use r36.4.0
When I run a container with jetson-containers run $(autotag nano_llm) and then
python3
import tensorrt
print(tensorrt.version)
10.4.0
I created a new installation on an SSD using the SDK Manager to see if the whisper error persists. When I then run the whisper plugin with Agent Studio, the following error message appears:
10:11:10 | INFO | loading Whisper model ‘base.en’ with TensorRT
/usr/local/lib/python3.10/dist-packages/whisper/init.py:150: FutureWarning: You are using torch.load with weights_only=False (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See pytorch/SECURITY.md at main · pytorch/pytorch · GitHub for more details). In a future release, the default value for weights_only will be flipped to True. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via torch.serialization.add_safe_globals. We recommend you start setting weights_only=True for any use case where you don’t have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.
checkpoint = torch.load(fp, map_location=device)
[03/31/2025-10:11:14] [TRT] [I] [MemUsageChange] Init CUDA: CPU +2, GPU +0, now: CPU 309, GPU 3523 (MiB)
[03/31/2025-10:11:14] [TRT] [V] Trying to load shared library libnvinfer_builder_resource.so.10.4.0
[03/31/2025-10:11:14] [TRT] [V] Loaded shared library libnvinfer_builder_resource.so.10.4.0
[03/31/2025-10:11:16] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +947, GPU +1038, now: CPU 1299, GPU 4562 (MiB)
[03/31/2025-10:11:16] [TRT] [V] CUDA lazy loading is enabled.
10:11:16 | ERROR | Exception occurred handling websocket message:
{ ‘add_plugin’: { ‘language_code’: ‘en_US’,
‘layout_node’: {‘x’: 10, ‘y’: 10},
‘model’: ‘base’,
‘name’: ‘WhisperASR’,
‘partial_transcripts’: 0.25,
‘type’: ‘WhisperASR’,
‘use_cache’: ‘true’}}
Traceback (most recent call last):
File “/opt/NanoLLM/nano_llm/web/server.py”, line 193, in on_message
callback(payload, payload_size=payload_size, msg_type=msg_type, msg_id=msg_id,
File “/opt/NanoLLM/nano_llm/agents/dynamic_agent.py”, line 442, in on_websocket
on_message(self, message)
File “/opt/NanoLLM/nano_llm/agents/dynamic_agent.py”, line 432, in on_message
if invoke_handler(obj, key, msg):
File “/opt/NanoLLM/nano_llm/agents/dynamic_agent.py”, line 414, in invoke_handler
response = func(**msg)
File “/opt/NanoLLM/nano_llm/agents/dynamic_agent.py”, line 58, in add_plugin
threading.Thread(target=self.add_plugin, kwargs={‘type’: type, ‘wait’: True, ‘state_dict’: state_dict, ‘layout_node’: layout_node, **kwargs}).run()
File “/usr/lib/python3.10/threading.py”, line 953, in run
self._target(*self._args, **self._kwargs)
File “/opt/NanoLLM/nano_llm/agents/dynamic_agent.py”, line 65, in add_plugin
plugin = DynamicPlugin(type, **init_kwargs)
File “/opt/NanoLLM/nano_llm/plugins/dynamic_plugin.py”, line 35, in new
instance = plugin(*args, **kwargs)
File “/opt/NanoLLM/nano_llm/plugins/speech/whisper_asr.py”, line 71, in init
self.model = load_trt_model(self.model_name, verbose=True)
File “/opt/whisper_trt/whisper_trt/model.py”, line 422, in load_trt_model
builder.build(path, verbose=verbose)
File “/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py”, line 116, in decorate_context
return func(*args, **kwargs)
File “/opt/whisper_trt/whisper_trt/model.py”, line 312, in build
“text_decoder_engine”: cls.build_text_decoder_engine().state_dict(),
File “/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py”, line 116, in decorate_context
return func(*args, **kwargs)
File “/opt/whisper_trt/whisper_trt/model.py”, line 206, in build_text_decoder_engine
engine = torch2trt.torch2trt(
File “/usr/local/lib/python3.10/dist-packages/torch2trt/torch2trt.py”, line 608, in torch2trt
torch.onnx.export(
File “/usr/local/lib/python3.10/dist-packages/torch/onnx/utils.py”, line 551, in export
_export(
File “/usr/local/lib/python3.10/dist-packages/torch/onnx/utils.py”, line 1648, in _export
graph, params_dict, torch_out = _model_to_graph(
File “/usr/local/lib/python3.10/dist-packages/torch/onnx/utils.py”, line 1170, in _model_to_graph
graph, params, torch_out, module = _create_jit_graph(model, args)
File “/usr/local/lib/python3.10/dist-packages/torch/onnx/utils.py”, line 1046, in _create_jit_graph
graph, torch_out = _trace_and_get_graph_from_model(model, args)
File “/usr/local/lib/python3.10/dist-packages/torch/onnx/utils.py”, line 950, in _trace_and_get_graph_from_model
trace_graph, torch_out, inputs_states = torch.jit._get_trace_graph(
File “/usr/local/lib/python3.10/dist-packages/torch/jit/_trace.py”, line 1497, in _get_trace_graph
outs = ONNXTracedModule(
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1562, in _call_impl
return forward_call(*args, **kwargs)
File “/usr/local/lib/python3.10/dist-packages/torch/jit/_trace.py”, line 141, in forward
graph, out = torch._C._create_graph_by_tracing(
File “/usr/local/lib/python3.10/dist-packages/torch/jit/_trace.py”, line 132, in wrapper
outs.append(self.inner(*trace_inputs))
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1562, in _call_impl
return forward_call(*args, **kwargs)
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1543, in _slow_forward
result = self.forward(*input, **kwargs)
File “/usr/local/lib/python3.10/dist-packages/torch2trt/flatten_module.py”, line 34, in forward
output = self.module(*args)
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1562, in _call_impl
return forward_call(*args, **kwargs)
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1543, in _slow_forward
result = self.forward(*input, **kwargs)
File “/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py”, line 116, in decorate_context
return func(*args, **kwargs)
File “/opt/whisper_trt/whisper_trt/model.py”, line 94, in forward
x = block(x, xa, mask)
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1562, in _call_impl
return forward_call(*args, **kwargs)
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1543, in _slow_forward
result = self.forward(*input, **kwargs)
File “/usr/local/lib/python3.10/dist-packages/whisper/model.py”, line 167, in forward
x = x + self.attn(self.attn_ln(x), mask=mask, kv_cache=kv_cache)[0]
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1562, in _call_impl
return forward_call(*args, **kwargs)
File “/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py”, line 1543, in _slow_forward
result = self.forward(*input, **kwargs)
File “/usr/local/lib/python3.10/dist-packages/whisper/model.py”, line 111, in forward
wv, qk = self.qkv_attention(q, k, v, mask)
File “/usr/local/lib/python3.10/dist-packages/whisper/model.py”, line 124, in qkv_attention
a = scaled_dot_product_attention(
TypeError: scaled_dot_product_attention(): argument ‘is_causal’ must be bool, not Tensor
Jetpack 6.2 L4T 36.4.3 is installed.
Everything worked perfectly with Jetpack 5. The error must be related to the scaled_dot_product_attention() function and the is_causal variable. If you set is_causal to a Boolean value, for example, False, the initialization process will complete. However, the error message “mask is unknown” occurs.
Thanks.