Situation: Download nano_llm docker image on Jetson Orin Nano Developer Kit - Jetpack 6.0 [L4T 36.2.0]
dustynv/nano_llm Tags | Docker Hub
docker pull dustynv/nano_llm:24.6-r36.2.
then execute
jetson-containers run $(autotag nano_llm) python3 -m nano_llm.agents.web_chat --api=mlc --model meta-llama/Meta-Llama-3-8B-Instruct --asr=riva --tts=piper
Expectation: The necessary images with docker container started llamaspeak with text LLM and ASR/TTS enabled.
Actual: I get the error below, and i’m kicked back out to the host prompt
@dusty_nv
From call stack. It seems like this commit introduce this problem.
added whisper_trt and VAD
05:28:09 | INFO | using chat template 'llama-3' for model Meta-Llama-3-8B-Instruct
05:28:09 | INFO | model 'Meta-Llama-3-8B-Instruct', chat template 'llama-3' stop tokens: ['<|end_of_text|>', '<|eot_id|>'] -> [128001, 128009]
05:28:09 | INFO | Warming up LLM with query 'What is 2+2?'
05:28:11 | INFO | Warmup response: 'Easy peasy!\n\nThe answer to 2+2 is... 4!<|eot_id|>'
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/opt/NanoLLM/nano_llm/agents/web_chat.py", line 327, in <module>
agent = WebChat(**vars(args))
File "/opt/NanoLLM/nano_llm/agents/web_chat.py", line 32, in __init__
super().__init__(**kwargs)
File "/opt/NanoLLM/nano_llm/agents/voice_chat.py", line 42, in __init__
self.vad = VADFilter(**kwargs).add(self.asr) if self.asr else None
File "/opt/NanoLLM/nano_llm/plugins/audio/vad_filter.py", line 35, in __init__
self.vad = load_vad()
File "/opt/whisper_trt/whisper_trt/vad.py", line 134, in load_vad
make_cache_dir()
File "/opt/whisper_trt/whisper_trt/cache.py", line 36, in make_cache_dir
os.makedirs(_CACHE_DIR)
File "/usr/lib/python3.10/os.py", line 225, in makedirs
mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/root/.cache/whisper_trt'
It seems like the following symbolic link cause the problem.
/root/.cache/whisper_trt → /data/models/whisper
I try to remove the whisper_trt symbolic link and it works around the problem.
root@jetbot-nx:~/.cache# ls -al
total 8
drwxr-xr-x 2 root root 4096 Jun 9 03:39 .
drwx------ 1 root root 4096 Jun 9 03:39 ..
lrwxrwxrwx 1 root root 20 Jun 9 03:39 whisper -> /data/models/whisper
lrwxrwxrwx 1 root root 20 Jun 9 03:39 whisper_trt -> /data/models/whisper