LLM on Jetson Nano 4GB B01

Robotics & Edge Computing Jetson Systems Jetson Nano
,
1

Hi All,
I am having Jetson Nano 4GB B01 development board. We want to add LLM on this device locally. I searched on the forum which shares information on NanoLLM, Phi-2 SLM etc. Which will be the best suitable forJetson Nano 4GB development board. Please guide us on which LLM should be used. Our main application will be to give prompts to LLM using voice commands and device should respond back with answers in chat/voice .

Your help is appreciated,

Thanks & Regards,
Polaris

3

@polaris-333 I would try building llama.cpp or ollama and using one of the SLMs in GGUF format (like Phi, ect). I have heard that llama.cpp/ollama can still compile on JetPack 4 / Ubuntu 18.04, but YMMV.

4

HI @dusty_nv Thanks for your reply. Can SLM like Phi be installed locally on the jetson nano?
Also can you share some links or tutorials on how to install the same?

Thanks & Regards,
Polaris.

5

Hi @polaris-333, I’ve not personally verified llama.cpp/ollama work on Nano 4GB and JetPack 4, but heard that other community members may have. I’d probably start by building llama.cpp with -DLLAMA_CUBLAS=on and then trying this quantized GGUF version of Phi-3:

6

Hi @dusty_nv can this models be used without internet?

7

Sorry for the delay @polaris-333 - yes presuming you are able to get them to run on Nano 4GB, they can be used without internet after they are downloaded, because all computation occurs onboard the device.

8

Hi @dusty. Thanks for your reply. I was able to run phi3 mini as well as tiny llama 1.1b using Ollama on Jetson Nano. I wanted to fine tune these models. Which will be the best method to use for fine tuning these models?

Regards,
Polaris.

9

That’s great to hear you were able to run those on Nano 4GB @polaris-333! So on Orin NX and AGX Orin 64GB, we have been using llama-factory for fine-tuning with LORA/qLORA. However I’m going to go out on a limb and think that is unlikely to work in only 4GB memory…but you seem to be having decent luck trying anyways!

Alternatively, do you have a PC with more RAM that you could run that on? Or I hear of people using spot instances on services like runpod/ect to run their trainings remotely. Or perhaps even in Google Colab you may be able to finagle it.

10

Hi @dusty_nv I am going through tutorials for fine tuning. Meanwhile I wanted to know if we can use Speech and Translation AI “NVIDIA Riva” on Jetson Nano 4GB B01 development board. I guess it can’t be used but just need some confirmation. Also which alternatives can be used for TTS and STT models for Jetson Nano?

Regards,
Polaris.

11

Hi @polaris-333, sorry for the delay - the Riva ASR/TTS models require more memory and are supported on JetPack 5/6. On Nano 4GB I would try using Piper-TTS and whisper-base or -small. I have used these on Orin Nano, but not sure about on older JetPack.

12

I’m trying to get an LLM running on a Jetson Nano B01 as well and would love to connect with someone who has experience with this. I’m still learning the ropes, but I’d be happy to discuss this with you. Please let me know if you’re available on email, Telegram, or WeChat.

13

Hi @YuxNvidia Try installing ollama on the device. Using ollama you can run various models. One other option is to install llama.cpp on the device. I have tried both and both work fine.

14

Thanks for you advice.I’m on vacation now, and I can’t do experiments with the equipment at school.Could u tell me which parameter models can work,please?such as gemma 2 2b or llama 8b?Thank you!