LLaMa 2 LLMs w/ NVIDIA Jetson and textgeneration-web-ui

Robotics & Edge Computing Jetson Systems Jetson Projects
64

Hi @dusty_nv - I recently joined the Jetson ecosystem (loving it so far)!

Would you consider providing some guidance on how to get Ollama to run on the Jetson lineup? Similarly to llama.cpp, it allows users to run models locally and has a rapidly growing community.

Ollama runs on Linux, but it doesn’t take advantage of the Jetson’s native CUDA support (so it technically works, but it is CPU only).

They also have an official Docker image on Docker Hub.

Would love your thoughts on how to get Ollama running on a Jetson! Thank you!

65

Hi @brandon_b, you can just try building it on Jetson and see if any errors occur, however if there are issues, given there is no CUDA support I’m not sure how worthwhile it will be, given there are a number of APIs that do support these models and CUDA and already build/run on Jetson (like llama.cpp, exllama, AutoGPTQ, AWQ, MLC, ect)

66

Hi @dusty_nv - thanks for getting back to me and sorry for not being a bit clearer.

Ollama definitely does support CUDA, but by default it depends on drivers that would be available via an external GPU (nvidia-smi or something like that).

I totally recognize that there are other local LLM projects out there. That said, I really believe that Ollama is perfect for people just getting started with AI development (i.e. the people who would buy a Jetson developer kit). It installs and runs on GPU with a one-liner and has automatic support for external-facing APIs. It also has a rapidly growing and engaged community.

I would be happy to support the development to make Ollama bypass its default CUDA dependency method and use the Jetson’s native CUDA support instead.

I just don’t know where to begin as I’m totally new to it all.

Would you be willing to look at the Ollama project and point me in the right direction?

67

Thanks @brandon_b, I can take working PR’s on jetson-containers to add it - Ollama appears to be based in Go, in which I am not versed. I’ll add it to the list to look into more though. It appears to use llama.cpp, there is a CUDA-enabled container for that here.

68

Hi ! @dusty_nv Thank you for the excellent work. I’ve been following your tutorials (just got my first Jetson - Nano 4GB) and I’m trying this models.

Does any work (or could work) with the Nano 4GB ? I already did try, but all fail with CUDA related problems. They even ask if I have a nvidia card… Do they work or is something I’m missing ? I tried minigpt4 and did try llamaspeak.
It’s a fresh image, with all the updates installed - Ubuntu 18.04 - with Pytorch installed and working.

Thank you

69

Hi @feiticeir0, I’m sorry, the original Jetson Nano doesn’t have enough memory to run these LLM’s, and many of the packages don’t support the older software. I do have a HuggingFace Transformers container built for JetPack 4 (dustynv/transformers:r32.7.1) which could in theory run the likes of BERT or DistilBERT.

Our CNN-based vision tutorials like Hello AI World, JetBot, and the Nano DLI continue to be supported on the original Jetson Nano which we have YouTube videos for here if you haven’t seen these:

70

Thank you @dusty_nv for the reply. I thought so, but I had to try.

Will check the transformers container and see if it can any of the models.
Again, thank you

71

@dusty_nv I used the newest textgeneration-webui docker image and mixtral doesnt work and i cant get superboogav2 to work for rag. I am still on xavier agx 16GB. I use the text-generation-webui:1.7-r35.4.1 docker image. I am on nvidia-l4t-core 35.4.1-20230801124926. Should i upgrade to 22.04 ubuntu and newest jetpack? Is it supporting the xavier 16GB, since i got supwerboogav2 to work on my laptop but with slow t/s, but i was able to install the desired packages. On the docker image not. Reddit - Dive into anything

72

Hi @robert.semmler1000, was the support for those features you mentioned recently added to text-generation-webui? It looks like that container was last updated a month ago, so I may need to rebuild it. Oogabooga will run on the Jetson’s with lower memory capacity, it just depends on which model size you are trying to load and with which API/quantization methods.

73

oh, thats what i wanted to try before mentioning it here. “Support for Mixtral was merged into Llama.cpp on December 13th.” I wanted to try it in your newest llama.cpp docker first but forgot. I wanted to use this TheBloke/laser-dolphin-mixtral-2x7b-dpo-GGUF · Hugging Face ( [laser-dolphin-mixtral-2x7b-dpo.Q4_K_M.gguf] (laser-dolphin-mixtral-2x7b-dpo.Q4_K_M.gguf with 10.28 GB Ram required)) on my current text-generation-webui. It gives the error message Traceback (most recent call last):

File “/opt/text-generation-webui/modules/ui_model_menu.py”, line 201, in load_model_wrapper etc.

I have problems finding version numbers of llama.cpp in text-generation-webui. But according to this it should be implemented (newest llama.cpp with python) and working How are you accessing it? | Hacker News .

And superboogav2 would be great to be working too to do rag.

your newest llama.cpp docker image seems to be working with my smaller laser dolphin mixtral.

74

OK gotcha, I just triggered the text-generation-webui container to get rebuilt (it should include the latest llama.cpp during that). llama.cpp is unversioned, but llama_cpp_python is (and you can check that version in the container with pip3)

75

I am trying to load TheBloke_Mistral-7B-Instruct-v0.2-AWQ, my first model using the AWQ format.
I got the error

  File "/opt/text-generation-webui/modules/models.py", line 307, in AutoAWQ_loader
    from awq import AutoAWQForCausalLM
ModuleNotFoundError: No module named 'awq'

I tried to pip install this module but pip could not find it.

( I updated the container to the newest release )

76

Hi @pontual, although I have container for AWQ, this is using a different library called AutoAWQ that I need to add support for - taking a look at it now.

77

OK, I made the Dockerfile for AutoAWQ and added it to the text-generation-webui container. I tested it in oobabooga on TheBloke_Mistral-7B-Instruct-v0.2-AWQ and it worked. It’s only for JetPack 6 because of the minimum CUDA version that AutoAWQ requires. The updated container is at dustynv/text-generation-webui:r36.2.0 (so pull that again if you already have it)

78

I was able to make it work, thank you!

For information, this format loads much faster than GPTQ on my Jetson Orin NX 16GB.

80

It seems like Jetson Nano 4GB can actually run ollama according to this:
Running Ollama 2 on NVIDIA Jetson Nano with GPU using Docker – Collabnix
Haven’t tried yet, but I will, as soon as I get some time :)

81

It’s not working. There is a couple of threads on Ollama GitHub issues

82

@shahizat @dusty_nv wanted to check up with u guys , if we can run LLAMA on Xavier NX and AGX boards or it only requires Orin series ? if so then can i follow up with @dusty_nv container based approach ?

83

@abhigoku10 oh yea, you can run Llama/ect on Xavier and JetPack 5, like in text-generation-webui, llama.cpp, exllama, AutoGPTQ, ect. I have containers built for JetPack 5 for these. Xavier doesn’t support all the latest optimizations and packages (because it is sm72 and Python 3.8), but it is more than enough to run the models with quite decent performance still.

84

I have a Jetson nano 8Gb, and i’m running the textgenerationwebui with a 4bit quantized model, i’m using the llama cpp loader and the openai extension, howerver, i’m getting 2 to 3 tokens/s which is very slow, i don’t know what i’m missing here?