i tried the llama.cpp docker file and and the 13b model is nearly same speed as 7b gptq… so great tipp. would i get even more speed if i would go (for me) the long journey of compiling it on the xavier? i tried but had problems with bitsandbytes. i get about 5 t/s in 13 b model with the speechless platypus… model. do you have some tipps for the settings of parameters except -n-gpu-layers 1000 ?
Do you mean re-compiling it from source when you are already running JetPack 5 on your Xavier? No, it’s unlikely to result in further speed-ups, baring any updates to the llama.cpp code itself. My Dockerfiles automatically trigger when updates are pushed to the upstream repos. The CUDA code for JetPack 5 containers is built with both sm_72 and sm_87 enabled, so it is optimized for Xavier too.
The next step up would be to use AWQ (which requires an SM_8x GPU like Orin, because it’s CUDA kernels use SM_8x intrinsics) or MLC - these packages are faster yet (by another ~30%) but more difficult to use, and you need to run the model quantizations yourself.
You could try the other GGUF/GGML quant variations available (the q4_0, q4_K, ect) - they vary in performance and accuracy. I typically just stick to what TheBloke recommends.
Hello, @dusty_nv, I’ve just tested Mistral-7B using LlamaSpeak (TheBloke/Mistral-7B-Instruct-v0.1-GGUF at main). It appears to outperform the Llama2 13B models, but I’m not certain about its accuracy in terms of ‘hallucination.’ It is definitely faster and provides decent answers. I assume it can be used with the previous generation of Nvidia Jetson boards, such as Xavier 16GB. People on Reddit are raising valid questions about when this trend will come to an end, when a new large language model outperforms the current state-of-the-art models))))
Could you give me a hint what I do wrong by following this? Docker
bitsandbytes doesnt seem to work if i just automatically let it install the correct version with autotag ( dustynv/bitsandbytes:r35.2.1). I m on a Xavier AGX if that matters on the newest jetpack version.
then I thought if I build this image with the build command it would build and install the dependencies automatically. I just have now a lot of big docker images with none of it working.
this version worked out of the box on my xavier agx 16GB LLaMa 2 LLMs w/ NVIDIA Jetson and textgeneration-web-ui - Hackster.io
why?
greetings
Hi @robert.semmler1000, what is the issue/error you are having?
TBH I’ve been thinking about removing bitsandbytes by default because it adds startup time anytime transformers is imported, and it’s slow, and there are heaps of GPTQ/GGUF/AWQ models out there with better quantization. Also…that bitsandbytes is from a fork with patches needed to work on Jetson/aarch64, and it’s fallen out of sync with the upstream version. So not sure if that’s related to your problem or not.
basically. but in general I dont know yet how to make textgeneration-webui work on my xavier agx 16GB. but I want to finetune and embed. I have a hard time working around using textgeneration-webui. the llama.cpp docker image worked great. then i wanted to use your textgen webui instead of the one in hackster.io but couldnt get it working with bitsandbytes as dependency. So I m looking now into doing most things with the llama.cpp docker image. Today I will try finetuning and embedding with LORA. hope it works with gpu support. Reddit - Dive into anything
I haven’t tried fine-tuning/LORAs and don’t know how much memory those consume. Are you able to start/run the text-generation-webui and view it in your browser with the commands from these pages?
First I did this:
git clone GitHub - dusty-nv/jetson-containers: Machine Learning Containers for NVIDIA Jetson and JetPack-L4T
cd jetson-containers
sudo apt update; sudo apt install -y python3-pip
pip3 install -r requirements.txt
then this:
cd jetson-containers
./run.sh $(./autotag text-generation-webui) where it suggested the )Docker) r35.2.1 version.
after downloading it gives an error because of bitsandbytes and wont start the webui server.
What’s the error and what’s the version of JetPack-L4T that your Jetson is running? (you can check this with cat /etc/nv_tegra_release outside of container)
$ cat /etc/nv_tegra_release
R35 (release), REVISION: 4.1, GCID: 33958178, BOARD: t186ref, EABI: aarch64, DATE: Tue Aug 1 19:57:35 UTC 2023
there is more if you need more:
bin /usr/local/lib/python3.8/dist-packages/bitsandbytes/libbitsandbytes_cuda114_nocublaslt.so
False
CUDA SETUP: CUDA runtime path found: /usr/local/cuda/lib64/libcudart.so.11.0
CUDA SETUP: Highest compute capability among GPUs detected: 7.2
CUDA SETUP: Detected CUDA version 114
/usr/local/lib/python3.8/dist-packages/bitsandbytes/cuda_setup/main.py:149: UserWarning: WARNING: Compute capability < 7.5 detected! Only slow 8-bit matmul is supported for your GPU!
warn(msg)
CUDA SETUP: Required library version not found: libbitsandbytes_cuda114_nocublaslt.so. Maybe you need to compile it from source?
CUDA SETUP: Defaulting to libbitsandbytes_cpu.so…
================================================ERROR=====================================
CUDA SETUP: CUDA detection failed! Possible reasons:
- CUDA driver not installed
- CUDA not installed
- You have multiple conflicting CUDA libraries
- Required library not pre-compiled for this bitsandbytes release!
CUDA SETUP: If you compiled from source, try again withmake CUDA_VERSION=DETECTED_CUDA_VERSIONfor example,make CUDA_VERSION=113.
CUDA SETUP: The CUDA version for the compile might depend on your conda install. Inspect CUDA version viaconda list | grep cuda.
================================================================================
CUDA SETUP: Something unexpected happened. Please compile from source:
git clone git@github.com:TimDettmers/bitsandbytes.git
cd bitsandbytes
CUDA_VERSION=114 make cuda11x_nomatmul
python setup.py install
CUDA SETUP: Setup Failed!
Aha, okay - thanks. I will add CUDA_VERSION=114 make cuda11x_nomatmul to the bitsandbytes dockerfile for Xavier.
In the meantime, can you try starting the container like this? It will remove bitsandbytes before starting the text-generation-webui server:
./run.sh $(./autotag text-generation-webui) /bin/bash -c '\
pip3 uninstall -y bitsandbytes && \
cd /opt/text-generation-webui && \
python3 server.py && \
--model-dir=/data/models/text-generation-webui \
--listen --verbose'
Thank you! The quick fix uninstalling bb worked too. TheBloke/Mistral-7B-OpenOrca-GGUF:mistral-7b-openorca.Q5_K_M.gguf was slow though 7t/s. Half the speed of just your llama.cpp docker image. My vram should be enough to slowly fine tune a 7b model. So this will be my next step. When everything works I will upgrade to a Orin for sure!
OK great. I updated/fixed the text-generation-webui build and the updated container is here:
docker pull dustynv/text-generation-webui:r35.2.1
Hi, @dusty_nv. I used rust docker image from jetson containers to try building from the source https://github.com/second-state/WasmEdge-WASINN-examples/tree/master/wasmedge-ggml-llama-interactive in order to take advantage of using Rust and Wasm to run LLama models. No Python dependencies makes this implementation really attractive. The authors have claimed that they will soon release support for Ubuntu 20.04 on aarch64, particularly on the Nvidia Jetson boards (free(): invalid pointer Aborted (core dumped) · Issue #48 · second-state/WasmEdge-WASINN-examples · GitHub). Meanwhile, I’ve attempted to run it on AMD64 machine equipped with 32 cores. Indeed, the Rust-based implementation shows impressive speed even when utilizing the CPU. It may be same as the llama.cpp implementation, though I’m not certain about it. It doesn’t provide token generation output in a real time as usual others do, instead, it shows the complete output at the end as one string. Further i want to test running using GPU [WASI-NN] ggml backend: enable cuBLAS and LLAMA_N_GL option. by hydai · Pull Request #2952 · WasmEdge/WasmEdge · GitHub
Hi all, if someone is interested to run LLaVA-1.5 (LLaVA - a Hugging Face Space by badayvedat) on the Jetson boards using llama.cpp. The model was taken here: mys/ggml_llava-v1.5-7b at main
./bin/llava -ngl 35 -m ./ggml-model-f16.gguf --mmproj ./mmproj-model-f16.gguf --image hoover.jpg
llm_load_tensors: VRAM used: 12603.02 MB
...................................................................................................
llama_new_context_with_model: n_ctx = 2048
llama_new_context_with_model: freq_base = 10000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: offloading v cache to GPU
llama_kv_cache_init: offloading k cache to GPU
llama_kv_cache_init: VRAM kv self = 1024.00 MB
llama_new_context_with_model: kv self size = 1024.00 MB
llama_new_context_with_model: compute buffer total size = 162.13 MB
llama_new_context_with_model: VRAM scratch buffer: 156.00 MB
llama_new_context_with_model: total VRAM used: 13783.02 MB (model: 12603.02 MB, context: 1180.00 MB)
The image displays a highway with a road sign situated in a desolate area near a cliff. The sign is white and green, indicating the highway exit and distance to the town of Hood, Dam. The highway has a straight road, with a lone car driving on it. The sign is positioned near a rocky mountain and a grassy area, giving a sense of remoteness to the area.
main: image encoded in 1668.78 ms by CLIP ( 2.90 ms per image patch)
llama_print_timings: load time = 6033.98 ms
llama_print_timings: sample time = 3.50 ms / 86 runs ( 0.04 ms per token, 24550.39 tokens per second)
llama_print_timings: prompt eval time = 2742.29 ms / 626 tokens ( 4.38 ms per token, 228.28 tokens per second)
llama_print_timings: eval time = 9998.41 ms / 86 runs ( 116.26 ms per token, 8.60 tokens per second)
llama_print_timings: total time = 18439.19 ms
You can also pass the prompt using -p parameter
./bin/llava -ngl 35 -m ./ggml-model-f16.gguf --mmproj ./mmproj-model-f16.gguf --image hoover.jpg -p 'What is written
"Hoover Dam"
main: image encoded in 1574.97 ms by CLIP ( 2.73 ms per image patch)
llama_print_timings: load time = 6059.44 ms
llama_print_timings: sample time = 0.34 ms / 7 runs ( 0.05 ms per token, 20833.33 tokens per second)
llama_print_timings: prompt eval time = 1289.05 ms / 623 tokens ( 2.07 ms per token, 483.30 tokens per second)
llama_print_timings: eval time = 781.45 ms / 7 runs ( 111.64 ms per token, 8.96 tokens per second)
llama_print_timings: total time = 7806.37 ms
'
Aha, thanks @shahizat, that’s a great development to have Llava supported in llama.cpp. I have found the llava-1.5 model to be much improved, and able to output JSON reliably. What I’ve been doing is integrating MLC/AWQ quantizations into this local_llm wrapper that delivers superior generation performance along with streaming KV caching for smooth chatting:
- llava-7b: 42.2 tokens/sec on Jetson AGX Orin
- llava-13b: 21.5 tokens/sec on Jetson AGX Orin
- CLIP vision: ~15 FPS on Jetson AGX Orin
This week I am in the process of integrating it all into ‘llamaspeak v2’ for interactive multimodal chat.
Yesterday, we published LLM/VLM/ViT benchmarks at jetson-ai-lab.com and made a bunch of exciting announcements about the AI roadmap!
@dusty_nv That set of benchmarks got me excited to try out some experiments, but I noticed that all the benchmarks are with a Jetson AGX Orin.
Is there any chance you could benchmark LLMs on a 16GB Jetson Orin NX? I know that the AGX is ~275 TOPS, and the NX is ~100 TOPS, but I don’t know if the tokens/second scale quite the same way. And honestly, your benchmarks are amazing, because I’m only getting ~4 tokens/sec on a 4090 RTX with an 8 bit quantized 13B parameter model!
I have a TuringPi motherboard, and I’ve been aching to get a Jetson Orin NX to put into it to create a smart-home conversational interface, probably using 8 bit quantized Mistral 7B or one of the 4 bit quantized 13B models, but I couldn’t find any good numbers for tokens/sec. for the Jetson Orin NX. :(
Thanks muchly for any guidance you might have!
– Morgan
Hi @cyberfox I actually don’t personally have an Orin NX 16GB yet either, but it’s been on my todo list to try because I think it would make a great platform for deploying the -13B models! These days I mainly use MLC/TVM for LLM inference, with llama.cpp as a backup - MLC is ~2x faster.
You already have a carrier/motherboard for the modules, but others without may be interested in this all-in-one Orin NX 16GB system from Seeed:
To give you an idea of the scaling - using llama.cpp on llama-2-7b with 4-bit quantization, AGX Orin gets 22.5 tokens/sec whereas Orin Nano gets 10.2 tokens/sec. I don’t have MLC numbers for Nano, but AGX gets 42.5 tokens/sec on llama-2-7b using MLC/TVM.