Hardware - GH200
Operating System: Ubuntu 24.04
There seems to be no real support for arm64. I read everywhere ARM64 support for TensorRT-LLM, TensorRT, vLLM, LLama.cpp but when i dig in there are no binaries which means I end up compiling which by the way doesn’t work; either CUDA doesn’t support it, torch doesn’t align, or a million other reasons so I wrote my own LLM Engine. It works BUT for some reason its single unable to get more than 38% out of the chipset. my engine uses batching (8 at a time) flash attn, and kv cache but im still only getting 12 tok/sec. i should be hitting like 50+/sec. If i call the gpu directly im getting almost 4 TB/sec this has something to do with transformers i think and limitations on arm64 but i can’t figure it out for the life of me. Im at the point my only option is writing my own kernel which isn’t attractive. this chipset has huge potential but its so early there is like zero support on arm64. Ive even tried prebuilt containers to learn they were CPU only on arm64.
Any suggestions? Has anyone else been successful running an llm-server on the arm64 gh200?
Hey, I have been using vLLM on arm64 with GH200 .
I had to rebuild them myself and upload it to Github Releases.
Link for binaries : Releases · dwani-ai/vllm-arm64 · GitHub
Link for the steps i used to build it : deploy/docs/v2 at main · dwani-ai/deploy · GitHub
docker image : dwani/vllm-arm64:latest
Hey, I was just thinking. I know of a fella, who first used an Nvidia. For his Standford Seminal Dissertation, if you want to know, AI=LLM? Why now, why LLMS. It took a young linguist at Standford using an Nvida Titan X to give us what we have today. Yeah, I know him. Here’s the link, it’s about everything you are doing today.
…as detailed in Sam Bowman’s 2016 Stanford dissertation Modeling Natural Language Semantics in Learned Representations…
...as detailed in *Sam Bowman's 2016 Stanford dissertation* [Modeling Natural Language Semantics in Learned Representations](https://nlp.stanford.edu/~manning/dissertations/Bowman-Sam-thesis-final-2016.pdf)...
type or paste code here
vLLM just released their official arm image at v0.10.2, it works flawlessly. Before that there have been 3rd party effort like lambda lab builds GitHub - LambdaLabsML/vllm-builder: vLLM container builder
Also ollama ARM support has been great since I got our GH200 since Sep 2024.
I think the key is using NGC prebuilt containers. NGC had pytorch arm containers since start of 2025. NIM support is still quite terrible, only around 7 out of 125 supports GH200.