I created a fork of the llama.cpp mainline to experiment with the latest LLM methodologies on Jetson Orin, including TurboQuant and MTP speculative decoding.
On Jetson Orin NX, I was able to achieve a 30–40% performance lift using MTP compared to llama.cpp mainline. For example:
- Gemma 4 E4B-Q4_K_M improved from ~13 tk/s to ~18 tk/s, and over 20 tk/s in some scenarios
- Gemma 4 12B-Q4_K_M improved from ~7.5 tk/s to >9.5 tk/s, frequently exceeding 12 tk/s
In contrast, I didn’t observe the same gains on an RTX A5000 — it actually regressed slightly.
More details can be found in my repos below. As a credit-card-sized device, the Orin NX is remarkably powerful. I don’t have access to an Orin Nano Super or AGX, so you’re welcome to test my fork on your own hardware and share results.