Weekend Home Lab: Qwen3.5 9B on Jetson Orin Nano Super with TurboQuant4 (100K token window)

Robotics & Edge Computing Jetson Systems Jetson Orin Nano
, , ,
1

Weekend Home Lab: Qwen3.5 9B on Jetson Orin Nano Super with TurboQuant4 (100K token window) ⚙️🧪

Pushing the Jetson Orin Nano Super to its limits is always fun. This weekend, I tested Qwen3.5 9B on the Orin Nano Super, and the power efficiency continues to impress.

I’m planning to pair this with OpenClaw next to see how well the setup holds under real agentic workloads.

Qwen 3.5 9B is smart. TurboQuant 3 is amazing in speed, while TurboQuant 4 feels more stable in long‑running tasks.

With TurboQuant 3, I can use a 128K token window.
With TurboQuant 4, I reduce the window to 100K for stability.

Gemma 4 E4B also runs on the Jetson Orin Nano. I’ll be testing both Qwen 3.5 9B and Gemma 4 E4B with OpenClaw to see which one delivers the most stable experience for agentic workflows.

If Gemma 4 E4B or E2B can run well with OpenClaw, that will be a good machine for running 7x24 local AI LLM.

I am so happy seeing Jetson using 7.2G/7.4G memory. That is almost fully utilized system resources.

#JetsonOrinNano #Qwen3 #Gemma4 #TurboQuant #AILabs #EdgeAI #HomeLab #OpenClaw #AIEngineering #WeekendBuild #tq3 #tq4 #llamacpp

image
image1137×718 148 KB

6

Hi,

We also verified that the Gemma 4 E4B can work on Jetson Orin Nano.

But Qwen 3.5 9B is a large model, usually we expect the model<=4B to run on Orin Nano.
Would you mind sharing some details about how you achieve this?
This should help others who want to run the same model on Orin Nano.

Thanks.

7

The followings are the important items

  1. unsloth/Qwen3.5-9B-GGUF · Hugging Face , I use Qwen3.5-9B-Q4_K_M.gguf. Q4 file. Q4_K_M is about 5.68GB.
  2. Llama cpp, turboquant fork, GitHub - TheTom/llama-cpp-turboquant: LLM inference in C/C++ · GitHub. TurboQuant can use less RAM for KV cache (context token)
  3. Jetpack 6 for Jetson and followed the memory optimized guide to free up memory.

I run this TurboQuant Llama CPP fork with this Qwen3.5 9B Q4 model.

TurboQuant 3 or 4 can work.
TQ3 can use full 128K context.
TQ4, I have to downside it to 100K context.

Note: For 9B models on 8GB Orin Nano, TQ is essential for limited memory.

nohup “$LLAMA_SERVER”
–model “$MODEL_PATH”
–port “$PORT”
–ctx-size “$CTX_SIZE”
–n-gpu-layers “$GPU_LAYERS”
–threads “$THREADS”
–cache-type-k turbo4
–cache-type-v turbo4
–host 0.0.0.0
–metrics > “$LOG_FILE” 2>&1 &

8

This TurboQuant Llama CPP fork with Gemma 4 e2b and e4b has crash issue with chat template in some cases. I spent few hours found out that crash chat template issue.

9

Hi,

Thanks for the update and testing.