Hey all, this setup has really been working out for me. 128K context so I can reasonably use this model for substantial coding and openclaw and it benches at:
Qwen3.5-35B-A3B — Performance Benchmark
Model: Qwen3.5-35B-A3B-UD-Q4_K_XL
Backend: llama-server (llama.cpp)
Date: 2026-04-01
Runs per task: 2 (mean ± stdev reported)
Hardware & Configuration
| Parameter | Value |
|---|---|
| Hardware | DGX Spark / GB10 (Blackwell SM121) |
| GPU offload | All layers (-ngl 999) |
| Context size | 131,072 tokens |
| Batch / ubatch size | 2,048 |
| KV cache type | Q8_0 (K and V) |
| Flash attention | Enabled |
| Memory mapping | Disabled (--no-mmap) |
| Quantisation | Q4_K_XL |
| Reasoning (chain-of-thought) | Disabled (--reasoning off) |
| Temperature | 0.0 (deterministic) |
Metrics are server-side timings from llama-server’s
timingsresponse field — not wall-clock estimates.
Results by Task
Tiny Prompt / Short Answer
| Metric | Mean | ± Stdev |
|---|---|---|
| Prompt tokens | 8 | — |
| Output tokens | 32 | 0 |
| Prefill speed (t/s) | 94.8 | 8.2 |
| Generation speed (t/s) | 48.2 | 0.3 |
| TTFT / prefill time (ms) | 85 | 7 |
| Total wall time (ms) | 778 | 20 |
Factual Q&A
| Metric | Mean | ± Stdev |
|---|---|---|
| Prompt tokens | 19 | — |
| Output tokens | 64 | 0 |
| Prefill speed (t/s) | 163.4 | 36.7 |
| Generation speed (t/s) | 47.8 | 0.2 |
| TTFT / prefill time (ms) | 119 | 27 |
| Total wall time (ms) | 1,576 | 166 |
Short Code Generation
| Metric | Mean | ± Stdev |
|---|---|---|
| Prompt tokens | 19 | — |
| Output tokens | 256 | 0 |
| Prefill speed (t/s) | 201.3 | 47.3 |
| Generation speed (t/s) | 55.2 | 0.1 |
| TTFT / prefill time (ms) | 97 | 23 |
| Total wall time (ms) | 4,846 | 108 |
Complex Code Generation
| Metric | Mean | ± Stdev |
|---|---|---|
| Prompt tokens | 46 | — |
| Output tokens | 512 | 0 |
| Prefill speed (t/s) | 440.6 | 12.7 |
| Generation speed (t/s) | 48.2 | 6.4 |
| TTFT / prefill time (ms) | 104 | 3 |
| Total wall time (ms) | 10,850 | 1,421 |
Multi-step Reasoning
| Metric | Mean | ± Stdev |
|---|---|---|
| Prompt tokens | 69 | — |
| Output tokens | 300 | 0 |
| Prefill speed (t/s) | 543.9 | 15.9 |
| Generation speed (t/s) | 52.6 | 0.1 |
| TTFT / prefill time (ms) | 127 | 4 |
| Total wall time (ms) | 5,857 | 41 |
Long-form Generation (~600 tokens)
| Metric | Mean | ± Stdev |
|---|---|---|
| Prompt tokens | 44 | — |
| Output tokens | 600 | 0 |
| Prefill speed (t/s) | 400.7 | 0.5 |
| Generation speed (t/s) | 52.4 | 0.0 |
| TTFT / prefill time (ms) | 110 | 0 |
| Total wall time (ms) | 11,571 | 21 |
Long Context / Summarisation
| Metric | Mean | ± Stdev |
|---|---|---|
| Prompt tokens | 300 | — |
| Output tokens | 400 | 0 |
| Prefill speed (t/s) | 1,221.4 | 45.0 |
| Generation speed (t/s) | 52.4 | 0.0 |
| TTFT / prefill time (ms) | 246 | 9 |
| Total wall time (ms) | 7,974 | 45 |
Summary Table
| Task | Prompt tok | Output tok | Prefill (t/s) | Generation (t/s) | TTFT (ms) | Wall (ms) |
|---|---|---|---|---|---|---|
| Tiny prompt | 8 | 32 | 94.8 | 48.2 | 85 | 778 |
| Factual Q&A | 19 | 64 | 163.4 | 47.8 | 119 | 1,576 |
| Short code gen | 19 | 256 | 201.3 | 55.2 | 97 | 4,846 |
| Complex code gen | 46 | 512 | 440.6 | 48.2 | 104 | 10,850 |
| Multi-step reasoning | 69 | 300 | 543.9 | 52.6 | 127 | 5,857 |
| Long-form generation | 44 | 600 | 400.7 | 52.4 | 110 | 11,571 |
| Long context / summarisation | 300 | 400 | 1,221.4 | 52.4 | 246 | 7,974 |
Overall
| Metric | Value |
|---|---|
| Mean generation speed | 51.0 t/s |
| Min generation speed | 47.8 t/s |
| Max generation speed | 55.2 t/s |
| Mean prefill speed | 438.0 t/s |
| Peak prefill speed (300-tok prompt) | 1,221.4 t/s |
| TTFT range | 85–246 ms |
Key observations
- Generation speed is highly stable — stdev is ≤ 0.3 t/s on most tasks, indicating consistent GPU utilisation with no thermal throttling.
- Prefill speed scales with prompt length — short prompts (8–19 tokens) prefill at 95–201 t/s; a 300-token prompt reaches 1,221 t/s, reflecting better GPU parallelism over larger batches.
- TTFT is low and predictable — 85–130 ms for typical prompts, rising to ~246 ms only for the 300-token context input.
- No KV cache reuse across requests (cache_hits = 0 on all runs) — a prefix cache or system-prompt cache could reduce TTFT further for repeated prompts.
Implementation Guide: Qwen3.5-35B-A3B via llama.cpp for Claude Code
Serves Qwen3.5-35B-A3B (MoE) locally using llama-server with its native Anthropic
Messages API, so Claude Code points directly at it — no proxy or shim required.
Target hardware: NVIDIA DGX Spark / GB10 (Blackwell, SM121 / CUDA 12.1)
Prerequisites
| Requirement | Notes |
|---|---|
| Ubuntu/Debian Linux | (other distros: adjust apt-get commands) |
| NVIDIA GPU (Blackwell / SM121) | The build flags are SM121-specific; adjust CMAKE_CUDA_ARCHITECTURES for other GPUs |
| CUDA toolkit installed | nvcc must be available, typically at /usr/local/cuda/bin/nvcc |
git |
For cloning llama.cpp |
uv (Python package manager) |
Used to download the model via huggingface_hub |
| ~25 GB free disk space | For the GGUF model file (~20 GB) |
| ~25 GB GPU VRAM | To run 35B MoE fully on GPU |
Step 1 — Install uv
uv manages the Python environment and model download dependencies.
curl -LsSf https://astral.sh/uv/install.sh | sh
# Restart your shell or run:
source ~/.local/bin/env
uv --version # verify
Step 2 — Build llama.cpp with CUDA support
2a. Install the required system library
sudo apt-get install libcurl4-openssl-dev
This enables llama-server’s built-in HuggingFace download flag (-hf).
2b. Clone the repository
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
2c. Set CUDA paths
export CUDACXX=/usr/local/cuda/bin/nvcc
export PATH=/usr/local/cuda/bin:$PATH
Verify nvcc location:
find /usr/local -name nvcc 2>/dev/null
If it’s in a different path, updateCUDACXXaccordingly.
2d. Configure the build
cmake -B build \
-DGGML_CUDA=ON \
-DLLAMA_CURL=ON \
-DGGML_CUDA_FA_ALL_QUANTS=ON \
-DCMAKE_CUDA_ARCHITECTURES=121 \
-DGGML_NATIVE=ON
| Flag | Purpose |
|---|---|
DGGML_CUDA=ON |
Enable CUDA GPU acceleration |
DLLAMA_CURL=ON |
Enable HuggingFace download support |
DGGML_CUDA_FA_ALL_QUANTS=ON |
Flash attention across all quantization types (needed for q8_0 KV cache) |
DCMAKE_CUDA_ARCHITECTURES=121 |
Target Blackwell SM121; change to 89 for Ada, 86 for Ampere, etc. |
DGGML_NATIVE=ON |
Optimize for the host CPU architecture |
2e. Compile and install
cmake --build build --config Release -j 20
sudo cp build/bin/llama-server /usr/local/bin/
sudo cp build/bin/llama-cli /usr/local/bin/
sudo cp build/bin/llama-bench /usr/local/bin/
The -j 20 uses 20 parallel jobs — tune to your CPU core count. Compilation takes
5–15 minutes.
Verify:
llama-server --version
Step 3 — Download the model
The model is Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf from the
unsloth/Qwen3.5-35B-A3B-GGUF repo on HuggingFace (~20 GB).
Run from the project directory
(~/data/ollama_configs/qwen-3-5-30b-moe-claude-code/):
cd ~/data/ollama_configs/qwen-3-5-30b-moe-claude-code
uv run --with "huggingface_hub>=0.24" --with hf_transfer \
python3 -c "
import huggingface_hub
from huggingface_hub import hf_hub_download
import os
os.environ['HF_HUB_ENABLE_HF_TRANSFER'] = '1'
hf_hub_download(
repo_id='unsloth/Qwen3.5-35B-A3B-GGUF',
filename='Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf',
local_dir=os.path.expanduser('~/data/ollama_configs/qwen-3-5-30b-moe-claude-code/qwen3.5-35b-a3b'),
)
print('Done.')
"
HF_HUB_ENABLE_HF_TRANSFER=1 uses the Rust-backed hf_transfer library for
maximum download speed.
The file will land at:
~/data/ollama_configs/qwen-3-5-30b-moe-claude-code/qwen3.5-35b-a3b/Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf
Alternatively, just run ./qwen35-serve.sh — it will download the model automatically
if not present (the download_model function is called by default; the current script
has it commented out, so uncomment the download_model call in main() if needed).
Step 4 — Start the server
cd ~/data/ollama_configs/qwen-3-5-30b-moe-claude-code
chmod +x qwen35-serve.sh
./qwen35-serve.sh
The script will:
- Check that
uvandllama-serverare present - Verify port 8080 is free
- Start
llama-serveron port 8080 (logs to/tmp/qwen35-serve.log) - Wait up to 5 minutes for the server health check to pass
- Run a smoke test against
/v1/messages - Print the Claude Code environment variables to use
- Continue tailing logs until you Ctrl+C
Server flags explained
| Flag | Value | Why |
|---|---|---|
--model |
path to .gguf |
The model to load |
--alias |
qwen3.5-35b |
The model name Claude Code sends in requests |
--host |
0.0.0.0 |
Accept connections from other machines on the network |
--port |
8080 |
Avoids conflict with Ollama’s default 11434 |
--ctx-size |
131072 |
128k token context window |
--batch-size / --ubatch-size |
2048 |
Prompt processing batch size |
--threads |
16 |
CPU threads for non-GPU work |
-ngl 999 |
— | Load all layers onto GPU (effectively “full GPU offload”) |
--no-mmap |
— | DGX Spark perf fix — mmap causes severe slowdowns on GB10; always disable |
--flash-attn on |
— | Flash attention; Blackwell (SM121) supports all quant types natively |
--cache-type-k q8_0 |
— | Critical — Qwen3.5 degrades with the default f16 KV cache; use q8_0 |
--cache-type-v q8_0 |
— | Same as above for value cache |
--temp |
0.6 |
Sampling temperature |
--top-p |
0.95 |
Nucleus sampling |
--top-k |
20 |
Top-k sampling |
--min-p |
0.01 |
Min-p sampling |
--repeat-penalty |
1.05 |
Mild repetition penalty |
--jinja |
— | Enable Jinja2 chat template rendering (required for Qwen’s chat format) |
--reasoning off |
— | Disables chain-of-thought — thinking mode burns tokens on every turn, killing Claude Code performance in agentic use |
Step 5 — Configure Claude Code
Set these environment variables before launching Claude Code. Add them to a wrapper
script or your shell profile.
unset ANTHROPIC_API_KEY
export ANTHROPIC_AUTH_TOKEN=llamacpp
export ANTHROPIC_BASE_URL=http://gx10-44bc.local:8080
export CLAUDE_CODE_ATTRIBUTION_HEADER=0
| Variable | Value | Why |
|---|---|---|
ANTHROPIC_API_KEY |
unset | Must be unset so Claude Code uses ANTHROPIC_AUTH_TOKEN instead |
ANTHROPIC_AUTH_TOKEN |
llamacpp |
Any non-empty string — llama-server accepts all tokens |
ANTHROPIC_BASE_URL |
http://<your-host>:8080 |
Points Claude Code at the local llama-server; replace hostname/IP |
CLAUDE_CODE_ATTRIBUTION_HEADER=0 |
0 |
Critical perf fix — Claude Code prepends an attribution header that busts the KV cache on every turn, causing ~90% slower inference. Setting this to 0 suppresses the header. |
Then start Claude Code:
claude --model qwen3.5-35b
The model name qwen3.5-35b must match the --alias flag used when starting the server.
Example wrapper script (multi-mode claude.sh)
#!/bin/bash
MODE="${1:-claude}"
if [ "$MODE" = "claude" ]; then
# Normal Anthropic cloud
claude "${@:2}"
elif [ "$MODE" = "qwen" ]; then
unset ANTHROPIC_API_KEY
export ANTHROPIC_AUTH_TOKEN=llamacpp
export ANTHROPIC_BASE_URL=http://gx10-44bc.local:8080
export CLAUDE_CODE_ATTRIBUTION_HEADER=0
claude --model qwen3.5-35b "${@:2}"
fi
Usage: ./claude.sh qwen starts Claude Code against the local Qwen model.
Verification
Check server health
curl http://localhost:8080/health
# Expected: {"status":"ok"}
Manual Anthropic API smoke test
curl http://localhost:8080/v1/messages \
-H "Content-Type: application/json" \
-H "x-api-key: llamacpp" \
-d '{
"model": "qwen3.5-35b",
"max_tokens": 64,
"messages": [{"role": "user", "content": "Say READY"}]
}'
View server logs
tail -f /tmp/qwen35-serve.log
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
llama-server not found |
Not built or not copied to /usr/local/bin |
Re-run Step 2 |
Port 8080 already in use |
Another process (including a previous server) is bound | lsof -i :8080 to find it, then kill it |
| Slow inference (~5–10 tok/s) | --no-mmap missing, or KV cache using f16 |
Confirm both --no-mmap and --cache-type-k/v q8_0 are set |
| KV cache thrash / slow per-turn | Attribution header not suppressed | Confirm CLAUDE_CODE_ATTRIBUTION_HEADER=0 is exported |
| Out of VRAM | Model too large for single GPU | Use a smaller quant (Q4_K_M instead of Q4_K_XL) or split across GPUs with --tensor-split |
| Download hangs or is slow | hf_transfer not active |
Ensure HF_HUB_ENABLE_HF_TRANSFER=1 is set before the download |
cmake can’t find nvcc |
CUDA not on PATH | export CUDACXX=/usr/local/cuda/bin/nvcc and re-run cmake |
Architecture summary
Claude Code CLI
│ (Anthropic Messages API)
▼
llama-server :8080
│ (native /v1/messages endpoint)
▼
Qwen3.5-35B-A3B-UD-Q4_K_XL.gguf
│ (fully loaded on GPU)
▼
NVIDIA GB10 (Blackwell SM121)
llama.cpp’s llama-server natively implements the Anthropic Messages API — no adapter,
proxy, or shim is required. Claude Code speaks directly to it.