Hey everyone! I wanted to share my setup for running Qwen3.5-27B (Claude 4.6 Opus reasoning-distilled, v2) locally with Claude Code using llama.cpp on both the NVIDIA Jetson Thor (GB10, 128 GB unified memory) and a desktop RTX 5090 (32 GB VRAM). This gives you a fully local, private agentic coding assistant, no API keys or cloud calls needed.
The model is Jackrong/Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-v2-GGUF, which includes vision support (mmproj included) and produces more efficient chain-of-thought reasoning than v1 (~24% shorter thinking, +31.6% more correct solutions per token).
Build llama.cpp and run llama-server
Install dependencies:
sudo apt update
sudo apt install libcurl4-openssl-dev libssl-dev build-essential cmake git
Clone and build:
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
mkdir build && cd build
For RTX 5090 (Blackwell, sm_120):
cmake .. -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="120" -DGGML_CUDA_FA_ALL_QUANTS=ON
For Nvidia Jetson Thor (sm_110a):
cmake .. -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="110a" -DGGML_CUDA_FA_ALL_QUANTS=ON
Then compile:
make -j$(nproc)
Download the Q4_K_M quantization and the vision projector:
wget -O models/Qwen3.5-27B-v2.Q4_K_M.gguf \
"https://huggingface.co/Jackrong/Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-v2-GGUF/resolve/main/Qwen3.5-27B.Q4_K_M.gguf"
wget -O models/mmproj-BF16.gguf \
"https://huggingface.co/Jackrong/Qwen3.5-27B-Claude-4.6-Opus-Reasoning-Distilled-v2-GGUF/resolve/main/mmproj-BF16.gguf"
With thinking/reasoning enabled (best for analytical tasks):
./build/bin/llama-server \
-m ./models/Qwen3.5-27B-v2.Q4_K_M.gguf \
--mmproj ./models/mmproj-BF16.gguf \
--temp 0.6 \
--top-p 0.95 \
--top-k 20 \
--min-p 0.00 \
-ngl 99 \
--flash-attn on \
--cache-type-k q8_0 --cache-type-v q8_0 \
--host 0.0.0.0 \
--port 8080
Output:
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant
<think>
'
srv init: init: chat template, thinking = 1
main: model loaded
main: server is listening on http://0.0.0.0:8080
main: starting the main loop...
srv update_slots: all slots are idle
Install and Configure Claude Code
curl -fsSL https://claude.ai/install.sh | bash
Fix KV cache invalidation (critical — without this, inference is ~90% slower):
Claude Code prepends a changing attribution header that invalidates the KV cache on every request. Fix it by editing ~/.claude/settings.json:
cat > ~/.claude/settings.json << 'EOF'
{
"env": {
"CLAUDE_CODE_ATTRIBUTION_HEADER": "0",
"CLAUDE_CODE_ENABLE_TELEMETRY": "0",
"CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1"
}
}
EOF
Important: Using
export CLAUDE_CODE_ATTRIBUTION_HEADER=0does NOT work — it must be in the settings.json file.
If Claude Code asks you to sign in, bypass it:
cat > ~/.claude.json << 'EOF'
{
"hasCompletedOnboarding": true,
"primaryApiKey": "sk-no-key-required"
}
EOF
Set the environment variables and launch Claude Code:
export ANTHROPIC_BASE_URL="http://localhost:8080"
export ANTHROPIC_API_KEY="sk-no-key-required"
cd ~/your-project
claude --model qwen3.5-27b
Output:
