Asus GX10 Stable: Hermes Twin Qwen3.6-35A-A3B + Qwen3.6-27B + ComfyUI

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10 Projects
, ,
1

Asus GX10 Stable llama.cpp Setup: 3 Instances, Qwen3.6, Hermes Agent, ComfyUI

After 2 months, I finally have the setup I needed, thanks to this forum and everyone’s shared ideas.

My Asus GX10 was slow, unstable, crashing every few hours, and with low speed 5 token/s generation. It was impossible to get any practical real-world use with it.

My path took me from Nemotron to practically every possible model that could run on this machine. What was most incredible was that solutions from LLMs to harness/drivers were being released every week as I built up the software stack bit by bit.

I’ve tried vLLM, TRT-LLM, and settled on llama.cpp due to its speed, GB10 support, and efficient memory usage.

I finally have a setup here that is ultra-stable, temps kept mostly below 80C, averaging around 60C, while maintaining 3 llama.cpp instances and still having enough VRAM for occasional ComfyUI.

  • Qwen3.6-35A-A3B #1: 60 token/s (128K Context)
  • Qwen3.6-35A-A3B #2: 60 token/s (128K Context)
  • Qwen3.6-27B: 25 token/s (128K Context)

A strong backup was Gemma-4 -31B and Gemma-4-26B-A4B

I hope if you were in the same place as me, this arrangement can help you out. This is mostly for running workflows, completing real-world work over web browser, tasks, APIs, and such. I don’t use it for coding or chat.

A key component to fixing the stability issue

  • Final BIOS update GX10DGX.0104.2026.0326.1657. This improved power stability.

  • RMA to Asus, which left the machine for burn-in for 3 days. It still shut off by itself after this point, but required much higher system stress.

  • Shifted the machine to vertical, power cable side down. This lowered temps by 20C.

  • Kept a laptop cooler 3 inches away blowing at the intake side, which was previously the bottom side. This finally stabilized the machine, even through the most extreme test cases.

Running Hermes Agent 24/7 with over 50 cron jobs. It’s a busy machine, not to mention my ad hoc requests.

  • Qwen3.6-35A-A3B #1: Dedicated to Hermes cron jobs
  • Qwen3.6-35A-A3B #2: Hermes Agent ad hoc requests/management
  • Qwen3.6-27B: Setup workflows, systems/software

llama.cpp build

cmake -B build \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda-13.1/bin/nvcc \
  -DGGML_CUDA=ON \
  -DGGML_CUDA_GRAPHS=ON \
  -DGGML_CUDA_FA_ALL_QUANTS=ON \
  -DCMAKE_CUDA_ARCHITECTURES="121a-real" \
  -DGGML_NATIVE=ON \
  -DGGML_CURL=ON \
  -DBUILD_SHARED_LIBS=OFF \
  -DCMAKE_C_FLAGS="-march=armv9-a+sve2+i8mm" \
  -DCMAKE_CXX_FLAGS="-march=armv9-a+sve2+i8mm"

cmake --build build --config Release -j 20 --target llama-cli llama-mtmd-cli llama-server llama-gguf-split

BIOS / Firmware

Item Details
BIOS Vendor American Megatrends International, LLC.
BIOS Version GX10DGX.0104.2026.0326.1657
BIOS Date 03/26/2026
Board Vendor ASUSTeK COMPUTER INC.
Board Name GX10

Kernel / OS

Item Version
OS Ubuntu 24.04.4 LTS (Noble Numbat)
Kernel 6.17.0-1014-nvidia (SMP PREEMPT_DYNAMIC)
Kernel Build #14-Ubuntu SMP PREEMPT_DYNAMIC Tue Mar 17 19:01:40 UTC 2026
GCC (kernel) gcc-13.3.0, GNU ld 2.42
Userland GCC 13.3.0-6ubuntu2~24.04.1
Make GNU Make 4.3
CMake 3.28.3
Git 2.43.0

NVIDIA Driver / CUDA

Item Version
NVIDIA Driver 595.58.03 (DKMS)
CUDA Toolkit 13.2 (driver cap) / 13.1 (nvcc: V13.1.115)
cuDNN 92,000
PyTorch CUDA cu130 (torch.version.cuda = 13.0)

NVIDIA GPU — Active Processes

PID Process VRAM Usage Port
3,007,439 llama-server (Qwen3.6-27B NVFP4 Q8) ~28.8 GB 10996
3,322,481 llama-server (Qwen3.6-35B-A3B NVFP4 Q8) ~27.5 GB 10993
3,323,919 llama-server (Qwen3.6-35B-A3B NVFP4 Q8) ~27.2 GB 10994

Total GPU VRAM in use: ~83.6 GB / 128 GB

Inference Server Settings (All Instances)

Setting Value
GPU Layers 999 (all layers on GPU)
Context Size 131,072
Flash Attention on
No MMAP enabled
KV Cache Unified enabled
Memory Lock (mlock) enabled
Temperature 0.7
Top-P 0.8
Top-K 20
Min-P 0.00
Threads 8
Batch Size 4096 (27B only)
Chat Template {"preserve_thinking": true}

Active Model Instances (2/5)

Port Alias Model Quantization Draft Spec KV Cache
10993 Qwen3.6-35B-A3B-GGUF Qwen3.6-35B-A3B-NVFP4-MTP Q8_0 + NVFP4 draft-mtp, max 3 BF16
10994 Qwen3.6-35B-A3B-GGUF-2 Qwen3.6-35B-A3B-NVFP4-MTP Q8_0 + NVFP4 draft-mtp, max 3 BF16
10996 Qwen3.6-27B-GGUF Qwen3.6-27B-NVFP4-MTP Q8_0 + NVFP4 draft-mtp + ngram-mod Q8_0

Python / Virtual Environment (Hermes Agent)

Item Version
Python 3.11.15
venv location /home/.../.hermes/hermes-agent/venv
Hermes Agent v0.14.0 (2026.5.16) — 453 commits behind update
OpenAI SDK 2.24.0

Key Python Packages

Package Version
anthropic 0.95.0
fastapi 0.133.1
uvicorn 0.41.0
openai 2.24.0
numpy 2.4.4
pillow 12.2.0
safetensors 0.7.0
sentencepiece 0.2.1
tokenizers 0.22.2
pydantic 2.12.5
PyYAML 6.0.3
Jinja2 3.1.6
click 8.4.1
httpx 0.28.1
websockets 15.0.1
aiohttp 3.13.3
einops 0.8.2
onnxruntime 1.24.4

ComfyUI (Stable Diffusion / Image Generation)

Item Version/Details
ComfyUI 0.24.0 (git: 84e092a — cursor-based pagination #14014)
ComfyUI Manager 4.2.1
Frontend Package 1.45.15
4

VERY Nice thanks for the feed back VERY valuable

Great setup — exactly what I’m planning to replicate.

A few questions:

1.	llama.cpp vs vLLM — did you test vLLM single node with EUGR image? Any stability or performance difference vs llama.cpp?
2.	MTP acceptance rate — what real-world acceptance rate do you get on the 35B-A3B outside of code tasks?
3.	Thermal — is vertical orientation + external fan mandatory for stable H24 operation or just precaution?
4.	KV cache — with ~44GB free, what’s your effective context window in practice on the 35B-A3B?
5.	ComfyUI concurrency — any tok/s degradation on the LLM instances when ComfyUI is actively generating?
6.	235B MoE vs dual 35B-A3B — have you tried Qwen3 235B MoE NVFP4 instead? Curious how deep reasoning compares on complex tasks.

Thanks

5

You can use my Cuda-llm-weight-share - LLM weight share for cuda across multiple processes to save VRAM and reuse the same model weight for multiple llama.cpp processes.

6

llama.cpp vs. vLLM

I haven’t tried EUGR, but I used AEON-Ultimate on vLLM and kept notes. Below are the best settings I was able to get with it.

I eventually returned to llama.cpp due to its improving support for CUDA, MTP, NVFP4, and Qwen models, while still having better launch speed and memory efficiency.

sudo docker run -d \
  --name aeon-ultimate \
  --gpus all \
  --ipc=host \
  --network=host \
  --ulimit memlock=-1:-1 \
  -e TORCH_CUDA_ARCH_LIST="12.0+PTX" \
  -e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True \
  -e VLLM_USE_FLASHINFER_MOE_FP4=0 \
  -e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
  -e TORCH_MATMUL_PRECISION=high \
  -e NVIDIA_FORWARD_COMPAT=1 \
  -e NVIDIA_DISABLE_REQUIRE=1 \
  -e ENABLE_NVFP4_SM100=0 \
  -e VLLM_TEST_FORCE_FP8_MARLIN=0 \
  -e VLLM_USE_FLASHINFER_SAMPLER=1 \
  -v /opt/models/aeon-ultimate-nvfp4:/models/aeon-ultimate:ro \
  -v /opt/models/qwen36-dflash:/models/dflash:ro \
  -v /opt/cache/vllm:/root/.cache/vllm \
  ghcr.io/aeon-7/vllm-aeon-ultimate-dflash:qwen36-v3 \
    vllm serve /models/aeon-ultimate \
    --host 0.0.0.0 \
    --port 10996 \
    --tensor-parallel-size 1 \
    --dtype auto \
    --quantization compressed-tensors \
    --kv-cache-dtype auto \
    --max-model-len 131072 \
    --max-num-seqs 8 \
    --max-num-batched-tokens 8192 \
    --gpu-memory-utilization 0.55 \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --load-format safetensors \
    --trust-remote-code \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_coder \
    --reasoning-parser qwen3 \
    --attention-backend flash_attn \
    --limit-mm-per-prompt '{"image": 4, "video": 2}' \
    --mm-encoder-tp-mode data \
    --mm-processor-cache-type shm \
    --speculative-config '{"method":"dflash","model":"/models/dflash","num_speculative_tokens":15}'

MTP Acceptance Rate

I did not measure the exact acceptance rate. Instead, I measured tokens per second inside the Hermes agent.

A draft value of 2–3 was the most optimal based on speed tests I ran, while also considering the other llama.cpp instances. The 35B model could draft up to 6, but that did not have much practical benefit. It was already fast enough, so drafting more was wasteful at that point.

MTP was used on the 35B model. On the 27B model, MTP was combined with ngram-mod for sequential speculation: MTP first, then n-gram.

This was mainly because the 27B model was bandwidth-intensive. The GB10’s bandwidth limitations meant I needed to work around the issue by limiting drafting while still trying to speed up token generation.

Thermal Management

Thermal management is required, not optional. I am convinced this device was made for a server room with controlled temperature.

Originally, I was inspired by others on this forum to 3D-print a custom rack for the GX10 with external fans. After having both Claude and OpenAI come up with independent designs, I concluded that the issue was related to how air flows through the device: primarily in through the bottom, then out through the back.

The issue is that, in the best case, airflow would have to be directed at a 45-degree angle toward the bottom of the device if I kept it horizontal. Ideally, the design would also need another fan moving air away from the back.

I remembered someone else on this forum mentioning that they set their device vertically. I did not believe it at first, but after seeing the airflow designs, I understood why. The device needs air to flow in through the bottom, which is always obstructed by whatever is underneath it. Even raising the device, which I tried, is not as good as placing it vertically because the air still cannot get direct flow into the device without an angle.

I used a laptop cooler, an old Thermaltake board with four fans, via USB. It was originally placed under the GX10, but I then moved it vertically against the wall. I adjusted the fans so they feed air into the bottom of the GX10, which is now on the side, while the other fans blow air away from the back. Both are parallel to the wall. It is not a strong cooler, but it is enough for this case.

KV Cache

Each llama.cpp instance runs with Parallel 1, to maximize single use speed, and so there is no division of the KV cache.

I tried different settings. Ultimately, setting something like Parallel 2 with a 256K context, which gets divided in two, was not as good as using Parallel 1 with a 128K KV cache.

Each instance is set to 128K, which allows for greater parallelism because each instance manages its own memory pool and CUDA usage.

ComfyUI Concurrency

Yes, even without ComfyUI, the three llama.cpp instances contend for GPU availability. However, this is still better than a single FIFO setup where everything has to wait for the GPU to become available.

Image creation may be a bit slower if other inference is running. I optimized the pipeline for the highest quality at low memory bandwidth.

One change I made was to unload two of the three llama.cpp instances on the weekends, specifically the two 35B instances. I run only the 27B model to create ComfyUI workflows and perform setup operations. This frees up 81 GB, allowing for video FLUX 2 + LTX 2.3 + Upscaler, while still keeping an orchestrator available in Hermes.

235B MoE vs. Dual 35B-A3B

I have not tried running Qwen3 235B MoE NVFP4 on a single GX10. Memory usage might be a challenge.

However, I do not recommend the 35B-A3B model for deep thinking. It is good for light investigations or for following a workflow created by a smarter model, such as the 27B. The 35B-A3B runs all my Hermes cron jobs, most have over 2000 lines of instructions, and it rarely misses.

7

Why not merge the 2 35Bs into a single vllm instance? The parallelization and batch processing should give you more throughput than separate llamacpps

8

I’d love to see a pic of your fan/cooling setup, or even a diagram or sketch. Thank you!

10

It’s a good point. Run a single vLLM, because it has batch processing, makes for much more efficient parallelism. When I last tested vLLM, results were slower than llama.cpp for single requests. That might still be the case but would need re-testing since the arrival of MTP and better use of GB10 (sm121) hardware capabilities.

If I recall, vLLM’s parallelism didn’t benefit until parallelism reached 4 or 8 requests. For my use-case (single-user), it doesn’t happen often enough to be useful.

Rather, being able to unload separate llama.cpp instances has other benefits. I find I run other experiments, and given I don’t have another spare server, I can unload idle models on the fly and free up vram when necessary without bringing down all models.

11

I hope this crude 3D model would show it…it’s been a while since I made 3d models..lol You have the laptop cooler sitting against the wall. There’s an air corridor that’s always kept clear.

I found it ideal to keep clear at least 2 feet in the front of the DGX (blue arrow) and at least 2 feet in the back (red arrow) to maintain optimal temperatures.

dgx-vertical
dgx-vertical1088×824 114 KB

One reference point I used is idle temperature. The better I can get the average idle temperature down, it kept the device from running too hot under load.

nvtop-gpu-idle
nvtop-gpu-idle1471×886 144 KB

12

Made an update over the weekend to Nvidia 610/CUDA13.3. Had to completely remove all existing NVIDIA drivers/CUDA and all dependencies for this update (My system had different versions). Seems to be stable for the last couple days and inference seems to be faster across the board. Recompiled latest (6/29) llama.cpp with CUDA 13.3. If I have time, I’ll post some tests. Here’s the current state:

Component Version
NVIDIA Driver 610.43.02 (via NVIDIA-SMI)
CUDA Toolkit 13.3.33 (V13.3.33)
CUDA SDK date 2026-05-19
cuBLAS 13.5.1.27
NVIDIA UMD (driver) 610.43.02
CUDA UMD (runtime) 13.3

Driver and CUDA toolkit are FINALLY matched — both on the 610/13.3 generation.