Slow Qwen 3.6 27B NVFP4 - recipe feedback

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
, ,
1

Hey everyone, I’m trying to set up a single DGX Spark for with Nemoclaw for offline LLM workloads. Based on another forum entry I found that people have success with using DFlash for fast inference. However, I’m having no luck. I use eugr’s repo for Spark-specific builds as detailed in the forum post, but I’m not seeing the benchmarked ~30-40 tokens/sec in real life usage. This is the recipe I’m using right now, but I’ve had a couple of variations with more SpecDecoding tokens and higher batch sizes, but no dice:

name: qwen36-27b-prismascout-dflash-agentic-tps
description: Qwen3.6-27B PrismaSCOUT DFlash Spark optimized for 6-7 concurrent agents

model: rdtand/Qwen3.6-27B-PrismaSCOUT-Blackwell-NVFP4-BF16-vllm
solo_only: true

container: vllm-node-dflash

build_args:
–apply-vllm-pr
“40898”
–tf5

mods:
mods/fix-qwen3.6-chat-template
mods/flashqla
defaults:
port: 8000
host: 0.0.0.0
tensor_parallel: 1
gpu_memory_utilization: 0.85
max_model_len: 262144
max_num_batched_tokens: 49152
max_num_seqs: 16
served_model_name: qwen/qwen3.6-27b
speculative_dflash: ‘{“method”:“dflash”,“model”:“z-lab/Qwen3.6-27B-DFlash”,“num_speculative_tokens”:7}’

env:
VLLM_ALLOW_LONG_MAX_MODEL_LEN: 1
PYTORCH_CUDA_ALLOC_CONF: expandable_segments:True
TORCH_MATMUL_PRECISION: high
NVIDIA_FORWARD_COMPAT: 1
NVIDIA_DISABLE_REQUIRE: 1
VLLM_MARLIN_USE_ATOMIC_ADD: 1

command: |
vllm serve rdtand/Qwen3.6-27B-PrismaSCOUT-Blackwell-NVFP4-BF16-vllm 
–served-model-name {served_model_name} 
–host {host} 
–port {port} 
-tp {tensor_parallel} 
–max-model-len {max_model_len} 
–max-num-batched-tokens {max_num_batched_tokens} 
–max-num-seqs {max_num_seqs} 
–gpu-memory-utilization {gpu_memory_utilization} 
–dtype bfloat16 
–kv-cache-dtype fp8 
–load-format fastsafetensors 
–enable-prefix-caching 
–enable-auto-tool-choice 
–reasoning-parser qwen3 
–tool-call-parser qwen3_coder 
–chat-template fixed_chat_template.jinja 
–speculative-config ‘{speculative_dflash}’ 
–attention-backend flash_attn

Can you give me some feedback? The results quality-wise are fine, but I’m getting around 10 tokens per sec, sometimes around 30 but mostly when running parallel calls/subagents. Am I misconfiguring something, or is this a Spark limitation? I’d be happy to get any kind of feedback.

2

I tried it today, no luck either. Could be a new image pushed just yesterday. My regular 27B nvfp4 on 0.20 release is essentially the same but with working q8 kv cache, this one has Turboquant integrated but not wired into the backend, at least to FlashAttention this model relies on.

3

A bit late to the post, but I had the same experience with this model. Dflash with 27b actually has lower token generation, while native MTP=3 on a single DGX can consistently generate 25–27 t/s. But in 35B, Dflash is quite good and generate up to 90-100 token/s

4

DFlash is broken in two major ways on vLLM today.

  1. DFlash Sliding Window Attention PR #40898 is not merged, and cannot be pulled into spark-vllm-docker due to upstream conflicting changes. The Z-Lab authors have left this in such a state, unattended, for a month now - not a great look.
  2. Recent change to a different structured output validation engine in vLLM has basically broken structured output for all diffusion models/drafters. That includes schema enforcement and - yes - tool calling.

I’m watching this closely. I made a PR which rebases the SWA PR on current HEAD (mine is [Model] TEMP Fix for DFlash SWA PR (#40898) by JDWarner · Pull Request #44807 · vllm-project/vllm · GitHub ) but there is no fix for problem 2 above as yet.

The clear conclusion is thus to use MTP for the moment.

5

For me dflash was about 10% faster (~30 t/s) than mtp, but I could never get it working with fp8. I use 27b for plans and review and 35b for the code and testing, I have them both running at the same time on different instances of vllm. To round it out I have a third vllm running qwen3-vl-2b just because 27b and 35b can do vision, but they are so slow at it in comparison. It’s my favorite thing about the spark to have them all up at once.

6

Well yeah ofc you’ll see better performance on an A3B model than dense 27B. But that’s regardless of MTP/DFlash.

7

Oh it can’t anymore? When I built this recipe, it was still possible. Good to know not to try to rebuild for now.

8

Good news: The vLLM devs reverted the change which was causing problem #2 in my above post earlier this evening.

Rebuild spark-vllm-docker using my draft PR which allows DFlash to be used with sliding window attention (barebones minimal necessary flags below):

./build-and-copy.sh --apply-vllm-pr 44807 --tf5

I just tested this and it’s working!

9

I built mine with 40898 as shown in my post, it works fine, my issue is even with that I only reach anything between 0 and 40, rarely 50tps, which seems low with all the tool usage of NemoClaw. I was just expecting more from a device that’s being advertised as an LLM inference target device.

10

40898 will fail to apply. You must use my draft PR in the the above command. Otherwise it will seem to work but DFlash will perform worse (acceptance) and scale much worse (throughput) as context increases.