Claude CLI - VLLM - Waiting for API Response - July 2026 Updates

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

I have a single ASUS GX10 and have been running eugr/spark-vllm-docker with a build of vllm 0.23 that fixed the Claude Code CLI issue from cli version 2.1.154 and higher and haven’t updated that since.

The model I’ve been running is saricles/Qwen3-Coder-Next-NVFP4-GB10 from hugging face and the startup command is below

ExecStart=/opt/spark-vllm-docker/launch-cluster.sh --solo exec vllm serve saricles/Qwen3-Coder-Next-NVFP4-GB10 --host 0.0.0.0 --port 8000 --max-model-len 262144 --gpu-memory-utilization 0.87 --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser qwen3_coder --kv-cache-dtype fp8 --enable-prefix-caching --enable-chunked-prefill

Ever since applying the July 2026 updates Claude CLI gives me “Waiting for API Response - will retry in Xm YYs - check your network” messages and it waits for a couple of minutes. While this is going on nvidia-smi shows the GPU is still loaded so I assume the model is still working.

I thought this might have been a regression in the Claude CLI when running local models so I reverted that back to a known working good one in 2.1.153 but the issue still occurred. I don’t think this is a network issue as I get that error if Claude CLI is running on the GX10 or on another system.

The only thing that has changed on the GX10 is the July 2026 updates that I applied via the DGX dashboard

I ran this problem through actual Claude and it suggested the issue was due to me not using MARLIN and recommended I change my config to as per below but this made no difference.

ExecStart=/opt/spark-vllm-docker/launch-cluster.sh --solo -e VLLM_MARLIN_USE_ATOMIC_ADD=1 -e TORCH_CUDA_ARCH_LIST=12.1 exec vllm serve saricles/Qwen3-Coder-Next-NVFP4-GB10 --host 0.0.0.0 --port 8000 --max-model-len 262144 --gpu-memory-utilization 0.87 --tensor-parallel-size 1 --enable-auto-tool-choice --tool-call-parser qwen3_coder --kv-cache-dtype fp8 --enable-prefix-caching --enable-chunked-prefill --moe-backend marlin

Does anyone have any ideas?

I installed OpenCode I don’t seem to get this sort of error using that, but I find the Claude CLI better, maybe because that is what I’m used to.

Is there a better way I should be starting spark-vllm-docker? I currently have it set to auto start on boot using SystemD.

Should I change to a newer model? I mainly use this local AI box for coding tasks.

Any help would be appreciated as I’m still relatively new to all this.

2

This may due the the model is still thinking claude doesn’t receive any stream for a long time.
You can try to extend the streaming timeout setting in your claude start up script:
export API_TIMEOUT_MS=1800000
export CLAUDE_STREAM_IDLE_TIMEOUT_MS=1800000
export CLAUDE_BYTE_STREAM_IDLE_TIMEOUT_MS=1800000

3

will give that a go. it is just strange that this wasn’t an issue before the last lot of nVidia updates

4

I encountered the issue earlier, I have to revert back to Claude Code v2.1.232 for it to work. I believe something from Claude Code’s recent updates broke the vLLM caching.

2.1.233: ~10x per-turn latency regression against custom ANTHROPIC_BASE_URL (vLLM) — prefix cache no longer hits · Issue #87227 · anthropics/claude-code · GitHub seems to capture the issue.

5

I reverted back further to 2.1.153 and still have the issue

6

This had no effect.

7

Make sure you have this too:
export CLAUDE_CODE_ATTRIBUTION_HEADER=0

8

yep I just found that in Qwen3.8 - How to Run Locally | Unsloth Documentation as I’m looking at changing my model to run the NVFP4 version of this. Going to try this on my existing model

also those env variables above have now changed the retry in from 2m to 29 minutes. that retry in my be controlled by those environment variables. It never waits that long though, so when the vllm api returns a response it must continue. There must be a trigger point to when Claude CLI displays that message, and those env variables do not seem to control that.

9

nope that didn’t fix it either

10

Why are you using the old vllm 0.23 instead of 0.27.2rc1?

11

I’m using the latest from spark-vllm-docker as of a few hours ago now. I’ve also switched my model out to qwen3.8

version 0.27.2rc1.dev360+ge85d1b69c.d20260821
model RadixArk/Qwen3.8-27B-NVFP4

Using this recipe - support qwen3.8-27b-nvfp4? · Issue #350 · eugr/spark-vllm-docker · GitHub

Quite a bit slower than the Qwen3-Coder-Next model I had before, but I’m getting less Claude Waiting For API messages now.

It seems to handle concurrent requests better. Previously I could only run one Claude instance , and if I tried to use Open WebUI on the same model, both would slow to almost standstill. I currently have one Claude instance, one OpenCode instance, and an Open WebUI chat going, slow but working. The vllm logs show 3 running requests and Avg generation at 40 to 60 tokens/s which I assume is the sum of all 3 running requests combined.

12

It slows down because qwen3.8-27b is a Dense model - each token polls the entire model. You need to use MoE then it will be faster

P. S. It seems that Qwen is planning a new MoE release next week.