vLLM API returns "Connection reset by peer"

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

Hi everyone,

I’m trying to deploy Qwen3.6-35B-A3B-NVFP4 using the latest vllm/vllm-openai Docker image on an ASUS Ascent GX10 (NVIDIA® GB10 Grace Blackwell).

I’m following NVIDIA’s official guide:

However, I cannot access the OpenAI-compatible API after starting the container.

Environment

  • Hardware: ASUS Ascent GX10

  • Platform: NVIDIA® GB10 Grace Blackwell

  • Docker

  • Image: vllm/vllm-openai:latest

  • Model: nvidia/Qwen3.6-35B-A3B-NVFP4

Docker command

docker run -d --gpus all -p 12001:8000 \
  --name qwen36-vllm \
  -e HF_TOKEN=<HF_TOKEN> \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --host 0.0.0.0 \
  --port 8000 \
  --tensor-parallel-size 1 \
  --trust-remote-code \
  --kv-cache-dtype fp8 \
  --attention-backend flashinfer \
  --moe-backend marlin \
  --gpu-memory-utilization 0.7 \
  --max-model-len 262144 \
  --max-num-seqs 4 \
  --max-num-batched-tokens 8192 \
  --enable-chunked-prefill \
  --async-scheduling \
  --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}' \
  --load-format fastsafetensors \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_xml \
  --enable-auto-tool-choice

Observed behavior

The container starts successfully and the model initialization begins.

However, the log appears to stop after the following messages and never progresses further:

INFO [parallel_state.py:1607] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://172.17.0.3:56269 backend=nccl
INFO [parallel_state.py:1942] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
INFO [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
WARNING min_p and logit_bias parameters won't work with speculative decoding.
INFO [gpu_model_runner.py:5209] Starting to load model nvidia/Qwen3.6-35B-A3B-NVFP4...
INFO Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
INFO Selected FlashInferFP8ScaledMMLinearKernel for ModelOptFp8LinearMethod
INFO DeepGEMM PDL enabled
INFO DeepGEMM E8M0 enabled
INFO Using Triton/FLA GDN prefill kernel
INFO Using 'MARLIN' NvFp4 MoE backend
INFO Using AttentionBackendEnum.FLASHINFER backend.

No additional log messages are produced after this point.

I never see messages such as:

  • Application startup complete

API request

curl http://localhost:12001/v1/models

returns:

curl: (56) Recv failure: Connection reset by peer

Questions

  1. Has anyone successfully deployed Qwen3.6-35B-A3B-NVFP4 on an ASUS Ascent GX10 (GB10 Grace Blackwell) using the official NVIDIA guide?

  2. Is it expected for the model loading to pause at this stage for a long time, or does this indicate that initialization has stalled?

  3. Are there any known compatibility issues between this model and the latest vllm/vllm-openai image?

Any suggestions would be greatly appreciated. Thanks!

2

@dotrantrung2003 your container is not launching properly, thus no Application startup complete on the console logs.

The curl error is also telling you the same. The container is not working!

3

@elsaco I’m following NVIDIA’s official guide: https://build.nvidia.com/spark/vllm/agent-ready-qwen35b

I agree that the API never becomes available, but the container itself does not appear to exit.

The container remains in the Up state. It just seems to hang during initialization.

From the logs, it consistently stops at the following line and no new log messages are produced afterwards:

(EngineCore pid=173) INFO 07-22 01:24:13 [gpu_model_runner.py:5209] Starting to load model nvidia/Qwen3.6-35B-A3B-NVFP4...
(EngineCore pid=173) INFO 07-22 01:24:13 [cuda.py:535] Using backend AttentionBackendEnum.FLASH_ATTN for vit attention
(EngineCore pid=173) INFO 07-22 01:24:13 [mm_encoder_attention.py:373] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(EngineCore pid=173) INFO 07-22 01:24:13 [__init__.py:631] Selected FlashInferFP8ScaledMMLinearKernel for ModelOptFp8LinearMethod
(EngineCore pid=173) INFO 07-22 01:24:13 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
(EngineCore pid=173) INFO 07-22 01:24:13 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
(EngineCore pid=173) INFO 07-22 01:24:13 [deep_gemm.py:120] DeepGEMM E8M0 enabled on current platform.
(EngineCore pid=173) INFO 07-22 01:24:13 [qwen_gdn_linear_attn.py:228] Using Triton/FLA GDN prefill kernel (requested=auto, head_k_dim=128).
(EngineCore pid=173) INFO 07-22 01:24:13 [nvfp4.py:239] Using 'MARLIN' NvFp4 MoE backend out of potential backends: ['FLASHINFER_TRTLLM', 'FLASHINFER_CUTEDSL', 'FLASHINFER_CUTEDSL_BATCHED', 'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'HUMMING', 'EMULATION'].
(EngineCore pid=173) INFO [cuda.py:416] Using AttentionBackendEnum.FLASHINFER backend.

Since it never progresses beyond this point, I never see the Application startup complete message and all API requests return Connection reset by peer.

I’ve attached the complete log from container startup. Would you mind taking a look to see if there’s anything suspicious or if this looks like a known issue?

log.txt (10.5 KB)

Thanks!

4

It looks like it hangs after trying to download the model. Are you downloading from huggingface or another network source, or do you have it locally? You can check if something is running in the container by running docker top <container_name>

5

“Connection reset by peer” almost always means the vLLM engine never finished initializing, so nothing is listening on 0.0.0.0:8000 inside the container. From your log it stops at model load + attention backend selection, which is consistent with a few common failure modes on GB10:

1. Wrong vLLM image for SM121. The upstream vllm/vllm-openai:latest does not include Blackwell/SM121 support out of the box. Use a GB10-tuned build (e.g., eugr/spark-vllm-docker with --tf5) or a vLLM wheel built for CUDA 13 / SM121.

2. Model download hang. nvidia/Qwen3.6-35B-A3B-NVFP4 is ~60–80GB. If the cache is cold and Hugging Face is slow, the engine sits at “Starting to load model…” for a long time. Mount a pre-downloaded model directory and check docker logs -f for network progress.

3. OOM during weight load. 35B NVFP4 should fit in 128GB UMA, but if you also allocate KV cache aggressively it can OOM silently. Try --max-model-len 8192 --gpu-memory-utilization 0.85 first.

4. --trust-remote-code + custom model architecture can crash on unsupported ops. Use the exact model variant recommended in the build.nvidia.com recipe.

Can you confirm which vLLM image tag you are running and whether the container logs ever show Application startup complete? That single line tells us whether the failure is startup vs. runtime.

6

Thank you for sharing your experience, my problem has been solved.