How to run NVFP4/DeepSeek-R1-0528-Qwen3-8B-FP4 using eugr/spark-vllm-docker

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

I build a vLLM image using eugr/spark-vllm-docker and I want to serve the NVFP4/DeepSeek-R1-0528-Qwen3-8B-FP4 on a single DGX Spark GB10.

I launched the container using the following command:

docker run -it \
  --runtime=nvidia \
  -e DISPLAY \
  -e QT_GRAPHICSSYSTEM=native \
  -e QT_X11_NO_MITSHM=1 \
  -e VLLM_USE_FLASHINFER_MOE_FP4=1 \
  -e GPU_MEMORY_UTIL=0.90 \
  -e PORT=8888 \
  -p 127.0.0.1:8888:8888 \
  -v /dev/shm:/dev/shm \
  -v /etc/localtime:/etc/localtime:ro \
  -v /tmp/.X11-unix:/tmp/.X11-unix \
  -v /var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket:ro \
  -v $HOME/.cache/huggingface:/root/.cache/huggingface \
  --rm \
  --name vllm-node-deepseek-r1-0528-qwen3-8b-fp4 \
  vllm-node:latest bash

and launch vllm with the following command:

export VLLM_USE_FLASHINFER_MOE_FP4=1

vllm serve nvfp4/deepseek-r1-0528-qwen3-8b-fp4 \
    --tensor-parallel-size 1 \
    --max-model-len 16384 \
    --gpu-memory-utilization 0.7 \
    --reasoning-parser deepseek_r1

However, I get the following error:


(EngineCore_DP0 pid=830) INFO 03-10 13:10:29 [core.py:103] Initializing a V1 LLM engine (v0.17.0rc1.dev168+gdc6b57846.d20260309) with config: model='nvfp4/deepseek-r1-0528-qwen3-8b-fp4', speculative_config=None, tokenizer='nvfp4/deepseek-r1-0528-qwen3-8b-fp4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=16384, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='deepseek_r1', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=nvfp4/deepseek-r1-0528-qwen3-8b-fp4, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': True, 'static_all_moe_layers': []}


(EngineCore_DP0 pid=830) ERROR 03-10 13:10:42 [core.py:1111]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2.py", line 517, in load_weights
(EngineCore_DP0 pid=830) ERROR 03-10 13:10:42 [core.py:1111]     weight_loader(param, loaded_weight)
(EngineCore_DP0 pid=830) ERROR 03-10 13:10:42 [core.py:1111]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/linear.py", line 1496, in weight_loader_v2
(EngineCore_DP0 pid=830) ERROR 03-10 13:10:42 [core.py:1111]     param.load_row_parallel_weight(loaded_weight=loaded_weight)
(EngineCore_DP0 pid=830) ERROR 03-10 13:10:42 [core.py:1111]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/parameter.py", line 222, in load_row_parallel_weight
(EngineCore_DP0 pid=830) ERROR 03-10 13:10:42 [core.py:1111]     loaded_weight = loaded_weight.narrow(
(EngineCore_DP0 pid=830) ERROR 03-10 13:10:42 [core.py:1111]                     ^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=830) ERROR 03-10 13:10:42 [core.py:1111] RuntimeError: start (0) + length (12288) exceeds dimension size (6144).

Q01: What can I do to resolve this error?

3

I cannot imagine why you would want to run that specific model (old, tiny) on the Spark, but maybe try setting --quantization modelopt_fp4 and see if that helps. It kind of looks like it isn’t detecting the right quantization… but I would recommend researching newer, more useful models to run instead. GPT-OSS-120B is a safe bet. Qwen3.5-122B-A10B is potentially even better.

4

Yes, there is no reason to use this particular model in 2026…

5

I already tried Qwen3-Next and Qwen3-VL and got them to run successfully on a single DGX Spark.

I want to do A/B model testing for math reasoning tasks.

I want to run DeepSeek-R1 and DeepSeek-V3.2 variants on the DGX Spark, so I can compare the outputs of both the DeepSeek and Qwen3 models for math reasoning tasks.

I have more DGX Spark units to run large models, but before I run it on a cluster, I want to make sure I can serve a small model first, before scaling the cluster and running a larger model.

Q02: Do you know which DeepSeek-R1 and DeepSeek-v3.2 is known to run with a DGX Spark? I see several models from NVFP4 and nvidia.

6

This is the NVFP4/DeepSeek-R1-0528-Qwen3-8B-FP4/config.json for the model

{
    "architectures": [
        "Qwen3ForCausalLM"
    ],
    "attention_bias": false,
    "attention_dropout": 0.0,
    "bos_token_id": 151643,
    "eos_token_id": 151645,
    "head_dim": 128,
    "hidden_act": "silu",
    "hidden_size": 4096,
    "initializer_range": 0.02,
    "intermediate_size": 12288,
    "max_position_embeddings": 131072,
    "max_window_layers": 36,
    "model_type": "qwen3",
    "num_attention_heads": 32,
    "num_hidden_layers": 36,
    "num_key_value_heads": 8,
    "rms_norm_eps": 1e-06,
    "rope_scaling": {
        "attn_factor": 0.8782488562869419,
        "factor": 4.0,
        "original_max_position_embeddings": 32768,
        "rope_type": "yarn"
    },
    "rope_theta": 1000000,
    "sliding_window": null,
    "tie_word_embeddings": false,
    "torch_dtype": "bfloat16",
    "transformers_version": "4.51.3",
    "use_cache": true,
    "use_sliding_window": false,
    "vocab_size": 151936,
    "quantization_config": {
        "ignore": [
            "lm_head"
        ],
        "quant_algo": "NVFP4",
        "kv_cache_scheme": "FP8",
        "producer": {
            "name": "modelopt",
            "version": "0.31.0"
        }
    }
}

It indicates that the model has been optimized for nvfp4

I tried -quantization nvfp4, etc, this type of error.

(APIServer pid=999) Value error, Quantization method specified in the model config (None) does not match the quantization method specified in the quantization argument (nvfp4). [type=value_error, input_value=ArgsKwargs((), {‘model’: …rocessor_plugin’: None}), input_type=ArgsKwargs]
(APIServer pid=999)
7

The model you want to run is not Deepseek-R1 anyway, it’s a distilled version, IOW Qwen3 model post-trained using Deepseek-R1. Real Deepseek models are all too large to run on a single Spark (or even on a Spark cluster unless you have 8 nodes).

The current Qwen models are Qwen3.5 - you may want to try those. There are recipes in the repository for those.

Also, I suggest using ./launch-cluster.sh to run the containers from this repo, even if you run it on a single node (just use --solo) switch. ./run-recipe.sh further simplifies running models and adds necessary mods/fixes needed for some models.

8

Which Qwen3.5-122b model would you recommend I use?

9

Intel/Qwen3.5-122B-A10B-int4-AutoRound.
On the cluster you can use native FP8 version from Qwen (there is a recipe for that too).

To use with spark-vllm-docker, you can just simply run:

./run-recipe.sh qwen3.5-122b-int4-autoround --solo
10

While using the Intel/Qwen3.5‑122B‑A10B‑int4‑AutoRound model based on @eugr’s recipe, the following message keeps appearing. What does this message mean?
(EngineCore_DP0 pid=343) /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fla/ops/utils.py:113: UserWarning: Input tensor shape suggests potential format mismatch: seq_len (48) < num_heads (64). This may indicate the inputs were passed in head-first format [B, H, T, ...] when head_first=False was specified. Please verify your input tensor format matches the expected shape [B, T, H, ...].

11

Ignore it, that appears for most new qwen models.