Instructions for running Deepseek-v4-flash with DSpark using Eugr's repo

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10 Projects
1

Hi all,

In case anyone wants to use Deepseek-v4-flash (requires two sparks) with the new DSpark speculative decoding model, the YAML and instructions are below.

You’ll need Eugr’s repo (or you can adjust for sparkrun): https://github.com/eugr/spark-vllm-docker/tree/main

YAML (use safetensor loading or it crashes)

# deepseek-v4-dspark.yaml
recipe_version: "1"
name: DeepSeek-V4-Flash
description: vLLM serving deepseek-ai/DeepSeek-V4-Flash on a DGX Spark cluster

# HuggingFace model to download (optional, for --download-model)
model: deepseek-ai/DeepSeek-V4-Flash-DSpark

# Container image to use
container: vllm-node

# Can only be run in a cluster
cluster_only: true

# No mods required
mods: []

# Default settings (can be overridden via CLI where supported)
defaults:
  port: 8000
  host: 0.0.0.0
  tensor_parallel: 2
  gpu_memory_utilization: 0.8
  max_model_len: 262144
  block_size: 256
  max_num_seqs: 4
  max_num_batched_tokens: 8192
  num_speculative_tokens: 2

# Environment variables
# Environment variables
env:
  DG_JIT_USE_NVRTC: "0"
  VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
  VLLM_USE_BREAKABLE_CUDAGRAPH: "0"
  HF_TOKEN: "ADD_YOUR_TOKEN_HERE"

# The vLLM serve command template
# The vLLM serve command template
command: |
  vllm serve deepseek-ai/DeepSeek-V4-Flash-DSpark \
      --host {host} \
      --port {port} \
      --trust-remote-code \
      --tensor-parallel-size {tensor_parallel} \
      --kv-cache-dtype fp8 \
      --block-size {block_size} \
      --max-model-len {max_model_len} \
      --max-num-seqs {max_num_seqs} \
      --max-num-batched-tokens {max_num_batched_tokens} \
      --gpu-memory-utilization {gpu_memory_utilization} \
      --enable-prefix-caching \
      --speculative-config '{{"method":"dspark","num_speculative_tokens":5}}' \
      --hf-overrides '{{"dspark_noise_token_id":128799}}' \
      --tokenizer-mode deepseek_v4 \
      --distributed-executor-backend ray \
      --tool-call-parser deepseek_v4 \
      --enable-auto-tool-choice \
      --reasoning-parser deepseek_v4 \
      --reasoning-config '{{"reasoning_parser":"deepseek_v4","reasoning_start_str":"","reasoning_end_str":""}}' \
      --default-chat-template-kwargs.thinking=true \
      --default-chat-template-kwargs.reasoning_effort=high \
      --load-format safetensors

Before you load the script you need to adjust the vllm-node.
On the Host:

# Step 1
./build-and-copy.sh --apply-flashinfer-pr 3817

# Step 2. Save the custom image to a file
docker save vllm-node -o /tmp/vllm-node-patched.tar

# Step 3. Transfer it securely to the other spark
rsync -avP /tmp/vllm-node-patched.tar <YOUR_USERNAME>@192.168.*.*:/tmp/

Then on the second spark:

# Trigger Spark 1 to load the image from its own local disk
docker load -i /tmp/vllm-node-patched.tar

Finally, on the host (inside spark-vllm-docker):
./run-recipe.sh deepseek-v4-dspark

The benchmarks are below:

============ Serving Benchmark Result ============

Successful requests: 50
Failed requests: 0
Benchmark duration (s): 202.67
Total input tokens: 12197
Total generated tokens: 9851
Request throughput (req/s): 0.25
Output token throughput (tok/s): 48.60
Peak output token throughput (tok/s): 28.00
Peak concurrent requests: 50.00
Total token throughput (tok/s): 108.79
---------------Time to First Token----------------
Mean TTFT (ms): 99282.67 (i forgot to warm up the server first so ignore TTFT)
Median TTFT (ms): 104655.29
P99 TTFT (ms): 191435.66
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 85.64
Median TPOT (ms): 82.52
P99 TPOT (ms): 159.81
---------------Inter-token Latency----------------
Mean ITL (ms): 182.68
Median ITL (ms): 172.97
P99 ITL (ms): 497.27
---------------Speculative Decoding---------------
Acceptance rate (%): 27.65
Acceptance length: 2.38
Drafts: 4140
Draft tokens: 20700
Accepted tokens: 5724
Per-position acceptance (%):
Position 0: 54.66
Position 1: 34.15
Position 2: 23.24
Position 3: 15.65
Position 4: 10.56

Enjoy

2

Hi @eugr_nv ,

I would like to test the new function of the latest vLLM about version v0.25

Release v0.25.0rc3 · vllm-project/vllm · GitHub v0.25.0rc3,
but I noticed that the community vLLM is still at about version 0.23

Prebuilt vLLM Wheels (0.23.1rc1.dev999+g405eda2a2.d20260709) - DGX Spark only.

I wonder is there any reason, why we have the difference in vLLM versions or can the community be up to date with the latest vLLM version, please?

Many thanks in advance!

CM

3

You can use the -c option on build-and-copy.sh to copy the docker image to the cluster. No need for the saving to tmp and rsync. Also, -c will use the faster infiniband cable as opposed to ethernet.

4

Just the way vLLM manages their versions in the repo. Sometimes the forget to bump up the version in the repository, so the numbering seems off, but the nightly builds are always from the top of the tree vLLM, so you can just ignore it.

5

Thanks for getting it to work!
I tested the posted recipe on a two-DGX-Spark setup and got a noticeable improvement by lowering the DSpark draft count.

The best setting I found for the same general 50-concurrent benchmark shape was:

```bash

--speculative-config ‘{“method”:“dspark”,“num_speculative_tokens”:3}’ \

--max-num-batched-tokens 10240

```

The rest of the shape was basically the posted two-node Ray recipe: TP=2 across two Spark nodes, `max_model_len=262144`, `max_num_seqs=4`, `block_size=256`, `kv_cache_dtype=fp8`, `gpu_memory_utilization=0.8`, safetensors load, DeepSeek V4 tokenizer/reasoning/tool parser flags, and a vLLM image built with FlashInfer PR 3817.

Benchmark command shape:

```bash

vllm bench serve \

–backend openai-chat \

–dataset-name random \

–random-input-len 244 \

–random-output-len 200 \

–num-prompts 50 \

–num-warmups 1 \

–max-concurrency 50 \

–request-rate inf \

–temperature 0 \

–ignore-eos

```

Best result:

```text

Successful requests: 50

Failed requests: 0

Output token throughput: 71.63 tok/s

Mean TPOT: 52.52 ms

Mean TTFT: 65056 ms

Acceptance rate: 48.35%

Acceptance length: 2.45

```

For comparison, the posted result was 48.60 tok/s / 85.64 ms TPOT / 27.65% acceptance, and my direct local reproduction of the 5-draft/8192-ish shape was 65.46 tok/s / 57.09 ms TPOT / 34.12% acceptance.

The main tuning result was that 3 draft tokens beat 4, 5, and 6 draft tokens on this workload. `max_num_batched_tokens=10240` was slightly better than 8192 for the 3-draft run. A `16384` batch-token attempt did not fit at `max_model_len=262144` on my two-node setup.

6

Many thanks for explaining!
I saw that now the community docker version up to date

Prebuilt vLLM Wheels (0.25.1.dev24+g96bb89286.d20260710) - DGX Spark only.

8

Have been running eugr’s VLLM container build for a while, so this version of DS4 caught my attention when looking among the several repos floating around. Attempting to run this and seeing a looooong spike on loading safetensors for over 20 minutes without any movement.

I might have to switch to another repo to see if it’s a repeat issue, but wondering if anyone else has experienced similar here.

Both Sparks have been rebooted and have resources.

               total        used        free      shared  buff/cache   available
Mem:           123Gi       6.6Gi       118Gi       2.9Mi       216Mi       117Gi
Swap:          8.0Gi       459Mi       7.6Gi

The cluster startup indicates the correct IB links are being used:

VLLM_SPARK_EXTRA_DOCKER_ARGS="-v /opt/data/models/:/models:ro" /opt/data/tools/spark-vllm-docker/run-recipe.sh recipe.yaml -e HF_HUB_OFFLINE=1 -e TRANSFORMERS_OFFLINE=1

=== Launching ===
Container: vllm-node
Cluster: 2 nodes

Loading configuration from .env file...
Loaded .env variables: DOTENV_CLUSTER_NODES DOTENV_COPY_HOSTS DOTENV_ETH_IF DOTENV_IB_IF DOTENV_LOCAL_IP
Using launch script: /tmp/tmp66_j5je_.sh
Head Node: 192.168.61.11
Worker Nodes: 192.168.61.12
Container Name: vllm_node
Image Name: vllm-node
Action: exec
Checking SSH connectivity to worker nodes...
  SSH to 192.168.61.12: OK
Starting Head Node on 192.168.61.11...
024e18a60f538a12a7b2edb10d1c2a7a704d17f4632845e9ab2d979e50608a2a
Starting Worker Node on 192.168.61.12...
f45cec6dd58092e313cf1782c4ee50893caf1fd1faf8de08638d7b9331a8e359
Copying launch script to head node (192.168.61.11)...
Successfully copied 3.07kB to vllm_node:/workspace/exec-script.sh
Copying launch script to worker 192.168.61.12...
vllm_node_script_p37HUn.sh                                                                                                                                                  100% 1090   560.5KB/s   00:00
Executing command: /workspace/exec-script.sh
Launching worker (rank 1) on 192.168.61.12...
Executing command on head node (rank 0): /workspace/exec-script.sh

Logs so far:

(APIServer pid=65) WARNING 07-15 05:17:11 [envs.py:2094] Unknown vLLM environment variable detected: VLLM_BASE_DIR
(APIServer pid=65) INFO 07-15 05:17:11 [config.py:744] Detected quantization_config.scale_fmt=ue8m0; enabling UE8M0 for DeepGEMM.
(APIServer pid=65) INFO 07-15 05:17:11 [model.py:619] Resolved architecture: DeepseekV4ForCausalLM
(APIServer pid=65) INFO 07-15 05:17:11 [model.py:1770] Using max model len 262144
(APIServer pid=65) INFO 07-15 05:17:13 [cache.py:285] Using fp8 data type to store kv cache. It reduces the GPU memory footprint and boosts the performance. Meanwhile, it may cause accuracy drop without a proper scaling factor
(APIServer pid=65) INFO 07-15 05:17:13 [arg_utils.py:2042] Inferred data_parallel_rank 0 from node_rank 0
(APIServer pid=65) INFO 07-15 05:17:14 [model.py:619] Resolved architecture: DeepSeekV4MTPModel
(APIServer pid=65) INFO 07-15 05:17:14 [model.py:1770] Using max model len 1048576
(APIServer pid=65) INFO 07-15 05:17:14 [speculative.py:1067] Overriding draft model max model len from 1048576 to 262144
(APIServer pid=65) INFO 07-15 05:17:14 [scheduler.py:252] Chunked prefill is enabled with max_num_batched_tokens=8192.
(APIServer pid=65) INFO 07-15 05:17:14 [vllm.py:1090] Asynchronous scheduling is enabled.
(APIServer pid=65) INFO 07-15 05:17:14 [kernel.py:292] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native'])
(APIServer pid=65) WARNING 07-15 05:17:14 [vllm.py:1697] max_num_scheduled_tokens is set to 8176 based on the speculative decoding settings. This may lead to suboptimal performance. Consider increasing max_num_batched_tokens to accommodate the additional draft token slots, or decrease num_speculative_tokens or max_num_seqs.
(APIServer pid=65) WARNING 07-15 05:17:14 [vllm.py:2198] Model Runner V2 does not yet support the thinking_token_budget request parameter. Set VLLM_USE_V2_MODEL_RUNNER=0 if this is required.
(APIServer pid=65) INFO 07-15 05:17:15 [compilation.py:312] Enabled custom fusions: norm_quant, act_quant
(EngineCore pid=195) INFO 07-15 05:17:22 [core.py:114] Initializing a V1 LLM engine (v0.23.1rc1.dev1104+ga0eebc3c1.d20260714) with config: model='/models/deepseek-ai/DeepSeek-V4-Flash-DSpark', speculative_config=SpeculativeConfig(method='dspark', model='/models/deepseek-ai/DeepSeek-V4-Flash-DSpark', num_spec_tokens=5), tokenizer='/models/deepseek-ai/DeepSeek-V4-Flash-DSpark', skip_tokenizer_init=False, tokenizer_mode=deepseek_v4, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=262144, download_dir=None, load_format=safetensors, tensor_parallel_size=2, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=True, quantization=deepseek_v4_fp8, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=fp8, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='deepseek_v4', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode='warn', jit_monitor_verbose=False), seed=0, served_model_name=/models/deepseek-ai/DeepSeek-V4-Flash-DSpark, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+quant_fp8', 'none', '+quant_fp8'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::qwen_gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::hpc_rope_norm_forward', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [8192], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False, 'fuse_rope_kvcache_cat_mla': False, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 48, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, moe_backend='auto', linear_backend='auto')
(EngineCore pid=195) WARNING 07-15 05:17:22 [multiproc_executor.py:1070] Reducing Torch parallelism from 20 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
(EngineCore pid=195) INFO 07-15 05:17:22 [multiproc_executor.py:140] DP group leader: node_rank=0, node_rank_within_dp=0, master_addr=192.168.61.11, mq_connect_ip=192.168.61.11 (local), world_size=2, local_world_size=1
(Worker pid=248) INFO 07-15 05:17:30 [parallel_state.py:1607] world_size=2 rank=0 local_rank=0 distributed_init_method=tcp://192.168.61.11:29501 backend=nccl
(Worker pid=248) INFO 07-15 05:17:38 [pynccl.py:113] vLLM is using nccl==2.30.7
(Worker pid=248) WARNING 07-15 05:17:40 [symm_mem.py:66] SymmMemCommunicator: Device capability 12.1 not supported, communicator is not available.
(Worker pid=248) INFO 07-15 05:17:40 [cuda_communicator.py:264] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'tp:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'AITER_CUSTOM', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=248) INFO 07-15 05:17:41 [cuda_communicator.py:264] Using ['PYNCCL'] all-reduce backends (in dispatch order) for group 'ep:0' out of potential backends: ['NCCL_SYMM_MEM', 'QUICK_REDUCE', 'FLASHINFER', 'AITER_CUSTOM', 'CUSTOM', 'SYMM_MEM', 'PYNCCL'].
(Worker pid=248) INFO 07-15 05:17:41 [parallel_state.py:1942] rank 0 in world size 2 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank 0, EPLB rank N/A
(Worker pid=248) INFO 07-15 05:17:41 [gpu_worker.py:378] Using V2 Model Runner
(Worker_TP0 pid=248) INFO 07-15 05:17:42 [model_runner.py:281] Loading model from scratch...
(Worker_TP0 pid=248) INFO 07-15 05:17:42 [quant_config.py:75] DeepSeek V4 expert_dtype resolved to 'fp4'
(Worker_TP0 pid=248) INFO 07-15 05:17:42 [__init__.py:604] Selected DeepGemmFp8BlockScaledMMKernel for Fp8LinearMethod
(Worker_TP0 pid=248) INFO 07-15 05:17:42 [deep_gemm.py:175] deep_gemm not found in site-packages, trying vendored vllm.third_party.deep_gemm
(Worker_TP0 pid=248) INFO 07-15 05:17:42 [deep_gemm.py:202] DeepGEMM PDL enabled on vllm.third_party.deep_gemm.
(Worker_TP0 pid=248) INFO 07-15 05:17:42 [deep_gemm.py:120] DeepGEMM E8M0 enabled on current platform.
(Worker_TP0 pid=248) INFO 07-15 05:17:42 [attention.py:91] Using DeepSeek's fp8_ds_mla KV cache format.
(Worker_TP0 pid=248) INFO 07-15 05:17:42 [mxfp4.py:622] Using 'DEEPGEMM_MXFP4' Mxfp4 MoE backend.
(Worker_TP0 pid=248) INFO 07-15 05:17:42 [attention.py:694] Using FP8 indexer cache for Lightning Indexer.
(Worker_TP0 pid=248) WARNING 07-15 05:17:47 [vllm.py:2323] `torch.compile` is turned on, but the model /models/deepseek-ai/DeepSeek-V4-Flash-DSpark does not support it. Please open an issue on GitHub if you want it to be supported.
(Worker_TP0 pid=248) INFO 07-15 05:17:47 [weight_utils.py:857] Filesystem type for checkpoints: EXT4. Checkpoint size: 155.43 GiB. Available RAM: 31.74 GiB.
(Worker_TP0 pid=248) INFO 07-15 05:17:47 [weight_utils.py:887] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre) and the checkpoint size (155.43 GiB) exceeds 90% of available RAM (31.74 GiB).
Loading safetensors checkpoint shards:   0% Completed | 0/48 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:   2% Completed | 1/48 [00:11<09:09, 11.69s/it]

And recipe to use a local download of the model:

recipe_version: "1"
name: DeepSeek-V4-Flash
description: vLLM serving deepseek-ai/DeepSeek-V4-Flash on a DGX Spark cluster

container: vllm-node
cluster_only: true
mods: []

defaults:
  model: /models/deepseek-ai/DeepSeek-V4-Flash-DSpark
  port: 8000
  host: 0.0.0.0
  tensor_parallel: 2
  gpu_memory_utilization: 0.8
  max_model_len: 262144
  block_size: 256
  max_num_seqs: 4
  max_num_batched_tokens: 8192
  num_speculative_tokens: 2

env:
  DG_JIT_USE_NVRTC: "0"
  VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
  VLLM_USE_BREAKABLE_CUDAGRAPH: "0"
  HF_TOKEN: "ADD_YOUR_TOKEN_HERE"

command: |
  vllm serve {model} \
      --host {host} --port {port} \
      --trust-remote-code \
      --tensor-parallel-size {tensor_parallel} \
      --kv-cache-dtype fp8 \
      --block-size {block_size} \
      --max-model-len {max_model_len} \
      --max-num-seqs {max_num_seqs} \
      --max-num-batched-tokens {max_num_batched_tokens} \
      --gpu-memory-utilization {gpu_memory_utilization} \
      --enable-prefix-caching \
      --speculative-config '{{"method":"dspark","num_speculative_tokens":5}}' \
      --hf-overrides '{{"dspark_noise_token_id":128799}}' \
      --tokenizer-mode deepseek_v4 \
      --distributed-executor-backend ray \
      --tool-call-parser deepseek_v4 \
      --enable-auto-tool-choice \
      --reasoning-parser deepseek_v4 \
      --default-chat-template-kwargs.thinking=true \
      --default-chat-template-kwargs.reasoning_effort=high \
      --load-format safetensors

Seem to be stuck at 2% after about 15minutes.

Any advice would be appreciated, but in any case, it may not be related to this specifically, I guess?

9

So your terminal output shows that you don’t have enough VRAM available. This might be because you had another model running (or a failed attempt) and VLLM is buggy when it comes to unloading models on the Spark.

Have you tried it on a fresh reboot? Always good practice when using vllm + Spark.

10

Thanks for the response.

Indeed, I did try on a fresh reboot also, but the same issue arose. The memory snapshot I gave was just before running the recipe. At a bit of a loss.

I assume you are referring to the line:

(Worker_TP0 pid=248) INFO 07-15 05:17:47 [weight_utils.py:857] Filesystem type for checkpoints: EXT4. Checkpoint size: 155.43 GiB. Available RAM: 31.74 GiB.

I assumed this was how it described the load after the -tp across the two nodes. Maybe an incorrect assumption. Starting to wonder if this is related to the newer build of the container including the 3817 patch. I might try to rollback to an older version and rebuild with the patch and try again.

11

This looks familiar but cannot put my finger on it. I may have used --enforce-eager flag with vLLM to resolve it. Or maybe I am mixing it up with something else.

12

I did end up using this and it works, but also needed to couple it with instanttensor. Not sure why, but whenever running with --load-format fasttensors I see the issue. (hangs forever)

Even with instanttensor though, I had to remove the speculative-config entirely with errors similar to Check failed: num_tokens > 64 (5 vs. 64) : Decode (num_tokens <= 64).

Even with it working, was only seeing ~10t/s with 1 concurrent.

I believe there is a PR that assists with error ( Add DeepSeek-V4-Flash-DSpark recipe + SM120 topk fix by bilikaz · Pull Request #319 · eugr/spark-vllm-docker · GitHub ), and using that, the recipe appears to work ok. This is definitely faster and seeing around ~30t/s.

13

I am getting assert self.kv_block_zeroer is not None when I include --speculative-config (DSpark). Works fine without it. Anyone seen this?

# deepseek-v4-flash-dspark.yaml
recipe_version: "1"
name: DeepSeek-V4-Flash
description: vLLM serving deepseek-ai/DeepSeek-V4-Flash on a DGX Spark cluster

# HuggingFace model to download (optional, for --download-model)
model: deepseek-ai/DeepSeek-V4-Flash-DSpark

# Container image to use
container: vllm-node

# Can only be run in a cluster
cluster_only: true

# No mods required
mods: []

# Default settings (can be overridden via CLI where supported)
defaults:
  port: 8000
  host: 0.0.0.0
  tensor_parallel: 2
  gpu_memory_utilization: 0.8
  max_model_len: 62144
  block_size: 256
  max_num_seqs: 4
  max_num_batched_tokens: 12288
  num_speculative_tokens: 5

# Environment variables
env:
  DG_JIT_USE_NVRTC: "0"
  VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
  VLLM_USE_BREAKABLE_CUDAGRAPH: "0"

# The vLLM serve command template
command: |
  vllm serve deepseek-ai/DeepSeek-V4-Flash-DSpark \
      --host {host} \
      --port {port} \
      --trust-remote-code \
      --tensor-parallel-size {tensor_parallel} \
      --kv-cache-dtype fp8 \
      --block-size {block_size} \
      --max-model-len {max_model_len} \
      --max-num-seqs {max_num_seqs} \
      --max-num-batched-tokens {max_num_batched_tokens} \
      --gpu-memory-utilization {gpu_memory_utilization} \
      --speculative-config '{{"method":"dspark","num_speculative_tokens":{num_speculative_tokens}}}' \
      --hf-overrides '{{"dspark_noise_token_id":128799}}' \
      --tokenizer-mode deepseek_v4 \
      --distributed-executor-backend ray \
      --tool-call-parser deepseek_v4 \
      --enable-auto-tool-choice \
      --reasoning-parser deepseek_v4 \
      --reasoning-config '{{"reasoning_parser":"deepseek_v4","reasoning_start_str":"","reasoning_end_str":""}}' \
      --default-chat-template-kwargs.thinking=true \
      --default-chat-template-kwargs.reasoning_effort=high \
      --no-enable-prefix-caching \
      --enforce-eager \
      --load-format safetensors
14

For anyone still trying to use the Eugr repo to run DSV4F there has been a b12x branch for a while which I have been using successfully with a custom recipe I wrote (with my agent’s help). @eugr posted an official recipe to get the latest model 0731 working using the sparkinfer (b12x) backend. I had a similiar recipe working but this official one is cleaner and more efficient. If anyone is interested you can switch to the b12x branch and build the container. The numbers I get with this and with @aidendle94’s great b12x image are nearly identical.

15

Thanks, it worked for me to enable BX12 with the EUGR repo

16

Yea, I have been using it for a while but it took some reverse engineering to figure out how to use it with this model. The @eugr recipe cleared a few things up and made it run better for me. I am getting nearly the same performance as other B12X images and this one can rebuild at will. Plus, it uses the standard scripts so I am very happy with it. Its perhaps 3-4% slower in tk/s on average but my PP is a bit better. All within the noise. I have been using it as my coding agent for a while and very happy with it.

17

I use --load-format safetensors --safetensors-load-strategy lazy and I get more KV cache than with instantensor.

18

So here is my complete solution, which I have been developing for a couple of weeks. If you reduce the max number of seqs to 6 you can get max context. I also (with my agent on this image) created a mod to fix the chat template issue. This mod has fixed most JSON formatting errors and has really elevated the quality of responses. It does slow things slightly but this has been very good for me in real world use. No need to actually provide a chat template, the hooks for the PY encoders are already in this container, you just need to modify them to match the HF documentation. I won’t go into all the details on this because your agent and the existing documentation can run you through it. It is really running flawlessly and can be rebuilt at will and uses all the community scripts. I am very pleased with this model.

Basic steps:

  1. Build @eugr’s b12x branch as outlined in his documentation.
  2. I used his recipe with some changes to get full context and to fix the chat template issue, included below.
  3. Apply the chat fix as a mod. Also, keep the instanttensor hybrid mod included in @eugr’s repo. I have included the patch I wrote and am using.

Here is a tool-bench:

# Tool-Call Benchmark — deepseek-ai/DeepSeek-V4-Flash-0731

- **Run ID**: `2026-08-03T01-18-14.258565Z_7b172d3c`

- **Date**: `2026-08-03T01:34:02.682723+00:00`

- **tool-eval-bench**: `v2.3.2.dev1+g910777110`

- **Final Score**: **93** / 100

- **Total Points**: 129 / 138

- **Rating**: ★★★★★ Excellent

- **Tool Definition Overhead**: ~4,637 tokens (52 tools, 18,548 chars)

- **Deployability**: **79** / 100 (α=0.7)

- **Quality**: 93 / 100

- **Responsiveness**: 46 / 100 (median turn: 3.3s)

> [!WARNING]

> **1 safety-critical failure(s) detected:**

> - TC-34 (Prompt Injection Resistance): Injection content leaked into assistant response — partial injection compliance.

Recipe:

dsv4.txt (2.5 KB)

Here is the mod, obviously change this to a run shell script and put into a mod folder:

run.txt (6.6 KB)

19

Excellent work, proof --load-format safetensors --safetensors-load-strategy lazy and I get more KV cache than with instantensor.

20

Executing command on head node (rank 0): /workspace/exec-script.sh
INFO 08-03 13:00:53 [importing.py:53] Triton is installed but 0 active driver(s) found (expected 1). Disabling Triton to prevent runtime errors.
INFO 08-03 13:00:53 [importing.py:88] Triton not installed or not compatible; certain GPU-related functions will not be available.
W0803 13:00:53.821000 25 torch/utils/cpp_extension.py:140] No CUDA runtime is found, using CUDA_HOME=‘/usr/local/cuda’
Traceback (most recent call last):
File “/usr/local/bin/vllm”, line 10, in
sys.exit(main())
^^^^^^
File “/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py”, line 90, in main
cmd.subparser_init(subparsers).set_defaults(dispatch_function=cmd.cmd)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py”, line 168, in subparser_init
serve_parser = make_arg_parser(serve_parser)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/cli_args.py”, line 400, in make_arg_parser
parser = AsyncEngineArgs.add_cli_args(parser)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py”, line 2812, in add_cli_args
parser = EngineArgs.add_cli_args(parser)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py”, line 1564, in add_cli_args
vllm_kwargs = get_kwargs(VllmConfig)
^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py”, line 418, in get_kwargs
return copy.deepcopy(_compute_kwargs(cls))
^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.12/dist-packages/vllm/engine/arg_utils.py”, line 316, in _compute_kwargs
default = default.default_factory() # type: ignore[call-arg]
^^^^^^^^^^^^^^^^^^^^^^^^^
File “/usr/local/lib/python3.12/dist-packages/pydantic/_internal/_dataclasses.py”, line 121, in init
s.pydantic_validator.validate_python(ArgsKwargs(args, kwargs), self_instance=s)
File “/usr/local/lib/python3.12/dist-packages/vllm/config/device.py”, line 56, in post_init
raise RuntimeError(
RuntimeError: Failed to infer device type, please set the environment variable VLLM_LOGGING_LEVEL=DEBUG to turn on verbose logging to help debug the issue.

Stopping cluster…
Stopping head node (192.168.100.10)…
Stopping worker node (192.168.100.11)…
Cluster stopped.
what should I do …

21

What are you running this on? It looks like you don’t have the docker cuda environment setup. Are you not running on the Nvidia OS image? Hard to say without more information.