What kind of performance do you get with the spark-vllm-docker build using the b12x branch?
| model | test | t/s | peak t/s | ttfr (ms) | est_ppt (ms) | e2e_ttft (ms) |
|---|---|---|---|---|---|---|
| deepseek-ai/DeepSeek-V4-Flash-0731 | pp2048 | 1832.52 ± 2.46 | 1119.84 ± 1.50 | 1117.59 ± 1.50 | 1119.84 ± 1.50 | |
| deepseek-ai/DeepSeek-V4-Flash-0731 | tg32 | 37.57 ± 4.84 | 38.78 ± 4.99 |
Here is my last benchy run, it is about what I was getting with the Aiden container. I don’t really chase benchmarks around too much. In practical use with c 2-4 I typically see 55+ tk/s. This has been a very good setup for me and I am going to keep it. I can rebuild the image at will and I have full insight into what is in the image. Thanks to @eugr for his b12x branch.
I agree with the “what’s in the image” bit. I like the idea of being able to build my own images, too! Thanks for sharing!
I used your recipe and mod, but 55 t/s is still out of reach. Any additional tips or tricks?
I am not sure, how are you measuring it? You can see my benchy numbers above, pretty much in alignment with the Aiden container I was using. I know others have tweaked like crazy and can get better numbers on the benchmarks but for me this is a better solution. The 55 tk/s is when I am using the agent in a coding environment and measuring the tokens at the interface so it is not a benchmark number but a practical use number. This model with this container has been very good for me both in speed and result quality.
EDIT: b12x build is now supported in spark-vllm-docker main branch. Updated instructions:
# Clone the repo if you haven't done it yet; run `git pull` if you have it already
git clone http://github.com/eugr/spark-vllm-docker
cd spark-vllm-docker
# Pull from dockerhub and distribute across nodes (alternatively you can build from source by adding `--rebuild-vllm`
./build-and-copy.sh --exp-b12x -c
# Run the model
./run-recipe.sh recipes/deepseek-v4-flash-0731.yaml
BTW, this version supports 3x Sparks with TP=3!
To use, just add --tp 3 to the recipe!
Hi Eugr!
Your spark-vllm-docker is absolutely amazing!✨
Because of your project, I’ve been able to use my DGX Sparks with confidence, and I’ve already expanded from two systems to four. Honestly, without your work, I probably would have stopped at two.
The new DeepSeek-V4-Flash-0731 worked almost effortlessly with your setup, and it reminded me once again how much I appreciate everything you’ve done.
I’m actually considering buying four more DGX Sparks.
Thank you for everything, and I wish you all the best in your new role!
I’ve merged b12x branch into main, and b12x build is a part of the nightly CI pipeline now. I’ve updated instructions above.
Thank you @eugr_nv this is the best model I run so far.
Using TP=3 I get 2.8 million tokens kv cache.
Draft speed jumped from 75 to 88 tokens per second.
For none coding tasks I get 55 tokens per second accept throughput, this should be more with coding.
For easy tasks it could reach 68 tokens per second accepted and 160 drafted for single stream using 12 dspark length. So I think 5 is the best choice.
Memory utilization is 116 to 117 gb on all three nodes.
Excellent work!!
For anyone interested, I’m sharing benchmark results from my environment with TP=2, TP=3, and TP=4.
More importantly than the benchmark scores themselves, I want to highlight how stable everything has been. It has been running flawlessly.
Thanks to the well-designed spark-vllm-docker, I can focus on what I actually want to do instead of spending time fighting with complex NCCL issues or configuration.
Thank you, spark-vllm-docker and @eugr_nv!
Thanks for sharing your results. I have been testing this model with eugr/spark-vllm-docker yesterday and I, too, really appreciate this project and its contributors.
I noticed that you are using tool-eval-bench v2.2.0. Just a heads up that the scoring has been tightened up, especially in the latest 2.4.x release. I was getting consistently around 88 scores with v2.3.1, but consistently around 82 (or lower) with v2.4.1. It took me a while to finally took the issue to my agent to find out why - “the stricter scoring was intentional and documented in the change log.” I thought I did something wrong with the recipe because I was applying the reasoning effort mod, using different temperatures and top_p, etc, before I fixed my model settings to compare different versions of tool-evan-bench. It just didn’t occur to me that my tool-eval-bench update would result in this dramatic drop of the score. Just FYi.
I am not sure if the author is ready to announce the new release (like he did for v2.3.0) for adoption. There were still active development commits yesterday.
Hi all,
I built the spark-vllm-docker using the command: ./build-and-copy.sh -c --exp-b12x.
Then I ran ./run-recipe.sh recipes/deepseek-v4-flash-0731.yaml -c and got this error:
./run-recipe.sh recipes/deepseek-v4-flash-0731.yaml -c
Recipe: DeepSeek-V4-Flash-0731
vLLM serving deepseek-ai/DeepSeek-V4-Flash-0731 on a dual Sparks using B12X docker
Using cluster nodes from .env: 192.168.177.13, 192.168.177.14
=== Launching ===
Container: vllm-node-b12x
Mods: mods/instanttensor-hybrid-draft-loader
Cluster: 2 nodes
Loading configuration from .env file...
Loaded .env variables: DOTENV_CLUSTER_NODES DOTENV_COPY_HOSTS DOTENV_ETH_IF DOTENV_IB_IF DOTENV_LOCAL_IP
Using launch script: /tmp/tmpwoz2s9x1.sh
Head Node: 192.168.177.13
Worker Nodes: 192.168.177.14
Container Name: vllm_node
Image Name: vllm-node-b12x
Action: exec
Checking SSH connectivity to worker nodes...
SSH to 192.168.177.14: OK
Verifying Docker image consistency across cluster nodes...
[HEAD] 192.168.177.13: sha256:43c0a36c33c90593e355852e9babedb87d3e8a145f03ea29c5699fbc837d683a
[WORKER] 192.168.177.14: sha256:43c0a36c33c90593e355852e9babedb87d3e8a145f03ea29c5699fbc837d683a (match)
Docker image consistency check passed.
Starting Head Node on 192.168.177.13...
9cb1aaec858f4874b97f7afe6fcbddb33bea6456dc79c63c2560a372003c0c19
Starting Worker Node on 192.168.177.14...
f5556f12b8b8a266c2fa4fb169d21cdf9c2ff295655aadb4cc2bd8ac58286096
Applying modifications to cluster nodes...
Applying mod 'instanttensor-hybrid-draft-loader' to 192.168.177.13...
Copying directory content to container...
Successfully copied 13.8kB to vllm_node:/workspace/mods/instanttensor-hybrid-draft-loader/
Running patch script on 192.168.177.13...
=== InstantTensor hybrid speculative-draft loader mod ===
[instanttensor-hybrid-draft-loader] /usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/__init__.py is compatible.
[instanttensor-hybrid-draft-loader] Patched /usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/__init__.py.
[instanttensor-hybrid-draft-loader] /usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/__init__.py is already patched.
[instanttensor-hybrid-draft-loader] Enabled with INSTANTTENSOR_DRAFT_LOADER=auto.
=== OK: target loads stay on InstantTensor; selected drafts use lazy safetensors ===
Applying mod 'instanttensor-hybrid-draft-loader' to 192.168.177.14...
Copying mod package to 192.168.177.14:/tmp/vllm_mod_pkg_1785950336_13357...
patch_model_loader.py 100% 6170 8.5MB/s 00:00
README.md 100% 2322 2.5MB/s 00:00
run.sh 100% 1503 950.0KB/s 00:00
Copying directory content to container...
Running patch script on 192.168.177.14...
=== InstantTensor hybrid speculative-draft loader mod ===
[instanttensor-hybrid-draft-loader] /usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/__init__.py is compatible.
[instanttensor-hybrid-draft-loader] Patched /usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/__init__.py.
[instanttensor-hybrid-draft-loader] /usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/__init__.py is already patched.
[instanttensor-hybrid-draft-loader] Enabled with INSTANTTENSOR_DRAFT_LOADER=auto.
=== OK: target loads stay on InstantTensor; selected drafts use lazy safetensors ===
Copying launch script to head node (192.168.177.13)...
Successfully copied 3.58kB to vllm_node:/workspace/exec-script.sh
Copying launch script to worker 192.168.177.14...
vllm_node_script_2mjFGb.sh 100% 1707 1.7MB/s 00:00
Executing command: /workspace/exec-script.sh
Launching worker (rank 1) on 192.168.177.14...
Executing command on head node (rank 0): /workspace/exec-script.sh
usage: vllm serve [model_tag] [options]
vllm serve: error: argument --compilation-config/-cc: expected one argument
Stopping cluster...
Stopping head node (192.168.177.13)...
Stopping worker node (192.168.177.14)...
Any ideas?
I used version 2.5 today. I think it is harder
Here is my results for original recipe
I tried the mod for fixing the thinking, but it was not useful
Then I removed the mod and made the temperature=0.95 and top_p =0.40. This is my best results. I might need to tune the temperature and top.p
In general it is good. With last setting.
Check the recipe. The line with compilation config or the line above it could have syntax problem
FYI. Latest Deepseek 4 Flash 0731 is now available via sparkrun.
https://sparkrun.dev | https://spark-arena.com
sparkrun update to get the latest sparkrun
sparkrun run @official/deepseek-v4-flash-0731-b12x-dspark-vllm to run on 2 nodes.
Brought to you by the Spark Arena Team (@raphael.amorim @dbsci @eugr)
Chat with us here or on Discord (see websites for links).
It looks like the ‘-c’ you have at the end of run-recipe is being passed to vllm. Try removing the -c from your launch command.
Yes, I notice two things. First, put the -c on the build and copy script as the last arg. I have had issues with it by not doing that but not in a long while so that is probably not your issue but I do it out of good habit. Your real issue is the run script should be ./run-recipe.sh deepseek-v4-flash-0731 and thats it. The -c is being passed onto the vLLM command and is unknown.
Ace thanks! I’ve got a giant 30 day training run going on so I am well behind on these.
As a note for those who (like me) have found that doing a sudo update / upgrade randomly breaks scripts for Eugr’s or Sparkrun, try adding this line:
env:
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
It’s literally taken me months to work out such a simple thing which is rather embarrasing.
I’ve no idea why an update makes them trigger the timeout.
I try to resetup again but start 0731 vllm show below
template_kwargs’: {‘thinking’: True, ‘reasoning_effort’: ‘high’}, ‘enable_auto_tool_choice’: True, ‘tool_call_parser’: ‘deepseek_v4’, ‘host’: ‘0.0.0.0’, ‘model’: ‘deepseek-ai/DeepSeek-V4-Flash-0731’, ‘tokenizer_mode’: ‘deepseek_v4’, ‘trust_remote_code’: True, ‘max_model_len’: -1, ‘load_format’: ‘instanttensor’, ‘attention_backend’: ‘B12X_MLA_SPARSE’, ‘reasoning_parser’: ‘deepseek_v4’, ‘master_addr’: ‘192.168.100.10’, ‘nnodes’: 2, ‘tensor_parallel_size’: 2, ‘block_size’: 256, ‘gpu_memory_utilization’: 0.85, ‘kv_cache_dtype’: ‘fp8’, ‘enable_prefix_caching’: True, ‘max_num_batched_tokens’: 8192, ‘max_num_seqs’: 8, ‘max_cudagraph_capture_size’: 64, ‘moe_backend’: ‘b12x’, ‘linear_backend’: ‘b12x’, ‘speculative_config’: {‘method’: ‘dspark’, ‘num_speculative_tokens’: 5, ‘draft_sample_method’: ‘probabilistic’, ‘attention_backend’: ‘B12X_MLA_SPARSE’}, ‘compilation_config’: {‘mode’: None, ‘debug_dump_path’: None, ‘cache_dir’: ‘’, ‘compile_cache_save_format’: ‘binary’, ‘backend’: ‘inductor’, ‘custom_ops’: [‘all’], ‘ir_enable_torch_wrap’: None, ‘splitting_ops’: None, ‘compile_mm_encoder’: False, ‘cudagraph_mm_encoder’: False, ‘encoder_cudagraph_token_budgets’: , ‘encoder_cudagraph_max_vision_items_per_batch’: 0, ‘encoder_cudagraph_max_frames_per_batch’: None, ‘compile_sizes’: None, ‘compile_ranges_endpoints’: None, ‘inductor_compile_config’: {‘enable_auto_functionalized_v2’: False, ‘combo_kernels’: True, ‘benchmark_combo_kernel’: True}, ‘inductor_passes’: {}, ‘cudagraph_mode’: <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, ‘cudagraph_num_of_warmups’: 0, ‘cudagraph_capture_sizes’: None, ‘cudagraph_copy_inputs’: False, ‘cudagraph_specialize_lora’: True, ‘use_inductor_graph_partition’: None, ‘pass_config’: {}, ‘max_cudagraph_capture_size’: None, ‘dynamic_shapes_config’: {‘type’: <DynamicShapesType.BACKED: ‘backed’>, ‘evaluate_guards’: False, ‘assume_32_bit_indexing’: False}, ‘local_cache_dir’: None, ‘fast_moe_cold_start’: None, ‘static_all_moe_layers’: }, ‘reasoning_config’: ReasoningConfig(reasoning_parser=‘deepseek_v4’, reasoning_start_str=‘’, reasoning_end_str=‘’)}
do I need upgrade nv driver or something wrong…