DeepSeekV4-Flash hybrid quant, 1x DGX Spark: antirez's optimized 128 GB MLX recipe ported to vLLM for GB10

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10 Projects
1

Hi all, just got a 2-bit hybrid quantization of DeepSeek-V4-Flash that fits on a single DGX Spark
and runs end-to-end on vLLM with coherent multi-token output. To the best of
my knowledge it’s the most capable open-weights model running on a single
Spark today, and I’d love help validating that claim across more units and
workloads.

Why this works: the constraint parallel with antirez/ds4 on MLX

A 128 GB Apple Silicon Mac and a DGX Spark sit in nearly the same envelope:
~128 GB of unified memory, no NVLink/UCX out to anything bigger, and a
dispatched-MoE architecture that wants that whole budget for weights + KV
cache + workspace.

Salvatore Sanfilippo (antirez) had already done
deep work attacking exactly this problem from the Mac side. He made a full
custom inference engine specifically for this DSV4-Flash hybrid quant setup
with his newly released antirez/ds4 C+Metal
project. He optimized for max performance in 128GB unified memory with
accuracy close to the original DeepSeek release:

Component Format bpw
Routed experts: gate / up IQ2_XXS ~2.06
Routed experts: down Q2_K ~2.62
Dense linears + attention FP8 E4M3 (block-128, UE8M0 scales) 8
Embeddings, lm_head, norms BF16 16

Total: ~85 GiB on disk, ~110 GiB resident during serving. The constraints on
Spark map almost 1:1 onto antirez’s MLX target, so the recipe lifted across
cleanly; the actual work was plumbing it through vLLM and getting Triton
kernels on SM121 to match the Metal ones bit-for-bit.

What you get

A public HF checkpoint and a one-shot installer:

curl -sSL https://raw.githubusercontent.com/Entrpi/ds4-spark-vllm/main/install.sh | bash

The script verifies the host is GB10/SM121/aarch64 with ≥118 GiB RAM, downloads
the ~85 GiB checkpoint with SHA256SUMS verification, pulls the community
lmxxf/vllm-deepseek-v4-dgx-spark
base image (without which none of this would have shipped — huge thanks to
that maintainer), starts vllm serve on :8000 with all the
correctness-critical flags baked in, and runs a first-token smoke test.

After it lands, the endpoint speaks the OpenAI completions/chat APIs:

curl -s http://localhost:8000/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"dsv4","prompt":"The capital of France is","max_tokens":20,"temperature":0}'

Caveats

  • Validated on one Spark. Other GB10 units should behave identically;
    B100/B200 likely work but unverified.
  • Tok/s is very slow currently (< 2 t/s), no optimizations and debug code
    still in place. Next steps are optimization work.
  • Base image is aarch64-only.
  • First cold start takes >10 min for flashinfer autotune; subsequent starts
    are faster.
  • MIT licensed end-to-end, matching upstream DeepSeek.

Links

Happy to answer questions, take feedback, or accept correctness regressions
as GitHub issues. Especially interested if anyone runs into different
behavior on a different Spark unit or with non-trivial workloads.

I just finished validating first inference, so still yet to put polish on it.

2

All right, I’m game. This seems pretty cool! Bringing it down.

Will do throughput, tool calling, and general comparison benchmarks. I don’t have the ability to run a less heavy quantized model on 2x, but can compare with a variety of tasks.

3

What speeds and context are you getting ? thinking about trying it on 2 nodes

4

On to it!

5

There is an error in the log at the end of vLLM weight loading, however, it continued past this point and is now autotuning.

(EngineCore pid=210) Traceback (most recent call last):
(EngineCore pid=210)   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/deepseek_v4.py", line 1683, in load_weights
(EngineCore pid=210)     _idx = _json.loads((_Path(_ckpt) / 'model.safetensors.index.json').read_text())['weight_map']
(EngineCore pid=210)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=210)   File "/usr/lib/python3.12/pathlib.py", line 1029, in read_text
(EngineCore pid=210)     with self.open(mode='r', encoding=encoding, errors=errors) as f:
(EngineCore pid=210)          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=210)   File "/usr/lib/python3.12/pathlib.py", line 1015, in open
(EngineCore pid=210)     return io.open(self, mode, buffering, encoding, errors, newline)
(EngineCore pid=210)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=210) FileNotFoundError: [Errno 2] No such file or directory: '/models/deepseek-v4-flash-ds4-q2/model.safetensors.index.json'
(EngineCore pid=210) INFO 05-10 04:48:49 [default_loader.py:384] Loading weights took 151.08 seconds
(EngineCore pid=210) INFO 05-10 04:48:54 [gpu_model_runner.py:4879] Model loading took 81.4 GiB memory and 154.601538 seconds

Again this actually didn’t kill the container, as the follow-up messages show.

Right before autotuning we get the KV cache log, which seems odd

(EngineCore pid=210) INFO 05-10 05:04:10 [gpu_worker.py:440] Available KV cache memory: 23.09 GiB
(EngineCore pid=210) INFO 05-10 05:04:10 [kv_cache_utils.py:1713] GPU KV cache size: 18,860 tokens
(EngineCore pid=210) INFO 05-10 05:04:10 [kv_cache_utils.py:1718] Maximum concurrency for 16,384 tokens per request: 28.68x

23.09 GB for KV cache on this model only yields 18k tokens? And then for the 16k max length set as the default it seems to think we can hit 28x concurrency?

I would have hoped for more overhead in KV cache for that amount of space… maybe something is being calculated wrong by vLLM here.

6

It seems like there are some basic benchmarks here: GitHub - lmxxf/deepseek-v4-deployment-on-dgx-spark · GitHub - not quite usable yet (especially in English) but it is nice to see some progress.

I tried getting DSV4 Flash via spark-vllm-docker up and running last night but only reached about 7-8 t/s with the “minimal” PR that @jasl is working on right now: [New Model][Nvidia] Add SM12x support for DeepSeek V4 Flash with essential fixes by jasl · Pull Request #41834 · vllm-project/vllm · GitHub

7

I finally realize the critical impact is the NCCL, so many protections for CUDA graph safety is unnecessary, so I need to revise my code, to remove those protections, which would impact performance.

Could you explain about the usable? Do you mean the speed or the quality?

8

In my case the token generation/prefill was too slow and I experienced 1-2 crashes right away.

I keep on coming back to the PR and try to evaluate the status quo every now and then – I’m sure things will be in much better shape next time I’ll give it a try.

9

Fix landed: the FileNotFoundError joshua hit was not cosmetic

@joshua.dale.warner — that [Errno 2] No such file or directory: '/models/deepseek-v4-flash-ds4-q2/model.safetensors.index.json' was an in-container patch defaulting to the dev-time checkpoint path instead of the HF-canonical name (DeepSeek-V4-Flash-IQ2XXS-Q2K-FP8-120GB-target/). The patch was wrapped in try/except, so the container kept running, but the consequence is bigger than the surface error suggests: that patch is what runs the norm-loading-order fix — when it silently fails, vLLM falls back to default-initializing *_norm.weight to 1.0, which loads the model with identity RMS-norm. Inference still produces plausible-looking English, but with degraded quality compared to the validated build.

Fixed in 5c21f84. To pick it up:

~/ds4-spark-vllm/install.sh --uninstall
curl -sSL https://raw.githubusercontent.com/Entrpi/ds4-spark-vllm/main/install.sh | bash

(Idempotent — no re-download.) Confirmed on my Spark: complete_load now reports direct-loaded 189 params via remap rules, defaulted 86 scale/bias/norm params, final unloaded count: 0 instead of the silent failure.

10

Crash may disappear after you upgrade the NCCL version.

And my preview branch should work for 1k-8k input, covering writing, translation, summarize. You should get a reasonable response time.

I plan to have a rest, my superpower drained (I mean my Codex weekly quota exceeded 🤡)

12

Thanks, I rebuilt and can confirm that error is now gone.

Again it’s in the autotuning phase. vLLM seems to be even sillier with the KV cache calculation now - 17,488 total tokens but it thinks concurrency of 257 would be fine with 16k tokens per request.

(EngineCore pid=179) INFO 05-10 18:33:17 [gpu_worker.py:440] Available KV cache memory: 21.42 GiB
(EngineCore pid=179) INFO 05-10 18:33:17 [kv_cache_utils.py:1713] GPU KV cache size: 17,488 tokens
(EngineCore pid=179) INFO 05-10 18:33:17 [kv_cache_utils.py:1718] Maximum concurrency for 16,384 tokens per request: 257.22x
13

New error. Rebuilt container and after autotune, the engine core fails to initialize. Then it loops back to try again because it’s set to auto-restart.

Managed to capture this from the logs before serve.log got overwritten in a new startup (this is right after autotune ends):

(EngineCore pid=179) INFO 05-10 18:50:03 [core.py:306] init engine (profile, create kv cache, warmup model) took 1902.76 s
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136] EngineCore failed to start.
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136] Traceback (most recent call last):
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1110, in run_engine_core
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]     return func(*args, **kwargs)
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 876, in __init__
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]     super().__init__(
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 145, in __init__
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]     self.scheduler: SchedulerInterface = Scheduler(
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]                                          ^^^^^^^^^^
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/async_scheduler.py", line 14, in __init__
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]     super().__init__(*args, **kwargs)
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/scheduler.py", line 228, in __init__
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]     self.kv_cache_manager = KVCacheManager(
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]                             ^^^^^^^^^^^^^^^
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_manager.py", line 137, in __init__
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]     self.coordinator = get_kv_cache_coordinator(
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]                        ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_coordinator.py", line 620, in get_kv_cache_coordinator
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]     return HybridKVCacheCoordinator(
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_coordinator.py", line 400, in __init__
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]     super().__init__(
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_coordinator.py", line 66, in __init__
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]     self.single_type_managers = tuple(
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]                                 ^^^^^^
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_coordinator.py", line 67, in <genexpr>
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]     get_manager_for_kv_cache_spec(
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/single_type_kv_cache_manager.py", line 1155, in get_manager_for_kv_cache_spec
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]     manager_class = spec_manager_map[type(kv_cache_spec)]
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136]                     ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=179) ERROR 05-10 18:50:03 [core.py:1136] KeyError: <class 'vllm.v1.kv_cache_interface.CompressorStateMLASpec'>
(EngineCore pid=179) Process EngineCore:
(EngineCore pid=179) Traceback (most recent call last):
(EngineCore pid=179)   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=179)     self.run()
(EngineCore pid=179)   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=179)     self._target(*self._args, **self._kwargs)
(EngineCore pid=179)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1140, in run_engine_core
(EngineCore pid=179)     raise e
(EngineCore pid=179)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1110, in run_engine_core
(EngineCore pid=179)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=179)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=179)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=179)     return func(*args, **kwargs)
(EngineCore pid=179)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=179)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 876, in __init__
(EngineCore pid=179)     super().__init__(
(EngineCore pid=179)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 145, in __init__
(EngineCore pid=179)     self.scheduler: SchedulerInterface = Scheduler(
(EngineCore pid=179)                                          ^^^^^^^^^^
(EngineCore pid=179)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/async_scheduler.py", line 14, in __init__
(EngineCore pid=179)     super().__init__(*args, **kwargs)
(EngineCore pid=179)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/sched/scheduler.py", line 228, in __init__
(EngineCore pid=179)     self.kv_cache_manager = KVCacheManager(
(EngineCore pid=179)                             ^^^^^^^^^^^^^^^
(EngineCore pid=179)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_manager.py", line 137, in __init__
(EngineCore pid=179)     self.coordinator = get_kv_cache_coordinator(
(EngineCore pid=179)                        ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=179)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_coordinator.py", line 620, in get_kv_cache_coordinator
(EngineCore pid=179)     return HybridKVCacheCoordinator(
(EngineCore pid=179)            ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=179)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_coordinator.py", line 400, in __init__
(EngineCore pid=179)     super().__init__(
(EngineCore pid=179)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_coordinator.py", line 66, in __init__
(EngineCore pid=179)     self.single_type_managers = tuple(
(EngineCore pid=179)                                 ^^^^^^
(EngineCore pid=179)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_coordinator.py", line 67, in <genexpr>
(EngineCore pid=179)     get_manager_for_kv_cache_spec(
(EngineCore pid=179)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/single_type_kv_cache_manager.py", line 1155, in get_manager_for_kv_cache_spec
(EngineCore pid=179)     manager_class = spec_manager_map[type(kv_cache_spec)]
(EngineCore pid=179)                     ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=179) KeyError: <class 'vllm.v1.kv_cache_interface.CompressorStateMLASpec'>
[rank0]:[W510 18:50:04.746506946 ProcessGroupNCCL.cpp:1575] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(APIServer pid=126) Traceback (most recent call last):
(APIServer pid=126)   File "/usr/local/bin/vllm", line 8, in <module>
(APIServer pid=126)     sys.exit(main())
(APIServer pid=126)              ^^^^^^
(APIServer pid=126)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/main.py", line 92, in main
(APIServer pid=126)     args.dispatch_function(args)
(APIServer pid=126)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/cli/serve.py", line 122, in cmd
(APIServer pid=126)     uvloop.run(run_server(args))
(APIServer pid=126)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=126)     return __asyncio.run(
(APIServer pid=126)            ^^^^^^^^^^^^^^
(APIServer pid=126)   File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
(APIServer pid=126)     return runner.run(main)
(APIServer pid=126)            ^^^^^^^^^^^^^^^^
(APIServer pid=126)   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=126)     return self._loop.run_until_complete(task)
(APIServer pid=126)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=126)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=126)   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=126)     return await main
(APIServer pid=126)            ^^^^^^^^^^
(APIServer pid=126)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 678, in run_server
(APIServer pid=126)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=126)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 692, in run_server_worker
(APIServer pid=126)     async with build_async_engine_client(
(APIServer pid=126)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=126)     return await anext(self.gen)
(APIServer pid=126)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=126)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client
(APIServer pid=126)     async with build_async_engine_client_from_engine_args(
(APIServer pid=126)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=126)     return await anext(self.gen)
(APIServer pid=126)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=126)   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args
(APIServer pid=126)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=126)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=126)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config
(APIServer pid=126)     return cls(
(APIServer pid=126)            ^^^^
(APIServer pid=126)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 146, in __init__
(APIServer pid=126)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=126)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=126)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=126)     return func(*args, **kwargs)
(APIServer pid=126)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=126)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 130, in make_async_mp_client
(APIServer pid=126)     return AsyncMPClient(*client_args)
(APIServer pid=126)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=126)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=126)     return func(*args, **kwargs)
(APIServer pid=126)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=126)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 900, in __init__
(APIServer pid=126)     super().__init__(
(APIServer pid=126)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 535, in __init__
(APIServer pid=126)     with launch_core_engines(
(APIServer pid=126)   File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=126)     next(self.gen)
(APIServer pid=126)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1119, in launch_core_engines
(APIServer pid=126)     wait_for_engine_startup(
(APIServer pid=126)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1178, in wait_for_engine_startup
(APIServer pid=126)     raise RuntimeError(
(APIServer pid=126) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
14

Seems to work now; the sha256sum check failed for me, so I downloaded the script and ran it with --skip-verify as it appears that only the README.md hash is incorrect for the model files.

I get about 2tok/s on my GB10, not really tested it beyond this, which repeats if I give it more tokens:

curl -s ``http://localhost:8000/v1/completions`` -H ‘Content-Type: application/json’ -d ‘{“model”:“dsv4”,“prompt”:“Starmer is …”,“max_tokens”:50,“temperature”:0}’ ….

a man of the left, but not a man of the far left. He is a man of the left, but not a man of the far left. He is a man of the left, but not a man of the far left.

15

Can anybody run Tool-Eval bench if they got it working?

18

Quick update with latest patches and next steps.

1. KV-cache. V4-Flash architecture enables ~7% of V3.2’s KV memory usage at 1M context (paper §2.3.4), but out-of-box on the lmxxf base image vLLM was admitting the compressor-state and SWA caches as if they were full-attention pools rather than fixed-size SSMs (paper §3.5.1)… costing ~9× concurrency at 16K context. Three small patches in Entrpi/vllm/tree/kv-layout-dsv4-compressor-state close this:

  • V1: CompressorStateMLASpec — bounds the compressor cache by sliding_window only.
  • V2: route DeepseekV4SWACache through the same spec.
  • V3: register the subclass in spec_manager_map (exact-type lookup; without this EngineCore crashed silently with KeyError, masked by the 22-min wait_for_engine_startup grace period).

Validated on Spark: KV pool 19.11 GiB / 15,608 tokens / 229× concurrency at 16K, up from ~25× pre-patch. Patches are PR-able upstream once we rebase onto current vllm/main.

2. Throughput. Single-stream decode is ~1.75 t/s as-is: bad, way below optimal, no MTP wired in yet.

First optimization fix in-flight right now:

FP8 W8A8 cache misses Three GB10-specific shapes are missing from jasl’s tuning aliases (PR #41834 / commit 52a715a98), all on the hot path:

  • N=32768, K=1024 — attn.wq_b (MLA Q up-proj, fires 43× per token)
  • N=4096, K=8192 — attn.wo_b (MLA output proj, fires 43× per token)
  • N=4096, K=2048 — mlp.shared_experts.down_proj (fires up to 42× per token)

Tuning sweep is running on Spark right now… 8 min in, ~2-2.5h total, output will be 3 JSONs that drop into the overlay. After deploy we get t/s before/after, and the JSONs are upstreamable as a discrete PR to vllm-project/vllm.

19

No luck so far. I seem to be stuck loading. Not sure how long I wait for this. I went and had breakfast, came back and its still loading. Is that normal?

EngineCore pid=214) [ds4_load_dbg] weight_loader=_passthrough_loader (Iq2XxsQ2KFusedMoEMethod.create_weights.<locals>._passthrough_loader)  E0: finite=65536/65536 |d|_min=2.632e-04 |d|_max=8.092e-04  E254: finite=65536/65536 |d|_min=2.832e-04 |d|_max=8.173e-04 nans=0
(EngineCore pid=214) [DS4_TRACE] apply() FIRST inst=261221056129280 layer_id=261221056128848 layer.prefix='?' x.shape=(8192, 4096) x.dtype=torch.bfloat16 topk_ids.shape=(8192, 6)
(EngineCore pid=214) [ds4_load_dbg] weight_loader=_passthrough_loader (Iq2XxsQ2KFusedMoEMethod.create_weights.<locals>._passthrough_loader)  E0: finite=65536/65536 |d|_min=2.837e-04 |d|_max=8.016e-04  E254: finite=65536/65536 |d|_min=2.732e-04 |d|_max=8.116e-04 nans=0

Also for something as experimental as this I perfer a lot more transparency when its loading, could we have an option to run as docker run -it rather than docker run -d

I manually edited install.sh to make this change.

20

Tuning sweep finished and I’ve done more vLLM patching work. Have also identified and am now working on fixing the top perf bottlenecks.

21

I’ll update the installer to cleanup prior state and give a clean install without this soon as I land the performance work.

22

You may have the same crash → container auto-restart issue I ran into. It’s hard to debug because the log file overwrites, you have to be watching as it happens.

If loading seems hung, checkdocker container list - if it says it was started hours ago but only has been up for less, that’s what is going on.