Hi, did anyone tried Tokenspeed? It’s more focused on agentic workflows but it’s very new.
TokenSpeed is a brand new inference engine purpose built for speed-of-light agentic workloads.
Read their blog to learn more about its advanced KV cache management, safe and efficient scheduler, and pluggable layered kernel system designed for...
TokenSpeed is a speed-of-light LLM inference engine.
They provide a Docker image which apparently is not reproducible based on the GitHub repo. The repo’s Dockerfile is based on their image.
Then you customize and apparently finish the build inside that image, before starting a server.
I can’t figure out if they have an ARM64 version or not, but if not, we’d be out of luck. And even if they did, I really dislike this method (it’s not open if we cannot reproduce the Docker image).
jasl
June 6, 2026, 11:50am
3
Since vLLM and SGLang are not interest on consumer Blackwell (their customers and sponsors are Datacenter-focused companies).
jasl
July 12, 2026, 1:48am
4
main ← jasl:codex/ds4-sm120-min-enable
Hi everyone — we've built the SM12x / DGX Spark GB10 DeepSeek-V4 path on [TokenS… peed](https://github.com/jasl/tokenspeed) (`sm12x-stable` branch) to the point where it's worth trying. The implementation there is cleaner than the vLLM fork's patch stack, and it's where our new work now lands. The vLLM fork stays available and maintained.
Same 2× DGX Spark GB10 pair, same fabric, same pinned llama-benchy, MTP2 + fp8 KV + prefix cache, C=1 × 3 runs. vLLM = our latest maintained fork head; TokenSpeed = `sm12x-stable` (torch 2.13, FlashInfer CUTLASS MXFP4 MoE):
| depth | ctx_pp t/s (TS / vLLM) | pp2048 (TS / vLLM) | tg128 peak (TS / vLLM) |
|---|---|---|---|
| 8192 | **2057 / 1866** | 1404 / 1406 | 30.3 / 41.5 |
| 16384 | **2062 / 1825** | 1329 / 1354 | 28.7 / 41.3 |
| 32768 | **1979 / 1737** | 1149 / 1224 | 33.3 / 45.3 |
**Where TokenSpeed leads: cold-context prefill, by ~10-14%** — the dominant cost in the long-context / 1M scenarios several of you have raised (@wingcomm's 536K indexer-bound prefill, @brianmiller's 121K cold prefill). pp2048-at-depth is at parity (94-100%). **Decode is currently behind (~70-74%)** — the CUTLASS MoE that wins prefill has a weaker small-M decode GEMM; a hybrid path (CUTLASS prefill + Triton decode, single weight residency) is in progress to close it. Being upfront: if your workload is decode-heavy at low concurrency, the vLLM fork is still faster today.
Other wins worth noting:
- **KV capacity +25%** — 1.90M vs 1.52M tokens at `--max-model-len 131072`, same config.
- **Tool calling: 45/45 requests engine-clean, zero 500s** (the intermittent MTP + thinking + `tool_choice` 500s some of you hit are an upstream vLLM reasoning-boundary bug the fork inherits). Serve with `--grammar-backend xgrammar`.
- Long-gen GSM8K 0.96, zero illegal-memory-access.
## Build (GB10 2-node; RTX PRO 6000 single-node similar)
```bash
git clone -b sm12x-stable https://github.com/jasl/tokenspeed
sudo apt install python3.12-dev libssl-dev
uv venv --seed .venv-ts && source .venv-ts/bin/activate
pip install torch==2.13.0
pip install -r tokenspeed-kernel/python/requirements/cuda.txt
# skips the 10-30 min cold CUTLASS-MoE JIT on first boot:
pip install flashinfer-jit-cache==0.6.14+cu130 --index-url https://flashinfer.ai/whl/cu130
# GB10 = 121, RTX PRO 6000 = 120:
TOKENSPEED_CUDA_ARCH=121 pip install -e tokenspeed-kernel/python --no-build-isolation
pip install --no-deps nvidia-nccl-cu13==2.30.4 # REQUIRED on multi-node, see below
pip install -e tokenspeed-scheduler --no-build-isolation
pip install -e python
pip uninstall -y torchvision torchaudio # unused; its torch-2.13 ABI trips transformers
```
Serve with `--moe-backend flashinfer_cutlass` (the prefill win; the registry otherwise auto-picks the Triton MoE) and `--grammar-backend xgrammar` for tool-calling.
Traps we hit so you don't have to:
- **`nvidia-nccl-cu13==2.30.4` is mandatory on multi-node host-staged RoCE.** 2.28.9 (torch's default), 2.29.7, and 2.30.7 all hit an NCCL graph-replay proxy-progress wedge that presents as serve hangs / late-wave request timeouts (this is likely behind several "2-node decode hang" reports in this thread). torch's dep re-resolves over manual pins on every editable install, so pin it with `--no-deps` **after** the kernel build. A minimal repro is headed to NVIDIA/nccl.
- `TOKENSPEED_CUDA_ARCH` must be set for the kernel build (no auto-detect → "no kernel image"); `rm -rf tokenspeed-kernel/python/objs` after changing it.
- `fast_hadamard_transform` must be built from the GitHub repo (`pip install --no-cache-dir --no-deps git+https://github.com/Dao-AILab/fast-hadamard-transform`) — the PyPI sdist is missing its `csrc/`.
Branches: `sm12x-stable` (validated) / `sm12x-preview` (gate-tested candidates) / `feat/sm12x-engine` (dev, may be unstable).
Benchy tables from your rigs are very welcome — especially the 8× RTX 5090 and 2× RTX PRO 6000 topologies we don't have here.
Spent two weeks adding SM12x support to TokenSpeed.
How does tokenspeed compare with vllm?
jasl
July 12, 2026, 8:09am
6
Because it’s too young, there are many things to do.
Compare my own vLLM fork
depth
ctx_pp t/s (TS / vLLM)
pp2048 (TS / vLLM)
tg128 peak (TS / vLLM)
8192
2057 / 1866 1404 / 1406
30.3 / 41.5
16384
2062 / 1825 1329 / 1354
28.7 / 41.3
32768
1979 / 1737 1149 / 1224
33.3 / 45.3
On the other hand, the arch is so clean that I believe catching up with vLLM isn’t difficult.
And Toolcall-15 100% pass, GSM8k 0.97, and MTP acceptance rate all higher than vLLM
renek
July 12, 2026, 8:50am
7
What’s the comparison to Atlas?
Any reason we should switch working that?
jasl
July 12, 2026, 8:53am
8
What is Atlas?
It’s an alternative, just like SGLang.
renek
July 12, 2026, 9:19am
9
Yeah than it’s similar to it, you can find it here in the forum, AI Framework build in Rust