NOTES:
This is a preview version. There’s going to be a more fleshed out version coming soon.
Vision is not supported yet, it is pending.
TP=4 is also pending and hopefully is going to deliver further speed improvements.
NVFP4 is actually supported, however it is not yet fully optimized so this will be officially released a bit later.
Also thanks to @vcruz305 for introducing me to EXL3 and the principles behind his recipe.
Any feedback, corrections, contributions etc are welcome.
veloGB10 is a one-man project by Stav Katsoulis — kernels, scheduler, transport, gates, docs, and releases are all done in one person’s limited time. Bug reports and well-formed issues are always free and welcome. If you need something specific and soon — a model port, a feature, tuning for your workload, TP>2 — special work requests are taken on at a price: open an issue describing the work and it will be quoted. This is also the most direct way to make the “next areas of research” above happen faster.
This looks extremely interesting. After vision support and prefix caching are done, I would test this out using 4bpw quant, c=16 performance, and check the performance impact if the PLE table being offloaded to SSD or over RDMA. Also, please update your tool eval bench to 2.7.0 as older version is scoring your recipe lower.
Prefix caching is implemented and enabled by default. Vision support is next, along with TP=4 support. 4bpw would require some kernel additions. c =16 probably on TP=4 should be somewhat usable, but that’s quite high. What would you use this for?
your PLE to SSD idea is good, I’ll try it. may give more space for concurrency. I suspect the impact may not be as high as people think.
I will upgrade to 2.7.0, however this is not a recipe. It is a full inference engine ;)
A lot of methods have already moved the PLE table to SSD to trade for more kv cache space in RAM.
Also, I have an AUR scanner which scans and notifies for malicious package. I am allocating c=12 for it which achieves around 200t/s prose combined. I also use Home Assistant voice commands, so single stream and c=16 are both important to me.
v0.7.1 is out: — TP=4 on the EXL3 path, image input, automatic shard shipping
TP=4 on the EXL3 path
--tp 4 now serves Qwen3.8-Flash-Next from EXL3 packs across three peer nodes plus the head: attention, GDN and the KV cache split by head, the routed experts dealt across the ranks, the vocab-parallel LM head, sharded MTP-head screening, a per-step lockstep agreement check and a watchdog. Measured on pure code: ~221 tok/s decode (min/max 173/260, p50/p90/p99 221/234/237) and ~3.2K tok/s prefill — TTFT 0.64 s at ~2.1K input tokens, 10.9 s at ~34.9K.
Image input, on every topology
Images are served end to end on the EXL3 path at TP=1, TP=2 and TP=4: the pack’s original bf16 vision tower is loaded (vision_tower_bf16.safetensors), each image is resized so its longer side is at most --image-max-edge (default 1024, aspect preserved), 3-axis mrope is applied at all six main-attention RoPE sites and the QSA indexer — so images work past the 2,051-token dense window — and image rows ship to the TP nodes as a digest-checked binary payload. Parts are returned in client order. Video and audio parts return 400 for now; video is planned.
Automatic shard shipping
This was already the default for all other model support and had been added for Qwen3.8-Flash-Next now. A multi-node run no longer needs the model copied to the peer machines. The head plans each rank’s shard and ships only that, through the TP blob cache: ~57.5 GiB per node at TP=2 and ~46.8 GiB at TP=4 for the EXL3 pack, and 66.0 GiB for NVFP4 (was 98.5 replicated). TP=2 output is token-identical to the previous path for both formats, and the live TP=4 bank is identical. --tp-sync-only runs a node that syncs and exits.
Plus minor bug fixes and optimizations.
Performance table as of today:
Note: I have seen decode jump momentarily to 292 tok/s on 4x super clean boxes.
Thanks for veloGB10!
We’ve spent the last few days benchmarking local models as coding-agent executors on one Spark and wanted to share how Flash-Next on veloGB10 did against the alternatives. Short version: it’s our top pick for single-stream speed and output
quality.
Setup
Agent: Oh My Pi (OMP), thinking off, 131k context
Tasks: five real tasks from our own Python repos, including CI tooling, a config/schema migration, a quota-gating regression suite with 7 planted bugs that must be caught, and a backend retry feature. Graded by hidden tests, scope checks and mutation tests.
veloGB10 v0.7.0 (commit 502db814), with the Flash-Next EXL3 3.05 bpw pack (revision 3b0d1493)
Compared with:
azampatti’s Qwen3.8-Flash-Next-125B-A5B-INT4-AutoRound on vLLM
We had a cloud reviewer judge both models’ diffs blind, on subtasks both passed, with each pair reviewed in both orders.
Flash-Next won 11, tied 8, lost 4 against Qwen3.6-35B, and was clearly ahead on readability and maintainability.
Flash-Next was the only model in the whole benchmark to solve our hardest logic subtask (a retry/backoff policy).
Qwen3.6-35B produces more volume through batching, but its code is weaker, and an added style guide didn’t close that gap.
Qwen3.8-27B passed only the easiest task single-stream. At 2 lanes it crashed (bug below).
So Flash-Next on veloGB10 is the model whose output we’d actually merge. What keeps it from also being our throughput pick is the one thing below.
Nice-to-have’s, in order of impact for us
Real batching on the EXL3 path at TP=1. Today 1, 2 and 3 lanes all give the same 95.8 tok/s, because the lanes take turns. Agent workloads are bursty: lanes wait on tool calls and tests. With even 2–3 lanes of real aggregate gain, Flash-Next would win on both quality and volume for us. Related ask: allowing k8v8/fp8 KV with --max-batch > 1, and the n-gram table on SSD, would free memory for those lanes.
Bug report: DFlash2 with 2 lanes. Qwen3.8-27B NVFP4-FULL + DFlash2 at --max-batch 2 on v0.7.0 panicked after about 110 s of agent traffic with assertion left == right failed: df2 ring nprev 512 != lane pos 713 at src/batch.rs:5628. The scheduler task died and the engine exited with status 70. Qwen3.6-35B + DFlash at 2 lanes was stable. Happy to send the full log and the binary hash, or open an issue.
A --host/bind flag. The server listens on all interfaces; we currently restrict it with systemd IP rules.
A Prometheus /metrics endpoint, like vLLM’s, for throughput and queue monitoring.
+1 for EXL3 4 bpw. A higher-precision Flash-Next is the first thing we’d test.
Thanks again! we’ll be happy to test new releases as well.
Funny that you mention batching, I was just working on this right now as I am aware that batching requires a serious upgrade.
I’ll look at: k8v8/fp8 KV with --max-batch > 1
n-gram table on SSD is planned
–host/bind will be added, I cant believe it’s not already there!
A Prometheus /metrics endpoint → Noted.
EXL3 4.05 bpw is planned already.
Qwen3.8-27B NVFP4-FULL + DFlash2 issue will be fixed. (full log is always helpful! please send over to the email address I’ll PM you.)
in fact i have planned a 3.5bpw + 4bpw support for Qwen3.8-27B as i suspect the quality and speed will be better on this. Frankly however with the real world performance I see with Flash-Next, I’m not sure I’ll be using 27b. ( I do want to fully support it though as there is a chance we may get a Qwen4.0-27b and I want to be ready to support it)
It’s my daily driver now with vllm 0.3.0 release, nvfp4 hybrid fp8 quant and I am really really really satisfied in java spring framework and react native frontend development. simply great till 500.000 token context at aprox 40-50 t/s and >1100 t/s prefill, toolbench 94
The theory says that byte for byte EXL3 quantization & optimization is better than straight up casting with minimal optimization. Most comparisons place the 3.05 or the 3.5 bpw EXL3 quantizations close or above the accuracy of NVFP4 respectively. A 4bpw quant on EXL3 is for sure better than NVFP4.
veloGB10 is extremely easy to try though and you can tell the difference by your self.
A 4bpw quant is coming soon though, do not expect the same speeds though, but it will definitely beat the vllm based recipe you mentioned
Agreed. 3.5 is largely considered to be equal or better than W4A16 NVFP4. 3.05 is better than W4A4 but that isnt saying much…4.05 is clearly better that all of em.
NOTE: both models have been updated to include the bf16 vision layers ( that’s +12% accuracy avg), so if you need vision please update. (no speed penalty as this was already working at bf16).
2) Concurrent requests on the EXL3 path (Qwen3.8-Flash-Next batching)
Aggregate generated tokens/s on a code workload (256-token requests, one lab box per node, untuned build, indicative absolute values), v0.7.1 behaviour → v0.7.2, at 4 / 8 / 16 busy requests:
4 requests
8 requests
16 requests
TP=1
91 → 97
89 → 129
88 → 168
TP=2
130 → 135
125 → 202
124 → 249
TP=4
159 → 173
152 → 238
154 → 293
IMPORTANT:
Two and three concurrent requests are not faster yet: their aggregate stays at the single-request rate (each request gets its turn). A packed multi-request speculative verify for that range is in development and is the next item on the list.
This need approx. 2 weeks of work to get perfect so watch this space for this.
3) Memory: the n-gram table can live on SSD
-ple-ram ram|ssd|auto (default auto; on/off still work).
Fixed Items
Tool-call history is rendered the way the model writes it: arguments that are a JSON object are parsed before the chat template runs, even for templates with an arguments is string branch (they used to render {"command": ...} inside <function=...>, a mixed XML/JSON history the model copied in long sessions; issue #6).
Tool-call parsing: text inside a parameter value that merely looks like a function tag (<function=x> or a bare function=x> line, for instance a file an agent is writing about this format) is payload and no longer produces a second, phantom tool call.
NVFP4 with DFlash2 or DSpark and --max-batch 2: the scheduler could stop with df2 ring nprev 512 != lane pos 713 when a second request arrived while the first was decoding (the drafter ring is shared). A request that loses the ring now continues on MTP.
NVFP4 --kv-cache k8v8 now serves any --max-batch (it was refused above one request). Plain batched decode runs the same attention kernel a lone request runs, so a request’s text does not depend on how many others are busy; tree verification and --mtp-lanes use new int8 readers. KV compaction after a tree-verified step on k8v8 used the bf16 copy kernel on int8 data; fixed.
A missing --model-dir is a clear error instead of a panic; TP shard shipping includes sidecar files the tensor index omits.
This is awesome. Thank you.
FYI: After about 2 hours on TP2 inference stopped. My bot reckons that the reason might be a desync on speculative decoding.
Thanks for this!
I have done a small soak test of up to 4 hours, but there is a 24 hour long soak test planned for Single, TP2 and TP4 as soon as i finalize all the code for this model. There’s still a few things that need to be sorted out
The only issue I noticed is with optel saying that the telemetry receiver is not turned on when it is enabled after I hit the clear button in VeloBenchmark.
I’ve just pulled your latest changes. Going to test max-batch 10. Awesome!
