# veloGB10 v0.7.0 RELEASE - Qwen3.8-Flash-Next TP=2 @ 186 tok/sec , SINGLE at 137 tok/sec on code **URL:** \ **Category:** DGX Spark / GB10 Projects\ **Created:** [September 30, 2026, 2:32pm UTC](https://forums.developer.nvidia.com/t/velogb10-v0-7-0-release-qwen3-8-flash-next-tp-2-186-tok-sec-single-at-137-tok-sec-on-code/384765 "2026-09-30T14:32:48Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![stav\_kats](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/stav_kats/32/524025_2.png) [@stav\_kats](https://forums.developer.nvidia.com/u/stav_kats)\ **Post date:** [September 30, 2026, 2:32pm UTC](https://forums.developer.nvidia.com/t/velogb10-v0-7-0-release-qwen3-8-flash-next-tp-2-186-tok-sec-single-at-137-tok-sec-on-code/384765/1 "2026-09-30T14:32:48Z")
Hello all. v0.7.0 is out. the headline: Qwen3.8-Flash-Next: ![Screenshot 2026-09-30 at 5.19.32 PM](https://global.discourse-cdn.com/nvidia/original/4X/c/7/d/c7db5b54a2dd9e7885ffc62bdf742a5f5a7d0383.png) Check this performance out on 2x Sparks: Get the model here: > **[doth4580/Qwen3.8-Flash-Next-EXL3-3.05bpw · Hugging Face](https://huggingface.co/doth4580/Qwen3.8-Flash-Next-EXL3-3.05bpw)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. Get the release here: > **[Release veloGB10 v0.7.0 · sf-stav/veloGB10](https://github.com/sf-stav/veloGB10/releases/tag/v0.7.0)** > > EXL3 packs are served directly. --model-dir now accepts an EXL3 (ExLlamaV3 trellis) pack, and the first model on that path is Qwen3.8-Flash-Next — 125B MoE, ~6B active, Qwen Sparse Attention, 51B n... Or build from source from here: > **[GitHub - sf-stav/veloGB10: An GB10-optimized inference engine](https://github.com/sf-stav/veloGB10)** > > An GB10-optimized inference engine More info: ![Screenshot 2026-09-30 at 3.40.58 PM](https://global.discourse-cdn.com/nvidia/original/4X/a/0/6/a0666daea8b46ab17dc5fb2497054e454f1fb7b1.jpeg) NOTES: This is a preview version. There’s going to be a more fleshed out version coming soon. Vision is not supported yet, it is pending. TP=4 is also pending and hopefully is going to deliver further speed improvements. NVFP4 is actually supported, however it is not yet fully optimized so this will be officially released a bit later. Also thanks to @vcruz305 for introducing me to EXL3 and the principles behind his recipe. Any feedback, corrections, contributions etc are welcome. veloGB10 is a **one-man project by [Stav Katsoulis](https://github.com/sf-stav)** — kernels, scheduler, transport, gates, docs, and releases are all done in one person’s limited time. Bug reports and well-formed issues are always free and welcome. If you need something specific and soon — a model port, a feature, tuning for your workload, TP\>2 — **special work requests are taken on at a price** : open an issue describing the work and it will be quoted. This is also the most direct way to make the “next areas of research” above happen faster. ---
**Author:** ![ykmran](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/ykmran/32/204322_2.png) [@ykmran](https://forums.developer.nvidia.com/u/ykmran)\ **Post date:** [September 30, 2026, 8:04pm UTC](https://forums.developer.nvidia.com/t/velogb10-v0-7-0-release-qwen3-8-flash-next-tp-2-186-tok-sec-single-at-137-tok-sec-on-code/384765/2 "2026-09-30T20:04:43Z")
This looks extremely interesting. After vision support and prefix caching are done, I would test this out using 4bpw quant, c=16 performance, and check the performance impact if the PLE table being offloaded to SSD or over RDMA. Also, please update your tool eval bench to 2.7.0 as older version is scoring your recipe lower. ---
**Author:** ![stav\_kats](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/stav_kats/32/524025_2.png) [@stav\_kats](https://forums.developer.nvidia.com/u/stav_kats)\ **Post date:** [September 30, 2026, 8:47pm UTC](https://forums.developer.nvidia.com/t/velogb10-v0-7-0-release-qwen3-8-flash-next-tp-2-186-tok-sec-single-at-137-tok-sec-on-code/384765/3 "2026-09-30T20:47:47Z")
Prefix caching is implemented and enabled by default. Vision support is next, along with TP=4 support. 4bpw would require some kernel additions. c =16 probably on TP=4 should be somewhat usable, but that’s quite high. What would you use this for? your PLE to SSD idea is good, I’ll try it. may give more space for concurrency. I suspect the impact may not be as high as people think. I will upgrade to 2.7.0, however this is not a recipe. It is a full inference engine ;) ---
**Author:** ![ykmran](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/ykmran/32/204322_2.png) [@ykmran](https://forums.developer.nvidia.com/u/ykmran)\ **Post date:** [September 30, 2026, 8:50pm UTC](https://forums.developer.nvidia.com/t/velogb10-v0-7-0-release-qwen3-8-flash-next-tp-2-186-tok-sec-single-at-137-tok-sec-on-code/384765/4 "2026-09-30T20:50:35Z")
A lot of methods have already moved the PLE table to SSD to trade for more kv cache space in RAM. Also, I have an AUR scanner which scans and notifies for malicious package. I am allocating c=12 for it which achieves around 200t/s prose combined. I also use Home Assistant voice commands, so single stream and c=16 are both important to me. ---
**Author:** ![stav\_kats](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/stav_kats/32/524025_2.png) [@stav\_kats](https://forums.developer.nvidia.com/u/stav_kats)\ **Post date:** [October 1, 2026, 6:42pm UTC](https://forums.developer.nvidia.com/t/velogb10-v0-7-0-release-qwen3-8-flash-next-tp-2-186-tok-sec-single-at-137-tok-sec-on-code/384765/6 "2026-10-01T18:42:46Z")
TP=4 almost ready. key numbers: Decode: 211 tok/s median prefill: ~ 3976 tok/s Peak decode: 292tok/s (!!!) VeloBenchmark 0.1.0 KEY METRICS `│ Decode rate median 211 tok/s` `│ Decode rate min / max 144 / 292 tok/s` `│ Decode p50 / p90 / p99 211 / 224 / 231 tok/s ` `│ First token (TTFT) p50 0.02s ` `│ TTFT p95 / p99 18ms / 18ms ` `│ First visible output p50 0.88s ` `│ ITL p50 / p90 / p99 0.13ms / 28.2ms / 30.6ms ` `│ TPOT (per output token) 4.9ms ` `│ Tokens total 2,676 ` `│ Tokens reasoning / output 132 / 2,544 ` `│ Reasoning share 5% ` `│ Thinking per output token x0.05 ` `│ Stability (sustain / peak) 95% ` `│ Prefill throughput avg ~ 3976 tok/s` Here’s a demo: ![velobench-analytics-db5cbf4d](https://global.discourse-cdn.com/nvidia/original/4X/7/f/7/7f79a67eaf51102067c828ef0b3a8f99f6ee477b.jpeg) Finishing up vision support and releasing. Next up: EXL3 4bpw support. Then, probably other models. ---
**Author:** ![stav\_kats](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/stav_kats/32/524025_2.png) [@stav\_kats](https://forums.developer.nvidia.com/u/stav_kats)\ **Post date:** [October 1, 2026, 7:04pm UTC](https://forums.developer.nvidia.com/t/velogb10-v0-7-0-release-qwen3-8-flash-next-tp-2-186-tok-sec-single-at-137-tok-sec-on-code/384765/7 "2026-10-01T19:04:18Z")
Agentic flow with ultra fast Qwe3.8-Flash-Next on 4x DGX Sparks ---
**Author:** ![stav\_kats](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/stav_kats/32/524025_2.png) [@stav\_kats](https://forums.developer.nvidia.com/u/stav_kats)\ **Post date:** [October 2, 2026, 9:07am UTC](https://forums.developer.nvidia.com/t/velogb10-v0-7-0-release-qwen3-8-flash-next-tp-2-186-tok-sec-single-at-137-tok-sec-on-code/384765/8 "2026-10-02T09:07:56Z")
**v0.7.1 is out: — TP=4 on the EXL3 path, image input, automatic shard shipping** ### **TP=4 on the EXL3 path** `--tp 4` now serves Qwen3.8-Flash-Next from EXL3 packs across three peer nodes plus the head: attention, GDN and the KV cache split by head, the routed experts dealt across the ranks, the vocab-parallel LM head, sharded MTP-head screening, a per-step lockstep agreement check and a watchdog. Measured on pure code: **~221 tok/s decode** (min/max 173/260, p50/p90/p99 221/234/237) and **~3.2K tok/s prefill** — TTFT 0.64 s at ~2.1K input tokens, 10.9 s at ~34.9K. ### **Image input, on every topology** Images are served end to end on the EXL3 path at TP=1, TP=2 and TP=4: the pack’s **original bf16 vision tower** is loaded (`vision_tower_bf16.safetensors`), each image is resized so its longer side is at most `--image-max-edge` (default 1024, aspect preserved), 3-axis mrope is applied at all six main-attention RoPE sites **and** the QSA indexer — so images work past the 2,051-token dense window — and image rows ship to the TP nodes as a digest-checked binary payload. Parts are returned in client order. **Video and audio parts return `400`** for now; video is planned. ### **Automatic shard shipping** This was already the default for all other model support and had been added for Qwen3.8-Flash-Next now. A multi-node run no longer needs the model copied to the peer machines. The head plans each rank’s shard and ships only that, through the TP blob cache: **~57.5 GiB per node at TP=2** and **~46.8 GiB at TP=4** for the EXL3 pack, and 66.0 GiB for NVFP4 (was 98.5 replicated). TP=2 output is token-identical to the previous path for both formats, and the live TP=4 bank is identical. `--tp-sync-only` runs a node that syncs and exits. Plus minor bug fixes and optimizations. Performance table as of today: Note: I have seen decode jump momentarily to 292 tok/s on 4x super clean boxes. ![Screenshot 2026-10-02 at 12.02.55 PM](https://global.discourse-cdn.com/nvidia/original/4X/4/2/6/4268c8ca7b6459112bb00ef945de31bf55db17f2.png) If you have any requests, now is the time! ---
**Author:** ![bddh](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@bddh](https://forums.developer.nvidia.com/u/bddh)\ **Post date:** [October 2, 2026, 11:48am UTC](https://forums.developer.nvidia.com/t/velogb10-v0-7-0-release-qwen3-8-flash-next-tp-2-186-tok-sec-single-at-137-tok-sec-on-code/384765/9 "2026-10-02T11:48:46Z")
Hi Stav, Thanks for veloGB10! We’ve spent the last few days benchmarking local models as coding-agent executors on one Spark and wanted to share how Flash-Next on veloGB10 did against the alternatives. Short version: it’s our top pick for single-stream speed and output quality. Setup - Agent: Oh My Pi (OMP), thinking off, 131k context - Tasks: five real tasks from our own Python repos, including CI tooling, a config/schema migration, a quota-gating regression suite with 7 planted bugs that must be caught, and a backend retry feature. Graded by hidden tests, scope checks and mutation tests. - veloGB10 v0.7.0 (commit 502db814), with the Flash-Next EXL3 3.05 bpw pack (revision 3b0d1493) - Compared with: - azampatti’s Qwen3.8-Flash-Next-125B-A5B-INT4-AutoRound on vLLM - Qwen3.6-35B-A3B NVFP4 + DFlash on veloGB10. - Qwen3.8-27B NVFP4 + DFlash2 on veloGB10. Single stream, whole tasks: ![image](https://global.discourse-cdn.com/nvidia/original/4X/9/6/3/9631bd697f1cfcfe49404c3395ec178f084669f1.png) Here we broke our tasks into small subtasks and ran through 60-minute windows, scored as subtasks that passed checks: ![image](https://global.discourse-cdn.com/nvidia/original/4X/7/b/c/7bc255009a2b01b3f988cb005befeb77f6eec768.png) We had a cloud reviewer judge both models’ diffs blind, on subtasks both passed, with each pair reviewed in both orders. - Flash-Next won 11, tied 8, lost 4 against Qwen3.6-35B, and was clearly ahead on readability and maintainability. - Flash-Next was the only model in the whole benchmark to solve our hardest logic subtask (a retry/backoff policy). - Qwen3.6-35B produces more volume through batching, but its code is weaker, and an added style guide didn’t close that gap. - Qwen3.8-27B passed only the easiest task single-stream. At 2 lanes it crashed (bug below). So Flash-Next on veloGB10 is the model whose output we’d actually merge. What keeps it from also being our throughput pick is the one thing below. Nice-to-have’s, in order of impact for us 1. Real batching on the EXL3 path at TP=1. Today 1, 2 and 3 lanes all give the same 95.8 tok/s, because the lanes take turns. Agent workloads are bursty: lanes wait on tool calls and tests. With even 2–3 lanes of real aggregate gain, Flash-Next would win on both quality and volume for us. Related ask: allowing k8v8/fp8 KV with --max-batch \> 1, and the n-gram table on SSD, would free memory for those lanes. 2. Bug report: DFlash2 with 2 lanes. Qwen3.8-27B NVFP4-FULL + DFlash2 at --max-batch 2 on v0.7.0 panicked after about 110 s of agent traffic with assertion `left == right` failed: df2 ring nprev 512 != lane pos 713 at src/batch.rs:5628. The scheduler task died and the engine exited with status 70. Qwen3.6-35B + DFlash at 2 lanes was stable. Happy to send the full log and the binary hash, or open an issue. 3. A --host/bind flag. The server listens on all interfaces; we currently restrict it with systemd IP rules. 4. A Prometheus /metrics endpoint, like vLLM’s, for throughput and queue monitoring. 5. +1 for EXL3 4 bpw. A higher-precision Flash-Next is the first thing we’d test. Thanks again! we’ll be happy to test new releases as well. ---
**Author:** ![stav\_kats](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/stav_kats/32/524025_2.png) [@stav\_kats](https://forums.developer.nvidia.com/u/stav_kats)\ **Post date:** [October 2, 2026, 1:28pm UTC](https://forums.developer.nvidia.com/t/velogb10-v0-7-0-release-qwen3-8-flash-next-tp-2-186-tok-sec-single-at-137-tok-sec-on-code/384765/10 "2026-10-02T13:28:00Z")
@bddh Thanks for this! Funny that you mention batching, I was just working on this right now as I am aware that batching requires a serious upgrade. I’ll look at: k8v8/fp8 KV with --max-batch \> 1 n-gram table on SSD is planned –host/bind will be added, I cant believe it’s not already there! A Prometheus /metrics endpoint → Noted. EXL3 4.05 bpw is planned already. Qwen3.8-27B NVFP4-FULL + DFlash2 issue will be fixed. (full log is always helpful! please send over to the email address I’ll PM you.) in fact i have planned a 3.5bpw + 4bpw support for Qwen3.8-27B as i suspect the quality and speed will be better on this. Frankly however with the real world performance I see with Flash-Next, I’m not sure I’ll be using 27b. ( I do want to fully support it though as there is a chance we may get a Qwen4.0-27b and I want to be ready to support it) ---
**Author:** ![jashic123](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@jashic123](https://forums.developer.nvidia.com/u/jashic123)\ **Post date:** [October 2, 2026, 6:26pm UTC](https://forums.developer.nvidia.com/t/velogb10-v0-7-0-release-qwen3-8-flash-next-tp-2-186-tok-sec-single-at-137-tok-sec-on-code/384765/11 "2026-10-02T18:26:10Z")
> [@bddh](#): > > 1. +1 for EXL3 4 bpw. A higher-precision Flash-Next I’d def test this. EXL3 3.05 is fast, but it also has more dmg that I’d like. ---
**Author:** ![TimoGoetze](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/timogoetze/32/537531_2.png) [@TimoGoetze](https://forums.developer.nvidia.com/u/TimoGoetze)\ **Post date:** [October 2, 2026, 8:12pm UTC](https://forums.developer.nvidia.com/t/velogb10-v0-7-0-release-qwen3-8-flash-next-tp-2-186-tok-sec-single-at-137-tok-sec-on-code/384765/12 "2026-10-02T20:12:39Z")
is the EXL3 Quant better in quality and really faster than this: [GitHub - blazux/qwen3.8-Flash-DGX: recipe for running Qwen3.8-Flash-Next on a single DGX Spark · GitHub](https://github.com/blazux/qwen3.8-Flash-DGX.git) ? It’s my daily driver now with vllm 0.3.0 release, nvfp4 hybrid fp8 quant and I am really really really satisfied in java spring framework and react native frontend development. simply great till 500.000 token context at aprox 40-50 t/s and \>1100 t/s prefill, toolbench 94 ---
**Author:** ![stav\_kats](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/stav_kats/32/524025_2.png) [@stav\_kats](https://forums.developer.nvidia.com/u/stav_kats)\ **Post date:** [October 2, 2026, 8:22pm UTC](https://forums.developer.nvidia.com/t/velogb10-v0-7-0-release-qwen3-8-flash-next-tp-2-186-tok-sec-single-at-137-tok-sec-on-code/384765/13 "2026-10-02T20:22:53Z")
The theory says that byte for byte EXL3 quantization & optimization is better than straight up casting with minimal optimization. Most comparisons place the 3.05 or the 3.5 bpw EXL3 quantizations close or above the accuracy of NVFP4 respectively. A 4bpw quant on EXL3 is for sure better than NVFP4. veloGB10 is extremely easy to try though and you can tell the difference by your self. A 4bpw quant is coming soon though, do not expect the same speeds though, but it will definitely beat the vllm based recipe you mentioned ---
**Author:** ![jashic123](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@jashic123](https://forums.developer.nvidia.com/u/jashic123)\ **Post date:** [October 3, 2026, 12:11am UTC](https://forums.developer.nvidia.com/t/velogb10-v0-7-0-release-qwen3-8-flash-next-tp-2-186-tok-sec-single-at-137-tok-sec-on-code/384765/14 "2026-10-03T00:11:31Z")
Agreed. 3.5 is largely considered to be equal or better than W4A16 NVFP4. 3.05 is better than W4A4 but that isnt saying much…4.05 is clearly better that all of em. ---
**Author:** ![stav\_kats](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/stav_kats/32/524025_2.png) [@stav\_kats](https://forums.developer.nvidia.com/u/stav_kats)\ **Post date:** [October 3, 2026, 8:20am UTC](https://forums.developer.nvidia.com/t/velogb10-v0-7-0-release-qwen3-8-flash-next-tp-2-186-tok-sec-single-at-137-tok-sec-on-code/384765/15 "2026-10-03T08:20:58Z")
## @jashic123 @TimoGoetze @bddh @ykmran v 0.7.2 is out. new: ## **1) Qwen3.8-Flash-Next at 4.05 bpw (EXL3)** you can now use the **4.05bpw** model from here: > **[doth4580/Qwen3.8-Flash-Next-EXL3-4.05bpw · Hugging Face](https://huggingface.co/doth4580/Qwen3.8-Flash-Next-EXL3-4.05bpw)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. or the **3.05bpw** model from here: > **[doth4580/Qwen3.8-Flash-Next-EXL3-3.05bpw · Hugging Face](https://huggingface.co/doth4580/Qwen3.8-Flash-Next-EXL3-3.05bpw)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. Here’s some results with the 4.05bpw build (measured at TP=4) ![Screenshot 2026-10-03 at 8.59.08 AM](https://global.discourse-cdn.com/nvidia/original/4X/8/b/c/8bc34b75a392bddfc41c6a72290115606a651c02.jpeg) NOTE: both models have been updated to include the bf16 vision layers ( that’s +12% accuracy avg), so if you need vision please update. (no speed penalty as this was already working at bf16). ## **2) Concurrent requests on the EXL3 path (Qwen3.8-Flash-Next batching)** Aggregate generated tokens/s on a code workload (256-token requests, one lab box per node, untuned build, indicative absolute values), v0.7.1 behaviour → v0.7.2, at 4 / 8 / 16 busy requests: | | **4 requests** | **8 requests** | **16 requests** | | --- | --- | --- | --- | | TP=1 | 91 → 97 | 89 → 129 | 88 → 168 | | TP=2 | 130 → 135 | 125 → 202 | 124 → 249 | | TP=4 | 159 → 173 | 152 → 238 | 154 → 293 | **IMPORTANT: Two and three concurrent requests are not faster yet:** their aggregate stays at the single-request rate (each request gets its turn). A packed multi-request speculative verify for that range is in development and is the next item on the list. This need approx. 2 weeks of work to get perfect so watch this space for this. ## **3) Memory: the n-gram table can live on SSD** `-ple-ram ram|ssd|auto` (default `auto`; `on`/`off` still work). ## ## **Fixed Items** - **Tool-call history** is rendered the way the model writes it: `arguments` that are a JSON object are parsed before the chat template runs, even for templates with an `arguments is string` branch (they used to render `{"command": ...}` inside ``, a mixed XML/JSON history the model copied in long sessions; issue #6). - **Tool-call parsing:** text inside a parameter value that merely looks like a function tag (`` or a bare `function=x>` line, for instance a file an agent is writing about this format) is payload and no longer produces a second, phantom tool call. - **NVFP4 with DFlash2 or DSpark and `--max-batch 2`:** the scheduler could stop with `df2 ring nprev 512 != lane pos 713` when a second request arrived while the first was decoding (the drafter ring is shared). A request that loses the ring now continues on MTP. - **NVFP4 `--kv-cache k8v8` now serves any `--max-batch`** (it was refused above one request). Plain batched decode runs the same attention kernel a lone request runs, so a request’s text does not depend on how many others are busy; tree verification and `--mtp-lanes` use new int8 readers. KV compaction after a tree-verified step on k8v8 used the bf16 copy kernel on int8 data; fixed. - A missing `--model-dir` is a clear error instead of a panic; TP shard shipping includes sidecar files the tensor index omits. See the CHANGELOG for more detail. Here’s an early test at c16: ![Screenshot 2026-10-03 at 11.23.24 AM](https://global.discourse-cdn.com/nvidia/original/4X/b/3/1/b31b77f95c883673bf987e9dc7ec83065be5f885.jpeg) ---
**Author:** ![omegadota2](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/omegadota2/32/537352_2.png) [@omegadota2](https://forums.developer.nvidia.com/u/omegadota2)\ **Post date:** [October 4, 2026, 4:10pm UTC](https://forums.developer.nvidia.com/t/velogb10-v0-7-0-release-qwen3-8-flash-next-tp-2-186-tok-sec-single-at-137-tok-sec-on-code/384765/16 "2026-10-04T16:10:51Z")
This is awesome. Thank you. FYI: After about 2 hours on TP2 inference stopped. My bot reckons that the reason might be a desync on speculative decoding. ---
**Author:** ![stav\_kats](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/stav_kats/32/524025_2.png) [@stav\_kats](https://forums.developer.nvidia.com/u/stav_kats)\ **Post date:** [October 4, 2026, 7:46pm UTC](https://forums.developer.nvidia.com/t/velogb10-v0-7-0-release-qwen3-8-flash-next-tp-2-186-tok-sec-single-at-137-tok-sec-on-code/384765/17 "2026-10-04T19:46:20Z")
Thanks for this! I have done a small soak test of up to 4 hours, but there is a 24 hour long soak test planned for Single, TP2 and TP4 as soon as i finalize all the code for this model. There’s still a few things that need to be sorted out ---
**Author:** ![sean.clearpixels](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@sean.clearpixels](https://forums.developer.nvidia.com/u/sean.clearpixels)\ **Post date:** [October 5, 2026, 3:02am UTC](https://forums.developer.nvidia.com/t/velogb10-v0-7-0-release-qwen3-8-flash-next-tp-2-186-tok-sec-single-at-137-tok-sec-on-code/384765/18 "2026-10-05T03:02:11Z")
@stav_kats Hey. This is really amazing work. I literaly [https://www.youtube.com/watch?v=9CS7j5I6aOc](https://www.youtube.com/watch?v=9CS7j5I6aOc) when I saw the following results with --max-batch 1 at TP=2 ```markdown | Test | pp t/s | tg t/s | TTFT (ms) | Total (ms) | Tokens | |---|---:|---:|---:|---:|---:| | pp2048 tg128 @ d0 | 387,069 | 105.7 | 778 | 1,216 | 2048+128 | | pp2048 tg128 @ d0 c2 | 1,378 | 81.1 | 1,782 | 1,186 | 2048+128 | | pp2048 tg128 @ d0 c4 | 1,185 | 75.2 | 3,639 | 1,135 | 2048+128 | | pp2048 tg128 @ d4096 | 429,312 | 105.8 | 2,369 | 1,223 | 2048+128 | | pp2048 tg128 @ d4096 c2 | 1,910 | 54.2 | 4,117 | 1,186 | 2048+128 | | pp2048 tg128 @ d4096 c4 | 1,711 | 42.2 | 7,778 | 1,273 | 2048+128 | | pp2048 tg128 @ d8192 | 457,144 | 109.5 | 3,885 | 1,189 | 2048+128 | | pp2048 tg128 @ d8192 c2 | 2,089 | 40.4 | 6,408 | 1,233 | 2048+128 | | pp2048 tg128 @ d8192 c4 | 1,947 | 30.9 | 11,539 | 1,253 | 2048+128 | ``` The only issue I noticed is with optel saying that the telemetry receiver is not turned on when it is enabled after I hit the clear button in VeloBenchmark. I’ve just pulled your latest changes. Going to test max-batch 10. Awesome! ---
**Author:** ![edp10961](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@edp10961](https://forums.developer.nvidia.com/u/edp10961)\ **Post date:** [October 5, 2026, 5:11am UTC](https://forums.developer.nvidia.com/t/velogb10-v0-7-0-release-qwen3-8-flash-next-tp-2-186-tok-sec-single-at-137-tok-sec-on-code/384765/19 "2026-10-05T05:11:49Z")
North Korea??? ---
**Author:** ![jashic123](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@jashic123](https://forums.developer.nvidia.com/u/jashic123)\ **Post date:** [October 5, 2026, 5:29am UTC](https://forums.developer.nvidia.com/t/velogb10-v0-7-0-release-qwen3-8-flash-next-tp-2-186-tok-sec-single-at-137-tok-sec-on-code/384765/20 "2026-10-05T05:29:06Z")
i filed an issue. issue#10. --tp-dh-shard off resolves the issue for me. ---
**Author:** ![stav\_kats](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/stav_kats/32/524025_2.png) [@stav\_kats](https://forums.developer.nvidia.com/u/stav_kats)\ **Post date:** [October 5, 2026, 6:17am UTC](https://forums.developer.nvidia.com/t/velogb10-v0-7-0-release-qwen3-8-flash-next-tp-2-186-tok-sec-single-at-137-tok-sec-on-code/384765/21 "2026-10-05T06:17:06Z")
Hey thanks for this! Yep, truth is I have not checked if telemetry is even wired up to the Flash-next paths. Will do this in a next version. [Next page](https://forums.developer.nvidia.com/t/velogb10-v0-7-0-release-qwen3-8-flash-next-tp-2-186-tok-sec-single-at-137-tok-sec-on-code/384765.md?page=2)