veloGB10 v0.4.0 RELEASE: Qwen 3.8-27b code sustained tok/s: 4x : 125 tok/s , 2x: 85/tok/s, 1x: 75tok/s

[stav_kats]

OK I have finally managed to release this.
I’m still not happy as there’s more performance left to work on (including potentially a novel drafting model for this). However I think this is good enough to release.

TP=4 CODE Sustained Demo:

TP=2 CODE Sustained Demo:

SINGLE CODE Sustained Demo:

Basic performance data:

The full repo is here:

https://github.com/sf-stav/veloGB10/tree/release-v0.4.0

Qwen 3.8 27b instructions here (Single, TP=2 and TP=4) :

QWEN_27B_SETUP.md

Some comparative stats:

More stats/graphs in the repo README.md

UPDATE: v0.5.0 is out. Vision support has now been added.

Opinions, criticism etc. etc. welcome

[stav_kats]

Things that I still need to do and/or are pending:

1. Fix the tencent/hy3 support (I have broken it during the last few weeks) ← DONE, FIXED
2. Add vision support for ALL qwen models. ← in progress
3. Long run stability testing ← DONE
4. Add support for the 397b model (includes Ornith 1.5)
5. Deepseek v4 Flash Dspark has started. It’s nowhere near complete and optimized. The goal for this one is to beat all competition in decode speed on 4x and 2x GB10.

If we see a 122b or any other Qwen 3.8 model that fits 1, 2x or 4x Spark, i’ll probably jump on it.
😐

[jetspark]

@stav_kats , did you try FP8? Thanks a lot for your work!

[emX0r]

@stav_kats

https://huggingface.co/doth4580/Qwen3.8-27B-DFlash2

thank you for your work. it looks really great. I’ll give it a try today. Have you tested also quality is it more or less same like the ones provided by MiaAI-Lab or should I expect worse/better?

[stav_kats]

Hi,

I have not had the time to test quality yet, please be my guest, I’m curious to know as well.

One thing to note (that I noticed yesterday) is that the Mia recipe uses the Radix quant of the model which is bigger, so presumably it contains selectively higher precision for a part of the weights. This suggests that quality may be higher with that model. One thing to remember is that the engine uses an optimized format for its models, so it’s not able to read the Radix model directly. If I have time, I may look into getting this to work, but I’d rather now focus on the DSV4 support (I want the engine to be able to support the flash + vision model that will come out this week hopefully).

[emX0r]

I’m getting strange error when trying to use it from my Pi.

" Error: 422 Failed to deserialize the JSON body into the target type: messages[1].content: invalid type: sequence, expected a string at line 1 column 6920 "

[stav_kats]

I’ll look into this!

[stav_kats]

@emX0r

I’m getting strange error when trying to use it from my Pi.

" Error: 422 Failed to deserialize the JSON body into the target type: messages[1].content: invalid type: sequence, expected a string at line 1 column 6920 "

I managed to repeat the problem:
hello

Error: 422 Failed to deserialize the JSON body into the target type: messages[1].content: invalid type: sequence, expected a string at line 1 column 11092

found the issue. Fixed. new binaries are available and fixes are merged.

it’s working now (verified in practice with pi)

get the latest code and build or the binary from here:

Release veloGB10 v0.4.2 · sf-stav/veloGB10

[JW2026]

Whats the TTFT and Prefill you are getting ?

[stav_kats]

on TP=4 I just got 318 ms for a cold session new prompt.
It will rise to several seconds if your context usage is high though.

[emX0r]

thank you for quick fix. I can confirm it works now. Just testing it more. One cosmetic thing which looks odd is the model identifies itself as 1b49c1e2ad0b7621e9a991bc4e2dd10380cfc175

When checking with curl:

**{**

  **"id":** "1b49c1e2ad0b7621e9a991bc4e2dd10380cfc175"**,**

  **"object":** "model"**,**

  **"created":** 1787574846,

  **"owned_by":** "rust_infer"

**}**

[JW2026]

@stav_kats

It will rise to several seconds if your context usage is high though.

Try 256K context.. it ranges from 30s to 220s …

Looks like fixed in 0.4.2

[JW2026]

@stav_kats

get the latest code and build or the binary from here:

Using build 0.4.2 - Basically a malform tools call . My workflow already clean that those via proxy but just FYI

[stav_kats]

will be checking this, as well as the prefill speed.

[stav_kats]

These report are EXTREMELY helpful!
Thanks!

[stav_kats]

New build out just for you (just joking)

• Hopefully the tools call issue should be fixed.

• The model should report it self better now.

• **Faster prefill/ttft. (there is more room for improvement which needs one or two new kernels so it’s going to take some time) .

  v0.5.0 — Vision support**

• Vision support. Image input is now supported end-to-end on a GPU vision tower (gpu_vision kernels), with a --vision-cpu escape hatch to the CPU reference path. PNG/JPEG/WebP/GIF decoding added. The engine now ships and requires the gpu_vision.ptx kernel artifact in addition to the existing PTX set.

• Better tool-call support. A single canonical serializer now handles streaming and non-streaming tool-call output identically, repairs malformed tool-call tags, and no longer drops or leaks text around tool-call boundaries. New tool-call compliance and serializer test suites.

• Prefill/TTFT optimizations. New opt-in prefill levers (tensor-core flash-attention prefill, v2 W4A4 prefill GEMM, GDN tensor-core chunked scan), all env-gated default off, so the default serving path is unchanged. Minor bug fixes and optimizations.

• Model-id fix. /v1/models and responses now report the model card’s base_model (e.g. Qwen/Qwen3.8-27B) instead of a local directory fragment. --model-name still overrides.

[stav_kats]

I’m getting ready to potentially support the upcoming Qwen3.8-Flash-Next model.

Hopefully the changes to the arch will not make it hard to support. The goal as always is best in class decode speed on TP=2 TP=4

Can’t wait to see what it is

[jbourny]

I’d like to try, but I’ve wasted so much time on projects like this that are actually completely buggy that it discouraged me, I expect feedback from other people who use it in real projects.

The last project that cut me off from the desire to waste time testing is Atlas, on paper it showed 2x higher flow rates than others, it even dared to set its record on spark arena but in reality it gets this flow without even looking at what it comes out, by testing in reality we have 50% of the time the same tokens in infinite loop so necessarily it goes faster but unusable.

There have also been other projects that cheat by compressing the weights to the maximum by completely losing the quality…

But maybe your project doesn’t have these problems, but I prefer to wait for more tests.

[stav_kats]

Hey @jbourny

Absolutely and fair enough!

I wasted a huge a mount of time on Atlas as well, and it was exactly for this reason I started my own.
I was sure that more performance can be squeezed out of the GB10 machines and I have to say that I also have gotten frustrated with most other solutions out there. Examples:

The latest vllm 1x Qwen3.8 27b for various 4bit variants are NOT acceptable, even though the theoretical throughput makes you think that this is a workable solution. There is NO 2x (TP=2) recipe for the 27b model that actually adds any value (they are mostly slower). The promoted 1x deepseek v4 flash recipe is unusable as after a few turns it spits out garbage. (he 2x spark recipes are cool though).

A few things that I did differently in my project:

Reduced scope:

1. I focus on very few models, namely Qwen 3.5/3.6/3.8, hy3 and I’m starting on deepseek.
2. The kernels and cuda logic are sm_121a specific (essentially exclusively for GB10).
3. Specific quants and model checkpoints only - there will never be generic support for arbitrary quants/configurations.

The above make the code very focused.

Stability:

Each release goes through a 8h/12h and 24h soak test, where I test continuous usage, with a variety, no breaks. The last one was done 3-4 days ago and the last few hours did produce the exact same results as the first, no drift.

About the weights compressing etc:

You can download and quantize the weights your self by downloading the original BF16 and then quantize with --quantize (this is a straight up, no frills nvfp4 quantization and is disclosed int he code, no tricks). In order to diffuse any such suspicions, I’ll be adding support for directly using models from respected providers.

As far as the performance goes: I have provided several graphs that show performance over time and depending on the type of task/output. I find these much more representative of reality, because you can see where the peaks and bottoms occur. In practice, especially the TP=2 and even more so the TP=4 modes feels very responsive and the token generation is in most cases fluid and for sure and obviously fast than anything that exists out there for the GB10. And as I mentioned, stability is there and tested. Atlas, while impressive, never offered a high tier product. It was always flakey, not clearly documented, no clear examples on how to exactly repeat the results etc. I have already done better that them, by show you guys the videos of exactly what I ran and how.

Lastly, while I respect your choice (I have felt the same way): Just FYI:

The release is 1 binary ( + 14 cuda files you don’t even need to look at). No need to build anything, download a super small archive (<30MB) (you also need he model) and run. You can within 15 mins have a Single,TP=2,TP=4 model running. Just to see how it does.

Here’s some results with the latest version of eval-tool-bench, single mode:

In any case, thanks for replying!

[JW2026]

@stav_kats

• Hopefully the tools call issue should be fixed.

I fixed the malform tag issue outside of the inference engine via a proxy, so it wasn’t really a concern for me, just thought I highlight it. Really appreciate the fast response and fixes all the same.

Cancelled my Anthropic Max 20 subscriptions , couldn’t afford it, so next install and update will have to go back to manual as I realized I was becoming overly dependant on Opus and Fable. :-P

• And of course Anthropics decides to give me issues with both Opus and Fable on the last 2 days after I cancelled subscription but before it ends. It almost seems like they were doing it deliberately.

@stav_kats

• Prefill/TTFT optimizations.

You have the bench for this ? One of the factors that will determine if I go ahead with the upgrades.

@stav_kats

• Model-id fix. /v1/models and responses now report the model card’s base_model (e.g. Qwen/Qwen3.8-27B) instead of a local directory fragment. --model-name still overrides.

Ah yes, this was a bit annoying , not really a show stopper. What I would REALLY like is " streaming telemetry data" - (TTFT, TPS, etc) instead that I could stream to my Dashboard, right now, it seems to be writing to a flat file.

@stav_kats

The goal as always is best in class decode speed on TP=2 TP=4

Make it the best in class for TP=1 .

@jbourny

I expect feedback from other people who use it in real projects.

I just had Hermes → Qwen 3.8 27B → veloGB10 (4.1) + proxy for fixing tags

1. Troubleshoot Hermes worker dropping jobs or not completing tasks.

• Prompt to Hermes “Do a RCA and close the haps with the workers”
• Took 1 hour and it found the issues with a misconfigurations of the workers endpoint. Close the gaps including setting up of various watchdogs.

2. Prompt “Check the latest stock price of ticker XX”

• Took 3 mins , " Done, sir. Ticker XX resolves to xxxxxxxxxxxxxxx plc on the NYSE (the xxxxxxx — flag it if you meant a different “XX”):

• Last: $120.47 USD (market closed Mon 24 Aug, 4:00 PM EDT; last print 4:02 PM)

• Change: −$0.75 (−0.62%) vs Friday’s close of $121.22

• Day range: $118.39 – $121.77 · Volume: ~4.45M

• 52-week range: $54.32 – $129.14

Source: Yahoo Finance quote API, pulled live just now. Want a chart, or the same for another ticker?

3. Prompt “Investigate my wireless setup, here are the addresses and credentials (gave it 3 IP addresses of my OpenWrt APs), check that the wireless is optimized”

• it detected a 4th AP, which is a wireless bridge, so I give it the additional IP and credentials.
• It gave me a overview of my Wireless network, highlighted a couple of warnings about DFS channels.

4. Additional msg to bot “Verify that the power levels are optimum for roaming.”

• It discovered one of my SSID for my IOT is using TKIP and some dropbear issues, ask if it should flip it to CCMP and if I want to fix the dropbear issue. I gave it the go ahead . It run on it’s own fiddling and finally reporting that it fixed the issues. It took 10 hours but there were no looping that I could see, basically just troubleshooting, etc.

Summary

1. I probably could have fixed these issues on my own in 10-20 mins once it has highlighted the issues.

2. The harness is basically Hermes + Telegram + skills

3. I was using 4.1 of the engine but I had to use my own proxy to clean the tag issues. That showed up early in my testing, so it ALMOST stop me from continuing with the testing.

4. In total, this run for 18 hours without issues . Responsiveness isn’t anywhere close to Qwen 3.6 35B but it works, and I wasn’t coding, just pure agentic workload.

@jbourny

But maybe your project doesn’t have these problems, but I prefer to wait for more tests.

My one cents , you don’t have to spend time testing his project and he didn’t have to spend time releasing it for anybody else. But where would we be otherwise ?