Hey all,
My name is Stav Katsoulis and I have been lurking for a long time here.
Today I’d like to introduce a project I have been working on.
It is a Rust based inference engine specifically optimized for the GB10.
Here’s what’s special about this:
- It has kernels written specifically to take advantage of the “retail” GB10 chipset for maximum possible speed.
- It is tuned to the maximum possible extent based on the physics of the hardware.
- It has an implementation of use of cluster (TP=2 only at the moment), specifically to increase performance, and not just memory.
- Easy install: you literally compile the rust and you get 1 binary, a couple of ptx files, and that’s pretty much it! No huge set of files. Just one binary and the cuda ptx files.
Here’s some notable performance stats:
NOTE: All tok/s numbers are single user, i.e. not aggregated.
(I’ll be doing aggregated numbers later what I find time.)
Single GB10:
Qweb3.6-27b-nvfp4-full ( i.e. 100% nvfp4 quant):
~40 tok/s
Qwen3.6-35B-A3B (nvfp4 full):
~110 tok/s
Qwen3.6-9B (nvfp4 full):
~80 tok/s
**
qwen3.5-0.8b-nvfp4-mixed:** ~227.8 tok/s**
qwen3.5-2b-nvfp4-mixed:** ~161.6 tok/s
qwen3.5-4b-nvfp4-mixed: ~100 tok/s
Two GB10s:
Qweb3.6-27b-nvfp4-full ( i.e. 100% nvfp4 quant):
~45-50 tok/s
Qwen3.6-35B-A3B (nvfp4 full):
~120+ tok/s
Qwen3.6-9B (nvfp4 full):
~90-100 tok/s
Models supported right now are:
Qwen3.5:
0.8b: mixed
2b: mixed
4b: mixed
9b: pure nvfp4
27b: pure nvfp4
122b: pure nvfp4 (the -gdn4 variant)
122b: mixed
Qwen3.6:
35b-A3B: pure nvfp4
27b: pure nvfp4
by mixed, usually this means that the MTP head is FP8 and everything else is nvfp4
by pure nvfp4 (or the -nvfp4-full models) it means that EVERY layer is nvfp4 (100% of the model has been quantized)
Note by “supported” I mean it as in they work. This is a one person project and there is NO support other than asking nicely or commissioning some work.
Repo is here:
Quantized models that work with this:
Note: I just released this in the last few minutes.
All models mentioned should be on Huggingface.
I’ll be updating the performance stats (some tok/s stats for older models are old).
any issues or problems let me know.
Planned models at MAX possible performance are:
High priority:
deepseek-v4-flash-dspark
hy3
Lower priority:
step-3.7-flash
qwen3.5 387b (not sure it’s worth it)
I’m hoping that by the time deepseek GA or 4.1 or whatever they call it I’ll be ready to directly be able to support the flash version.
- Stav
