Hey GB10 crowd. I am not here to oversell an idea, but given how my 5090 has been running Neroued’s Ninfer inference engine ( GitHub - Neroued/ninfer: High-performance single-GPU inference for selected model checkpoints and GPUs. · GitHub ), I contemplated a GB10 variant and settled on porting atop lkarlsund’s 6000 ( GitHub - lkarlslund/ninfer6000: High-performance single-GPU inference for selected model checkpoints and GPUs. · GitHub ) fork.
I present for you an alternative Qwen 3.8 Flash Next focused port of the Ninfer engine made for 5090s.
Starting with basic monkey patching of the Origin repo and some tweaking to align better to GB10’s SMs and other considerations, moving over to re-use lkarlsund’s setup and pre-work on q38fn made the work with the assistance of Anthropic Opus 5.5. Locally, Pi agent backed by Ninfer on the 5090 running on a Gigabyte GB10 working to deliver the smartest tweaks possible, cherry picking good ideas from a slew of other inference engines. My hope this spurs adoption of such ideas in vLLM or other more mainstream inference engines.
ninfer-gb10 runs Qwen3.8 Flash-Next 125B-A6B on a single DGX Spark. One user gets about 60 tokens per second. Four users get 121 tokens per second. The speed holds flat as prompts grow from 15,000 to 60,000 tokens.
On the same weights and the same hardware, it is 26% faster than the upstream fork at four concurrent requests. For long prompts, it generates output 25-35% faster than DGPP.
Every kernel is checked against an independent math oracle. Ideas that did not pay were measured and removed. Five validated fixes went back upstream. PRs welcome.