This has been a very interesting thread to follow along. I have downloaded and run both the OP’s dgx-vllm and eugr’s excellent spark-vllm-docker. However, while I can see the memory advantage, I am not really seeing a real world performance improvements. I have a single DGX Spark and using a Qwen3-Coder-Next-FP8 recipe I am getting usable performance and accuracy with my existing Opencode toolchain to run documentation, integration test writing and codebase research.
Like many others I bought into the NVFP4 marketing hype when I purchase my Spark and experienced waves of buyers remorse as I realised it was much more bleeding edge than I was expecting. However I have experienced this before with other technology lifecycles and I am really appreciative of all the work this community is putting into fulfilling the promise of this device. I too think its a bit of an iphone like watershed moment, with the backing of multiple vendors, and the right software stack it really has potential.
So far I have tried
- GadflyII/Qwen3-Coder-Next-NVFP4
- nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4
- nvidia/Qwen3-Next-80B-A3B-Instruct-NVFP4
In an OpenWebUI chat window they all seem to work. From my personal experience with writing tasks Qwen3 Next 80B is a step up in sophistication with the quality of its responses. Not as good as Kimi K2.5 but usable for many tasks. The avorok Qwen3-Next-80B-A3B-Instruct-NVFP4 outlined here is giving me 50 ish tokens a second which feels performant enough for a self hosted model running on your desktop and consuming less than 100w.
However if I add --enable-auto-tool-choice --tool-call-parser <parser> to vllm and run simple tool calls with Opencode the “marginal” degradation in performance is noticeable. None of the models feel as responsive as Qwen3-Coder-Next-FP8 and vary from error-prone to unusable.
Reading comments like the following leaves me all the more confused.
flash3 – If you are confused too - it seems that Avarok has solved a problem that does not exist anymore (software fallback for e2m1), only exists in older versions of CUTLASS and is “solved” in newer ones by just using cpu instruction sets that the compiler does not want you to use if you haven’t said the magic word “family”.
along with…
uger – Just in case you haven’t seen the other thread, this doesn’t require any special builds and works with any recent vLLM build, including our community Docker out of the box.
I guess my question is: Where do you think all of this heading? Will NVFP4 get more performant and accurate with time as the software stack catches up and the models are tuned to this quantisation – or is the accuracy likely to remain how it is and all we can expect is a speed improvement or just less memory overhead? I ask because this feels like a huge rabbit hole to go down and I have a setup that while not what I was expecting is adequate. This community clearly has a wealth of expertise – so what future improvements do you think we should be expecting from this work? What contributions are helpful from the less technical or newer members?
Thanks