Is worth Running NVFP4 Quantization?

Accelerated Computing DGX Spark / GB10 User Forum DGX Spark / GB10
1

In the meantime Nvidia refresh/updates NVFP4 Quantization | DGX Spark recipe I’ve put a small recipe gist:52274dbd5c72648018acd89747983d5b · GitHub

I’ve tried various models to convert from, the biggest closest one was Qwen3 27B but need to check tensorRT llm errors i got, similar situation for Qwen3.6 27B. Will post more once have something reproducible as like the instructions from the gist.

Are you running Quantization in the GB10? Is it worth it? or more a learning exercise? I’m focusing on distilled or/and 30B parameter models then moving to eval then…

2

Hi, yes, quantizing on the GB10 is very powerful. However, if you want to preserve model quality, you can’t quantize every layer to NVFP4 , you need to do it carefully.

I successfully quantized a 27B mode in NVFP4 < here >.

If you’re interested, I also optimized the throughput using the DSpark paper, achieving around 18–20 tokens/s without any noticeable quality loss: < here >

3

Thanks for sharing. I’ll definitely look on the layers aspect, i’m still learning

4

Thank you for the links.. Certainly worth trying.
Could you share a complete recipe if possible?

5

Sure, dgx-spark-playbooks/nvidia/nvfp4-quantization/README.md at 05a1991fb5e093a59e6cf5b49a58ed55d559c6b4 · kristianpaul/dgx-spark-playbooks · GitHub

6

It is not easy to quantize a model. Some cloude providers are serving abysmal NVFP4 quantized GLM-5.2 instances, while Unsloth’s IQ1_S quantization comfortably beating them. The secret sauce is to know which layer to quantize how.