Hi everyone, we’ve seen a lot of questions about optimizing vLLM’s inference performance using NVFP4, and GB10’s underlying NVFP4 capability more generally. We’ll answer some of those questions here.
First, we’re working to improve NVFP4 performance in vLLM on several fronts.
CUTLASS back end improvements. We are regularly contributing fixes and changes to FlashInfer and vLLM upstream. Over the past couple of weeks, FlashInfer and CUTLASS have significantly improved the CUTLASS backend for NVFP4. These changes are now in FlashInfer 0.6.8.
Optimized Kernels for SM12.x. Luke Alonso recently published an optimized set of MoE and GEMM kernels for SM 120/121 using cuteDSL/TileIR. These have been integrated into FlashInfer top-of-tree, and a PR for integration in vLLM is currently open — we are working to land that as quickly as possible given strong community interest.
We’re also working on Multi-Token Prediction (MTP), to further improve performance. We’ll share more updates on this soon.
Once complete, these improvements will be available both in NVIDIA-provided containers and upstream frameworks, ensuring developers can benefit regardless of deployment path. We will continue to share updates as these enhancements are rolled out.
Note that performance improvements with NVFP4 vary by model and workload. While reduced precision enables higher throughput, actual speedup depends on factors such as model architecture, kernel utilization, and framework behavior.
There have been some questions about GB10’s NVFP4 instructions. The SM 12.x used in GB10 has 5th Generation Blackwell Tensor Cores, with support for block-scaled fp4 (NVFP4) at the hardware level. This capability is expressed in the instruction set differently across SM 12.x and SM 10.x.
- SM 10.x uses the tcgen05.* instruction family for its 5th Gen operations
- SM 12.x uses the mma.* instruction family — the same naming lineage as earlier generations — because it carries forward ISA compatibility with prior architectures while adding 5th Gen capabilities on top
Thus, the difference in PTX instruction naming reflects ISA compatibility conventions, not generational capability.. While the instruction sets will remain different, we are committed to delivering the best possible performance of the hardware capabilities with SM12.x.
Finally, thanks for all of your feedback on this topic. While we can’t respond to every comment, we read it all, and your feedback directly helps us improve NVFP4 performance and usability. We’ll share more updates here as they become available.