Lama.cpp ggml - nvfp4 quantization type support might arrive soon

chrm · March 8, 2026, 9:55pm

Hi,

Maybe, interesting and relevant for someone

github.com/ggml-org/llama.cpp

ggml : add NVFP4 quantization type support (#19769)

master ← richarddd:feat/nvfp4

opened 09:50PM - 20 Feb 26 UTC

richarddd

+704 -51

I'm not super experienced with the ggml/gguf internals so feedback is very welco…me. Note on AI usage: Claude Opus 4.6 was used for navigating the codebase, debugging, and writing parts of the code. All changes have been reviewed and tested manually. Open to reworking anything that doesn't meet the project's standards. This adds support for NVIDIA's NVFP4 quantization format (FP4 E2M1 weights, UE4M3 per-block scale, 16 elements per block). This is the format produced by NVIDIA ModelOpt's NVFP4 algo. The main difference is the scale encoding (UE4M3 vs E8M0). What's in here: - New GGML_TYPE_NVFP4 type, block struct, UE4M3 conversion helpers, reference quantize/dequantize - convert_hf_to_gguf.py detects NVFP4 ModelOpt models and repacks into the GGUF block format - CPU backend: scalar dot product + ARM NEON - gguf-py: type constant, quant/dequant, endian conversion - Tests added to test-backend-ops and test-quantize-fns Tested with models from https://huggingface.co/NVFP4 Apple M5 MacBook (CPU, NEON) Ran llama-bench and a basic server smoke test. Would appreciate help with that if someone has a good baseline to compare against. Here is a [Qwen3-4B](https://huggingface.co/richarddavison/Qwen3-4B-NVFP4-GGUF) model to test with.

I am looking forward to test this.

Kidtronic · March 9, 2026, 11:58am

Great find… looks like Georgi approved code this morning!

I have also really been looking forward to this.

Topic		Replies	Views
GLM4.6V NVFP4 existing? DGX Spark / GB10	7	450	January 22, 2026
Llama.cpp - NVFP4 native support on Blackwell DGX Spark / GB10 llama	1	605	April 29, 2026
PSA: State of FP4/NVFP4 Support for DGX Spark in VLLM DGX Spark / GB10	232	11245	May 3, 2026
Your GPU does not have native support for FP4 computation but FP4 quantization is being used DGX Spark / GB10	5	1481	January 30, 2026
Marlin Fix: NVFP4 Actually Works on SM121 (DGX Spark) DGX Spark / GB10 Projects jetson , nemotron	16	2038	April 26, 2026
NVFP4 quantization on the GP10 error DGX Spark / GB10 llama , deepseek	3	380	November 14, 2025
Introducing NVFP4 for Efficient and Accurate Low-Precision Inference Technical Blog	4	495	January 26, 2026
New bleeding-edge vLLM Docker Image: avarok/vllm-nvfp4-gb10-sm120 DGX Spark / GB10 Projects	35	3015	December 31, 2025
Custom FP4 CUDA Kernel - 129 TFLOPS on DGX Spark with Pre-Quantized Weight Cache CUDA Programming and Performance cublas	4	572	February 25, 2026
What's the best speed we can get with Qwen 3.6 27B without quantizing? DGX Spark / GB10	26	10061	April 29, 2026

Lama.cpp ggml - nvfp4 quantization type support might arrive soon

Related topics