Transformers documentation
NVFP4
NVFP4
NVFP4 is NVIDIA’s 4-bit floating-point format. Each block of 16 weights shares an FP8 scale, and each tensor has an FP32 scale on top. Transformers quantizes the weights while the model loads and quantizes activations on the fly, so prefill runs as a W4A4 matmul on Blackwell tensor cores and decode runs as a W4A16 matmul. Use it to run 4-bit models on a Blackwell GPU.
The table below summarizes what NVFP4 supports in Transformers.
| Config | NVFP4Config |
quant_method | nvfp4 |
| Workflow | Quantize while loading |
| Bits | 4 |
| Hardware | One CUDA GPU with compute capability 10.0+ (Blackwell) |
| Save | ❌ |
| Fine-tune | ❌ |
| Auto-compile | ✅ |
Install the NVFP4 kernels
Transformers downloads the NVFP4 kernels from the kernels-community/nvfp4-gemm repository on the Hub. Install the kernels library. The kernel needs a build that matches your PyTorch and CUDA versions.
pip install -U "transformers[kernels]"Quantize a model with NVFP4
Create a NVFP4Config and pass it to from_pretrained() with a device_map that places the whole model on one CUDA GPU. The source checkpoint must have floating-point weights.
from transformers import AutoModelForCausalLM, NVFP4Config
quantization_config = NVFP4Config()
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen3-8B",
quantization_config=quantization_config,
device_map="cuda",
)NVFP4 replaces each torch.nn.Linear layer that has no bias and whose in_features and out_features are both divisible by 16. Other linear layers keep their original precision. NVFP4Config has no parameters that change how the weights are quantized.
Limitations
- NVFP4 needs a CUDA GPU with compute capability 10.0 or later. Loading without a CUDA GPU or on an older GPU raises a
RuntimeError. - NVFP4 runs on one GPU. A
device_mapthat spans several GPUs, or tensor parallelism, raises aValueError. - NVFP4 can’t offload to the CPU or disk. A
device_mapthat includes"cpu"or"disk", or nodevice_mapat all, raises aValueError. - NVFP4 can’t load pre-quantized NVFP4 checkpoints. Loading one raises a
ValueError. - NVFP4 models can’t be saved or fine-tuned.
Resources
- Introducing NVFP4 for Efficient and Accurate Low-Precision Inference blog post
- kernels-community/nvfp4-gemm Hub kernel