Transformers documentation

Fine-grained FP8

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

Fine-grained FP8

Fine-grained FP8 stores weights in FP8 (E4M3) with one scale per 2D block of weights, 128x128 by default, and quantizes activations to FP8 at runtime with one scale per token for each 128-value group of input channels. It’s the format DeepSeek-V3 and many other FP8 checkpoints on the Hub use. Use it to run FP8 checkpoints, or to quantize a model to FP8 while it loads, on Ada, Hopper, or Blackwell GPUs and on Intel XPUs.

The table below summarizes what Fine-grained FP8 supports in Transformers.

ConfigFineGrainedFP8Config
quant_methodfp8, mxfp8
WorkflowQuantize while loading, load pre-quantized
Bits8
HardwareCUDA (compute capability 8.9+), XPU
Save✅
Fine-tune✅
Auto-compile✅

Install the Fine-grained FP8 kernels

Transformers downloads the FP8 matmul kernel from the kernels-community/finegrained-fp8 repository on the Hub. Install the kernels and Accelerate libraries.

pip install -U "transformers[kernels]" accelerate

Quantize a model with Fine-grained FP8

Create a FineGrainedFP8Config and pass it to from_pretrained().

from transformers import AutoModelForCausalLM, FineGrainedFP8Config

quantization_config = FineGrainedFP8Config()
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-8B",
    quantization_config=quantization_config,
    device_map="cuda",
)

Configure Fine-grained FP8

These FineGrainedFP8Config parameters change how the weights and activations are quantized.

ParameterDefaultEffect
weight_block_size(128, 128)Size of the weight block that shares one scale. Set it to None for one scale per weight tensor.
activation_scheme"dynamic""dynamic" computes activation scales at runtime. "static" uses one stored activation scale per layer, read from a pre-quantized checkpoint.
scale_fmt"float"Storage format of the weight scales. "float" stores float32 scales. "ue8m0" stores 1-byte power-of-two scales (torch.float8_e8m0fnu), as in DeepSeek-V4 checkpoints, and needs PyTorch 2.7 or later.
expert_dtype"fp8"Format of the mixture-of-experts weights. "fp4" loads the FP4-packed experts in DeepSeek-V4 checkpoints.
modules_to_convertNoneEmbedding tables to load in FP8 from a pre-quantized checkpoint, matched by module name suffix.

Load a pre-quantized FP8 checkpoint

Checkpoints quantized with Fine-grained FP8 carry their quantization config, so load them without a FineGrainedFP8Config.

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3-8B-FP8", device_map="cuda")

Fine-grained FP8 also loads MXFP8 checkpoints, whose config sets quant_method="mxfp8". MXFP8 stores FP8 weights with one power-of-two scale, saved as a byte, per block of 32 weights.

Use the DeepGEMM fast path

On Hopper (SM90) and Blackwell (SM100) GPUs, FP8 linear layers dispatch to the DeepGEMM kernels from kernels-community/deep-gemm when weight_block_size=(128, 128) and activation_scheme="dynamic". DeepGEMM is faster than the default Triton kernel on the shapes it supports. The kernels install above enables it.

DeepGEMM compiles its kernels with the system nvcc the first time it runs, so you need a full CUDA toolkit, not a runtime-only install. Hopper needs CUDA 12.3 or later and Blackwell needs CUDA 12.9 or later. Transformers looks for the toolkit in CUDA_HOME, then CUDA_PATH, then the directory of nvcc on your PATH, then /usr/local/cuda. Set CUDA_HOME to the toolkit root if Transformers doesn’t find it.

If DeepGEMM can’t load or declines a layer, Transformers logs a warning and falls back to the Triton kernel. Set TRANSFORMERS_DISABLE_DEEPGEMM_LINEAR=1 to always use the Triton kernel for linear layers.

On Blackwell, DeepGEMM only reads UE8M0 scales. Linear layers with float32 scales (scale_fmt="float") use the Triton kernel instead.

For mixture-of-experts layers, DeepGEMM is opt-in. Pass experts_implementation="deepgemm" to from_pretrained(), or "deepgemm_megamoe" on Blackwell. See the Experts backends guide for every option.

Limitations

  • Quantizing while loading needs a CUDA GPU or an XPU. Without one, loading raises a RuntimeError. A pre-quantized checkpoint loads without an accelerator, but Transformers dequantizes it to bf16 and logs a warning.
  • On CUDA, FP8 compute needs compute capability 8.9 or higher, like an RTX 4090 or H100. On an older GPU, Transformers dequantizes the model to bf16 and logs a warning.
  • Quantizing while loading can’t offload to the CPU or disk. A device_map that includes "cpu" or "disk" alongside other devices raises a ValueError. A pre-quantized checkpoint can offload.
  • DeepGEMM is turned off when one process spreads the model across several CUDA devices, and FP8 layers use the Triton kernel instead. Selecting experts_implementation="deepgemm" in that setup raises a RuntimeError. Run tensor or expert parallelism, with one device per process, to keep DeepGEMM.
  • On Blackwell, the "deepgemm" experts backend raises a NotImplementedError for float32 scales. Use a scale_fmt="ue8m0" checkpoint, or the "grouped_mm" or "batched_mm" experts backends.
  • The "grouped_mm" and "batched_mm" experts backends don’t support activation_scheme="static" and raise a NotImplementedError.
  • scale_fmt="ue8m0" needs PyTorch 2.7 or later. Older versions raise a RuntimeError.

Resources

Update on GitHub