Transformers documentation

GPTQ

Hugging Face's logo
Join the Hugging Face community

and get access to the augmented documentation experience

to get started

GPTQ

GPTQ quantizes a model one layer at a time and uses a small calibration dataset to update the remaining weights so they compensate for each layer’s rounding error. Transformers runs GPTQ through Optimum and the GPT-QModel library. Use it to quantize a model to 2 to 8 bits with calibration while it loads, or to load one of the many GPTQ checkpoints on the Hub.

The table below summarizes what GPTQ supports in Transformers.

ConfigGPTQConfig
quant_methodgptq
WorkflowQuantize while loading, with calibration
Bits2, 3, 4, 8
HardwareCUDA, XPU, CPU
Save✅
Fine-tune✅
Auto-compile❌

Install GPTQ

Install Optimum 1.24.0 or later and GPT-QModel 1.4.3 or later.

pip install -U "optimum>=1.24.0"
pip install -U "gptqmodel>=1.4.3" --no-build-isolation

Quantize a model with GPTQ

Create a GPTQConfig with a dataset to calibrate on, and pass it to from_pretrained(). The weights are quantized after they load. GPTQ recommends fp16, so the example below sets dtype=torch.float16.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, GPTQConfig

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B")
quantization_config = GPTQConfig(bits=4, dataset="c4", tokenizer=tokenizer)
model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-0.6B",
    quantization_config=quantization_config,
    dtype=torch.float16,
    device_map="auto",
)

dataset is required to quantize a model. Pass the name of a dataset from the GPTQ paper, "wikitext2", "c4", or "c4-new", or a list of strings. The tokenizer prepares the dataset. If you don’t pass one, GPTQ loads the tokenizer from the model’s repository.

Calibration runs the model block by block, so it takes much longer than loading. Before you quantize a model, check the Hub for an existing GPTQ checkpoint.

Load a GPTQ checkpoint

Pass the checkpoint to from_pretrained() without a GPTQConfig. Transformers reads the quantization settings from the checkpoint’s config.

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-1.5B-Instruct-GPTQ-Int4",
    device_map="auto",
)

To change the kernel or buffer size, pass a GPTQConfig with the new backend or max_input_length. Transformers overrides only these two loading attributes and keeps the rest of the checkpoint’s settings.

Configure GPTQ

These GPTQConfig parameters change how the weights are quantized. See GPTQConfig for the rest.

ParameterDefaultEffect
bitsrequiredNumber of bits per weight. Supports 2, 3, 4, and 8.
group_size128Number of weights that share a scale. -1 uses one scale per column.
damp_percent0.1Fraction of the average Hessian diagonal added for numerical stability. Must be between 0 and 1.
desc_actFalseQuantizes columns in order of decreasing activation size (act-order). Can improve accuracy and slow down inference. Setting it to True turns off act_group_aware.
symTrueUses symmetric quantization. Set to False for asymmetric quantization.
modules_in_block_to_quantizeNoneLists of module names in each block to quantize, in order. Linear layers not listed stay unquantized. With the default, every linear layer in the block is quantized. Requires Optimum 1.15.0 or later.

Choose a kernel backend

GPT-QModel runs the quantized layers through one of several kernels. The backend parameter picks one and defaults to "auto", which lets GPT-QModel choose a kernel for your hardware and checkpoint. The example below loads a checkpoint with the ExLlamaV2 kernel.

from gptqmodel import BACKEND
from transformers import AutoModelForCausalLM, GPTQConfig

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen2.5-1.5B-Instruct-GPTQ-Int4",
    quantization_config=GPTQConfig(bits=4, backend=BACKEND.EXLLAMA_V2),
    device_map="auto",
)

See the GPT-QModel backend list for the available values.

Limitations

  • GPTQ only supports text models. Loading a model whose main input isn’t input_ids raises a RuntimeError.
  • Quantizing a model with disk offloading in its device_map raises a ValueError.
  • Loading without GPT-QModel installed raises an ImportError. AutoGPTQ isn’t supported.
  • Without a device_map, the model loads on the CPU.

Resources

Update on GitHub