PEFT documentation
Helper methods
Helper methods
A collection of helper functions for PEFT.
Checking if a model is a PEFT model
peft.helpers.check_if_peft_model
< source >( model_name_or_path: str ) → bool
Check if the model is a PEFT model.
Temporarily Rescaling Adapter Scale in LoraLayer Modules
peft.helpers.rescale_adapter_scale
< source >( model: Modulemultiplier: float | int )
Parameters
- model — The model containing
LoraLayermodules whose scaling is to be adjusted. - multiplier (float or int) —
The multiplier that rescales the
scalingattribute. Must be of type float or int.
Raises
ValueError
ValueError— If the model does not contain anyLoraLayerinstances, indicating that the model does not support scaling.
Context manager to temporarily rescale the scaling of the LoRA adapter in a model.
The original scaling values are restored when the context manager exits. This context manager works with the transformers and diffusers models that have directly loaded LoRA adapters.
For LoRA, applying this context manager with multiplier in [0, 1] is strictly equivalent to applying wise-ft (see #1940 for details). It can improve the performances of the model if there is a distribution shift between the training data used for fine-tuning, and the test data used during inference.
Warning: It has been reported that when using Apple’s MPS backend for PyTorch, it is necessary to add a short sleep time after exiting the context before the scales are fully restored.
Context manager to disable input dtype casting in the forward method of LoRA layers
peft.helpers.disable_input_dtype_casting
< source >( model: Moduleactive: bool = True )
Context manager disables input dtype casting to the dtype of the weight.
Context manager to enable DoRA caching (faster at inference time but requires more memory)
Context manager to enable DoRA caching, which improves speed of DoRA inference at the expense of memory.
With active caching, the materialized LoRA weight (B @ A) and the weight norm (base weight + LoRA weight) are cached.
Even within the caching context, if the model is in training mode, caching is disabled. When the model switches to training mode, the cache will be cleared.
Example:
>>> from peft.helpers import DoraCaching
>>> model.eval() # put in eval model for caching to work
>>> with DoraCaching(): # use as a context manager
... output = model(inputs)
>>> dora_caching = DoraCaching()
>>> dora_caching(enabled=True) # permanently enable caching
>>> output = model(inputs)
>>> dora_caching(enabled=False) # permanently disable caching
>>> output = model(inputs)Creating a dummy LoRA configuration for hotswapping
See starting with a dummy adapter for the workflow and command-line script.
peft.helpers.create_dummy_lora_config
< source >( configs: Sequence ) → LoraConfig
Parameters
- configs (
Sequence[LoraConfig]) — Nonempty sequence of compatible LoRA configurations for the same base model. Each must have explicit, nonemptytarget_modules; strings (including regexes and"all-linear") and inferred targets are not supported. Standard LoRA and rsLoRA are supported, without extra saved modules, biases, or parameters.
Returns
LoraConfig
A configuration with zero-output initialization and a uniform maximum rank.
Raises
TypeError or ValueError
TypeError— If any configuration is not a LoRA configuration.ValueError— If no configurations are supplied, or their settings are unsupported or incompatible.
Create a no-op LoRA configuration that reserves capacity for hotswapping.
The purpose of this helper function is to create a dummy LoRA config which can be used to initialze a LoRA model before hotswapping different LoRA adapters (see https://huggingface.co/docs/peft/main/en/package_reference/hotswap). One issue with hotswapping is that the first LoRA adapter that is loaded needs to install LoRA weights on all layers that the later, hotswapped LoRAs will target. If these LoRAs target disjunct layers, none of them can serve as the first LoRA. This helper function creates a dummy LoRA that targets the union of the layers of all the individual LoRAs. It also pre-allocates LoRA weights with a sufficiently large rank.
The generated LoRA config is not intended to be used for further training.
No base model or adapter weights are loaded. Layer filters and exclusions are removed to cover every adapter, which can over-allocate LoRA weights on additional layers.
Instantiate the result with get_peft_model, then call prepare_model_for_compiled_hotswap before compiling. The
dummy adapter is initialized such that it is a no-op, i.e. it does not need to be disabled to run inference on the
base model.
Note that some LoraConfig arguments are not supported and will result in an error. This is because it would not
be possible to combine those configs into one dummy config.
There is a helper script in the PEFT repository to create the dummy LoRA config from the command line:
python scripts/create-dummy-lora-for-hotswap.py adapter0 adapter1 [...] --output-dir dummy-lora
KappaTune target selection
KappaTuneSelector and find_kappa_target_modules implement a general target selection process from the KappaTune paper.
The method identifies modules with higher flexibility (higher output differential entropy) and lower specialization (lower sensitivity to specific input directions).
These properties make the selected modules good candidates for mitigating catastrophic forgetting in any adaptation method that adds trainable parameters, including LoRA, DoRA, LoHa, AdaLoRA, and even direct fine-tuning of the original weights.
class peft.helpers.KappaTuneSelector
< source >( model: Modulemax_dim_size_to_analyze: int = 16384moe_param_suffixes: tuple[str, ...] | None = Noneshow_progress: bool = True )
Lightweight utility to compute per-module / per-parameter condition numbers and return the best LoRA targets.
Supports:
- Classic nn.Linear modules (target_modules in LoraConfig)
- Modern fused MoE weights stored as 3D nn.Parameter (gate_up_proj / down_proj, gate_proj / up_proj, etc.) used in Llama-4, Qwen2_MoE, Qwen3_MoE, Mixtral, OLMoE and similar models. These are returned via target_parameters.
Notes:
- Condition-number computation requires running SVD and can take several minutes on very large models. A progress
bar can be shown/disabled via
show_progress.
peft.find_kappa_target_modules
< source >( model: Moduletop_p: float = 0.2max_dim_size_to_analyze: int = 16384moe_param_suffixes: tuple[str, ...] | None = Noneshow_progress: bool = True )
Parameters
- model (nn.Module) — Base model whose weights will be analyzed for condition numbers.
- top_p (float, optional) — Select the top fraction of candidate modules/parameters with the lowest condition numbers.
- max_dim_size_to_analyze (int, optional) — Upper bound on the maximum matrix dimension analyzed via SVD. Defaults to 16384.
- moe_param_suffixes (Optional[tuple[str, …]], optional) —
Parameter-name suffixes used to identify fused MoE tensors that should be returned via
target_parameters. If None, sensible defaults are used. - show_progress (bool, optional) — Whether to display a progress bar while computing condition numbers (SVD-based) across candidate tensors/modules. Disable in CI or other non-interactive environments. Defaults to True.
One-liner convenience function for KappaTune target selection. Returns both target_modules and target_parameters.