Transformers

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.22.2).

Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

This model was contributed to Hugging Face Transformers on 2026-11-12.

InternLM3

InternLM3 is an 8B instruction-tuned language model developed by Shanghai AI Laboratory. Its architecture is the same as Llama. Its config names the biases of the query, key and value projections qkv_bias and those of the attention output projection and MLP layers bias; InternLM3Config loads them as attention_bias and mlp_bias, and requires them to be equal, as they are in the released checkpoints.

The auto classes load it with the Llama implementation (LlamaModel and LlamaForCausalLM), so only InternLM3Config is specific to InternLM3.

The example below demonstrates how to generate text with the AutoModelForCausalLM class.

from transformers import AutoModelForCausalLM, AutoTokenizer


tokenizer = AutoTokenizer.from_pretrained("internlm/internlm3-8b-instruct")
model = AutoModelForCausalLM.from_pretrained("internlm/internlm3-8b-instruct", device_map="auto")

messages = [{"role": "user", "content": "Explain the Pythagorean theorem in one sentence."}]
inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True)
output = model.generate(**inputs.to(model.device), max_new_tokens=64)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1] :], skip_special_tokens=True))

InternLM3Config

class transformers.InternLM3Config

< source >

( transformers_version: str | None = Nonearchitectures: list[str] | None = Noneoutput_hidden_states: bool | None = Falsereturn_dict: bool | None = Truedtype: str | torch.dtype | None = Nonechunk_size_feed_forward: int = 0is_encoder_decoder: bool = Falseid2label: dict[int, str] | dict[str, str] | None = Nonelabel2id: dict[str, int] | dict[str, str] | None = Noneproblem_type: Literal['regression', 'single_label_classification', 'multi_label_classification'] | None = Nonevocab_size: int = 128512hidden_size: int = 4096intermediate_size: int = 10240num_hidden_layers: int = 48num_attention_heads: int = 32num_key_value_heads: int | None = 2hidden_act: str = 'silu'max_position_embeddings: int = 32768initializer_range: float = 0.02rms_norm_eps: float = 1e-05use_cache: bool = Truepad_token_id: int | None = 2bos_token_id: int | None = 1eos_token_id: int | list[int] | None = 2tie_word_embeddings: bool = Falserope_parameters: transformers.modeling_rope_utils.RopeParameters | dict | None = Noneattention_bias: bool = Falseattention_dropout: float | int = 0.0mlp_bias: bool = Falsehead_dim: int | None = 128 )

Parameters

vocab_size (int, optional, defaults to 128512) — Vocabulary size of the model. Defines the number of different tokens that can be represented by the input_ids.
hidden_size (int, optional, defaults to 4096) — Dimension of the hidden representations.
intermediate_size (int, optional, defaults to 10240) — Dimension of the MLP representations.
num_hidden_layers (int, optional, defaults to 48) — Number of hidden layers in the Transformer decoder.
num_attention_heads (int, optional, defaults to 32) — Number of attention heads for each attention layer in the Transformer decoder.
num_key_value_heads (int, optional, defaults to 2) — This is the number of key_value heads that should be used to implement Grouped Query Attention. If num_key_value_heads=num_attention_heads, the model will use Multi Head Attention (MHA), if num_key_value_heads=1 the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out this paper. If it is not specified, will default to num_attention_heads.
hidden_act (str, optional, defaults to silu) — The non-linear activation function (function or string) in the decoder. For example, "gelu", "relu", "silu", etc.
max_position_embeddings (int, optional, defaults to 32768) — The maximum sequence length that this model might ever be used with.
initializer_range (float, optional, defaults to 0.02) — The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
rms_norm_eps (float, optional, defaults to 1e-05) — The epsilon used by the rms normalization layers.
use_cache (bool, optional, defaults to True) — Whether or not the model should return the last key/values attentions (not used by all models). Only relevant if config.is_decoder=True or when the model is a decoder-only generative model.
pad_token_id (int, optional, defaults to 2) — Token id used for padding in the vocabulary.
bos_token_id (int, optional, defaults to 1) — Token id used for beginning-of-stream in the vocabulary.
eos_token_id (Union[int, list[int]], optional, defaults to 2) — Token id used for end-of-stream in the vocabulary.
tie_word_embeddings (bool, optional, defaults to False) — Whether to tie weight embeddings according to model’s tied_weights_keys mapping.
rope_parameters (Union[~modeling_rope_utils.RopeParameters, dict], optional) — Dictionary containing the configuration parameters for the RoPE embeddings. The dictionary should contain a value for rope_theta and optionally parameters used for scaling in case you want to use RoPE with longer max_position_embeddings.
attention_bias (bool, optional, defaults to False) — Whether to use a bias in the query, key, value and output projection layers during self-attention.
attention_dropout (Union[float, int], optional, defaults to 0.0) — The dropout ratio for the attention probabilities.
mlp_bias (bool, optional, defaults to False) — Whether to use a bias in up_proj, down_proj and gate_proj layers in the MLP layers.
head_dim (int, optional, defaults to 128) — The attention head dimension. If None, it will default to hidden_size // num_attention_heads

This is the configuration class to store the configuration of a LlamaModel. It is used to instantiate a Internlm3 model according to the specified arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of the internlm/internlm3-8b-instruct

Configuration objects inherit from PreTrainedConfig and can be used to control the model outputs. Read the documentation from PreTrainedConfig for more information.

>>> from transformers import AutoModelForCausalLM, InternLM3Config

>>> # Initializing an InternLM3-8B style configuration
>>> configuration = InternLM3Config()

>>> # Initializing a model (which uses the Llama implementation) from that configuration
>>> model = AutoModelForCausalLM.from_config(configuration)

>>> # Accessing the model configuration
>>> configuration = model.config

Update on GitHub

←I-BERT IQuestCoder→