MedcellStudios/OLM3Nano

OLM3 Nano

OLM3 Nano is a small (~1B parameter) decoder-only causal language model, trained from scratch on the FineWeb corpus.

Model details

Architecture	Decoder-only Transformer with RoPE positional embeddings
Parameters	~1.02B
Hidden size	2048
Layers	16
Attention heads	16
Vocabulary size	50304
Max context length	2048 tokens
Positional encoding	Rotary (RoPE), θ = 10000
Normalization	RMSNorm
Weight tying	Input embeddings and output (LM head) are tied
Training data	FineWeb
Checkpoint step	14086

Tokenizer

This model was trained with the GPT-2 tokenizer (as used by tiktoken's "gpt2" encoding). Use GPT2TokenizerFast / AutoTokenizer from this repo, or tiktoken.get_encoding("gpt2") directly.

Usage

This model uses custom modeling code, so trust_remote_code=True is required.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "MedcellStudios/OLM3Nano"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.float32,
).to("cuda")
model.eval()

prompt = "Hello! How are you?"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to("cuda")

with torch.no_grad():
    output = model.generate(
        input_ids,
        max_new_tokens=80,
        do_sample=True,
        temperature=0.8,
        top_k=40,
        repetition_penalty=1.2,
        no_repeat_ngram_size=3,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0, input_ids.shape[1]:], skip_special_tokens=True))

Intended use and limitations

OLM3 Nano is a small research/hobby-scale language model. It is not instruction-tuned or aligned, and its outputs should not be treated as factual, safe, or suitable for production use without further fine-tuning and evaluation. Given its small parameter count and training scale, expect frequent repetition, factual errors, and limited reasoning ability compared to larger models.

Known issue: over-memorized personality section

After training and deploying this model on our website, we noticed that the model had over-memorized the personality section of its SFT data. As a result, some responses can be inconsistent — the model may repeat fixed personality-related phrasing verbatim rather than generating a natural, context-appropriate reply. We're aware of this and plan to address it in a future fine-tuning pass with more varied personality examples; in the meantime, treat personality-flavored outputs with some skepticism.

License

Apache 2.0.