💫🧮 Luciole KD Math
Collection
Knowledge Distillation versions of Luciole LLM tuned for math • 7 items • Updated • 1
Model Card for Luciole-8B-KD-Math-Base
- Model Description
- Uses
- Bias, Risks, and Limitations
- Example Code in Python
- Loading Intermediate Checkpoints
- Training Details
- Citation
- Acknowledgements
- License
- Contact
Model Description
Luciole-8B-KD-Math-Base is a causal language model developed by LINAGORA and the OpenLLM-France consortium as a part of the OpenLLM France project, funded by BPI France through the France 2030 program. It was trained on the GENCI supercomputer Jean Zay, managed by IDRIS.
Luciole-8B-KD-Math-Base was trained through knowledge distillation (KD) following the Minitron approach (Sreenivas et al., 2024. LLM Pruning and Distillation in Practice: The Minitron Approach). We began by continually pretraining Luciole-23B-Base on a math-heavy mix of around 50 billion tokens to create a teacher for knowledge distillation. The teacher was then pruned to a roughly 14 billion parameter model that was subsequently trained through knowledge distillation on around 112 tokens of the mix used for correction. A second stage of pruning and distillation was applied, still using the corrected 23B model as the teacher, to create Luciole-8B-KD-Math-Base.
This goal of this project was to see whether we could reuse Luciole-23B-Base to create a smaller model that improves on Luciole-8B-Base in mathematics and maintains as much of its general knowledge as possible, without requiring full pretraining.
- Developed by: LINAGORA
- Language(s) (NLP): Luciole-8B-KD-Math has been evaluated on English and French. It may also contains capacities in other languages from the Luciole-23B-Base training dataset.
- License: Apache 2.0
Uses
Direct Use
Luciole-8B-KD-Math-Base is a foundation language model trained solely to predict the most probable next word in a sequence. It is designed as the first brick in a more complex training pipeline that would include multitask training on diverse instructions or focused fine-tuning on select downstream tasks, as well as possible alignment for human preferences.
Downstream Use
While the teacher correction mix used for distillation of Luciole-8B-KD-Math-Base contains mostly English data, Luciole-23-Base underwent multilingual training. Luciole-8B-KD-Math-Base can thus be fine-tuned for downstream tasks centered on the generation of multilingual text, with a special focus on French and English.
Out-of-Scope Use
Luciole-8B-KD-Math-Base is not intended to generate text directly for end use cases. It must be fine-tuned first. Its pretraining is optimized for multilingual performance, especially in French and English, and might perform less well on other languages without additional training. While trained on a much higher proportion of code data than Luciole-23-Base, it is still not optimized for code generation tasks.
Bias, Risks, and Limitations
Luciole-23-Base was trained on large amounts of web data. Additionally, due to the scarcity of French textual non-web data published under open licenses, much of the French training data came from older works in the public domain that carry biases from other time periods. While we made efforts to reduce toxic and offensive content in the Luciole Training Dataset, Luciole-8B-KD-Math-Base may still generate such content due to its close relation to Luciole-23-Base. Filtering of the Luciole Training Dataset is an ongoing project to which we welcome contributions.
Recommendations
To limit the generation of undesirable content, it is advised to fine-tune Luciole-8B-KD-Math-Base through instruction and preference tuning (DPO, RLHF, etc.).
Example Code in Python
Load the model
Load the model (quantized version on GPU if possible, for efficient inference):
import transformers
model_name = "OpenLLM-France/Luciole-8B-KD-Math-Base"
tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)
model = transformers.AutoModelForCausalLM.from_pretrained(model_name,
device_map="auto",
load_in_4bit=True # For efficient inference, if quantization is supported by the GPU card
)
Sentence completion
Wrap the model in a text generation pipeline, and specify some generation parameters:
pipeline = transformers.pipeline("text-generation", model=model, tokenizer=tokenizer)
generation_kwargs = dict(
num_return_sequences=1, # Number of variants to generate.
return_full_text= False, # Do not include the prompt in the generated text.
do_sample=True,
temperature=1.0, top_p=1, top_k=None, # Sampling parameters.
max_new_tokens=200, # Maximum length for the output text (in number of tokens).
)
Try 1-shot question answering:
prompt = """\
Quelle est la capitale de l'Espagne ? Madrid\n\
Quelle est la capitale de la France ?\
"""
completions = pipeline(prompt, **generation_kwargs)
for completion in completions:
print(prompt + "[…]" + completion['generated_text'])
This will print something like:
Quelle est la capitale de l'Espagne ? Madrid
Quelle est la capitale de la France ?[…] Paris
Quelle est la capitale du Brésil ? Brasilia
Quelle est la capitale de la Belgique ? Bruxelles
Quelle est la capitale de l'Italie ? Rome
...
Training Details
Training Data
Luciole-8B-KD-Math-Base was trained on a math-focused mixture built from subsets of OpenLLM-France/Luciole-Training-Dataset and OpenLLM-France/Luciole-PostTraining-Dataset-1.1, together with open datasets from NVIDIA's Nemotron collections.
| Domain | Luciole datasets | NVIDIA Nemotron |
|---|---|---|
| Math | FineMath-4plus, Nemotron-Post-Training-Dataset-v2 (math) | Nemotron-Math-v2, Nemotron-SFT-Math-v3 |
| General | FineWeb-Edu (en), Wikipedia (en), FineWeb2-HQ (fr) | – |
| Code | OpenCodeReasoning, Nemotron-Post-Training-Dataset-v2 (code) | Nemotron-Competitive-Programming-v1 |
| Science | Common Pile PubMed, OpenThoughts3 (science), Nemotron-Science-v1 (MCQ)¹ | Nemotron-Science-v1 (RQA) |
| Instruction | – | Nemotron-SFT-Instruction-Following-Chat-v2 (reasoning on/off) |
| STEM | Common Pile arXiv papers, Nemotron-Post-Training-Dataset-v2 (STEM) | – |
¹ From Luciole-PostTraining-Dataset-1.1; all other Luciole subsets are from Luciole-Training-Dataset.
The data was cleaned before training (length filtering and deduplication). The final proportions of the training mixture are:
| Category | Share |
|---|---|
| Math | 42.94% |
| General | 26.73% |
| Code | 19.40% |
| Science | 5.20% |
| Instruction | 3.07% |
| STEM | 2.66% |
Training consisted of three phases, all with a context length of 32,768 tokens:
- Teacher correction: 50 billion tokens of continued pretraining of Luciole-23B-Base
- Stage 1: pruning 23B → 14B, then 112 billion tokens of distillation
- Stage 2: pruning 14B → 8B, then 112 billion tokens of distillation
Training Procedure
Luciole-8B-KD-Math-Base is a causal decoder-only model. It was obtained from the math-corrected Luciole-23B with two rounds of structured pruning and knowledge distillation, following the Minitron approach. The corrected 23B model is the teacher in both stages.
Pruning. Pruning used the mcore_minitron mode of NVIDIA Model Optimizer 0.42.0, calibrated on the math split of Nemotron-Post-Training-Dataset-v2. At each stage, the architecture was chosen among candidates that fit the parameter budget, based on short distillation runs on a shortlist of candidates.
Distillation. The student was trained with forward KL divergence between the teacher and student token distributions (logit distillation only, with no cross-entropy loss on ground-truth tokens).
Training ran on 128 H100 80GB GPUs (32 nodes) for about 30,500 GPU hours on the Jean Zay supercomputer.
Teacher correction used NVIDIA's NeMo framework version 2.3.1, the same stack used to pretrain Luciole. Pruning and distillation used NVIDIA Model Optimizer 0.42.0 with Megatron-Bridge 0.3.1.
Neural Network Architecture
Luciole-8B-KD-Math-Base keeps the decoder-only architecture of Luciole-23B-Base, with reduced depth, hidden size, and FFN width. It has 8.36 billion parameters, with the following hyperparameters:
| Hyperparameter | Value |
|---|---|
| Vocabulary size (# tokens) | 128,000 |
| # layers | 28 |
| # attention heads | 48 |
| # key-value heads | 8 |
| Head dimension | 128 |
| Hidden size | 3,840 |
| Intermediate size | 27,136 |
| MLP Activation | relu2 |
| RoPE base | 2,000,000 |
Training Hyperparameters
The details of the teacher-correction phase are listed below. For the distillation phases, only the values that differ from teacher correction are listed. Stages 1 and 2 use exactly the same recipe and differ only in the student initialization.
1. Teacher correction (continued pretraining of Luciole-23B-Base)
| Hyperparameter | Value |
|---|---|
| Initialization | Luciole-23B-Base (context-extension checkpoint) |
| Total # samples | 1,525,888 (50B tokens) |
| Total # steps | 11,921 |
| Context length | 32,768 |
| Batch size | 128 |
| Learning rate schedule | Warmup + Cosine annealing |
| Maximum Learning rate | 6e-5 |
| Final Learning rate | 0 |
| Weight decay | 0.1 |
| Precision | Hybrid FP8 (tensorwise), with the first and last 4 layers in bfloat16 |
| Tensor Parallelism (with 128 GPUs) | 4 |
| Pipeline Parallelism (with 128 GPUs) | 2 |
| Context Parallelism (with 128 GPUs) | 2 |
| Data Parallelism (with 128 GPUs) | 8 |
2. Stage 1 distillation (23B → 14B) and 3. Stage 2 distillation (14B → 8B)
| Hyperparameter | Value |
|---|---|
| Student initialization | Stage 1: 14B model pruned from the corrected 23B; Stage 2: 8B model pruned from the distilled 14B |
| Teacher | Corrected Luciole-23B (both stages) |
| Loss | Forward KL divergence on logits |
| Total # samples | 3,417,984 (112B tokens) per stage |
| Total # steps | 26,703 per stage |
| Learning rate schedule | Warmup + Cosine annealing |
| Maximum Learning rate | 1e-4 |
| Final Learning rate | 1e-5 |
| Gradient clipping | 1 |
| Optimizer | AdamW (β₁=0.9, β₂=0.95, ε=1e-8) |
| Precision | bfloat16 |
Training Convergence and Evaluation
Training loss
Training loss curves for each phase are released at
metadata/training_logs
├── ConvergenceCurve_phase1-teacher-correction.csv -- training logs for phase 1 (teacher correction of the 23B model)
├── ConvergenceCurve_phase2-distillation-14b.csv -- training logs for phase 2 (distillation 23B → 14B)
└── ConvergenceCurve_phase3-distillation-8b.csv -- training logs for phase 3 (distillation 14B → 8B)
Training loss for each phase (cross-entropy for teacher correction, KL divergence for distillation):
Evaluation
Luciole-8B-KD-Math-Base breaks away from Luciole-8B-Base, Llama-3.1-8B and EuroLLM-9B to obtain scores closer to Gemma2-9B and Olmo2-7B on the following five mathematical benchmarks:
The figure below shows that Luciole-8B-KD-Math-Base (dark blue) maintains almost all of the mathematical performance of the corrected 23B teacher (dark gray), largely surpassing the mathematical peformance of Luciole-8B-Base (red) and even Luciole-23B-Base (light gray).
At the same time, it manages to maintain performance close to Luciole-8B-Base on general benchmarks as illustrated below.
Citation
Coming soon.
Acknowledgements
We gratefully acknowledge BPI France for funding the OpenLLM France project under the call "Communs numériques pour l’intelligence artificielle générative" ("Digital commons for generative artificial intelligence") as a part of the France 2030 program.
Training of Luciole-8B-KD-Math-Base was made possible by computing AI and storage resources by GENCI at IDRIS thanks to the grant A0181016189 on the supercomputer Jean Zay’s H100 partition. We gratefully acknowledge support from GENCI and IDRIS and from Stephane Requena (GENCI) and Pierre-François Lavallée (IDRIS) in particular.
The pruning and distillation pipeline behind Luciole-8B-KD-Math-Base was carried out by Niyar Barman with the support of other LINAGORA members (in alphabetical order): Akshay Chaturvedi, Olivier Gouvert, Julie Hunter, Jean-Pierre Lorré, and Jérôme Louradour.
License
Copyright (c) 2025-2026 LINAGORA SAS. Luciole-8B-KD-Math-Base is developed by LINAGORA within the OpenLLM-France consortium (PROJET FRANCE2030 OpenLLM-France, funded by Bpifrance) and released under the Apache License 2.0.
Contact
- Downloads last month
- 772