"HOM2" GPT2 (base) and GPT2Moe
Collection
Trained on fra_latn, glg_latn, spa_latn, por_latn, and ita_latn (homogeneous set 2) โข 6 items โข Updated
GPT2 (base) trained on a balanced set of languages Balanced model training set (1000mb with byte premium scaling for each language) Balanced tokenizer training set (100mb with byte premium scaling for each language)
Based on 1 W&B runs named gptbase_hom2_1000mb in sjzahra-ri-se/huggingface.
Loss: 1 run(s); warmup: 1; GPU telemetry: 1 run(s).
Training hardware: Trained on a single H100 GPU.
French, Spanish, Portuguese, Italian, Galician
| Parameter | Value |
|---|---|
| Runs included | 1 |
| Training hardware | Trained on a single H100 GPU |
| Total epochs | 9.99979 |
| Total training steps | 375,482 |
| Total training FLOPs | 6.2790e+18 |
| Learning rate | 0.0001 |
| Train batch size | 64 |
| Eval batch size | 64 |
| Gradient accumulation | 1 |
| Weight decay | 0.01 |
| Warmup steps | 37548 |
| Warmup ratio | 0 |
| Optimizer | adamw_torch_fused |
| Scheduler | linear |
| Max sequence length | 20 |
| Precision | False |
| Seed | 43 |
| GPU utilization | mean 65.4%, peak 100.0% |
W&B runs: gptbase_hom2_1000mb
Generated 2026-11-17T17:25:40+00:00.