"HET3" GPT2 (base) and GPT2MoE
Collection
Trained on est_latn, arb_arab, ell_grek, tur_latn, and fra_latn (heterogeneous set 3). โข 6 items โข Updated
GPT2Moe (custom) trained on a balanced set of languages Balanced model training set (1000mb with byte premium scaling for each language) Balanced tokenizer training set (100mb with byte premium scaling for each language)
Based on 7 W&B runs named gpt2moe_het3_1000mb in sjzahra-ri-se/huggingface.
Loss: 4 run(s); warmup: 4; GPU telemetry: 5 run(s).
Training hardware: Trained on a single H100 GPU.
Greek, Turkish, Estonian, French, Arabic
| Parameter | Value |
|---|---|
| Runs included | 7 |
| Training hardware | Trained on a single H100 GPU |
| Total epochs | 9.99976 |
| Total training steps | 385,460 |
| Total training FLOPs | 1.2893e+19 |
| Learning rate | 0.0001 |
| Train batch size | 8 |
| Eval batch size | 4 |
| Gradient accumulation | 8 |
| Weight decay | 0.01 |
| Warmup steps | 38546 |
| Warmup ratio | 0 |
| Optimizer | adamw_torch |
| Scheduler | linear |
| Max sequence length | 20 |
| Precision | False |
| Seed | 42 |
| GPU utilization | mean 45.5%, peak 98.0% |
W&B runs: 49mpjskm, 8nzdvov9, 9mtd1f31, ku8cxhvp, m9fmugix, o87giw3v, pi3bqaey
Generated 2026-11-17T17:20:49+00:00.