gpt2moe_dom_hom2_1000mb
GPT2MoE (custom) trained on a set of languages where French dominates the training data French is dominant in the model training set (3000mb of French and 500mb (both with byte premium scaling) for all other languages) Balanced tokenizer training set (100mb with byte premium scaling for each language)
Training results
Based on 5 W&B runs named gpt2moe_dom_hom2_1000mb in sjzahra-ri-se/huggingface.
Loss: 3 run(s); warmup: 3; GPU telemetry: 5 run(s).
Training hardware: Trained on a single H100 GPU.
Languages
French, Spanish, Portuguese, Italian, Galician
Training metadata
| Parameter | Value |
|---|---|
| Runs included | 5 |
| Training hardware | Trained on a single H100 GPU |
| Total epochs | 9.99995 |
| Total training steps | 398,698 |
| Total training FLOPs | 1.3336e+19 |
| Learning rate | 0.0001 |
| Train batch size | 8 |
| Eval batch size | 4 |
| Gradient accumulation | 8 |
| Weight decay | 0.01 |
| Warmup steps | 39869 |
| Warmup ratio | 0 |
| Optimizer | adamw_torch |
| Scheduler | linear |
| Max sequence length | 20 |
| Precision | False |
| Seed | 42 |
| GPU utilization | mean 34.7%, peak 52.0% |
W&B runs: 30lay6qw, 6k09cfp3, nri0o7zh, rq85zlyn, x58u3d1r
Generated 2026-11-17T17:18:35+00:00.
- Downloads last month
- 11
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐ Ask for provider support
Collection including NLP-RISE/gpt2moe_dom_hom2_1000mb
Collection
Trained on fra_latn, glg_latn, spa_latn, por_latn, and ita_latn (homogeneous set 2) โข 6 items โข Updated
