You need to agree to share your Hugging Face information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

gpt2moe_dom_het3_1000mb

GPT2MoE (custom) trained on a set of languages where French dominates the training data French is dominant in the model training set (3000mb of French and 500mb (both with byte premium scaling) for all other languages) Balanced tokenizer training set (100mb with byte premium scaling for each language)

Training results

Based on 8 W&B runs named gpt2moe_dom_het3_1000mb in sjzahra-ri-se/huggingface.

Training charts

Loss: 4 run(s); warmup: 4; GPU telemetry: 6 run(s).

Training hardware: Trained on a single H100 GPU.

Languages

Greek, Turkish, Estonian, French, Arabic

Training metadata

Parameter Value
Runs included 8
Training hardware Trained on a single H100 GPU
Total epochs 10
Total training steps 413,730
Total training FLOPs 1.3839e+19
Learning rate 0.0001
Train batch size 8
Eval batch size 4
Gradient accumulation 8
Weight decay 0.01
Warmup steps 41243 (3 runs); 41373 (1 runs)
Warmup ratio 0
Optimizer adamw_torch
Scheduler linear
Max sequence length 20
Precision False
Seed 42
GPU utilization mean 47.2%, peak 99.0%

W&B runs: 1xtjeh57, 83laip56, 8n1tm99p, 9evboldx, qdvag8dz, v9uc5vk7, wsl6jckw, y2vq2ufn

Generated 2026-11-17T17:16:18+00:00.

Downloads last month
6
Safetensors
Model size
0.2B params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Collection including NLP-RISE/gpt2moe_dom_het3_1000mb