gptbase_dom_het_tok_1000mb

GPT2 (base) trained on a set of languages where English dominates the model and tokenizer training data sets English is dominant in the model training set (3000mb of English and 500mb (both with byte premium scaling) for all other languages) English is dominant in the tokenizer training set (300mb of English and 50mb (both with byte premium scaling) for all other languages)

Training results

Based on 1 W&B runs named gptbase_dom_het_tok_1000mb in sjzahra-ri-se/huggingface.

Training charts

Loss: 1 run(s); warmup: 1; GPU telemetry: 1 run(s).

Training hardware: Trained on a single H100 GPU.

Languages

Greek, Turkish, Estonian, English, Arabic

Training metadata

Parameter Value
Runs included 1
Training hardware Trained on a single H100 GPU
Total epochs 9.99975
Total training steps 360,651
Total training FLOPs 6.0309e+18
Learning rate 0.0001
Train batch size 64
Eval batch size 64
Gradient accumulation 1
Weight decay 0.01
Warmup steps 36065
Warmup ratio 0
Optimizer adamw_torch_fused
Scheduler linear
Max sequence length 20
Precision False
Seed 43
GPU utilization mean 97.0%, peak 100.0%

W&B runs: dy9jdmqz

Generated 2026-11-17T17:24:04+00:00.

Downloads last month
40
Safetensors
Model size
0.1B params
Tensor type
F32
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Collection including NLP-RISE/gptbase_dom_het_tok_1000mb