gptbase_het3_1000mb / README.md
i-be-snek's picture
Update model card from W&B
01ac60a verified
|
Raw History Blame Contribute Delete
1.54 kB
metadata
language:
  - el
  - tr
  - et
  - fr
  - ar
tags:
  - generated-from-wandb
  - gptbase
metrics:
  - loss

gptbase_het3_1000mb

GPT2 (base) trained on a balanced set of languages Balanced model training set (1000mb with byte premium scaling for each language) Balanced tokenizer training set (100mb with byte premium scaling for each language)

Training results

Based on 1 W&B runs named gptbase_het3_1000mb in sjzahra-ri-se/huggingface.

Training charts

Loss: 1 run(s); warmup: 1; GPU telemetry: 1 run(s).

Training hardware: Trained on a single H100 GPU.

Languages

Greek, Turkish, Estonian, French, Arabic

Training metadata

Parameter Value
Runs included 1
Training hardware Trained on a single H100 GPU
Total epochs 9.99974
Total training steps 385,460
Total training FLOPs 6.4458e+18
Learning rate 0.0001
Train batch size 64
Eval batch size 64
Gradient accumulation 1
Weight decay 0.01
Warmup steps 38546
Warmup ratio 0
Optimizer adamw_torch_fused
Scheduler linear
Max sequence length 20
Precision False
Seed 43
GPU utilization mean 98.7%, peak 100.0%

W&B runs: buh64hkk

Generated 2026-11-17T17:25:27+00:00.