gptbase_het3_1000mb / README.md
i-be-snek's picture
Update model card from W&B
01ac60a verified
|
Raw History Blame Contribute Delete
1.54 kB
---
language:
- el
- tr
- et
- fr
- ar
tags:
- generated-from-wandb
- gptbase
metrics:
- loss
---
# gptbase_het3_1000mb
<!-- collection-note:start -->
GPT2 (base) trained on a balanced set of languages
Balanced model training set (1000mb with byte premium scaling for each language)
Balanced tokenizer training set (100mb with byte premium scaling for each language)
<!-- collection-note:end -->
<!-- wandb-aggregate:start -->
## Training results
Based on **1** W&B runs named `gptbase_het3_1000mb` in [sjzahra-ri-se/huggingface](https://wandb.ai/sjzahra-ri-se/huggingface).
![Training charts](wandb-plots/training-eval-warmup-gpu.png)
Loss: **1** run(s); warmup: **1**; GPU telemetry: **1** run(s).
**Training hardware:** Trained on a single H100 GPU.
### Languages
Greek, Turkish, Estonian, French, Arabic
### Training metadata
| Parameter | Value |
|---|---|
| Runs included | 1 |
| Training hardware | Trained on a single H100 GPU |
| Total epochs | 9.99974 |
| Total training steps | 385,460 |
| Total training FLOPs | 6.4458e+18 |
| Learning rate | 0.0001 |
| Train batch size | 64 |
| Eval batch size | 64 |
| Gradient accumulation | 1 |
| Weight decay | 0.01 |
| Warmup steps | 38546 |
| Warmup ratio | 0 |
| Optimizer | adamw_torch_fused |
| Scheduler | linear |
| Max sequence length | 20 |
| Precision | False |
| Seed | 43 |
| GPU utilization | mean 98.7%, peak 100.0% |
W&B runs: [buh64hkk](https://wandb.ai/sjzahra-ri-se/huggingface/runs/buh64hkk)
_Generated 2026-11-17T17:25:27+00:00._
<!-- wandb-aggregate:end -->