Download TestTokenizerSize.py from CMSManhattan/LlamaRoboticsTokenizer: direct link, hf CLI and curl.
- Browser
- Download file 194 Bytes
-
https://huggingface.co/CMSManhattan/LlamaRoboticsTokenizer/resolve/main/TestTokenizerSize.py
- Command line
-
hf download hf://CMSManhattan/LlamaRoboticsTokenizer/TestTokenizerSize.py
-
curl -L -o TestTokenizerSize.py https://huggingface.co/CMSManhattan/LlamaRoboticsTokenizer/resolve/main/TestTokenizerSize.py
194 Bytes
| from transformers import AutoTokenizer | |
| tok = AutoTokenizer.from_pretrained(".") | |
| print("Vocab size:", len(tok)) | |
| print("pad_token_id:", tok.pad_token_id) | |
| print("eos_token_id:", tok.eos_token_id) | |