|
Download README.md from CMSManhattan/JiRack-Router-Tokenizer-65K: direct link, hf CLI and curl.
- Browser
- Download file 5.5 kB
-
https://huggingface.co/CMSManhattan/JiRack-Router-Tokenizer-65K/resolve/main/README.md
- Command line
-
hf download hf://CMSManhattan/JiRack-Router-Tokenizer-65K/README.md
-
curl -L -o README.md https://huggingface.co/CMSManhattan/JiRack-Router-Tokenizer-65K/resolve/main/README.md
5.5 kB
| license: apache-2.0 | |
| language: | |
| - multilingual | |
| tags: | |
| - tokenizer | |
| - bpe | |
| - byte-level-bpe | |
| - chatml | |
| - routing | |
| - moe | |
| Enjoy β We extend the JiRack Models Ecosystem! π | |
| - The Best RAG with Mixture-of-Experts via CMS Manhattan RAG System with JiRack Ternary Experts for CPU Solution. | |
| - JiRackTernary_1b model https://huggingface.co/kgrabko/JiRackTernary_1b | |
| # JiRack Router Tokenizer Pro | |
| **JiRack Pro Tokenizer - 347 active language editions of Wikipedia** | |
| **High-performance custom Byte-Level BPE tokenizer** trained on the full Wikipedia dump across multiple languages. | |
| Developed specifically for intelligent routing models and Mixture-of-Experts systems. | |
| ## Model Details | |
| - **Developer**: Konstantin Grabko (JiRack) | |
| - **License**: Apache License 2.0 | |
| - **Training Data**: Full Wikipedia multilingual dump | |
| - **Vocabulary Size**: 65,536 tokens | |
| - **Special Tokens**: 128 reserved tokens (including 40+ domain routing tokens) | |
| ## Key Features | |
| - Correctly placed `<|unk|>` token at ID 0 | |
| - Full native support for **ChatML** format (`<|im_start|>` / `<|im_end|>`) | |
| - Large set of specialized routing tokens (`__CODING__`, `__MATH__`, `__PYTHON__`, `__SCIENCE__`, etc.) | |
| - Support JiRack Robotics Technology with tags (`<|action_start|>` / `<|action_end|>`) | |
| - Support JiRack vision , images , audio-visual | |
| - Strong multilingual performance | |
| # Basic system and dialogue tokens | |
| "<|unk|>", | |
| "<|endoftext|>", | |
| "<|padding|>", | |
| "<|im_start|>", | |
| "<|im_end|>", | |
| # Core roles | |
| "<|im_start|>system", | |
| "<|im_start|>user", | |
| "<|im_start|>assistant", | |
| # Additional useful tokens | |
| "<|im_start|>tool", | |
| "<|im_start|>function", | |
| # Reasoning block | |
| "<|im_start|>thought", | |
| # Tool calls | |
| "<|tool_call|>", | |
| "<|tool_response|>", | |
| # Multimodality and audio-visual block | |
| "<|image|>", | |
| "<|video|>", | |
| "<|sound|>", | |
| "<|voice|>", | |
| "<|listening|>", | |
| "<|vision|>", | |
| # Emotional state (Mood) | |
| "<|mood_happy|>", | |
| "<|mood_sad|>", | |
| "<|mood_angry|>", | |
| "<|mood_neutral|>", | |
| # --- FIM (Fill-in-the-Middle) tokens from StarCoder --- | |
| "<|fim_prefix|>", | |
| "<|fim_middle|>", | |
| "<|fim_suffix|>", | |
| # Robotics (trajectory/command boundaries) | |
| "<|action_start|>", | |
| "<|action_end|>", | |
| # Highlights | |
| - **Architecture**: Byte-Level BPE (Byte-level Byte Pair Encoding) which natively prevents Out-Of-Vocabulary (OOV) tokens. | |
| - **Form Factor**: Fully wrapped into Hugging Face `PreTrainedTokenizerFast` with native `ByteLevel` decoders for clean cyrillic representation. | |
| - **Chat Standard**: Out-of-the-box support for **ChatML** formatting (`<|im_start|>` / `<|im_end|>`). | |
| - **Domain Specialization**: Pre-baked atomic routing tokens like `__CODING__` and `__PYTHON__` etc. | |
| ### CMS Manhattan Service & Support | |
| - Jirack patent guards your technology for competitors | |
| - Redesign Llama , Qwen , Gemma to Ternary model | |
| - Re-tain and replace embeddings for Llama , Qwen , Gemma to extend langeages to 347 | |
| - Accelerate inference via high compression 256K tokenizer and replace multiplication with sum operations via Ternary weights | |
| ## Usage | |
| ```python | |
| from transformers import AutoTokenizer | |
| tokenizer = AutoTokenizer.from_pretrained("CMSManhattan/JiRack-Router-Tokenizer-65K") | |
| text = "<|im_start|>user\n__CODING__ __PYTHON__ Write a merge sort function.<|im_end|>" | |
| print(tokenizer.encode(text)) | |
| ``` | |
| ### JiRack Pretrain Dataset | |
| https://huggingface.co/datasets/CMSManhattan/JiRack-Pretrain-Dataset | |
| ```bash | |
| python train_jirack_accelerate.py | |
| Processing jirack_pretrain_chunk_0.pt: 27%|ββββββββββββββββββββββ | 268/1000 [22:02:45<60:12:33, 296.11s/it, loss=6.3145, avg_loss=7.0132, ppl=1111.16] | |
| Processing jirack_pretrain_chunk_0.pt: 87%|βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ | 866/1000 [75:57:47<13:13:29, 355.29s/it, loss=2.8616, avg_loss=5.9877, ppl=398.52] | |
| ``` | |
| ### Benchmark for tokens quality . | |
| ```bash | |
| === Text after ChatML Template === | |
| <|im_start|>system | |
| You are a precise router model.<|im_end|> | |
| <|im_start|>user | |
| __CODING__ __PYTHON__ Write a merge sort function in Python.<|im_end|> | |
| === Tokens (IDs) === | |
| [5, 326, 5095, 944, 396, 23348, 1021, 7831, 5869, 141, 4, 326, 6, 326, 29, 348, 44, 26876, 396, 52698, 6521, 2031, 460, 7524, 141, 4, 326] | |
| === Decoding Token by Token === | |
| [transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway. | |
| 5 -> '<|im_start|>system' | |
| 326 -> ' | |
| ' | |
| 5095 -> 'You' | |
| 944 -> ' are' | |
| 396 -> ' a' | |
| 23348 -> ' precise' | |
| 1021 -> ' ro' | |
| 7831 -> 'uter' | |
| 5869 -> ' model' | |
| 141 -> '.' | |
| 4 -> '<|im_end|>' | |
| 326 -> ' | |
| ' | |
| 6 -> '<|im_start|>user' | |
| 326 -> ' | |
| ' | |
| 29 -> '__CODING__' | |
| 348 -> ' ' | |
| 44 -> '__PYTHON__' | |
| 26876 -> ' Write' | |
| 396 -> ' a' | |
| 52698 -> ' merge' | |
| 6521 -> ' sort' | |
| 2031 -> ' function' | |
| 460 -> ' in' | |
| 7524 -> ' Python' | |
| 141 -> '.' | |
| 4 -> '<|im_end|>' | |
| 326 -> ' | |
| ' | |
| ``` | |
| ## π§ Contact & Licensing | |
| For joint ventures, hardware integration, or licensing inquiries: | |
| - **Email:** grabko@cmsmanhattan.com | |
| - **Phone:** +1 (516) 777-0945 | |
| - **Location:** New York, USA | |
| ## π§ Copyright 2026 CMS Manhattan . All rights reserved | |