日本語向け SentencePiece Unigram トークナイザ v2 系列。語彙 32,000〜128,000 の6サイズ。改行などを U+E000 でエスケープする可逆ラッパー（escape-e000-v2）付き。語彙ブラウザ・比較: https://j-llm.org/tokenizer/
J-LLM
community
AI & ML interests
Japanese Language Models, Tokenizers, Small Language Models, LLM Training, Efficient Inference
Recent Activity
models 11
j-llm/Qwen3.5-2B-SpeedX
Text Generation • 2B • Updated • 274
j-llm/action1.5-policy
48.9M • Updated • 11 • 2
j-llm/j128-tokenizer-v2
Updated
j-llm/j96-tokenizer-v2
Updated
j-llm/j72-tokenizer-v2
Updated
j-llm/j64-tokenizer-v2
Updated
j-llm/j48-tokenizer-v2
Updated
j-llm/j32-tokenizer-v2
Updated
j-llm/action1
Reinforcement Learning • Updated • 8 • 2
j-llm/Auralis
Updated • 3
datasets 0
None public yet