Arabic
arabic
tokenizer
morphology
nlp
dialect
df-arc / tokenizer.json

Commit History

  1. vocab_size Increased from 64000 to 128000

    Verified signature
    34fc84b

  1. Upload custom Unigram tokenizer (v1)

    Verified signature
    ea23987

  1. Release v1.1: PMI Phrase Merging & Smart Morphology

    Verified signature
    bc06a6e

  1. Upload folder using huggingface_hub

    Verified signature
    3b90e9e