codegen-scala / tokenizer.json

Commit History

  1. 50k vocab size instead of 200k to match gpt-2

    2a1f4da

  1. Upload tokenizer

    ba44dad