Spaces:

dispatchAI
/

tokenizer-visualizer

Runtime error

App Files Files Community

tokenizer-visualizer / README.md

3morixd

Upload README.md with huggingface_hub

0285034 verified 4 months ago

preview code

Raw History Blame Contribute Delete

506 Bytes

A newer version of the Gradio SDK is available: 6.31.0

Upgrade

metadata

title: Tokenizer Visualizer
emoji: 🔤
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: 4.44.0
app_file: app.py
pinned: false
license: apache-2.0
tags:
  - tokenizer
  - visualization
  - mcp
  - dispatchai

🔤 dispatchAI Tokenizer Visualizer

MCP Server — visualize how tokenizers split text.

Tools

tokenize_text(text, method) — Tokenize with chosen method
compare_methods(text) — Compare token counts across methods

🚀 dispatchAI