Spaces:

dispatchAI
/

tokenizer-visualizer

Runtime error

App Files Files Community

tokenizer-visualizer / README.md

3morixd

Upload README.md with huggingface_hub

0285034 verified 4 months ago

preview code

Raw History Blame Contribute Delete

506 Bytes

---
title: Tokenizer Visualizer
emoji: 🔤
colorFrom: blue
colorTo: purple
sdk: gradio
sdk_version: "4.44.0"
app_file: app.py
pinned: false
license: apache-2.0
tags: [tokenizer, visualization, mcp, dispatchai]
---
# 🔤 dispatchAI Tokenizer Visualizer

**MCP Server** — visualize how tokenizers split text.

## Tools
- `tokenize_text(text, method)` — Tokenize with chosen method
- `compare_methods(text)` — Compare token counts across methods

🚀 [dispatchAI](https://huggingface.co/dispatchAI)

	---
	title: Tokenizer Visualizer
	emoji: 🔤
	colorFrom: blue
	colorTo: purple
	sdk: gradio
	sdk_version: "4.44.0"
	app_file: app.py
	pinned: false
	license: apache-2.0
	tags: [tokenizer, visualization, mcp, dispatchai]
	---

	# 🔤 dispatchAI Tokenizer Visualizer

	MCP Server — visualize how tokenizers split text.

	## Tools
	- `tokenize_text(text, method)` — Tokenize with chosen method
	- `compare_methods(text)` — Compare token counts across methods

	🚀 [dispatchAI](https://huggingface.co/dispatchAI)