| --- |
| license: apache-2.0 |
| language: |
| - en |
| tags: |
| - text-generation |
| - cli |
| - shell |
| - command-line |
| - sft |
| - instruction-following |
| pipeline_tag: text-generation |
| widget: |
| - text: "Instruction: List all files in the current directory\nCommand:" |
| example_title: List files |
| - text: "Instruction: Find all Python files\nCommand:" |
| example_title: Find Python files |
| - text: "Instruction: Show disk usage\nCommand:" |
| example_title: Disk usage |
| --- |
| |
| # Tiny-LLM CLI SFT (54M) |
|
|
| A **54 million parameter** language model fine-tuned for CLI command generation. |
|
|
| ## Model Description |
|
|
| This model is a Supervised Fine-Tuned (SFT) version of [jonmabe/tiny-llm-54m](https://huggingface.co/jonmabe/tiny-llm-54m), trained to generate Unix/Linux shell commands from natural language instructions. |
|
|
| ### Training Data |
|
|
| - **Geddy's NL2Bash dataset**: ~2,300 natural language to bash command pairs |
| - **NL2Bash benchmark**: Standard benchmark for command translation |
| - **Synthetic examples**: Additional generated pairs |
| - **Total**: ~13,000 training pairs |
|
|
| ### Training Details |
|
|
| | Parameter | Value | |
| |-----------|-------| |
| | Base Model | tiny-llm-54m | |
| | Training Steps | 2,000 | |
| | Best Checkpoint | Step 1,000 | |
| | Best Val Loss | 1.2456 | |
| | Learning Rate | 5e-5 | |
| | Batch Size | 16 | |
| | Hardware | NVIDIA RTX 5090 | |
| | Training Time | ~9 minutes | |
|
|
| ## Architecture |
|
|
| - **Parameters**: 54.93M |
| - **Layers**: 12 |
| - **Hidden Size**: 512 |
| - **Attention Heads**: 8 |
| - **Intermediate Size**: 1408 |
| - **Max Position**: 512 |
| - **Vocabulary**: 32,000 tokens |
| - **Features**: RoPE, RMSNorm, SwiGLU, Weight Tying |
|
|
| ## Usage |
|
|
| ### Prompt Format |
|
|
| ``` |
| Instruction: <natural language description> |
| Command: |
| ``` |
|
|
| ### Example |
|
|
| ```python |
| from model import TinyLLM |
| import torch |
| |
| # Load model |
| checkpoint = torch.load("best_model.pt", map_location="cpu") |
| model = TinyLLM(checkpoint["config"]["model"]) |
| model.load_state_dict(checkpoint["model_state_dict"]) |
| model.eval() |
| |
| # Generate |
| prompt = "Instruction: Find all Python files modified in the last day\nCommand:" |
| # ... tokenize and generate |
| ``` |
|
|
| ## Limitations |
|
|
| ⚠️ **Known Issues:** |
| - Tokenizer decode shows raw BPE tokens (Ġ = space, Ċ = newline) |
| - Model generates fragments of correct commands but output can be noisy |
| - Needs more training steps for reliable generation |
| - Small model size limits command complexity |
|
|
| ## Improvement Plan |
|
|
| 1. **Fix tokenizer decode** - Proper BPE to text conversion |
| 2. **Longer training** - 5,000-10,000 steps |
| 3. **Data quality** - Curate cleaner training pairs |
| 4. **Lower LR** - More stable convergence with 1e-5 |
|
|
| ## License |
|
|
| Apache 2.0 |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{tiny-llm-cli-sft-2026, |
| author = {Jon Mabe}, |
| title = {Tiny-LLM CLI SFT: Small Language Model for Command Generation}, |
| year = {2026}, |
| publisher = {HuggingFace}, |
| url = {https://huggingface.co/jonmabe/tiny-llm-cli-sft} |
| } |
| ``` |
|
|
