Instructions to use atasoglu/gemma4-e2b-it-ConvFill-TR with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use atasoglu/gemma4-e2b-it-ConvFill-TR with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="atasoglu/gemma4-e2b-it-ConvFill-TR") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("atasoglu/gemma4-e2b-it-ConvFill-TR") model = AutoModelForMultimodalLM.from_pretrained("atasoglu/gemma4-e2b-it-ConvFill-TR", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=256) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use atasoglu/gemma4-e2b-it-ConvFill-TR with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "atasoglu/gemma4-e2b-it-ConvFill-TR" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "atasoglu/gemma4-e2b-it-ConvFill-TR", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/atasoglu/gemma4-e2b-it-ConvFill-TR
- SGLang
How to use atasoglu/gemma4-e2b-it-ConvFill-TR with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "atasoglu/gemma4-e2b-it-ConvFill-TR" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "atasoglu/gemma4-e2b-it-ConvFill-TR", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "atasoglu/gemma4-e2b-it-ConvFill-TR" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "atasoglu/gemma4-e2b-it-ConvFill-TR", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Unsloth Desktop
- Docker Model Runner
How to use atasoglu/gemma4-e2b-it-ConvFill-TR with Docker Model Runner:
docker model run hf.co/atasoglu/gemma4-e2b-it-ConvFill-TR
Model Card for gemma4-e2b-it-ConvFill-TR
Turkish finetune of google/gemma-4-E2B-it for the conversational infill task described in the ConvFill paper. This repository has the merged bf16 weights.
Other formats: ONNX (q4f16, WebGPU), GGUF (llama.cpp).
Model Details
Model Description
Deploying responsive, multi-turn conversational voice agents with large language models poses a critical challenge: cloud-based foundation models can reason, retrieve information and use tools, but their latency disrupts natural conversation, while small models answer quickly but lack those capabilities. Conversational infill is a task where a small, local model generates prompt, contextually appropriate dialogue and seamlessly incorporates delayed, external knowledge produced in parallel by a foundation model backend. This model is a Turkish version of that idea: it is the small local model (the "Talker"), and it expects the backend answer to arrive in a knowledge turn while it is already speaking.
- Finetuned from model: google/gemma-4-E2B-it
- Language: Turkish
- License: Gemma
Model Sources
- Paper: Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents
- Original English models and code: vysri/conversational-infill
- Dataset: atasoglu/convfill-dataset-tr-10k (Turkish version of zenglhardt/convfill-dataset)
Uses
Direct Use
Use it as the fast local half of a voice assistant: it answers immediately with short fillers, then continues with the knowledge produced by a larger model. It is not a general chat model; see the paper for how the knowledge turn is used.
How to Get Started with the Model
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
repo = "atasoglu/gemma4-e2b-it-ConvFill-TR"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto")
prompt = (
"<bos><|turn>system\nSen Türkçe konuşan sesli bir asistansın.<turn|>\n"
"<|turn>user\nYarın İstanbul'da hava nasıl olacak?<turn|>\n"
"<|turn>model\n"
)
ids = tok(prompt, add_special_tokens=False, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=160, eos_token_id=[1, 106, 50], do_sample=False)
print(tok.decode(out[0, ids.input_ids.shape[1]:]))
Training Details
Training Data
ConvFill-TR-10k: 9,992 synthetic Turkish conversations written for a two-model voice assistant. Each conversation is cut into turn-level examples; the loss is computed on the model turns (fillers, <sil> and the final answer), not on the prompt. About 307k examples were used for one epoch; examples longer than 1024 tokens were dropped.
Training Procedure
Finetuned with Unsloth as 4-bit QLoRA on unsloth/gemma-4-E2B-it. The adapter was then merged into bf16 weights (the 4-bit base was dequantized locally first).
| Hyperparameter | Value |
|---|---|
| Base model | unsloth/gemma-4-E2B-it (4-bit) |
| LoRA | r = 64, alpha = 64, dropout = 0 |
| LoRA targets | attention (q, k, v, o) and MLP (gate, up, down) projections of the language model, plus the per-layer input projections |
| Optimizer | AdamW (fused), weight decay 0.01, gradient clipping 1.0 |
| Learning rate | 2e-4, 5% warmup, cosine decay to 0 |
| Batch size | 16 (4 per step x 4 gradient accumulation) |
| Precision | bf16 |
| Epochs / steps | 1 epoch, 19,201 steps |
| Max sequence length | 1024 tokens |
| Seed | 42 |
| Hardware / time | single consumer GPU, about 5.5 hours, peak 8.2 GB VRAM |
Evaluation
Bits per byte (bpb) of the model turns on held-out conversations, lower is better:
| Split | All | <sil> / filler |
Knowledge answer |
|---|---|---|---|
| Validation subset | 0.3661 | 0.4642 | 0.3317 |
| Test | 0.3627 | 0.4676 | 0.3258 |
For comparison, vngrs/Kumru-2B finetuned the same way reached 0.4113 on the validation split.
Use Restrictions
You must not use any of the Gemma Services:
for the restricted uses set forth in the Gemma Prohibited Use Policy at ai.google.dev/gemma/prohibited_use_policy, which is hereby incorporated by reference into this Agreement; or in violation of applicable laws and regulations.
See the Gemma terms for details.
Bias, Risks, and Limitations
This model is not explicitly tuned for guardrailed behavior. It is trained on synthetic, machine-translated conversations, so it can repeat their style and errors. It only talks and relays knowledge it is given; facts must come from the backend model, not from the Talker itself. Turkish only.
Citation
@misc{srinivas2026thinkingspeakinginferencetimeknowledge,
title={Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents},
author={Vidya Srinivas and Zachary Englhardt and Vikram Iyer and Shwetak Patel},
year={2026},
eprint={2511.07397},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2511.07397},
}
- Downloads last month
- 66
Model tree for atasoglu/gemma4-e2b-it-ConvFill-TR
Dataset used to train atasoglu/gemma4-e2b-it-ConvFill-TR
Paper for atasoglu/gemma4-e2b-it-ConvFill-TR
Evaluation results
- Bits per byte (all model turns) on ConvFill-TR-10k (test)test set self-reported0.363