Model Card for gemma4-e2b-it-ConvFill-TR

Turkish finetune of google/gemma-4-E2B-it for the conversational infill task described in the ConvFill paper. This repository has the merged bf16 weights.

Other formats: ONNX (q4f16, WebGPU), GGUF (llama.cpp).

Model Details

Model Description

Deploying responsive, multi-turn conversational voice agents with large language models poses a critical challenge: cloud-based foundation models can reason, retrieve information and use tools, but their latency disrupts natural conversation, while small models answer quickly but lack those capabilities. Conversational infill is a task where a small, local model generates prompt, contextually appropriate dialogue and seamlessly incorporates delayed, external knowledge produced in parallel by a foundation model backend. This model is a Turkish version of that idea: it is the small local model (the "Talker"), and it expects the backend answer to arrive in a knowledge turn while it is already speaking.

Model Sources

Uses

Direct Use

Use it as the fast local half of a voice assistant: it answers immediately with short fillers, then continues with the knowledge produced by a larger model. It is not a general chat model; see the paper for how the knowledge turn is used.

How to Get Started with the Model

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "atasoglu/gemma4-e2b-it-ConvFill-TR"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.bfloat16, device_map="auto")

prompt = (
    "<bos><|turn>system\nSen Türkçe konuşan sesli bir asistansın.<turn|>\n"
    "<|turn>user\nYarın İstanbul'da hava nasıl olacak?<turn|>\n"
    "<|turn>model\n"
)
ids = tok(prompt, add_special_tokens=False, return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=160, eos_token_id=[1, 106, 50], do_sample=False)
print(tok.decode(out[0, ids.input_ids.shape[1]:]))

Training Details

Training Data

ConvFill-TR-10k: 9,992 synthetic Turkish conversations written for a two-model voice assistant. Each conversation is cut into turn-level examples; the loss is computed on the model turns (fillers, <sil> and the final answer), not on the prompt. About 307k examples were used for one epoch; examples longer than 1024 tokens were dropped.

Training Procedure

Finetuned with Unsloth as 4-bit QLoRA on unsloth/gemma-4-E2B-it. The adapter was then merged into bf16 weights (the 4-bit base was dequantized locally first).

Hyperparameter Value
Base model unsloth/gemma-4-E2B-it (4-bit)
LoRA r = 64, alpha = 64, dropout = 0
LoRA targets attention (q, k, v, o) and MLP (gate, up, down) projections of the language model, plus the per-layer input projections
Optimizer AdamW (fused), weight decay 0.01, gradient clipping 1.0
Learning rate 2e-4, 5% warmup, cosine decay to 0
Batch size 16 (4 per step x 4 gradient accumulation)
Precision bf16
Epochs / steps 1 epoch, 19,201 steps
Max sequence length 1024 tokens
Seed 42
Hardware / time single consumer GPU, about 5.5 hours, peak 8.2 GB VRAM

Evaluation

Bits per byte (bpb) of the model turns on held-out conversations, lower is better:

Split All <sil> / filler Knowledge answer
Validation subset 0.3661 0.4642 0.3317
Test 0.3627 0.4676 0.3258

For comparison, vngrs/Kumru-2B finetuned the same way reached 0.4113 on the validation split.

Use Restrictions

You must not use any of the Gemma Services:

for the restricted uses set forth in the Gemma Prohibited Use Policy at ai.google.dev/gemma/prohibited_use_policy, which is hereby incorporated by reference into this Agreement; or in violation of applicable laws and regulations.

See the Gemma terms for details.

Bias, Risks, and Limitations

This model is not explicitly tuned for guardrailed behavior. It is trained on synthetic, machine-translated conversations, so it can repeat their style and errors. It only talks and relays knowledge it is given; facts must come from the backend model, not from the Talker itself. Turkish only.

Citation

@misc{srinivas2026thinkingspeakinginferencetimeknowledge,
      title={Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents},
      author={Vidya Srinivas and Zachary Englhardt and Vikram Iyer and Shwetak Patel},
      year={2026},
      eprint={2511.07397},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2511.07397},
}
Downloads last month
66
Safetensors
Model size
5B params
Tensor type
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for atasoglu/gemma4-e2b-it-ConvFill-TR

Finetuned
(511)
this model
Quantizations
2 models

Dataset used to train atasoglu/gemma4-e2b-it-ConvFill-TR

Paper for atasoglu/gemma4-e2b-it-ConvFill-TR

Evaluation results