pipecat-ai/voice-reader

Pipecat Voice Reader (rc6)

⚠️ Experimental. A release candidate. Outputs, names and calibration may change between releases, and its readings are cues for an LLM to weigh, not verdicts to act on alone.

A small CPU model that hears how a user sounds: tone of voice, sarcasm, energy, laughter and speech rate. Pipecat runs it on up to the last 8 s of a user's speech each time they stop talking, and writes the reading at the head of their message to the LLM as RCS perception:

([tone frustrated=0.71, neutral=0.22],[arousal value=0.34],[laughter value=0.02],[speech_rate value=3.1],[sarcasm_voice value=0.63]) Oh, great.

It hears the voice, and Pipecat's Tone Reader reads the words. Pipecat reports the model's sarcasm as sarcasm_voice, beside the sarcasm the Tone Reader reads from the words.

Use

With Pipecat 1.16 or later:

uv add "pipecat-ai[expressivity]"

from pipecat.expressive.processors.voice_reader import PipecatVoiceReader

pipeline = Pipeline([transport.input(), stt, PipecatVoiceReader(), user_aggregator, llm, tts, transport.output(), assistant_aggregator])

Set PipelineParams(enable_expressive_conversation=True) so the LLM is taught to read it. Each Pipecat version pins the release it was tested with: revision= pins another, variant="tiny" loads the smaller model and sad=True adds the sad signal. See the reference, and the example bot for running it beside the Tone Reader in one reading.

Outputs

Output	Kind	Meaning
tone	distribution	how the voice sounds: positive, neutral, sad or frustrated
sarcasm	chance	a listener would hear the turn as sarcastic, words included; rarely above 0.5
laughter	chance	a listener would hear laughter in the turn, or a clearly laughing voice
sad	chance	a listener would hear sadness
arousal	level, 0–1	vocal energy
upset	level, 0–1	how clearly the speaker is upset
speech_rate	syllables per second	how fast they speak

Every value leaves the ONNX graph final, with its calibration applied. Each config.json declares the outputs, the lengths the graph reads (read_secs: 4, 6 or 8 s, as the turn needs) and the sarcasm threshold (0.066 for base, 0.068 for tiny). Pipecat's reader writes them under these names, except sarcasm, which it writes as sarcasm_voice. It adds sad when asked and leaves out upset, which PipecatVoiceClassifier answers.

Variants

Folder	variant	int8 ONNX	Read time on 1 thread, x86 server: turn up to 4 s / 6 s / 8 s	Choose it for
voice-reader-base/	"base" (default)	28.6 MB	~50–75 / ~75–115 / ~100–160 ms	the best tone reading, and sadness in conversation
voice-reader-tiny/	"tiny"	14.2 MB	~35 / ~55–60 / ~75–85 ms	one-vCPU deployments: sarcasm about as well, half the CPU

Each folder has model.int8.onnx, model.onnx (fp32), config.json and MANIFEST.json (SHA-256 of every file).

Evaluation

Labels on real speech come from two audio LLM judges (Gemini 3.1 Pro preview and OpenAI gpt-audio-1.5). A turn counts as sarcastic only when both say so. Test speakers never appear in training. Each turn is read whole, as cut for training; see the first limitation for how a live pipeline differs.

	base	tiny
Sarcasm AUROC, conversation (3,945 turns chosen to be hard, 203 sarcastic)	0.923	0.910
Sarcasm at the shipped threshold, same turns: recall / false alarms	0.86 / 14.9 %	0.78 / 12.7 %
Sarcasm at 0.5, same turns: recall / false alarms	0.47 / 1.1 %	0.42 / 0.8 %
Random conversation turns over the shipped threshold / over 0.5 (512 turns, 0.8 % sarcastic)	2.3 % / 0.2 %	2.3 % / 0.0 %
Tone macro-F1: conversation / Emolia / CREMA-D (acted)	0.736 / 0.677 / 0.874	0.732 / 0.638 / 0.880
sad AUROC: conversation (any judge heard sadness) / Emolia	0.759 / 0.933	0.744 / 0.925
Upset, mean AUROC over three sets	0.875	0.857
Arousal CCC	0.89	0.88
Laughter AUROC against Gemini, two conversation sets (4,366 turns, 769 with laughter; 5,778 turns, 1,360)	0.920 / 0.963	0.919 / 0.959
Laughter at 0.5, same sets: recall, false alarms	0.59, 1.0 % / 0.81, 1.5 %	0.58, 1.1 % / 0.79, 1.5 %
Laughter at 0.5, Emolia (4,512 clips, 500 with laughter): recall, false alarms	0.43, 1.3 %	0.44, 1.3 %
Random conversation turns at or over 0.5 on laughter (530 and 565 turns; Gemini heard laughter in 5.1 % and 4.1 %)	4.3 % / 3.7 %	4.5 % / 3.0 %
CREMA-D clips at or over 0.5 on laughter (349 acted clips, none with laughter)	0.0 %	0.0 %

Limitations

• A pipeline that reads at every pause reads worse than this card. Pipecat asks at each pause in a turn with that stretch of audio alone and keeps the highest of each value. Replayed that way over the same test turns, base's sarcasm ranking (AUPRC) falls from 0.60 to 0.50 on one conversation set and from 0.35 to 0.24 on the other, and tone macro-F1 by 0.01 to 0.03. Laughter hardly moves.
• Noise, rooms and narrow channels cost accuracy. Through background noise at 5 to 25 dB, a synthetic room and a phone or low-bitrate channel, base loses 0.04 to 0.05 of sarcasm AUPRC and 0.02 of tone macro-F1, and its recall of sad turns falls from 0.75 to 0.69 on one conversation set and from 0.52 to 0.40 on the other. tiny, trained on such copies, still loses 0.02 to 0.04 and 0.03, and was tested with noise recordings it trained on.
• Sarcasm is a weak cue. At 0.5 it catches under half of the sarcasm both judges heard on turns picked to be hard, and a tenth or less on other sets. The shipped threshold marks about 2 in 100 random turns, most of them sincere. It is weak on synthetic speech, too.
• Sadness in conversation is rare (2–6 % of turns), so most single-turn flags are wrong. It is a graded cue, not a measure of depression or anyone's mental health. Tone can overcall sad: in conversation half or more of base's sad calls are turns the judges agreed were another tone (recall 0.58–0.81, precision 0.31–0.47).
• Laughter is scored against the judge that taught it, not human listeners, and misses quiet laughs: at 0.5 it catches 0.5 to 0.8 of the laughter Gemini hears in conversation and 0.43 in podcast and video speech. One human reference: of 29 HarperValleyBank turns its transcribers marked as laughter, none of them trained on, base reads 19 at 0.5 or more and tiny 16, against 0.3 % and 0.5 % of the 12,620 turns they left unmarked. It fires on a clearly laughing voice as well as on a laugh, and still on about 1 in 20 turns Gemini heard crying in.
• Very short turns are read with the audio just before them, and can read high on laughter.
• English only. Labels are LLM judgments of the audio, not human ratings.

What changed since rc5

Only laughter: it now means laughter a listener hears in the turn, refit on what Gemini heard and calibrated against it. Every other value is exactly rc5's.

• A laugh inside a spoken turn now reads. Against Gemini on test turns, base's recall at 0.5 goes from 0.06 to 0.59 on one conversation set and from 0.05 to 0.81 on the other, with false alarms at 1.0 % and 1.5 % (rc5: 1.6 % on both).
• A sob isn't a laugh. On 176 test turns Gemini heard crying and no laughter in, base reads 4 % at 0.5 or more and tiny 5 %.

Training

base is the average of four Whisper encoders initialised from LAION's whisper-tiny-base-emotion-voice-burst (CC BY 4.0) and cut to an 8 s window, with every head retrained on the averaged encoder. tiny is distilled from an earlier base model, then fine-tuned on its training data plus degraded copies of the judged part (noise from MUSAN's noise set, CC BY 4.0; a synthetic room; Opus at 6 to 16 kbit/s, mu-law or an 8 kHz band-limit). In rc6 each size's laughter head is refit on its frozen encoder and grafted onto the rc5 model.

Training data: judged conversations from Open Yap 1K and otoSpeech, HarperValleyBank (CC BY 4.0), the English YODAS part of Emilia from laion/Emolia (CC BY 4.0) and CREMA-D (ODbL / DbCL), plus pseudo-labelled windows. MUStARD++ (CC BY-NC) and EmoNet-Voice Bench were used for evaluation only.