Matcha-TTS-PL — four synthetic voices, modern Polish, English words

Formerly published as machinekind/Matcha-TTS-PL-v2 (old links redirect here).

Fine-tuned from Matcha-TTS-PL-Pretrain (pretraining on Polish audiobooks: Wolne Lektury + AZON): a small, fast, non-autoregressive Matcha-TTS (20.9 M parameters, real-time on a small GPU or in a browser) distilled from VoxCPM2, a 2B-parameter TTS, onto synthetic voices. One model per voice (each with its own fine-tuned HiFi-GAN vocoder), all in this repo:

voice	gender	UTMOS ↑	Whisper WER ↓	similarity ↑	VoxCPM2 teacher (UTMOS / similarity)	questions: same direction as the teacher, mean difference	default tempo	designed from the description
male_voxcpm_1	male	3.40	1.3 %	0.850	3.68 / 0.883	8/8, 3.3 st	0.8	Male, teenager, energetic, optimistic, ready to help, speaks crip Polish.
female_voxcpm_1	female	3.27	2.0 %	0.836	3.82 / 0.884	6/8, 3.1 st	0.8	Female, teenager, energetic, optimistic, confident, ready to help, speaks crip Polish, mid/high-pitch timbre, mid/fast tempo.
male_voxcpm_2	male	3.59	0.7 %	0.850	3.87 / 0.874	7/8, 4.0 st	0.8	Voice of a dog, fast, energetic, crazy, mid/high pitch.
female_voxcpm_2	female	3.39	0.7 %	0.907	3.69 / 0.919	6/8, 4.0 st	0.8	Female, teenager, energetic, optimistic, confident, ready to help, speaks crip Polish, mid/high-pitch timbre, mid/fast tempo.

12 held-out sentences, Mac CPU, 4 ODE steps, temperature 0.8, model + its own vocoder; question test: final pitch slope of 8 questions (4 wh-, 4 yes/no) vs the teacher reading the same questions. The voices are synthetic — designed from a text description with VoxCPM2 (OpenBMB, Apache-2.0) and frozen as prompt caches; not recordings or clones of real people.

Note on female_voxcpm_1. The VoxCPM2 reference clip this voice was frozen from has strong p pops (low-frequency thumps on a p before a vowel), and the model learned them. Its output is therefore high-passed at 90 Hz (zero-phase, 4th-order Butterworth): onnx/voices_index.json and onnx/female_voxcpm_1/voices.json carry "highpass_hz": 90, and the Space applies it automatically. Apply the same filter when you use this voice's ONNX graph or checkpoint directly (see the quick start). The other voices need no post-filter.

• Modern Polish of a humanoid assistant — about 15 h of training speech per voice, the same texts for every voice: assistant speech in 18 everyday domains, questions of every kind, modern corpora (NKJP, Wikinews, KPWr, ParlaMint, OpenAssistant, Wikibooks, Wikivoyage) and reviewed YouTube transcripts; two thirds of it in multi-sentence passages (10–25 s), so prosody is planned across sentences. First-person forms are 50/50 masculine and feminine in the texts.
• English words inside Polish sentences — "Po meetingu wyślę ci feedback na maila" is read with English phonemes: English words are detected automatically and phonemized by espeak-ng en-us, the rest by espeak-ng pl (english/).
• A HiFi-GAN fine-tuned on each voice, with real ground truth (the teacher's audio).

Try it in the browser (voice selector): spaces/machinekind/matcha-tts-pl · training data: datasets/machinekind/polish-synthetic-speech-voxcpm2. Defaults for every voice: temperature 0.8, tempo 0.8 (length_scale; at 1.0 the models speak 10–19 % slower than the teacher), style token 8 = neutral.

Files

browser / ONNX (acoustic model + vocoder in one graph, 4 / 2 ODE steps)	PyTorch Matcha checkpoint (speaker 20)	HiFi-GAN generator
onnx/matcha_pl_t4.onnx, onnx/matcha_pl_t2.onnx, onnx/voices.json	model/matcha_pl_v2.ckpt	vocoder/generator_male_voxcpm_1
onnx/female_voxcpm_1/matcha_pl_t4.onnx, onnx/female_voxcpm_1/matcha_pl_t2.onnx, onnx/female_voxcpm_1/voices.json	model/matcha_pl_v2_female_voxcpm_1.ckpt	vocoder/female_voxcpm_1/generator
onnx/male_voxcpm_2/matcha_pl_t4.onnx, onnx/male_voxcpm_2/matcha_pl_t2.onnx, onnx/male_voxcpm_2/voices.json	model/matcha_pl_v2_male_voxcpm_2.ckpt	vocoder/male_voxcpm_2/generator
onnx/female_voxcpm_2/matcha_pl_t4.onnx, onnx/female_voxcpm_2/matcha_pl_t2.onnx, onnx/female_voxcpm_2/voices.json	model/matcha_pl_v2_female_voxcpm_2.ckpt	vocoder/female_voxcpm_2/generator

onnx/voices_index.json lists the voices with their paths and defaults. ONNX inputs: x (phoneme ids), x_lengths, scales=[temperature, length_scale], spk_emb (float32 [1, 64]: speakers["0"].emb + a style embedding from the voice's voices.json); outputs wav (22.05 kHz), wav_lengths. The first voice keeps its original paths.

path	what
english/	English detection and the mixed phonemizer (Python and browser), see below
samples/ (male_voxcpm_1), samples/<voice>/	the 12 held-out sentences, model and teacher
data/	style token map, speaker map
ATTRIBUTION.md	every text source of the training data, and the base model's data

Quick start (ONNX, Python)

import re, sys, json, numpy as np, onnxruntime as ort, soundfile as sf
sys.path.append("english")                          # english_detect.py + english_words.txt (optional fallback: pip install wordfreq)
from english_detect import mark_english
from matcha.text import text_to_sequence            # Matcha-TTS with the tts-pl cleaners: paste english/polish_cleaners.py and
from matcha.utils.utils import intersperse          # english/polish_mixed_cleaners.py into matcha/text/cleaners.py
text = mark_english("Po meetingu wyślę ci feedback na maila? A w weekend zrobię update.")
text = re.sub(r"\s+\?", "?", text)                # keep '?' glued to the word, as in the training text
ids = np.array(intersperse(text_to_sequence(text, ["polish_mixed_cleaners"])[0], 0), np.int64)
vj = json.load(open("onnx/voices.json"))           # other voices: onnx/<voice>/voices.json, onnx/<voice>/matcha_pl_t4.onnx
spk = np.array(vj["speakers"]["0"]["emb"], np.float32) + np.array(vj["styles"]["8"]["emb"], np.float32)   # 8 = neutral style (optional)
sess = ort.InferenceSession("onnx/matcha_pl_t4.onnx")
wav, n = sess.run(None, {"x": ids[None], "x_lengths": np.array([ids.size]), "scales": np.array([0.8, 0.8], np.float32), "spk_emb": spk[None]})
out = wav[0, :n[0]]
if vj.get("defaults", {}).get("highpass_hz"):   # female_voxcpm_1: remove the learned "p" pops
    from scipy.signal import butter, sosfiltfilt
    out = sosfiltfilt(butter(4, vj["defaults"]["highpass_hz"], "highpass", fs=22050, output="sos"), out)
sf.write("out.wav", out, 22050)   # scales = [temperature, length_scale (tempo)]

Text front end: mark_english() wraps English words in braces (Po {meeting}u wyślę ci {feedback}.), the cleaner phonemizes braced spans with espeak-ng en-us and the rest with pl (with the Polish ending of an inflected English word glued on unstressed, and one-letter prepositions before English words handled by rule). You can brace any word yourself. Questions: keep the ? glued to the word ("kuchni?"), as in the training text — the model follows VoxCPM2's question intonation (wh-questions fall, yes/no questions rise); a space before ? makes wh-questions rise. The browser port produces the training phonemes character for character (tested on 300 sentences).

Training

• Base: Matcha-TTS-PL-Pretrain (its target model), new speaker row initialised from the closest real reader; same mel statistics, style tokens and phoneme table (all en-us phones were already in the symbol table, now trained).
• Data: machinekind/polish-synthetic-speech-voxcpm2 — about 8 000 clips (≈ 15 h) per voice, generated by VoxCPM2 with each frozen voice (anchor + continuation, 10 diffusion steps), every clip gated by Whisper (text match), UTMOS ≥ 3.2, speaker similarity ≥ 0.80 and pitch, up to 3 takes per sentence. Source texts reviewed and minimally completed by Claude (cut sentences finished, numbers written as words).
• Matcha fine-tune per voice: male_voxcpm_1 10 k steps (first data version), the others 7.5 k steps; batch 64, lr 5e-5, bf16, RTX 6000 Ada (≈ 1 h per voice); quality plateaus from about 5 k steps.
• Vocoder per voice: teacher-forced mels of the voice's final Matcha checkpoint for all its training clips paired with the VoxCPM2 audio, HiFi-GAN universal v1 fine-tuned 15 k steps, lr 2e-5 (male_voxcpm_2: the 5 k checkpoint, which scored best).

Limitations

• One voice per model (no blending). The rhythm is more regular than the teacher's: Matcha's deterministic duration predictor averages timing.
• English detection uses a curated list (≈ 1 900 words and brands, Polish lookalikes excluded); an unlisted English word is read the Polish way unless you put it in braces.
• Synthetic voice: label generated speech as synthetic; don't present the voices as real people.

Licence

CC BY-SA 4.0 (training data includes CC BY-SA texts; the base model is CC BY-SA 4.0). VoxCPM2 is Apache-2.0; espeak-ng (runtime phonemizer) is GPL-3.0. See ATTRIBUTION.md.