| --- |
| license: mit |
| base_model: Aratako/MioCodec-25Hz-24kHz |
| base_model_relation: adapter |
| language: |
| - ja |
| - en |
| tags: |
| - speaker-embedding |
| - webgpu |
| - browser |
| - voice-cloning |
| --- |
| |
| # MioCodec's speaker encoder, in one file for the browser |
|
|
| The global-embedding path of |
| [Aratako/MioCodec-25Hz-24kHz](https://huggingface.co/Aratako/MioCodec-25Hz-24kHz)'s |
| encoder — reference audio in, a 128-dimensional speaker embedding out — |
| collected into a single f32 safetensors so a browser can fetch it in one |
| request. Built for [voxshot](https://github.com/m96-chan/voxshot). |
|
|
| **This is a repackaging, not a new model.** No weight here was trained; every |
| tensor is copied from one of the two sources below. |
|
|
| | | | |
| | --- | ---: | |
| | `encoder-weights.safetensors` | 117,303,232 B, 99 tensors, 29,322,838 params | |
| | `encoder-weights.json` | 8,905 B — tensor index and provenance | |
|
|
| ## Why it exists |
|
|
| The weights for this one path live in **two different places**: the |
| `GlobalEncoder` (50 tensors) is in MioCodec's own checkpoint, while the WavLM |
| front end it runs on top of is a torchaudio download. A browser should not |
| have to know that, nor fetch a 378 MB `.pth` to use 4 MB of it. So the path is |
| collected once, ahead of time, into one file. |
|
|
| Only the **global** branch is included. MioCodec's encoder also produces |
| content tokens — that half is not here, and is not needed for speaker |
| embedding. |
|
|
| ## What the path is |
|
|
| ``` |
| 24 kHz mono |
| → symmetric zero pad |
| → resample 24k→16k (torchaudio's polyphase filter, precomputed |
| here as a conv1d kernel [2, 1, 23], stride 3) |
| → WavLM feature extractor (7 strided convs, GroupNorm on the first, GELU) |
| → LayerNorm + Linear 512→768 |
| → positional conv (grouped k=128, weight-norm already folded) |
| → transformer layers 1 and 2 (12 heads; bucketed, gated relative-position |
| bias — the global branch reads only these two) |
| → mean of the two layers |
| → ConvNeXt backbone, 4 blocks, 768→384 |
| → attentive statistics pooling → Linear 768→128 → LayerNorm |
| → 128-dim embedding |
| ``` |
|
|
| Two details a reimplementation gets wrong and this file cannot tell you, so |
| they are written down here: |
|
|
| - torchaudio's Base+ WavLM is **post-norm**, so an encoder-level LayerNorm runs |
| between the positional-conv residual and layer 1. |
| - the pooling softmax is over **time**, not channels. |
|
|
| ## Naming |
|
|
| WavLM tensors keep torchaudio's own names under a `wavlm.` prefix, the |
| `GlobalEncoder` tensors keep MioCodec's names under `global_encoder.`, and the |
| precomputed resampler is `resample.kernel`. `encoder-weights.json` lists every |
| tensor with its shape. |
|
|
| The positional convolution's weight normalization is **already folded** into |
| its weight — there is no separate `weight_g` / `weight_v` to combine. |
|
|
| ## Provenance |
|
|
| Produced by [`spike/miocodec/export_encoder_weights.py`](https://github.com/m96-chan/voxshot/blob/main/spike/miocodec/export_encoder_weights.py) from: |
|
|
| ``` |
| Aratako/MioCodec-25Hz-24kHz model.safetensors |
| 523,087,956 bytes |
| sha256 60483759cde136451d53ff5a2f7e283c015758ab540de9037b5d4519366a7705 |
| |
| torchaudio WAVLM_BASE_PLUS wavlm_base_plus.pth |
| 377,604,347 bytes |
| sha256 136a3e720c04f2c77bf7a4dc6a3868b14d5a2c145a988114b733cb1a8428be98 |
| ``` |
|
|
| Both digests are recorded in `encoder-weights.json` and cross-checked at export |
| time against the golden this port is verified with, so the weights and the |
| checkpoints they came from cannot drift apart unnoticed. |
|
|
| ## Accuracy |
|
|
| The TypeScript port that consumes this file was checked stage by stage against |
| the reference encoder running under torch, on two real reference clips and one |
| synthetic input. Every stage agrees to 1e-4 relative to its own peak, except |
| four ConvNeXt stages on one clip which reach 1e-4 in the reference's *own* |
| f32-versus-f64 comparison — conditioning of that clip, not port error. The |
| final embedding agrees to 7.3e-7. |
|
|
| ## License and attribution |
|
|
| MIT, which both sources carry. |
|
|
| - **MioCodec**: [Aratako/MioCodec-25Hz-24kHz](https://huggingface.co/Aratako/MioCodec-25Hz-24kHz) by Chihiro Arata, MIT. |
| - **WavLM Base+**: originally published by the authors of *WavLM* (Chen et al., 2022) under the MIT License and redistributed by torchaudio under the same license. |
| [Source](https://github.com/microsoft/unilm/tree/65f15af2a307ebb64cfb25adf54375b002e6fe8d/wavlm#pre-trained-models) · |
| [License](https://github.com/microsoft/unilm/blob/65f15af2a307ebb64cfb25adf54375b002e6fe8d/LICENSE) |
|
|
| ```bibtex |
| @article{chen2022wavlm, |
| title = {WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing}, |
| author = {Chen, Sanyuan and Wang, Chengyi and Chen, Zhengyang and others}, |
| journal = {IEEE Journal of Selected Topics in Signal Processing}, |
| year = {2022} |
| } |
| ``` |
|
|
