File size: 11,993 Bytes
86dc2b6
 
 
 
 
 
 
 
 
 
 
f2fdb1f
86dc2b6
 
eca55dc
f2fdb1f
 
 
2f6bbfe
86dc2b6
 
 
eca55dc
86dc2b6
 
 
eca55dc
2f6bbfe
 
 
f2fdb1f
 
 
 
2f6bbfe
3edbc10
2f6bbfe
f2fdb1f
 
 
17ca7cd
f2fdb1f
3edbc10
 
 
f2fdb1f
 
 
2f6bbfe
 
 
 
f2fdb1f
 
 
 
 
 
 
 
 
2f6bbfe
209f8a9
 
f2fdb1f
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
86dc2b6
eca55dc
86dc2b6
 
 
 
 
 
 
 
 
 
 
4af0bfb
eca55dc
86dc2b6
eca55dc
86dc2b6
eca55dc
 
86dc2b6
eca55dc
 
86dc2b6
 
eca55dc
86dc2b6
 
 
eca55dc
86dc2b6
 
 
 
 
 
 
 
 
 
eca55dc
 
86dc2b6
 
 
 
4af0bfb
86dc2b6
eca55dc
86dc2b6
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
4af0bfb
86dc2b6
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
eca55dc
 
86dc2b6
eca55dc
86dc2b6
 
4af0bfb
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
---
library_name: transformers
license: mit
tags:
  - audio
  - feature-extraction
  - custom_code
  - self-supervised-learning
  - audio-embeddings
  - best-rq-2
  - audioset
  - arxiv:2606.30700
---
# BEST-RQ-2

[![BEST-RQ-2 paper: arXiv 2606.30700](https://img.shields.io/badge/arXiv-2606.30700-b31b1b.svg)](https://arxiv.org/abs/2606.30700)

> **Recommended: [BEST-RQ-2.2-base-2M](https://huggingface.co/ltuncay/BEST-RQ-2.2-base-2M)** has the highest reported overall X-ARES scores in this comparison (2M training steps, `overlap50` evaluation).

BEST-RQ-2 is a self-supervised audio encoder trained on AudioSet for
a configured budget of **200,000 optimizer steps**. It produces **768-dimensional** clip and frame
embeddings from mono **16 kHz** waveforms, and supports downstream fine-tuning.

This repository contains the trained encoder, its preprocessing configuration,
and the custom Transformers implementation. No installation of the research
repository is needed.

## X-ARES results

Scores on [X-ARES](https://arxiv.org/abs/2505.16369) (0–100, higher is better).
Audio-JEPA, BEST-RQ (Conformer), BEST-RQ (ViT), and the original BEST-RQ-2
variants below use the **same AudioSet split for 200,000 steps**.
**BEST-RQ-2.2 (2M)** uses **2,000,000 steps**, with batch size 128 per GPU
on two GPUs (effective batch 256). The pretrained baselines are shown for comparison.

| Model | Speech | Music | Environment | Global Mean | Mean of Means | Hugging Face model |
| --- | ---: | ---: | ---: | ---: | ---: | --- |
| data2vec | 50.62 | 23.24 | 15.41 | 37.83 | 29.76 | [data2vec-audio-base](https://huggingface.co/facebook/data2vec-audio-base) |
| wav2vec 2.0 | 41.79 | 34.94 | 29.52 | 37.84 | 35.42 | [wav2vec2-large-100k-voxpopuli](https://huggingface.co/facebook/wav2vec2-large-100k-voxpopuli) |
| Whisper | 49.19 | 38.67 | 28.61 | 42.75 | 38.82 | [whisper-base](https://huggingface.co/openai/whisper-base) |
| Audio-JEPA | 29.64 | 44.27 | 25.61 | 31.18 | 33.17 | [Audio-JEPA-base](https://huggingface.co/ltuncay/Audio-JEPA-base) |
| BEST-RQ (Conformer) | 40.43 | 35.58 | 30.81 | 37.43 | 35.60 | Separate codebase (recipe unavailable) |
| BEST-RQ (ViT) | 32.87 | 41.62 | 34.50 | 34.88 | 36.33 | [BEST-RQ-ViT](https://huggingface.co/ltuncay/BEST-RQ-ViT) |
| BEST-RQ-2 (Interspeech 2026) | 38.49 | 54.40 | 46.39 | 43.21 | 46.43 | [BEST-RQ-2](https://huggingface.co/ltuncay/BEST-RQ-2) |
| BEST-RQ-2.1 | 52.60 | 62.23 | 53.38 | 54.59 | 56.07 | [BEST-RQ-2.1-base](https://huggingface.co/ltuncay/BEST-RQ-2.1-base) |
| BEST-RQ-2.2 | 53.78 | 63.90 | 55.71 | 56.11 | 57.80 | [BEST-RQ-2.2-base](https://huggingface.co/ltuncay/BEST-RQ-2.2-base) |
| BEST-RQ-2.2 (2M) | 58.62 | 65.74 | 61.80 | 58.77 | 62.05 | [BEST-RQ-2.2-base-2M](https://huggingface.co/ltuncay/BEST-RQ-2.2-base-2M) |
| BEST-RQ-2.2 (2M, overlap50) | **60.87** | **65.78** | **65.90** | **61.13** | **64.18** | [BEST-RQ-2.2-base-2M](https://huggingface.co/ltuncay/BEST-RQ-2.2-base-2M) |

Global Mean averages all benchmark task scores. Mean of Means gives equal
weight to the Speech, Music, and Environment category means.

The two 2M rows report the same training checkpoint with different extraction
settings: [original evaluation](https://wandb.ai/tuncay-ludovic/X-ARES/runs/cw0rrt7o)
and [50% overlapping windows](https://wandb.ai/tuncay-ludovic/X-ARES/runs/a7a3fy9z).
Their scores use the logged **Combined** metric: equal weight for the linear-probe
and k-NN aggregates, excluding unavailable tasks separately for each method.
The earlier rows retain previously reported scores. Extraction and evaluation
settings differ across runs, so this is not a controlled training-duration ablation.

Audio-JEPA scores were supplied by the author for [run `jp6l70l6`](https://wandb.ai/tuncay-ludovic/audio%20embeddings/runs/jp6l70l6). The other 200k-model and pretrained-baseline scores are reported in the [project README](https://github.com/LudovicTuncay/audio-embeddings/blob/bb88bf790b1dcf8251c6b38e7a4766534adf33d3/README.md). These are reported research results, not a new benchmark run of the Transformers exports.

The Audio-JEPA row refers to the newer **16 kHz, 200,000-step** run, not the original ICME model (**32 kHz, 100,000 steps**). The author reports better results for this newer checkpoint.

## HEAR results

Scores on [HEAR](https://arxiv.org/abs/2203.03022) (0–100, higher is better),
using `overlap50_two_phase` extraction. All five project models cover all 19 tasks.
BEST-RQ-2.2 (2M) was trained for 2,000,000 steps; the other rows use 200,000 steps.

| Model | Speech | Music | Environment | Global Mean | Mean of Means | Coverage |
| --- | ---: | ---: | ---: | ---: | ---: | --- |
| Audio-JEPA | 36.53 | 74.20 | 51.85 | 56.42 | 54.19 | 19/19 |
| BEST-RQ-2 | 50.74 | 79.46 | 69.21 | 67.69 | 66.47 | 19/19 |
| BEST-RQ-2.1 | 64.74 | 86.47 | 70.83 | 75.50 | 74.02 | 19/19 |
| BEST-RQ-2.2 | 65.76 | 85.13 | 76.19 | 76.66 | 75.69 | 19/19 |
| BEST-RQ-2.2 (2M) | 70.18 | 85.38 | 81.82 | 79.64 | 79.13 | 19/19 |

The 2M scores come from [this W&B evaluation](https://wandb.ai/tuncay-ludovic/HEAR/runs/hear-v2-aa56c55deb6099f6); its checkpoint hash
matches the released 2M checkpoint. The earlier rows retain the reported
2026-09-19 W&B comparison. [Score sources and task-level 2M results](https://github.com/LudovicTuncay/audio-embeddings/blob/master/docs/data/best_rq_2_2_2m_results.json).
These are arithmetic means of primary task scores, with equal task weighting for
Global Mean and equal domain weighting for Mean of Means; they are not the
challenge's standardized ranking. These evaluations precede the Transformers export.

## Model and training

| Property | Value |
| --- | --- |
| Architecture | 12-layer Transformer, 768 dimensions, 12 attention heads |
| Input frontend | 128-bin mel spectrogram with a linear patch projection |
| Patch shape | 16 mel bins by 16 time frames |
| Transformer | Sinusoidal positional embeddings, LayerNorm, GELU MLP |
| Training data | AudioSet |
| Training objective | Masked prediction of frozen codebook targets |
| Masking ratio | 40–60% |
| Checkpoint step metadata | Not recorded in this older safetensors file |
| Exported weight dtype | float32 |
| Extraction policy | `native_one_phase` |

## Load the model

Install the runtime dependencies in your Python environment:

```bash
pip install "torch>=2.9.1" "torchaudio>=2.9.1" "timm>=0.9" "einops>=0.7" "transformers>=4.57,<6"
```

Use matching PyTorch and torchaudio versions. GPU installations may require the
appropriate PyTorch build for your CUDA version.

```python
import torch
from transformers import AutoFeatureExtractor, AutoModel

model_name = "ltuncay/BEST-RQ-2"
processor = AutoFeatureExtractor.from_pretrained(model_name, trust_remote_code=True)
model = AutoModel.from_pretrained(model_name, trust_remote_code=True).eval()
audio = torch.zeros(processor.sampling_rate)  # Replace with real mono audio.
inputs = processor(audio, sampling_rate=processor.sampling_rate, return_tensors="pt")
with torch.inference_mode():
    outputs = model(**inputs)
clip_embeddings = outputs.pooler_output
frame_embeddings = outputs.last_hidden_state
output_dim = model.config.encoder_kwargs["embed_dim"]
```

Resample to `processor.sampling_rate` and downmix stereo before preprocessing.
The extractor performs padding only; spectrogram/convolution features are computed
inside the model. Pass the returned sample attention mask for variable durations.
Outputs include a frame attention mask and timestamps in milliseconds (-1 for padding).
`pooler_output` uses the saved HEAR extraction preset, using a single phase with no context-window overlap by default.
Frame features average frequency patches; they are not the raw frequency-time ViT grid.

## Fine-tuning

For fine-tuning call `model.train()`, attach a task head and optimize its parameters
alongside the model. Save with `model.save_pretrained(path)` and
`processor.save_pretrained(path)`. This encoder export excludes pretraining predictors,
quantizers, teachers and optimizer state. Continue self-supervised research training
with the original Lightning code and checkpoints.

## Reproducibility and provenance

`revision` is optional; pin both loaders to the same full commit hash for reproducibility.
Custom Python code is included in this repository and requires `trust_remote_code=True`.
The weights use safetensors. See `export_manifest.json` for source and validation details.
Source: the existing **AECC 2026 submission** in this repository, pinned at
[`a03eeb5c4433f4bf7a7e6e8b4724af862789959c`](https://huggingface.co/ltuncay/BEST-RQ-2/tree/a03eeb5c4433f4bf7a7e6e8b4724af862789959c).
The export uses its `BEST-RQ-2.safetensors` and matching `config.yaml`.
The saved configuration specifies a 200,000-step training budget, but this older
safetensors file has no saved-step metadata. The actual checkpoint step therefore
remains unspecified in the export manifest. Source file hashes are recorded there.

## AECC 2026 compatibility

The [original submission](https://huggingface.co/ltuncay/BEST-RQ-2/tree/aecc-2026-submission)
is preserved at revision `aecc-2026-submission`, including its model card, checkpoint,
technical report and xares-llm integration. The legacy files also remain available
on `main` for existing consumers.

This Transformers export uses the **same pretrained encoder weights**. Its public
outputs follow the shared BEST-RQ-2 family extraction policy: frequency-averaged
frame embeddings and clip embeddings with non-overlapping context windows by default.
The original AECC wrapper returned spectrogram patch tokens with sequential audio
chunking, so the two public interfaces are not interchangeable for reproducing
benchmark results. Use the archived revision for the original evaluation setup.

A comparison against the archived frontend and encoder on a 10-second waveform
found exact spectrogram and patch-projection agreement, with maximum encoder
absolute difference `1.67e-6` (within floating-point tolerance). The validation
manifest records this comparison.

The exported model matched the pre-export encoder exactly on the exporter's
variable-duration test batch. These are integration checks, not a downstream
benchmark evaluation of this release.

## Intended uses and limitations

Use these embeddings as features for audio research, classification, retrieval,
or downstream fine-tuning. This is an encoder: it does not generate transcripts,
audio, or class labels without a downstream model.

AudioSet training does not establish performance on every language, acoustic
domain, demographic group, or downstream task. Evaluate the model on your target
data. No new downstream benchmark scores are claimed for this export.

The feature extractor pads waveforms and checks the sample rate; it does not
resample or downmix. Unequal-length clips are processed individually after padding
is removed. The output embeddings follow the saved windowing policy rather than
exposing the raw spectrogram token grid.

Browse the [BEST-RQ-2 family collection](https://huggingface.co/collections/ltuncay/best-rq-2-family-6aad761396dfe31da48ff91a).

## Source and citation

Research code: [audio-embeddings](https://github.com/LudovicTuncay/audio-embeddings).
BEST-RQ-2 belongs to the BEST-RQ-2 family. Cite the family paper when using it:

```bibtex
@inproceedings{tuncay2026best,
  title={BEST-RQ-2: Contextualize-Then-Predict, a Two-Step Approach for Self-Supervised Audio Representations},
  author={Tuncay, Ludovic K and Labb{\'e}, Etienne and Pellegrini, Thomas},
  booktitle={Interspeech 2026},
  year={2026}
}
```

## License

The model weights and bundled implementation are released under the MIT license.
See `LICENSE` and `CODE_LICENSE`.

Default inference uses `native_one_phase` (no context-window overlap and one encoding phase). Reported benchmark results retain their stated evaluation presets; to reproduce overlapping two-phase evaluation, set `model.config.extraction_preset = "overlap50_two_phase"`.