Feature Extraction
Transformers
Safetensors
audio_embeddings
audio
custom_code
self-supervised-learning
audio-embeddings
best-rq-2
audioset
Instructions to use ltuncay/BEST-RQ-ViT with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use ltuncay/BEST-RQ-ViT with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="ltuncay/BEST-RQ-ViT", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModel model = AutoModel.from_pretrained("ltuncay/BEST-RQ-ViT", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
fix: default to single-phase extraction without overlap
Browse files- README.md +4 -2
- config.json +1 -1
- configuration_audio.py +1 -1
README.md
CHANGED
|
@@ -100,7 +100,7 @@ challenge's standardized ranking. These evaluations precede the Transformers exp
|
|
| 100 |
| Masking ratio | 40–60% |
|
| 101 |
| Saved training step | 200,000, verified from checkpoint metadata |
|
| 102 |
| Exported weight dtype | float32 |
|
| 103 |
-
| Extraction policy | `
|
| 104 |
|
| 105 |
## Load the model
|
| 106 |
|
|
@@ -133,7 +133,7 @@ Resample to `processor.sampling_rate` and downmix stereo before preprocessing.
|
|
| 133 |
The extractor performs padding only; spectrogram/convolution features are computed
|
| 134 |
inside the model. Pass the returned sample attention mask for variable durations.
|
| 135 |
Outputs include a frame attention mask and timestamps in milliseconds (-1 for padding).
|
| 136 |
-
`pooler_output` uses the saved HEAR extraction preset,
|
| 137 |
Frame features average frequency patches; they are not the raw frequency-time ViT grid.
|
| 138 |
|
| 139 |
## Fine-tuning
|
|
@@ -194,3 +194,5 @@ BEST-RQ-ViT is the ViT baseline in the BEST-RQ-2 study. For that comparison, cit
|
|
| 194 |
|
| 195 |
The model weights and bundled implementation are released under the MIT license.
|
| 196 |
See `LICENSE` and `CODE_LICENSE`.
|
|
|
|
|
|
|
|
|
| 100 |
| Masking ratio | 40–60% |
|
| 101 |
| Saved training step | 200,000, verified from checkpoint metadata |
|
| 102 |
| Exported weight dtype | float32 |
|
| 103 |
+
| Extraction policy | `native_one_phase` |
|
| 104 |
|
| 105 |
## Load the model
|
| 106 |
|
|
|
|
| 133 |
The extractor performs padding only; spectrogram/convolution features are computed
|
| 134 |
inside the model. Pass the returned sample attention mask for variable durations.
|
| 135 |
Outputs include a frame attention mask and timestamps in milliseconds (-1 for padding).
|
| 136 |
+
`pooler_output` uses the saved HEAR extraction preset, using a single phase with no context-window overlap by default.
|
| 137 |
Frame features average frequency patches; they are not the raw frequency-time ViT grid.
|
| 138 |
|
| 139 |
## Fine-tuning
|
|
|
|
| 194 |
|
| 195 |
The model weights and bundled implementation are released under the MIT license.
|
| 196 |
See `LICENSE` and `CODE_LICENSE`.
|
| 197 |
+
|
| 198 |
+
Default inference uses `native_one_phase` (no context-window overlap and one encoding phase). Reported benchmark results retain their stated evaluation presets; to reproduce overlapping two-phase evaluation, set `model.config.extraction_preset = "overlap50_two_phase"`.
|
config.json
CHANGED
|
@@ -19,7 +19,7 @@
|
|
| 19 |
"pos_embed_type": "sincos",
|
| 20 |
"qkv_bias": true
|
| 21 |
},
|
| 22 |
-
"extraction_preset": "
|
| 23 |
"feature_encoder_kwargs": {},
|
| 24 |
"hidden_size": 768,
|
| 25 |
"model_target": "src.models.best_rq_module.BestRQModule",
|
|
|
|
| 19 |
"pos_embed_type": "sincos",
|
| 20 |
"qkv_bias": true
|
| 21 |
},
|
| 22 |
+
"extraction_preset": "native_one_phase",
|
| 23 |
"feature_encoder_kwargs": {},
|
| 24 |
"hidden_size": 768,
|
| 25 |
"model_target": "src.models.best_rq_module.BestRQModule",
|
configuration_audio.py
CHANGED
|
@@ -45,7 +45,7 @@ class AudioEmbeddingConfig(PretrainedConfig):
|
|
| 45 |
feature_encoder_kwargs: dict[str, Any] | None = None,
|
| 46 |
sampling_rate: int = 16000,
|
| 47 |
spectrogram_adjustment_mode: str = "pad",
|
| 48 |
-
extraction_preset: str = "
|
| 49 |
**kwargs: Any,
|
| 50 |
) -> None:
|
| 51 |
super().__init__(**kwargs)
|
|
|
|
| 45 |
feature_encoder_kwargs: dict[str, Any] | None = None,
|
| 46 |
sampling_rate: int = 16000,
|
| 47 |
spectrogram_adjustment_mode: str = "pad",
|
| 48 |
+
extraction_preset: str = "native_one_phase",
|
| 49 |
**kwargs: Any,
|
| 50 |
) -> None:
|
| 51 |
super().__init__(**kwargs)
|