YAMNet, ONNX (float16 weights)
YAMNet (tensorflow/models, research/audioset/yamnet,
by Manoj Plakal and Dan Ellis, Google): a MobileNetV1 trained on AudioSet that scores each 0.96 s log-mel patch for 521
AudioSet classes, a sigmoid each. This is its network over patches, converted for
@audio/neural-tag, which computes the patches in
JavaScript as upstream's features.py does and runs this file through onnxruntime (Node, or a browser's
onnxruntime-web).
| File | Size | SHA-256 |
|---|---|---|
yamnet.fp16.onnx |
7.5 MB (7,495,175 bytes) | 0a1f3b050ebfdfa651e78fb12714c3c1fe750ceebd58529818a1ddddec515688 |
Source
- Code: tensorflow/models at
c14bf9ad91962cf189f9f58db2132c06247fcd53, the last commit to touchresearch/audioset/yamnet:yamnet.py,features.py,params.py,yamnet_class_map.csv. - Weights:
yamnet.h5, as upstream's README links it (15,296,092 bytes, SHA-25613c3308955bbfaef262f175ac9c40e47b134573a93984f009220dd7cc12a1744).
Graph
| name | shape | ||
|---|---|---|---|
| input | patches |
[n, 96, 64] float32 | log-mel patches: 96 frames of 10 ms ร 64 mel bands, as features.py frames them |
| output | scores |
[n, 521] float32 | a sigmoid per class, in yamnet_class_map.csv's order |
The patches, at 16 kHz mono (features.py, params.py): the waveform padded with zeros to at least 15,600 samples and
then to a whole hop; frames of 400 samples every 160, periodic Hann, |FFT 512|; 64 HTK-mel bands 125 to 7500 Hz
(tf.signal.linear_to_mel_weight_matrix); ln(mel + 0.001); 96 frames every 48 (0.96 s every 0.48 s).
import { tag } from '@audio/neural-tag'
let tags = await tag(samples, 44100) // [{ name: 'Speech', id: '/m/09x0r', index: 0, score }, โฆ]
Conversion and verification
scripts/export.py in @audio/neural-tag builds the graph from yamnet.h5's arrays (h5py, onnx; opset 17): the 14
convolution layers of yamnet.py, each batch norm (center, no scale, epsilon 1e-4) folded into its convolution in
float64, TensorFlow's 'same' padding written out per layer, global average pooling, the dense layer and the sigmoid;
3,729,481 parameters. The weights of over 1024 values are stored as float16 and Cast to float32 in the graph (biases
and the first three kernels stay float32): computed in float32, the file half the float32 export's 14.9 MB.
Against the Keras model (upstream's yamnet.py with yamnet.h5 loaded, TensorFlow 2.21, tf-keras 2.21) on the same 49
patches of seven test signals (speech, plucked notes, a swept tone, white noise, a beep, silence, nothing):
| max |score difference| | mean | top class | |
|---|---|---|---|
| float32 export | 1.6e-6 | 1.5e-8 | the same in 49 of 49 |
| this file | 3.6e-3 | 3.0e-6 | the same in 49 of 49 |
End to end, @audio/neural-tag's JavaScript features through this file against upstream's yamnet_frames_model on the
same waveform: 2.3e-3 at most on 12.3 s of speech, 3.6e-3 on plucked notes, within 1.4e-3 on the rest; the same top
classes. The features themselves are within 2.4e-7 of upstream's ops computed in float64.
Accuracy, as upstream states it
On AudioSet's 20,366-segment evaluation set, over the 521 classes: balanced average d-prime 2.318, balanced mAP 0.306, balanced average lwlrap 0.393 (upstream's README). Six of AudioSet's 527 classes were dropped "on the recommendation of our Fairness reviewers": the gendered Speech and Singing, Battle cry, Funny music. "YAMNet's classifier outputs have not been calibrated across classes, so you cannot directly treat the outputs as probabilities" (the TF Hub model card, Limitations).
Licence and attribution
Apache License 2.0 (LICENSE: tensorflow/models' licence at the commit above, which covers /research).
YAMNet's code and weights are Google's, "Copyright 2019 The TensorFlow Authors"; the same model on Kaggle (formerly TF
Hub google/yamnet/1) states Apache 2.0. Modified from the original weights: converted to ONNX, batch norms folded, the
weights stored as float16. The class names and ids are the AudioSet ontology's, which Google makes available under
CC BY-SA 4.0; AudioSet's labels are CC BY 4.0.
@inproceedings{gemmeke2017audioset,
title = {Audio Set: An ontology and human-labeled dataset for audio events},
author = {Gemmeke, Jort F. and Ellis, Daniel P. W. and Freedman, Dylan and Jansen, Aren and Lawrence, Wade and Moore, R. Channing and Plakal, Manoj and Ritter, Marvin},
booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
year = {2017}
}
@article{howard2017mobilenets,
title = {MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications},
author = {Howard, Andrew G. and Zhu, Menglong and Chen, Bo and Kalenichenko, Dmitry and Wang, Weijun and Weyand, Tobias and Andreetto, Marco and Adam, Hartwig},
journal = {arXiv:1704.04861},
year = {2017}
}