| | --- |
| | license: apache-2.0 |
| | license_link: >- |
| | https://github.st.com/AIS/stm32ai-modelzoo/raw/master/audio_event_detection/LICENSE.md |
| | pipeline_tag: audio-classification |
| | --- |
| | # Quantized Yamnet |
| |
|
| | ## **Use case** : `AED` |
| |
|
| | # Model description |
| |
|
| | Yamnet is a very well-known audio classification model, pre-trained on Audioset and released by Google. The default model outputs embedding vectors of size 1024. |
| |
|
| | As the default Yamnet is a bit too large to fit on most microcontrollers (over 3M parameters), we provide in this model zoo a much downsized version of Yamnet which outputs embeddings of size 256. |
| |
|
| | We now also provide the original Yamnet (named Yamnet-1024 in this repo), with its original 3.2 million parameters, for use on the STM32N6. |
| |
|
| | Additionally, the default Yamnet provided by Google expects waveforms as input and has specific custom layers to perform conversion to mel-spectrogram and patch extraction. |
| | These custom layers are not included in Yamnet-256 or Yamnet-1024, as STEDGEAI cannot convert them to C code, and more efficient implementations of these operations already exist on microcontrollers. |
| | Thus, Yamnet-256 and Yamnet-1024 expect mel-spectrogram patches of size 64x96, format (n_mels, n_frames) |
| |
|
| | The model is quantized in int8 using tensorflow lite converter for Yamnet-256, and ONNX quantizer for Yamnet-1024. |
| |
|
| | We provide Yamnet-256s for two different datasets : ESC-10, which is a small research dataset, and FSD50K, a large generalist dataset using the audioset ontology. |
| | For FSD50K, the model is trained to detect a small subset of the classes included in the dataset. This subset is : Knock, Glass, Gunshots and gunfire, Crying and sobbing, Speech. |
| |
|
| | The inference time & footprints are very similar in both cases, with the FSD50K model being very slightly smaller and faster. |
| |
|
| | ## Network information |
| |
|
| | Yamnet-256 |
| |
|
| | | Network Information | Value | |
| | |-------------------------|-----------------| |
| | | Framework | TensorFlow Lite | |
| | | Parameters Yamnet-256 | 130 K | |
| | | Quantization | int8 | |
| | | Provenance | https://tfhub.dev/google/yamnet/1 | |
| |
|
| | Yamnet-1024 |
| |
|
| | | Network Information | Value | |
| | |-------------------------|-----------------| |
| | | Framework | TensorFlow Lite | |
| | | Parameters Yamnet-1024 | 3.2 M | |
| | | Quantization | int8 | |
| | | Provenance | https://tfhub.dev/google/yamnet/1 | |
| |
|
| |
|
| | ## Network inputs / outputs |
| |
|
| |
|
| | The network expects spectrogram patches of 96 frames and 64 mels, of shape (64, 96, 1). |
| | Additionally, the original Yamnet converts waveforms to spectrograms by using an FFT and window size of 25 ms, a hop length of 10ms, and by clipping frequencies between 125 and 7500 Hz. |
| |
|
| | Yamnet-256 outputs embedding vectors of size 256. If you use the model zoo scripts to perform transfer learning, a classification head with the specified number of classes will automatically be added to the network. |
| |
|
| | Yamnet-1024 is the original yamnet without the TF preprocessing layers attached, and outputs embedding vectors of size 1024. If you use the model zoo scripts to perform transfer learning, a classification head with the specified number of classes will automatically be added to the network. |
| |
|
| |
|
| | ## Recommended platforms |
| |
|
| | For Yamnet-256 |
| | | Platform | Supported | Recommended | |
| | |----------|-----------|-----------| |
| | | STM32U5 |[x]|[x]| |
| | | STM32N6 |[x]|[x]| |
| |
|
| | For Yamnet-1024 |
| | | Platform | Supported | Recommended | |
| | |----------|-----------|-----------| |
| | | STM32N6 |[x]|[x]| |
| |
|
| |
|
| |
|
| | # Performances |
| |
|
| | ## Metrics |
| |
|
| | * Measures are done with default STEDGEAI configuration with enabled input / output allocated option. |
| |
|
| | * `tl` stands for "transfer learning", meaning that the model backbone weights were initialized from a pre-trained model, then only the last layer was unfrozen during the training. |
| |
|
| | ### Reference **NPU** memory footprint based on ESC-10 dataset |
| | |Model | Dataset | Format | Resolution | Series | Internal RAM (KiB) | External RAM (KiB) | Weights Flash (KiB) | STEdgeAI Core version | |
| | |----------|------------------|--------|-------------|------------------|------------------|---------------------|-------|-------------------------| |
| | | [Yamnet 256](https://github.com/STMicroelectronics/stm32ai-modelzoo/tree/main/audio_event_detection/yamnet/esc10/yamnet_e256_64x96_tl/yamnet_e256_64x96_tl_int8.tflite) | esc-10 | Int8 | 64x96x1 | STM32N6 | 144 | 0.0 | 137.33 | 3.0.0 | |
| | | [Yamnet 1024](https://github.com/STMicroelectronics/stm32ai-modelzoo/tree/main/audio_event_detection/yamnet/esc10/yamnet_e1024_64x96_tl/yamnet_e1024_64x96_tl_qdq_int8.onnx) | esc-10 | Int8 | 64x96x1 | STM32N6 | 144 | 0.0 | 3159.2 | 3.0.0 | |
| |
|
| | ### Reference **NPU** inference time based on ESC-10 dataset |
| | | Model | Dataset | Format | Resolution | Board | Execution Engine | Inference time (ms) | Inf / sec | STEdgeAI Core version | |
| | |--------|------------------|--------|-------------|------------------|------------------|---------------------|-------|-------------------------| |
| | | [Yamnet 256](https://github.com/STMicroelectronics/stm32ai-modelzoo/tree/main/audio_event_detection/yamnet/esc10/yamnet_e256_64x96_tl/yamnet_e256_64x96_tl_int8.tflite) | esc-10 | Int8 | 64x96x1 | STM32N6570-DK | NPU/MCU | 0.93 | 1075.27 | 3.0.0 | |
| | | [Yamnet 1024](https://github.com/STMicroelectronics/stm32ai-modelzoo/tree/main/audio_event_detection/yamnet/esc10/yamnet_e1024_64x96_tl/yamnet_e1024_64x96_tl_qdq_int8.onnx) | esc-10 | Int8 | 64x96x1 | STM32N6570-DK | NPU/MCU | 9.12 | 109.64 | 3.0.0 | |
| |
|
| |
|
| | ### Reference **MCU** memory footprint based on ESC-10 dataset |
| | | Model | Format | Resolution | Series | Activation RAM (kB) | Runtime RAM (kB) | Weights Flash (kB) | Code Flash (kB) | Total RAM (kB) | Total Flash (kB) | STEdgeAI Core version | |
| | |-------------------|--------|------------|---------|----------------|-------------|---------------|------------|-------------|-------------|-----------------------| |
| | |[Yamnet 256](https://github.com/STMicroelectronics/stm32ai-modelzoo/tree/main/audio_event_detection/yamnet/esc10/yamnet_e256_64x96_tl/yamnet_e256_64x96_tl_int8.tflite) | Int8 | 64x96x1 | B-U585I-IOT02A | 109.57 | 0.99 | 135.91 | 31.19 | 110.56 | 167.1 | 3.0.0 | |
| | |[Yamnet 1024](https://github.com/STMicroelectronics/stm32ai-modelzoo/tree/main/audio_event_detection/yamnet/esc10/yamnet_e1024_64x96_tl/yamnet_e1024_64x96_tl_qdq_int8.onnx) | Int8 | 64x96x1 | STM32N6 | 144.0 | 1.77 | 3159.2 | 184.74 | 145.77 | 3343.94 | 3.0.0 | |
| |
|
| | ### Reference inference time based on ESC-10 dataset |
| | | Model | Format | Resolution | Board | Execution Engine | Frequency | Inference time | STEdgeAI Core version | |
| | |-------------------|--------|------------|------------------|------------------|--------------|-----------------|-----------------------| |
| | | [Yamnet 256](https://github.com/STMicroelectronics/stm32ai-modelzoo/tree/main/audio_event_detection/yamnet/esc10/yamnet_e256_64x96_tl/yamnet_e256_64x96_tl_int8.tflite) | Int8 | 64x96x1 | B-U585I-IOT02A | 1 CPU | 160 MHz | 279.99 ms | 3.0.0 |
| | |[Yamnet 1024](https://github.com/STMicroelectronics/stm32ai-modelzoo/tree/main/audio_event_detection/yamnet/esc10/yamnet_e1024_64x96_tl/yamnet_e1024_64x96_tl_qdq_int8.onnx) | Int8 | 64x96x1 | STM32N6 | 1 CPU + 1 NPU | 800MhZ/1000MhZ | 9.12 ms | 3.0.0 |
| |
|
| |
|
| | ### Accuracy with ESC-10 dataset |
| |
|
| | A note on clip-level accuracy : In a traditional AED data processing pipeline, audio is converted to a spectral representation (in this model zoo, mel-spectrograms), which is then cut into patches. Each patch is fed to the inference network, and a label vector is output for each patch. The labels on these patches are then aggregated based on which clip the patch belongs to, to form a single aggregate label vector for each clip. Accuracy is then computed on these aggregate label vectors. |
| |
|
| | The reason this metric is used instead of patch-level accuracy is because patch-level accuracy varies immensely depending on the specific manner used to cut spectrogram into patches, and also because clip-level accuracy is the metric most often reported in research papers. |
| |
|
| | | Model | Format | Resolution | Clip-level Accuracy | |
| | |-------|--------|------------|----------------| |
| | | [Yamnet 256](https://github.com/STMicroelectronics/stm32ai-modelzoo/tree/main/audio_event_detection/yamnet/esc10/yamnet_e256_64x96_tl/yamnet_e256_64x96_tl.keras) | float32 | 64x96x1 | 94.9% | |
| | | [Yamnet 256](https://github.com/STMicroelectronics/stm32ai-modelzoo/tree/main/audio_event_detection/yamnet/esc10/yamnet_e256_64x96_tl/yamnet_e256_64x96_tl_int8.tflite) | int8 | 64x96x1 | 94.9% | |
| | | [Yamnet 1024](https://github.com/STMicroelectronics/stm32ai-modelzoo/tree/main/audio_event_detection/yamnet/esc10/yamnet_e1024_64x96_tl/yamnet_e1024_64x96_tl.keras) | float32 | 64x96x1 | 100.0% | |
| | | [Yamnet 1024](https://github.com/STMicroelectronics/stm32ai-modelzoo/tree/main/audio_event_detection/yamnet/esc10/yamnet_e1024_64x96_tl/yamnet_e1024_64x96_tl_qdq_int8.onnx) | int8 | 64x96x1 | 100.0% | |
| |
|
| |
|
| |
|
| | ### Accuracy with FSD50K dataset - Domestic AED use case |
| | In this use case, the model is trained to detect a small subset of the classes included in the dataset. This subset is : Knock, Glass, Gunshots and gunfire, Crying and sobbing, Speech. |
| |
|
| | A note on clip-level accuracy : In a traditional AED data processing pipeline, audio is converted to a spectral representation (in this model zoo, mel-spectrograms), which is then cut into patches. Each patch is fed to the inference network, and a label vector is output for each patch. The labels on these patches are then aggregated based on which clip the patch belongs to, to form a single aggregate label vector for each clip. Accuracy is then computed on these aggregate label vectors. |
| |
|
| | The reason this metric is used instead of patch-level accuracy is because patch-level accuracy varies immensely depending on the specific manner used to cut spectrogram into patches, and also because clip-level accuracy is the metric most often reported in research papers. |
| |
|
| | **IMPORTANT NOTE** : The accuracy for the model with the "unknown class" added is significantly lower when performing inference on PC. This is because this additional class regroups a lot (appromiatively 194 in this specific case) of other classes, and thus drags performance down a bit. |
| |
|
| | However, contrary to what the numbers might suggest online performance on device is much improved in practice by this addition, in this specific case. |
| |
|
| | Note that accuracy with unknown class is lower. This is normal |
| | | Model | Format | Resolution | Clip-level Accuracy | |
| | |-------|--------|------------|----------------| |
| | | [Yamnet 256 without unknown class](https://github.com/STMicroelectronics/stm32ai-modelzoo/tree/main/audio_event_detection/yamnet/fsd50k/yamnet_e256_64x96_tl/without_unknown_class/yamnet_e256_64x96_tl.keras) | float32 | 64x96x1 | 86.0% | |
| | | [Yamnet 256 without unknown class](https://github.com/STMicroelectronics/stm32ai-modelzoo/tree/main/audio_event_detection/yamnet/fsd50k/yamnet_e256_64x96_tl/without_unknown_class/yamnet_e256_64x96_tl_int8.tflite) | float32 | 64x96x1 | 87.0% | |
| | | [Yamnet 256 with unknown class](https://github.com/STMicroelectronics/stm32ai-modelzoo/tree/main/audio_event_detection/yamnet/fsd50k/yamnet_e256_64x96_tl/with_unknown_class/yamnet_e256_64x96_tl.keras) | float32 | 64x96x1 | 73.0% | |
| | | [Yamnet 256 with unknown class](https://github.com/STMicroelectronics/stm32ai-modelzoo/tree/main/audio_event_detection/yamnet/fsd50k/yamnet_e256_64x96_tl/with_unknown_class/yamnet_e256_64x96_tl_int8.tflite) | int8 | 64x96x1 | 73.9% | |
| |
|
| | ## Retraining and Integration in a simple example: |
| |
|
| | Please refer to the stm32ai-modelzoo-services GitHub [here](https://github.com/STMicroelectronics/stm32ai-modelzoo-services) |
| |
|
| |
|
