|
--- |
|
language: |
|
- en |
|
license: mit |
|
base_model: openai/whisper-small |
|
tags: |
|
- generated_from_trainer |
|
metrics: |
|
- wer |
|
model-index: |
|
- name: whisper-small-singlish-122k |
|
results: |
|
- task: |
|
type: automatic-speech-recognition |
|
dataset: |
|
name: NSC |
|
type: NSC |
|
metrics: |
|
- name: WER |
|
type: WER |
|
value: 9.69 |
|
--- |
|
|
|
# Whisper-small-singlish-122k. |
|
|
|
This model is a [openai/whisper-small](https://huggingface.co/openai/whisper-small), fine-tuned on a subset (122k samples) of the [National Speech Corpus](https://www.imda.gov.sg/how-we-can-help/national-speech-corpus). |
|
|
|
The following results on the evaluation set (43,788k samples) are reported: |
|
|
|
- Loss: 0.171377 |
|
- WER: 9.69 |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
- **Developed by:** [jensenlwt](https://huggingface.co/jensenlwt) |
|
- **Model type:** automatic-speech-recognition |
|
- **License:** MIT |
|
- **Finetuned from model:** [openai/whisper-small](https://huggingface.co/openai/whisper-small) |
|
|
|
## Uses |
|
|
|
The model is intended as exploration exercise to develop better ASR model for Singapore English (singlish). |
|
|
|
The recommended audio usage for testing should be: |
|
|
|
1. Involves local Singapore slang, dialect, names, and terms etc. |
|
2. Involves Singaporean accent. |
|
|
|
### Direct Use |
|
|
|
To use the model in an application, you can make use of `transformers`: |
|
|
|
```python |
|
# Use a pipeline as a high-level helper |
|
from transformers import pipeline |
|
|
|
pipe = pipeline("automatic-speech-recognition", model="jensenlwt/whisper-small-singlish-122k") |
|
``` |
|
|
|
### Out-of-Scope Use |
|
|
|
- Long form audio |
|
- Broken Singlish (typically from older generation) |
|
- Poor quality audio (audio samples are recorded in a controlled environment) |
|
- Conversation (as the model is not trained on conversation) |
|
|
|
## Training Details |
|
|
|
### Training Data |
|
|
|
We made use of the [National Speech Corpus](https://www.imda.gov.sg/how-we-can-help/national-speech-corpus) for training. |
|
In specific, we made use of **Part 2** – which is a series of audio samples of prompted read speech recordings that involves local named entities, slang, and dialect. |
|
|
|
To train, I make used of the first 300 transcripts in the corpus, which is around 122k samples from ~161 speakers. |
|
|
|
### Training Procedure |
|
|
|
The model is fine-tuned with occasional interruptions to adjust batch size to maximise GPU utilisation. |
|
In addition, I also end training early if eval_loss does not decrease in two evaluation steps as per previous training experience. |
|
|
|
#### Training Hyperparameters |
|
|
|
The following hyperparameters are used: |
|
|
|
- **batch_size**: 128 |
|
- **gradient_accumulation_steps**: 1 |
|
- **learning_rate**: 1e-5 |
|
- **warmup_steps**: 500 |
|
- **max_steps**: 5000 |
|
- **fp16**: true |
|
- **eval_batch_size**: 32 |
|
- **eval_step**: 500 |
|
- **max_grad_norm**: 1.0 |
|
- **generation_max_length**: 225 |
|
|
|
#### Training Results |
|
|
|
| Steps | Epoch | Train Loss | Eval Loss | WER | |
|
|:-----:|:--------:|:----------:|:---------:|:------------------:| |
|
| 500 | 0.654450 | 0.7418 | 0.3889 | 17.968250 | |
|
| 1000 | 1.308901 | 0.2831 | 0.2519 | 11.880948 | |
|
| 1500 | 1.963351 | 0.1960 | 0.2038 | 9.948440 | |
|
| 2000 | 2.617801 | 0.1236 | 0.1872 | 9.420248 | |
|
| 2500 | 3.272251 | 0.0970 | 0.1791 | 8.539280 | |
|
| 3000 | 3.926702 | 0.0728 | 0.1714 | 8.207827 | |
|
| 3500 | 4.581152 | 0.0484 | 0.1741 | 8.145801 | |
|
| 4000 | 5.235602 | 0.0401 | 0.1773 | 8.138047 | |
|
|
|
The model with the lowest evaluation loss is used as the final checkpoint. |
|
|
|
### Testing Data, Factors & Metrics |
|
|
|
#### Testing Data |
|
|
|
To test the model, I made use of the last 100 transcripts (held-out test set) in the corpus, which is around 43k samples. |
|
|
|
### Results |
|
|
|
| Model | WER | |
|
|:----------------------------:|:-----:| |
|
| fine-tuned-122k-whisper-small| 9.69% | |
|
|
|
#### Summary |
|
|
|
The overall model is not perfect, but if audio is spoken clearly, the model is able to transcribe Singaporean terms and slang accurately. |
|
|
|
### Compute Infrastructure |
|
|
|
Trained on VM instance provisioned on [jarvislabs.ai](https://jarvislabs.ai/). |
|
|
|
#### Hardware |
|
|
|
- Single A6000 GPU |
|
|
|
## Model Card Authors [optional] |
|
|
|
[More Information Needed] |
|
|
|
## Model Card Contact |
|
|
|
[Low Wei Teck](mailto:[email protected]) |
