|
--- |
|
license: apache-2.0 |
|
language: |
|
- vi |
|
tags: |
|
- automatic-speech-recognition |
|
- common-voice |
|
- hf-asr-leaderboard |
|
- robust-speech-event |
|
datasets: |
|
- mozilla-foundation/common_voice_7_0 |
|
model-index: |
|
- name: xls-asr-vi-40h-1B |
|
results: |
|
- task: |
|
name: Speech Recognition |
|
type: automatic-speech-recognition |
|
dataset: |
|
name: Common Voice 7.0 |
|
type: mozilla-foundation/common_voice_7_0 |
|
args: vi |
|
metrics: |
|
- name: Test WER (with LM) |
|
type: wer |
|
value: 25.846 |
|
- name: Test CER (with LM) |
|
type: cer |
|
value: 12.961 |
|
- task: |
|
name: Speech Recognition |
|
type: automatic-speech-recognition |
|
dataset: |
|
name: Common Voice 8.0 |
|
type: mozilla-foundation/common_voice_8_0 |
|
args: vi |
|
metrics: |
|
- name: Test WER (with LM) |
|
type: wer |
|
value: 31.158 |
|
- name: Test CER (with LM) |
|
type: cer |
|
value: 16.179 |
|
--- |
|
|
|
|
|
# xls-asr-vi-40h-1B |
|
|
|
This model is a fine-tuned version of [facebook/wav2vec2-xls-r-1b](https://huggingface.co/facebook/wav2vec2-xls-r-1b) on 40 hours of FPT Open Speech Dataset (FOSD) and Common Voice 7.0. |
|
|
|
### Benchmark WER result: |
|
| | [VIVOS](https://huggingface.co/datasets/vivos) | [COMMON VOICE 7.0](https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0) | [COMMON VOICE 8.0](https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0) |
|
|---|---|---|---| |
|
|without LM| 25.93 | 34.21 | |
|
|with 4-grams LM| 24.11 | 25.84 | 31.158 | |
|
|
|
### Benchmark CER result: |
|
| | [VIVOS](https://huggingface.co/datasets/vivos) | [COMMON VOICE 7.0](https://huggingface.co/datasets/mozilla-foundation/common_voice_7_0) | [COMMON VOICE 8.0](https://huggingface.co/datasets/mozilla-foundation/common_voice_8_0) |
|
|---|---|---|---| |
|
|without LM| 9.24 | 19.94 | |
|
|with 4-grams LM| 10.37 | 12.96 | 16.179 | |
|
|
|
## Evaluation |
|
|
|
Please use the eval.py file to run the evaluation |
|
|
|
```python |
|
python eval.py --model_id geninhu/xls-asr-vi-40h-1B --dataset mozilla-foundation/common_voice_7_0 --config vi --split test --log_outputs |
|
``` |
|
|
|
## Training procedure |
|
|
|
### Training hyperparameters |
|
|
|
The following hyperparameters were used during training: |
|
- learning_rate: 5e-05 |
|
- train_batch_size: 16 |
|
- eval_batch_size: 16 |
|
- seed: 42 |
|
- gradient_accumulation_steps: 2 |
|
- total_train_batch_size: 32 |
|
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08 |
|
- lr_scheduler_type: linear |
|
- lr_scheduler_warmup_steps: 1500 |
|
- num_epochs: 10.0 |
|
- mixed_precision_training: Native AMP |
|
|
|
### Training results |
|
|
|
| Training Loss | Epoch | Step | Validation Loss | Wer | |
|
|:-------------:|:-----:|:-----:|:---------------:|:------:| |
|
| 4.6222 | 1.85 | 1500 | 5.9479 | 0.5474 | |
|
| 1.1362 | 3.7 | 3000 | 7.9799 | 0.5094 | |
|
| 0.7814 | 5.56 | 4500 | 5.0330 | 0.4724 | |
|
| 0.6281 | 7.41 | 6000 | 2.3484 | 0.5020 | |
|
| 0.5472 | 9.26 | 7500 | 2.2495 | 0.4793 | |
|
| 0.4827 | 11.11 | 9000 | 1.1530 | 0.4768 | |
|
| 0.4327 | 12.96 | 10500 | 1.6160 | 0.4646 | |
|
| 0.3989 | 14.81 | 12000 | 3.2633 | 0.4703 | |
|
| 0.3522 | 16.67 | 13500 | 2.2337 | 0.4708 | |
|
| 0.3201 | 18.52 | 15000 | 3.6879 | 0.4565 | |
|
| 0.2899 | 20.37 | 16500 | 5.4389 | 0.4599 | |
|
| 0.2776 | 22.22 | 18000 | 3.5284 | 0.4537 | |
|
| 0.2574 | 24.07 | 19500 | 2.1759 | 0.4649 | |
|
| 0.2378 | 25.93 | 21000 | 3.3901 | 0.4448 | |
|
| 0.217 | 27.78 | 22500 | 1.1632 | 0.4565 | |
|
| 0.2115 | 29.63 | 24000 | 1.7441 | 0.4232 | |
|
| 0.1959 | 31.48 | 25500 | 3.4992 | 0.4304 | |
|
| 0.187 | 33.33 | 27000 | 3.6163 | 0.4369 | |
|
| 0.1748 | 35.19 | 28500 | 3.6038 | 0.4467 | |
|
| 0.17 | 37.04 | 30000 | 2.9708 | 0.4362 | |
|
| 0.159 | 38.89 | 31500 | 3.2045 | 0.4279 | |
|
| 0.153 | 40.74 | 33000 | 3.2427 | 0.4287 | |
|
| 0.1463 | 42.59 | 34500 | 3.5439 | 0.4270 | |
|
| 0.139 | 44.44 | 36000 | 3.9381 | 0.4150 | |
|
| 0.1352 | 46.3 | 37500 | 4.1744 | 0.4092 | |
|
| 0.1369 | 48.15 | 39000 | 4.2279 | 0.4154 | |
|
| 0.1273 | 50.0 | 40500 | 4.1691 | 0.4133 | |
|
|
|
|
|
### Framework versions |
|
|
|
- Transformers 4.16.0.dev0 |
|
- Pytorch 1.10.1+cu102 |
|
- Datasets 1.17.1.dev0 |
|
- Tokenizers 0.11.0 |
|
|
