SER_wav2vec2-large-xlsr-53_240303

This model is a fine-tuned version of facebook/wav2vec2-large-xlsr-53 on a Speech Emotion Recognition (en) dataset.

This dataset includes the 4 most popular datasets in English: Crema, Ravdess, Savee, and Tess, containing a total of over 12,000 .wav audio files. Each of these four datasets includes 6 to 8 different emotional labels.

It achieves the following results on the evaluation set:

Loss: 1.7923
Accuracy: 0.2408
Precision: 0.2324
Recall: 0.2466
F1: 0.2226

For a better performance version, please refer to

hughlan1214/Speech_Emotion_Recognition_wav2vec2-large-xlsr-53_240304_SER_fine-tuned2.0

Model description

The model was obtained through feature extraction using facebook/wav2vec2-large-xlsr-53 and underwent several rounds of fine-tuning. It predicts the 7 types of emotions contained in speech, aiming to lay the foundation for subsequent use of human micro-expressions on the visual level and context semantics under LLMS to infer user emotions in real-time.

Although the model was trained on purely English datasets, post-release testing showed that it also performs well in predicting emotions in Chinese and French, demonstrating the powerful cross-linguistic capability of the facebook/wav2vec2-large-xlsr-53 pre-trained model.

emotions = ['angry', 'disgust', 'fear', 'happy', 'neutral', 'sad', 'surprise']

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 0.0001
train_batch_size: 4
eval_batch_size: 4
seed: 42
gradient_accumulation_steps: 3
total_train_batch_size: 12
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
lr_scheduler_warmup_ratio: 0.1
num_epochs: 8

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	Precision	Recall	F1
1.9297	1.0	101	1.9452	0.1233	0.0306	0.1468	0.0454
1.9114	2.0	202	1.9115	0.1773	0.1501	0.1803	0.1323
1.7863	3.0	303	1.8564	0.2081	0.1117	0.2193	0.1336
1.8439	4.0	404	1.8590	0.2042	0.2196	0.2156	0.1755
1.9361	5.0	505	1.8375	0.2081	0.2617	0.2213	0.1573
1.7572	6.0	606	1.8081	0.2100	0.2018	0.2214	0.1841
1.6715	7.0	707	1.8131	0.2389	0.2263	0.2442	0.2129
1.6687	8.0	808	1.7923	0.2408	0.2324	0.2466	0.2226

Framework versions

Transformers 4.38.1
Pytorch 2.2.1
Datasets 2.17.1
Tokenizers 0.15.2

hughlan1214
/

SER_wav2vec2-large-xlsr-53_fine-tuned_1.0