xlm-r-base-amazon-massive-intent

This model is a fine-tuned version of xlm-roberta-base on Amazon Massive dataset (only en-US subset). It achieves the following results on the evaluation set:

• Loss: 0.5439
• Accuracy: 0.8775
• F1: 0.8775

Results

domain	train-accuracy	test-accuracy
alarm	0.967	0.9846
audio	0.7458	0.659
calendar	0.9797	0.3181
cooking	0.9714	0.9571
datetime	0.9777	0.9402
email	0.9727	0.9296
general	0.8952	0.5949
iot	0.9329	0.9122
list	0.9792	0.9538
music	0.9355	0.8837
news	0.9607	0.8764
play	0.9419	0.874
qa	0.9677	0.8591
recommendation	0.9515	0.8764
social	0.9671	0.8932
takeaway	0.9192	0.8478
transport	0.9425	0.9193
weather	0.9895	0.93

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 16
• eval_batch_size: 16
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 5

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	F1
2.734	1.0	720	1.1883	0.7196	0.7196
1.2774	2.0	1440	0.7162	0.8342	0.8342
0.6301	3.0	2160	0.5817	0.8672	0.8672
0.4901	4.0	2880	0.5555	0.8770	0.8770
0.3398	5.0	3600	0.5439	0.8775	0.8775

Framework versions

• Transformers 4.22.1
• Pytorch 1.12.1+cu113
• Datasets 2.5.1
• Tokenizers 0.12.1

Citation

@article{kubis2023back,
  title={Back Transcription as a Method for Evaluating Robustness of Natural Language Understanding Models to Speech Recognition Errors},
  author={Kubis, Marek and Sk{\'o}rzewski, Pawe{\l} and Sowa{\'n}ski, Marcin and Zi{\k{e}}tkiewicz, Tomasz},
  journal={arXiv preprint arXiv:2310.16609},
  year={2023}
  eprint={2310.16609},
  archivePrefix={arXiv},
}