metadata

language:
  - en
license: mit
tags:
  - generated_from_trainer
  - nlu
  - intent-classification
datasets:
  - AmazonScience/massive
metrics:
  - accuracy
  - f1
base_model: xlm-roberta-base
model-index:
  - name: xlm-r-base-amazon-massive-intent
    results:
      - task:
          type: intent-classification
          name: intent-classification
        dataset:
          name: MASSIVE
          type: AmazonScience/massive
          split: test
        metrics:
          - type: f1
            value: 0.8775
            name: F1

xlm-r-base-amazon-massive-intent

This model is a fine-tuned version of xlm-roberta-base on Amazon Massive dataset (only en-US subset). It achieves the following results on the evaluation set:

Loss: 0.5439
Accuracy: 0.8775
F1: 0.8775

Results

domain	train-accuracy	test-accuracy
alarm	0.967	0.9846
audio	0.7458	0.659
calendar	0.9797	0.3181
cooking	0.9714	0.9571
datetime	0.9777	0.9402
email	0.9727	0.9296
general	0.8952	0.5949
iot	0.9329	0.9122
list	0.9792	0.9538
music	0.9355	0.8837
news	0.9607	0.8764
play	0.9419	0.874
qa	0.9677	0.8591
recommendation	0.9515	0.8764
social	0.9671	0.8932
takeaway	0.9192	0.8478
transport	0.9425	0.9193
weather	0.9895	0.93

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 2e-05
train_batch_size: 16
eval_batch_size: 16
seed: 42
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: linear
num_epochs: 5

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	F1
2.734	1.0	720	1.1883	0.7196	0.7196
1.2774	2.0	1440	0.7162	0.8342	0.8342
0.6301	3.0	2160	0.5817	0.8672	0.8672
0.4901	4.0	2880	0.5555	0.8770	0.8770
0.3398	5.0	3600	0.5439	0.8775	0.8775

Framework versions

Transformers 4.22.1
Pytorch 1.12.1+cu113
Datasets 2.5.1
Tokenizers 0.12.1

Citation

@article{kubis2023back,
  title={Back Transcription as a Method for Evaluating Robustness of Natural Language Understanding Models to Speech Recognition Errors},
  author={Kubis, Marek and Sk{\'o}rzewski, Pawe{\l} and Sowa{\'n}ski, Marcin and Zi{\k{e}}tkiewicz, Tomasz},
  journal={arXiv preprint arXiv:2310.16609},
  year={2023}
  eprint={2310.16609},
  archivePrefix={arXiv},
}