---
license: mit
tags:
- generated_from_trainer
metrics:
- precision
- recall
- f1
- accuracy
model-index:
- name: fedcsis-slot_baseline-xlm_r-en
  results: []
datasets:
- cartesinus/leyzer-fedcsis
language:
- en
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# fedcsis-slot_baseline-xlm_r-en

This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on the 
[leyzer-fedcsis](https://huggingface.co/cartesinus/leyzer-fedcsis) dataset.

Results on test set:
- Precision: 0.7767
- Recall: 0.7991
- F1: 0.7877
- Accuracy: 0.9425

It achieves the following results on the evaluation set:
- Loss: 0.1097
- Precision: 0.9705
- Recall: 0.9723
- F1: 0.9714
- Accuracy: 0.9859

## Model description

More information needed

## Intended uses & limitations

More information needed

## Training and evaluation data

More information needed

## Training procedure

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 2e-05
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 5

### Training results

| Training Loss | Epoch | Step | Validation Loss | Precision | Recall | F1     | Accuracy |
|:-------------:|:-----:|:----:|:---------------:|:---------:|:------:|:------:|:--------:|
| 1.2866        | 1.0   | 814  | 0.3188          | 0.8661    | 0.8672 | 0.8666 | 0.9250   |
| 0.1956        | 2.0   | 1628 | 0.1299          | 0.9409    | 0.9471 | 0.9440 | 0.9736   |
| 0.1063        | 3.0   | 2442 | 0.1196          | 0.9537    | 0.9607 | 0.9572 | 0.9810   |
| 0.0558        | 4.0   | 3256 | 0.0789          | 0.9661    | 0.9697 | 0.9679 | 0.9854   |
| 0.0367        | 5.0   | 4070 | 0.0824          | 0.9685    | 0.9690 | 0.9687 | 0.9848   |
| 0.031         | 6.0   | 4884 | 0.0887          | 0.9712    | 0.9728 | 0.9720 | 0.9859   |
| 0.0233        | 7.0   | 5698 | 0.0829          | 0.9736    | 0.9744 | 0.9740 | 0.9872   |
| 0.0139        | 8.0   | 6512 | 0.0879          | 0.9743    | 0.9747 | 0.9745 | 0.9876   |
| 0.007         | 9.0   | 7326 | 0.0978          | 0.9740    | 0.9734 | 0.9737 | 0.9870   |
| 0.0076        | 10.0  | 8140 | 0.1015          | 0.9723    | 0.9726 | 0.9725 | 0.9860   |
| 0.026         | 11.0   | 814  | 0.1264          | 0.9732    | 0.9620 | 0.9676 | 0.9829   |
| 0.0189        | 12.0   | 1628 | 0.0975          | 0.9732    | 0.9711 | 0.9722 | 0.9861   |
| 0.0099        | 13.0   | 2442 | 0.1080          | 0.9721    | 0.9715 | 0.9718 | 0.9866   |
| 0.0052        | 14.0   | 3256 | 0.1052          | 0.9706    | 0.9715 | 0.9710 | 0.9860   |
| 0.0031        | 15.0   | 4070 | 0.1097          | 0.9705    | 0.9723 | 0.9714 | 0.9859   |

### Per slot evaluation on test set

| slot_name | precision | recall | f1 | tc_size |
|-----------|-----------|--------|----|---------|
| album | 0.7000 | 0.8750 | 0.7778 | 8 |
| album_name | 0.9091 | 0.6250 | 0.7407 | 16 |
| album_type | 0.1842 | 0.5385 | 0.2745 | 13 |
| album_type_1a | 0.0000 | 0.0000 | 0.0000 | 10 |
| album_type_an | 0.0000 | 0.0000 | 0.0000 | 20 |
| all_lang | 0.5556 | 0.7143 | 0.6250 | 7 |
| artist | 0.7500 | 0.7857 | 0.7674 | 42 |
| av_alias | 0.8333 | 0.5263 | 0.6452 | 19 |
| caption | 0.8065 | 0.7576 | 0.7813 | 33 |
| category | 0.8571 | 1.0000 | 0.9231 | 18 |
| channel | 0.6786 | 0.8085 | 0.7379 | 47 |
| channel_id | 0.7826 | 0.9000 | 0.8372 | 20 |
| count | 0.5714 | 1.0000 | 0.7273 | 4 |
| date | 0.8333 | 0.7500 | 0.7895 | 40 |
| date_day | 1.0000 | 1.0000 | 1.0000 | 4 |
| date_month | 1.0000 | 1.0000 | 1.0000 | 8 |
| device_name | 0.8621 | 0.7576 | 0.8065 | 33 |
| email | 1.0000 | 1.0000 | 1.0000 | 16 |
| event_name | 0.5467 | 0.5325 | 0.5395 | 77 |
| file_name | 0.7333 | 0.7857 | 0.7586 | 14 |
| file_size | 1.0000 | 1.0000 | 1.0000 | 1 |
| filename | 0.7083 | 0.7391 | 0.7234 | 23 |
| filter | 0.8333 | 0.9375 | 0.8824 | 16 |
| from | 1.0000 | 1.0000 | 1.0000 | 33 |
| hashtag | 1.0000 | 0.6000 | 0.7500 | 10 |
| img_query | 0.9388 | 0.9246 | 0.9316 | 199 |
| label | 0.2500 | 1.0000 | 0.4000 | 1 |
| location | 0.8319 | 0.9171 | 0.8724 | 205 |
| mail | 1.0000 | 1.0000 | 1.0000 | 2 |
| massage | 0.0000 | 0.0000 | 0.0000 | 1 |
| mesage | 0.0000 | 0.0000 | 0.0000 | 1 |
| message | 0.5856 | 0.5285 | 0.5556 | 123 |
| mime_type | 0.6667 | 1.0000 | 0.8000 | 2 |
| name | 0.9412 | 0.8767 | 0.9078 | 73 |
| pathname | 0.7805 | 0.6809 | 0.7273 | 47 |
| percent | 1.0000 | 0.9583 | 0.9787 | 24 |
| phone_number | 1.0000 | 1.0000 | 1.0000 | 48 |
| phone_type | 1.0000 | 0.9375 | 0.9677 | 16 |
| picture_url | 1.0000 | 1.0000 | 1.0000 | 14 |
| playlist | 0.7219 | 0.8134 | 0.7649 | 134 |
| portal | 0.8108 | 0.7692 | 0.7895 | 39 |
| power | 1.0000 | 1.0000 | 1.0000 | 1 |
| priority | 0.6667 | 1.0000 | 0.8000 | 2 |
| purpose | 1.0000 | 1.0000 | 1.0000 | 8 |
| query | 0.6706 | 0.6064 | 0.6369 | 94 |
| rating | 0.9167 | 0.9167 | 0.9167 | 12 |
| review_count | 0.8750 | 0.7778 | 0.8235 | 9 |
| section | 0.9091 | 0.9091 | 0.9091 | 22 |
| seek_time | 0.6667 | 1.0000 | 0.8000 | 2 |
| sender | 0.6000 | 0.6000 | 0.6000 | 10 |
| sender_address | 0.6364 | 0.8750 | 0.7368 | 8 |
| song | 0.5476 | 0.6133 | 0.5786 | 75 |
| src_lang_de | 0.8765 | 0.9467 | 0.9103 | 75 |
| src_lang_en | 0.6604 | 0.6481 | 0.6542 | 54 |
| src_lang_es | 0.8132 | 0.9024 | 0.8555 | 82 |
| src_lang_fr | 0.8795 | 0.9125 | 0.8957 | 80 |
| src_lang_it | 0.8590 | 0.9437 | 0.8993 | 71 |
| src_lang_pl | 0.7910 | 0.8833 | 0.8346 | 60 |
| state | 1.0000 | 1.0000 | 1.0000 | 1 |
| status | 0.5455 | 0.5000 | 0.5217 | 12 |
| subject | 0.6154 | 0.5581 | 0.5854 | 86 |
| text_de | 0.9091 | 0.9091 | 0.9091 | 77 |
| text_en | 0.5909 | 0.5843 | 0.5876 | 89 |
| text_es | 0.7935 | 0.8111 | 0.8022 | 90 |
| text_esi | 0.0000 | 0.0000 | 0.0000 | 1 |
| text_fr | 0.9125 | 0.8588 | 0.8848 | 85 |
| text_it | 0.8205 | 0.9014 | 0.8591 | 71 |
| text_multi | 0.3333 | 1.0000 | 0.5000 | 1 |
| text_pl | 0.8167 | 0.7656 | 0.7903 | 64 |
| time | 0.8750 | 1.0000 | 0.9333 | 7 |
| to | 0.8927 | 0.9186 | 0.9054 | 172 |
| topic | 0.4000 | 0.6667 | 0.5000 | 3 |
| translator | 0.7991 | 0.9777 | 0.8794 | 179 |
| trg_lang_de | 0.8116 | 0.8615 | 0.8358 | 65 |
| trg_lang_en | 0.8841 | 0.8841 | 0.8841 | 69 |
| trg_lang_es | 0.8906 | 0.8769 | 0.8837 | 65 |
| trg_lang_fr | 0.8676 | 0.9365 | 0.9008 | 63 |
| trg_lang_general | 0.8235 | 0.7368 | 0.7778 | 19 |
| trg_lang_it | 0.8254 | 0.8667 | 0.8455 | 60 |
| trg_lang_pl | 0.8077 | 0.8630 | 0.8344 | 73 |
| txt_query | 0.5714 | 0.7059 | 0.6316 | 17 |
| username | 0.6875 | 0.7333 | 0.7097 | 15 |
| value | 0.7500 | 0.8571 | 0.8000 | 14 |

### Framework versions

- Transformers 4.27.4
- Pytorch 1.13.1+cu116
- Datasets 2.11.0
- Tokenizers 0.13.2

## Citation

If you use this model, please cite the following:
```
@inproceedings{kubis2023caiccaic,
	author={Marek Kubis and Paweł Skórzewski and Marcin Sowański and Tomasz Ziętkiewicz},
	pages={1319–1324},
	title={Center for Artificial Intelligence Challenge on Conversational AI Correctness},
	booktitle={Proceedings of the 18th Conference on Computer Science and Intelligence Systems},
	year={2023},
	doi={10.15439/2023B6058},
	url={http://dx.doi.org/10.15439/2023B6058},
	volume={35},
	series={Annals of Computer Science and Information Systems}
}

```