0 1 2
---
license: mit
tags:
- generated_from_trainer
metrics:
- accuracy
model-index:
- name: deberta-v3-large
  results: []
---

# deberta-v3-large-sentiment

This model is a fine-tuned version of [microsoft/deberta-v3-large](https://huggingface.co/microsoft/deberta-v3-large) on an [tweet_eval](https://huggingface.co/datasets/tweet_eval)  dataset.

## Model description

Test set results:

| Model                      | Emotion       | Hate          | Irony         | Offensive     | Sentiment     | 
| -------------              | ------------- | ------------- | ------------- | ------------- | ------------- | 
| deberta-v3-large           | **86.3**      | **61.3**      | **87.1**      | **86.4**      | **73.9**      | 
| BERTweet                   | 79.3          | -             | 82.1          | 79.5          | 73.4          | 
| RoB-RT                     | 79.5          | 52.3          | 61.7          | 80.5          | 69.3          | 

[source:papers_with_code](https://paperswithcode.com/sota/sentiment-analysis-on-tweeteval)


## Intended uses & limitations

Classifying attributes of interest on tweeter like data. 

## Training and evaluation data

[tweet_eval](https://huggingface.co/datasets/tweet_eval)  dataset.

## Training procedure

Fine tuned and evaluated with [run_glue.py]()

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-06
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- gradient_accumulation_steps: 2
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 50
- num_epochs: 10.0

### Training results

| Training Loss | Epoch | Step  | Validation Loss | Accuracy |
|:-------------:|:-----:|:-----:|:---------------:|:--------:|
| 1.0614        | 0.07  | 100   | 1.0196          | 0.4345   |
| 0.8601        | 0.14  | 200   | 0.7561          | 0.6460   |
| 0.734         | 0.21  | 300   | 0.6796          | 0.6955   |
| 0.6753        | 0.28  | 400   | 0.6521          | 0.7000   |
| 0.6408        | 0.35  | 500   | 0.6119          | 0.7440   |
| 0.5991        | 0.42  | 600   | 0.6034          | 0.7370   |
| 0.6069        | 0.49  | 700   | 0.5976          | 0.7375   |
| 0.6122        | 0.56  | 800   | 0.5871          | 0.7425   |
| 0.5908        | 0.63  | 900   | 0.5935          | 0.7445   |
| 0.5884        | 0.7   | 1000  | 0.5792          | 0.7520   |
| 0.5839        | 0.77  | 1100  | 0.5780          | 0.7555   |
| 0.5772        | 0.84  | 1200  | 0.5727          | 0.7570   |
| 0.5895        | 0.91  | 1300  | 0.5601          | 0.7550   |
| 0.5757        | 0.98  | 1400  | 0.5613          | 0.7525   |
| 0.5121        | 1.05  | 1500  | 0.5867          | 0.7600   |
| 0.5254        | 1.12  | 1600  | 0.5595          | 0.7630   |
| 0.5074        | 1.19  | 1700  | 0.5594          | 0.7585   |
| 0.4947        | 1.26  | 1800  | 0.5697          | 0.7575   |
| 0.5019        | 1.33  | 1900  | 0.5665          | 0.7580   |
| 0.5005        | 1.4   | 2000  | 0.5484          | 0.7655   |
| 0.5125        | 1.47  | 2100  | 0.5626          | 0.7605   |
| 0.5241        | 1.54  | 2200  | 0.5561          | 0.7560   |
| 0.5198        | 1.61  | 2300  | 0.5602          | 0.7600   |
| 0.5124        | 1.68  | 2400  | 0.5654          | 0.7490   |
| 0.5096        | 1.75  | 2500  | 0.5803          | 0.7515   |
| 0.4885        | 1.82  | 2600  | 0.5889          | 0.75     |
| 0.5111        | 1.89  | 2700  | 0.5508          | 0.7665   |
| 0.4868        | 1.96  | 2800  | 0.5621          | 0.7635   |
| 0.4599        | 2.04  | 2900  | 0.5995          | 0.7615   |
| 0.4147        | 2.11  | 3000  | 0.6202          | 0.7530   |
| 0.4233        | 2.18  | 3100  | 0.5875          | 0.7625   |
| 0.4324        | 2.25  | 3200  | 0.5794          | 0.7610   |
| 0.4141        | 2.32  | 3300  | 0.5902          | 0.7460   |
| 0.4306        | 2.39  | 3400  | 0.6053          | 0.7545   |
| 0.4266        | 2.46  | 3500  | 0.5979          | 0.7570   |
| 0.4227        | 2.53  | 3600  | 0.5920          | 0.7650   |
| 0.4226        | 2.6   | 3700  | 0.6166          | 0.7455   |
| 0.3978        | 2.67  | 3800  | 0.6126          | 0.7560   |
| 0.3954        | 2.74  | 3900  | 0.6152          | 0.7550   |
| 0.4209        | 2.81  | 4000  | 0.5980          | 0.75     |
| 0.3982        | 2.88  | 4100  | 0.6096          | 0.7490   |
| 0.4016        | 2.95  | 4200  | 0.6541          | 0.7425   |
| 0.3966        | 3.02  | 4300  | 0.6377          | 0.7545   |
| 0.3074        | 3.09  | 4400  | 0.6860          | 0.75     |
| 0.3551        | 3.16  | 4500  | 0.6160          | 0.7550   |
| 0.3323        | 3.23  | 4600  | 0.6714          | 0.7520   |
| 0.3171        | 3.3   | 4700  | 0.6538          | 0.7535   |
| 0.3403        | 3.37  | 4800  | 0.6774          | 0.7465   |
| 0.3396        | 3.44  | 4900  | 0.6726          | 0.7465   |
| 0.3259        | 3.51  | 5000  | 0.6465          | 0.7480   |
| 0.3392        | 3.58  | 5100  | 0.6860          | 0.7460   |
| 0.3251        | 3.65  | 5200  | 0.6697          | 0.7495   |
| 0.3253        | 3.72  | 5300  | 0.6770          | 0.7430   |
| 0.3455        | 3.79  | 5400  | 0.7177          | 0.7360   |
| 0.3323        | 3.86  | 5500  | 0.6943          | 0.7400   |
| 0.3335        | 3.93  | 5600  | 0.6507          | 0.7555   |
| 0.3368        | 4.0   | 5700  | 0.6580          | 0.7485   |
| 0.2479        | 4.07  | 5800  | 0.7667          | 0.7430   |
| 0.2613        | 4.14  | 5900  | 0.7513          | 0.7505   |
| 0.2557        | 4.21  | 6000  | 0.7927          | 0.7485   |
| 0.243         | 4.28  | 6100  | 0.7792          | 0.7450   |
| 0.2473        | 4.35  | 6200  | 0.8107          | 0.7355   |
| 0.2447        | 4.42  | 6300  | 0.7851          | 0.7370   |
| 0.2515        | 4.49  | 6400  | 0.7529          | 0.7465   |
| 0.274         | 4.56  | 6500  | 0.7390          | 0.7465   |
| 0.2674        | 4.63  | 6600  | 0.7658          | 0.7460   |
| 0.2416        | 4.7   | 6700  | 0.7915          | 0.7485   |
| 0.2432        | 4.77  | 6800  | 0.7989          | 0.7435   |
| 0.2595        | 4.84  | 6900  | 0.7850          | 0.7380   |
| 0.2736        | 4.91  | 7000  | 0.7577          | 0.7395   |
| 0.2783        | 4.98  | 7100  | 0.7650          | 0.7405   |
| 0.2304        | 5.05  | 7200  | 0.8542          | 0.7385   |
| 0.1937        | 5.12  | 7300  | 0.8390          | 0.7345   |
| 0.1878        | 5.19  | 7400  | 0.9150          | 0.7330   |
| 0.1921        | 5.26  | 7500  | 0.8792          | 0.7405   |
| 0.1916        | 5.33  | 7600  | 0.8892          | 0.7410   |
| 0.2011        | 5.4   | 7700  | 0.9012          | 0.7325   |
| 0.211         | 5.47  | 7800  | 0.8608          | 0.7420   |
| 0.2194        | 5.54  | 7900  | 0.8852          | 0.7320   |
| 0.205         | 5.61  | 8000  | 0.8803          | 0.7385   |
| 0.1981        | 5.68  | 8100  | 0.8681          | 0.7330   |
| 0.1908        | 5.75  | 8200  | 0.9020          | 0.7435   |
| 0.1942        | 5.82  | 8300  | 0.8780          | 0.7410   |
| 0.1958        | 5.89  | 8400  | 0.8937          | 0.7345   |
| 0.1883        | 5.96  | 8500  | 0.9121          | 0.7360   |
| 0.1819        | 6.04  | 8600  | 0.9409          | 0.7430   |
| 0.145         | 6.11  | 8700  | 1.1390          | 0.7265   |
| 0.1696        | 6.18  | 8800  | 0.9189          | 0.7430   |
| 0.1488        | 6.25  | 8900  | 0.9718          | 0.7400   |
| 0.1637        | 6.32  | 9000  | 0.9702          | 0.7450   |
| 0.1547        | 6.39  | 9100  | 1.0033          | 0.7410   |
| 0.1605        | 6.46  | 9200  | 0.9973          | 0.7355   |
| 0.1552        | 6.53  | 9300  | 1.0491          | 0.7290   |
| 0.1731        | 6.6   | 9400  | 1.0271          | 0.7335   |
| 0.1738        | 6.67  | 9500  | 0.9575          | 0.7430   |
| 0.1669        | 6.74  | 9600  | 0.9614          | 0.7350   |
| 0.1347        | 6.81  | 9700  | 1.0263          | 0.7365   |
| 0.1593        | 6.88  | 9800  | 1.0173          | 0.7360   |
| 0.1549        | 6.95  | 9900  | 1.0398          | 0.7350   |
| 0.1675        | 7.02  | 10000 | 0.9975          | 0.7380   |
| 0.1182        | 7.09  | 10100 | 1.1059          | 0.7350   |
| 0.1351        | 7.16  | 10200 | 1.0933          | 0.7400   |
| 0.1496        | 7.23  | 10300 | 1.0731          | 0.7355   |
| 0.1197        | 7.3   | 10400 | 1.1089          | 0.7360   |
| 0.1111        | 7.37  | 10500 | 1.1381          | 0.7405   |
| 0.1494        | 7.44  | 10600 | 1.0252          | 0.7425   |
| 0.1235        | 7.51  | 10700 | 1.0906          | 0.7360   |
| 0.133         | 7.58  | 10800 | 1.1796          | 0.7375   |
| 0.1248        | 7.65  | 10900 | 1.1332          | 0.7420   |
| 0.1268        | 7.72  | 11000 | 1.1304          | 0.7415   |
| 0.1368        | 7.79  | 11100 | 1.1345          | 0.7380   |
| 0.1228        | 7.86  | 11200 | 1.2018          | 0.7320   |
| 0.1281        | 7.93  | 11300 | 1.1884          | 0.7350   |
| 0.1449        | 8.0   | 11400 | 1.1571          | 0.7345   |
| 0.1025        | 8.07  | 11500 | 1.1538          | 0.7345   |
| 0.1199        | 8.14  | 11600 | 1.2113          | 0.7390   |
| 0.1016        | 8.21  | 11700 | 1.2882          | 0.7370   |
| 0.114         | 8.28  | 11800 | 1.2872          | 0.7390   |
| 0.1019        | 8.35  | 11900 | 1.2876          | 0.7380   |
| 0.1142        | 8.42  | 12000 | 1.2791          | 0.7385   |
| 0.1135        | 8.49  | 12100 | 1.2883          | 0.7380   |
| 0.1139        | 8.56  | 12200 | 1.2829          | 0.7360   |
| 0.1107        | 8.63  | 12300 | 1.2698          | 0.7365   |
| 0.1183        | 8.7   | 12400 | 1.2660          | 0.7345   |
| 0.1064        | 8.77  | 12500 | 1.2889          | 0.7365   |
| 0.0895        | 8.84  | 12600 | 1.3480          | 0.7330   |
| 0.1244        | 8.91  | 12700 | 1.2872          | 0.7325   |
| 0.1209        | 8.98  | 12800 | 1.2681          | 0.7375   |
| 0.1144        | 9.05  | 12900 | 1.2711          | 0.7370   |
| 0.1034        | 9.12  | 13000 | 1.2801          | 0.7360   |
| 0.113         | 9.19  | 13100 | 1.2801          | 0.7350   |
| 0.0994        | 9.26  | 13200 | 1.2920          | 0.7360   |
| 0.0966        | 9.33  | 13300 | 1.2761          | 0.7335   |
| 0.0939        | 9.4   | 13400 | 1.2909          | 0.7365   |
| 0.0975        | 9.47  | 13500 | 1.2953          | 0.7360   |
| 0.0842        | 9.54  | 13600 | 1.3179          | 0.7335   |
| 0.0871        | 9.61  | 13700 | 1.3149          | 0.7385   |
| 0.1162        | 9.68  | 13800 | 1.3124          | 0.7350   |
| 0.085         | 9.75  | 13900 | 1.3207          | 0.7355   |
| 0.0966        | 9.82  | 14000 | 1.3248          | 0.7335   |
| 0.1064        | 9.89  | 14100 | 1.3261          | 0.7335   |
| 0.1046        | 9.96  | 14200 | 1.3255          | 0.7360   |


### Framework versions

- Transformers 4.20.0.dev0
- Pytorch 1.9.0
- Datasets 2.2.2
- Tokenizers 0.11.6
