distilbert-base-german-cased-finetuned-amazon-reviews

This model is a fine-tuned version of distilbert-base-german-cased on the amazon_reviews_multi dataset. It achieves the following results on the evaluation set:

• Loss: 3.8874

Model description

This model is a fine-tuned version of distilbert-base-german-cased using the dataset from amazon_reviews_multi (available in Huggin Face). The purpose is to extend the model's domain, which, once fine-tuned, will be modified for the fill-in-the-gaps task. It's related to my other model (fine-tuned-spanish-bert) as a comparison of both performances.

Intended uses & limitations

The use is limited to school use and the limitations have to do with the size of the dataset, since it does not allow for a large contribution, a larger dataset would have to be used to get a larger contribution.

Training and evaluation data

I did a training that gives the training and validation set loss. (It takes a lot of time. If you're using colab, I recommend to use less Epochs because the result does not change too much, and even though the loss is quite high, the performance of the model based on the perplexity is not that bad) Also, I checked the perplexity, which is one good measure for Languages Models. The value of the perplexity is considerabily good: 48'78.

• Evaluation: I checked the performance of my model in the notebook provided, just by generating examples.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 64
• eval_batch_size: 64
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 10

Training results

Training Loss	Epoch	Step	Validation Loss
4.625	1.0	141	4.2747
4.3013	2.0	282	4.1549
4.1841	3.0	423	4.0902
4.1208	4.0	564	3.9958
4.0475	5.0	705	3.9710
4.0116	6.0	846	3.9100
3.9988	7.0	987	3.9194
3.9641	8.0	1128	3.9381
3.9661	9.0	1269	3.8631
3.944	10.0	1410	3.8668

Framework versions

• Transformers 4.27.0
• Pytorch 1.13.1+cu116
• Tokenizers 0.13.2