metadata
library_name: transformers
license: other
base_model: trl-lib/qwen1.5-0.5b-sft
tags:
- alignment-handbook
- trl
- simpo
- generated_from_trainer
- trl
- simpo
- generated_from_trainer
datasets:
- yakazimir/ultrafeedback_binarized
model-index:
- name: qwen_cUNL_entropy
results: []
qwen_cUNL_entropy
This model is a fine-tuned version of trl-lib/qwen1.5-0.5b-sft on the yakazimir/ultrafeedback_binarized dataset. It achieves the following results on the evaluation set:
- Loss: 0.5196
- Rewards/chosen: -7.4572
- Rewards/rejected: -8.6117
- Rewards/accuracies: 0.7285
- Rewards/margins: 1.1545
- Logps/rejected: -8.6117
- Logps/chosen: -7.4572
- Logits/rejected: 0.5435
- Logits/chosen: 0.4914
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-06
- train_batch_size: 2
- eval_batch_size: 4
- seed: 42
- distributed_type: multi-GPU
- gradient_accumulation_steps: 16
- total_train_batch_size: 32
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 3.0
Training results
| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.8222 | 0.2141 | 400 | 0.8285 | -1.7118 | -1.9086 | 0.5534 | 0.1968 | -1.9086 | -1.7118 | 0.3777 | 0.2894 |
| 0.5698 | 0.4282 | 800 | 0.5834 | -4.3085 | -4.8785 | 0.6899 | 0.5700 | -4.8785 | -4.3085 | 0.4424 | 0.3680 |
| 0.5645 | 0.6422 | 1200 | 0.5407 | -5.3672 | -6.1336 | 0.7196 | 0.7664 | -6.1336 | -5.3672 | 0.5839 | 0.4867 |
| 0.4723 | 0.8563 | 1600 | 0.5308 | -6.0239 | -6.7829 | 0.7188 | 0.7590 | -6.7829 | -6.0239 | 0.4449 | 0.3580 |
| 0.5671 | 1.0704 | 2000 | 0.5245 | -6.1299 | -6.9744 | 0.7270 | 0.8445 | -6.9744 | -6.1299 | 0.5458 | 0.4536 |
| 0.5184 | 1.2845 | 2400 | 0.5194 | -6.2767 | -7.2502 | 0.7300 | 0.9736 | -7.2502 | -6.2767 | 0.5423 | 0.4595 |
| 0.4823 | 1.4986 | 2800 | 0.5166 | -6.4303 | -7.3916 | 0.7285 | 0.9613 | -7.3916 | -6.4303 | 0.4681 | 0.4003 |
| 0.5627 | 1.7127 | 3200 | 0.5134 | -6.6572 | -7.6688 | 0.7352 | 1.0116 | -7.6688 | -6.6572 | 0.5174 | 0.4489 |
| 0.5355 | 1.9267 | 3600 | 0.5093 | -6.3599 | -7.3630 | 0.7352 | 1.0031 | -7.3630 | -6.3599 | 0.4672 | 0.4010 |
| 0.3968 | 2.1408 | 4000 | 0.5234 | -7.4930 | -8.6276 | 0.7248 | 1.1346 | -8.6276 | -7.4930 | 0.5678 | 0.5128 |
| 0.4135 | 2.3549 | 4400 | 0.5203 | -7.4952 | -8.6565 | 0.7240 | 1.1613 | -8.6565 | -7.4952 | 0.4661 | 0.4203 |
| 0.4277 | 2.5690 | 4800 | 0.5189 | -7.3524 | -8.5007 | 0.7270 | 1.1483 | -8.5007 | -7.3524 | 0.5701 | 0.5143 |
| 0.3999 | 2.7831 | 5200 | 0.5187 | -7.4281 | -8.5789 | 0.7292 | 1.1507 | -8.5789 | -7.4281 | 0.5522 | 0.4986 |
| 0.3855 | 2.9972 | 5600 | 0.5195 | -7.4572 | -8.6117 | 0.7285 | 1.1545 | -8.6117 | -7.4572 | 0.5435 | 0.4914 |
Framework versions
- Transformers 4.44.2
- Pytorch 2.2.2+cu121
- Datasets 2.18.0
- Tokenizers 0.19.1