tinyllama-1.1b-sum-dpo-full_LR5e-8_BS64_4epochs_old
This model is a fine-tuned version of martimfasantos/tinyllama-1.1b-sum-sft-full_old on the openai/summarize_from_feedback dataset. It achieves the following results on the evaluation set:
- Loss: 0.6803
- Rewards/chosen: -0.1265
- Rewards/rejected: -0.1560
- Rewards/accuracies: 0.6036
- Rewards/margins: 0.0295
- Logps/rejected: -78.7771
- Logps/chosen: -71.3634
- Logits/rejected: -2.9512
- Logits/chosen: -2.9570
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-08
- train_batch_size: 8
- eval_batch_size: 8
- seed: 42
- distributed_type: multi-GPU
- gradient_accumulation_steps: 8
- total_train_batch_size: 64
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: cosine
- lr_scheduler_warmup_ratio: 0.1
- num_epochs: 4
Training results
| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.6931 | 0.0689 | 100 | 0.6932 | -0.0001 | 0.0001 | 0.4793 | -0.0001 | -63.1744 | -58.7172 | -3.1574 | -3.1630 |
| 0.6932 | 0.1378 | 200 | 0.6931 | 0.0001 | 0.0001 | 0.4956 | 0.0000 | -63.1716 | -58.7029 | -3.1576 | -3.1633 |
| 0.693 | 0.2068 | 300 | 0.6932 | 0.0001 | 0.0002 | 0.4724 | -0.0001 | -63.1577 | -58.7002 | -3.1575 | -3.1632 |
| 0.693 | 0.2757 | 400 | 0.6931 | 0.0003 | 0.0003 | 0.5007 | 0.0000 | -63.1547 | -58.6827 | -3.1569 | -3.1625 |
| 0.6927 | 0.3446 | 500 | 0.6931 | 0.0006 | 0.0004 | 0.5128 | 0.0002 | -63.1359 | -58.6518 | -3.1563 | -3.1619 |
| 0.6922 | 0.4135 | 600 | 0.6930 | 0.0009 | 0.0005 | 0.5358 | 0.0004 | -63.1295 | -58.6249 | -3.1544 | -3.1600 |
| 0.692 | 0.4824 | 700 | 0.6928 | 0.0015 | 0.0008 | 0.5516 | 0.0007 | -63.0973 | -58.5609 | -3.1522 | -3.1578 |
| 0.6911 | 0.5513 | 800 | 0.6926 | 0.0018 | 0.0006 | 0.5634 | 0.0012 | -63.1172 | -58.5317 | -3.1497 | -3.1553 |
| 0.6903 | 0.6203 | 900 | 0.6923 | 0.0019 | 0.0002 | 0.5641 | 0.0017 | -63.1634 | -58.5242 | -3.1456 | -3.1513 |
| 0.6899 | 0.6892 | 1000 | 0.6920 | 0.0016 | -0.0008 | 0.5676 | 0.0024 | -63.2556 | -58.5502 | -3.1411 | -3.1467 |
| 0.6898 | 0.7581 | 1100 | 0.6916 | 0.0011 | -0.0021 | 0.5802 | 0.0032 | -63.3925 | -58.6040 | -3.1359 | -3.1415 |
| 0.689 | 0.8270 | 1200 | 0.6913 | 0.0000 | -0.0038 | 0.5753 | 0.0038 | -63.5565 | -58.7099 | -3.1316 | -3.1371 |
| 0.6881 | 0.8959 | 1300 | 0.6910 | -0.0015 | -0.0061 | 0.5804 | 0.0046 | -63.7902 | -58.8624 | -3.1268 | -3.1325 |
| 0.6874 | 0.9649 | 1400 | 0.6907 | -0.0037 | -0.0088 | 0.5825 | 0.0051 | -64.0628 | -59.0799 | -3.1213 | -3.1269 |
| 0.6867 | 1.0338 | 1500 | 0.6903 | -0.0063 | -0.0124 | 0.5843 | 0.0061 | -64.4169 | -59.3381 | -3.1142 | -3.1198 |
| 0.6857 | 1.1027 | 1600 | 0.6899 | -0.0097 | -0.0166 | 0.5876 | 0.0069 | -64.8429 | -59.6860 | -3.1081 | -3.1137 |
| 0.6843 | 1.1716 | 1700 | 0.6895 | -0.0148 | -0.0227 | 0.5804 | 0.0078 | -65.4468 | -60.1953 | -3.1013 | -3.1070 |
| 0.6842 | 1.2405 | 1800 | 0.6890 | -0.0219 | -0.0309 | 0.5871 | 0.0089 | -66.2668 | -60.9047 | -3.0944 | -3.1001 |
| 0.6802 | 1.3094 | 1900 | 0.6886 | -0.0263 | -0.0362 | 0.5920 | 0.0098 | -66.7954 | -61.3438 | -3.0883 | -3.0940 |
| 0.6824 | 1.3784 | 2000 | 0.6881 | -0.0324 | -0.0436 | 0.5939 | 0.0112 | -67.5355 | -61.9519 | -3.0814 | -3.0871 |
| 0.6799 | 1.4473 | 2100 | 0.6875 | -0.0387 | -0.0510 | 0.5992 | 0.0123 | -68.2835 | -62.5824 | -3.0754 | -3.0811 |
| 0.6793 | 1.5162 | 2200 | 0.6872 | -0.0420 | -0.0551 | 0.5913 | 0.0131 | -68.6940 | -62.9161 | -3.0698 | -3.0755 |
| 0.6797 | 1.5851 | 2300 | 0.6868 | -0.0485 | -0.0626 | 0.5918 | 0.0141 | -69.4427 | -63.5627 | -3.0623 | -3.0680 |
| 0.6792 | 1.6540 | 2400 | 0.6863 | -0.0512 | -0.0663 | 0.5939 | 0.0151 | -69.8102 | -63.8365 | -3.0547 | -3.0604 |
| 0.6775 | 1.7229 | 2500 | 0.6860 | -0.0552 | -0.0710 | 0.5946 | 0.0158 | -70.2800 | -64.2325 | -3.0488 | -3.0546 |
| 0.6768 | 1.7919 | 2600 | 0.6856 | -0.0598 | -0.0766 | 0.5936 | 0.0169 | -70.8443 | -64.6883 | -3.0412 | -3.0469 |
| 0.675 | 1.8608 | 2700 | 0.6851 | -0.0654 | -0.0832 | 0.5948 | 0.0178 | -71.4996 | -65.2471 | -3.0345 | -3.0402 |
| 0.6736 | 1.9297 | 2800 | 0.6847 | -0.0707 | -0.0896 | 0.5983 | 0.0189 | -72.1448 | -65.7864 | -3.0286 | -3.0344 |
| 0.6773 | 1.9986 | 2900 | 0.6844 | -0.0746 | -0.0943 | 0.6020 | 0.0196 | -72.6052 | -66.1758 | -3.0225 | -3.0283 |
| 0.6724 | 2.0675 | 3000 | 0.6841 | -0.0793 | -0.0997 | 0.6029 | 0.0204 | -73.1465 | -66.6415 | -3.0158 | -3.0216 |
| 0.674 | 2.1365 | 3100 | 0.6837 | -0.0824 | -0.1036 | 0.6029 | 0.0212 | -73.5381 | -66.9540 | -3.0112 | -3.0169 |
| 0.6764 | 2.2054 | 3200 | 0.6834 | -0.0857 | -0.1076 | 0.6066 | 0.0219 | -73.9390 | -67.2856 | -3.0047 | -3.0105 |
| 0.6749 | 2.2743 | 3300 | 0.6831 | -0.0887 | -0.1113 | 0.6069 | 0.0226 | -74.3103 | -67.5846 | -2.9991 | -3.0049 |
| 0.6746 | 2.3432 | 3400 | 0.6828 | -0.0921 | -0.1154 | 0.6055 | 0.0233 | -74.7230 | -67.9247 | -2.9944 | -3.0002 |
| 0.6718 | 2.4121 | 3500 | 0.6824 | -0.0962 | -0.1204 | 0.6069 | 0.0242 | -75.2213 | -68.3350 | -2.9890 | -2.9948 |
| 0.672 | 2.4810 | 3600 | 0.6822 | -0.1013 | -0.1261 | 0.6048 | 0.0248 | -75.7936 | -68.8439 | -2.9844 | -2.9902 |
| 0.6733 | 2.5500 | 3700 | 0.6820 | -0.1048 | -0.1302 | 0.6032 | 0.0254 | -76.1958 | -69.1902 | -2.9800 | -2.9858 |
| 0.6715 | 2.6189 | 3800 | 0.6817 | -0.1077 | -0.1336 | 0.6046 | 0.0260 | -76.5409 | -69.4776 | -2.9765 | -2.9823 |
| 0.6709 | 2.6878 | 3900 | 0.6816 | -0.1102 | -0.1366 | 0.6020 | 0.0264 | -76.8374 | -69.7330 | -2.9729 | -2.9787 |
| 0.6696 | 2.7567 | 4000 | 0.6814 | -0.1132 | -0.1400 | 0.6032 | 0.0268 | -77.1831 | -70.0346 | -2.9698 | -2.9756 |
| 0.6687 | 2.8256 | 4100 | 0.6812 | -0.1154 | -0.1427 | 0.6048 | 0.0273 | -77.4501 | -70.2526 | -2.9670 | -2.9729 |
| 0.6692 | 2.8946 | 4200 | 0.6810 | -0.1166 | -0.1443 | 0.6073 | 0.0277 | -77.6081 | -70.3715 | -2.9649 | -2.9708 |
| 0.6742 | 2.9635 | 4300 | 0.6809 | -0.1184 | -0.1463 | 0.6027 | 0.0279 | -77.8100 | -70.5513 | -2.9629 | -2.9687 |
| 0.6652 | 3.0324 | 4400 | 0.6808 | -0.1191 | -0.1473 | 0.6090 | 0.0282 | -77.9141 | -70.6218 | -2.9606 | -2.9664 |
| 0.6659 | 3.1013 | 4500 | 0.6807 | -0.1206 | -0.1490 | 0.6046 | 0.0284 | -78.0785 | -70.7742 | -2.9587 | -2.9645 |
| 0.666 | 3.1702 | 4600 | 0.6805 | -0.1225 | -0.1512 | 0.6062 | 0.0288 | -78.3027 | -70.9582 | -2.9569 | -2.9628 |
| 0.6644 | 3.2391 | 4700 | 0.6805 | -0.1237 | -0.1527 | 0.6059 | 0.0290 | -78.4454 | -71.0785 | -2.9557 | -2.9615 |
| 0.6685 | 3.3081 | 4800 | 0.6804 | -0.1246 | -0.1536 | 0.6053 | 0.0291 | -78.5441 | -71.1674 | -2.9547 | -2.9605 |
| 0.6651 | 3.3770 | 4900 | 0.6803 | -0.1250 | -0.1542 | 0.6039 | 0.0293 | -78.6030 | -71.2072 | -2.9539 | -2.9598 |
| 0.6689 | 3.4459 | 5000 | 0.6803 | -0.1254 | -0.1547 | 0.6062 | 0.0293 | -78.6476 | -71.2503 | -2.9530 | -2.9588 |
| 0.6653 | 3.5148 | 5100 | 0.6802 | -0.1256 | -0.1552 | 0.6050 | 0.0296 | -78.6955 | -71.2721 | -2.9525 | -2.9583 |
| 0.6664 | 3.5837 | 5200 | 0.6803 | -0.1261 | -0.1556 | 0.6046 | 0.0295 | -78.7380 | -71.3226 | -2.9519 | -2.9577 |
| 0.6687 | 3.6527 | 5300 | 0.6803 | -0.1265 | -0.1559 | 0.6064 | 0.0294 | -78.7701 | -71.3572 | -2.9516 | -2.9574 |
| 0.6641 | 3.7216 | 5400 | 0.6803 | -0.1266 | -0.1560 | 0.6059 | 0.0294 | -78.7822 | -71.3690 | -2.9514 | -2.9573 |
| 0.6637 | 3.7905 | 5500 | 0.6803 | -0.1265 | -0.1559 | 0.6053 | 0.0295 | -78.7736 | -71.3579 | -2.9516 | -2.9575 |
| 0.6694 | 3.8594 | 5600 | 0.6802 | -0.1265 | -0.1561 | 0.6036 | 0.0296 | -78.7869 | -71.3611 | -2.9515 | -2.9574 |
| 0.6684 | 3.9283 | 5700 | 0.6803 | -0.1266 | -0.1560 | 0.6071 | 0.0294 | -78.7792 | -71.3707 | -2.9512 | -2.9571 |
| 0.6668 | 3.9972 | 5800 | 0.6803 | -0.1265 | -0.1560 | 0.6036 | 0.0295 | -78.7771 | -71.3634 | -2.9512 | -2.9570 |
Framework versions
- Transformers 4.41.2
- Pytorch 2.1.2
- Datasets 2.20.0
- Tokenizers 0.19.1
- Downloads last month
- 7