metadata
license: other
library_name: peft
tags:
- trl
- dpo
- generated_from_trainer
base_model: google/gemma-2b
model-index:
- name: gemma-2b-lora-distilabel-intel-orca-dpo-pairs
results: []
gemma-2b-lora-distilabel-intel-orca-dpo-pairs
This model is a fine-tuned version of google/gemma-2b on an unknown dataset. It achieves the following results on the evaluation set:
- Loss: 0.4752
- Rewards/chosen: -0.2074
- Rewards/rejected: -2.7558
- Rewards/accuracies: 0.8491
- Rewards/margins: 2.5483
- Logps/rejected: -309.6141
- Logps/chosen: -258.4032
- Logits/rejected: -29.9596
- Logits/chosen: -27.7808
Model description
More information needed
Intended uses & limitations
More information needed
Training and evaluation data
More information needed
Training procedure
Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 1e-05
- train_batch_size: 2
- eval_batch_size: 2
- seed: 42
- gradient_accumulation_steps: 4
- total_train_batch_size: 8
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 250
- num_epochs: 5
- mixed_precision_training: Native AMP
Training results
| Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0.6326 | 0.19 | 250 | 0.5095 | -0.4198 | -1.0814 | 0.8348 | 0.6616 | -292.8703 | -260.5270 | -29.7690 | -27.6954 |
| 0.4753 | 0.39 | 500 | 0.4478 | -0.4809 | -1.9324 | 0.8507 | 1.4515 | -301.3807 | -261.1383 | -29.6874 | -27.5844 |
| 0.4466 | 0.58 | 750 | 0.4318 | -0.1824 | -1.8487 | 0.8503 | 1.6663 | -300.5433 | -258.1532 | -29.6629 | -27.5793 |
| 0.4287 | 0.78 | 1000 | 0.4400 | -0.1281 | -2.0702 | 0.8507 | 1.9420 | -302.7580 | -257.6101 | -30.0317 | -27.8922 |
| 0.4417 | 0.97 | 1250 | 0.4321 | 0.1125 | -1.7668 | 0.8495 | 1.8792 | -299.7242 | -255.2044 | -30.0155 | -27.875 |
| 0.4085 | 1.17 | 1500 | 0.4355 | -0.1108 | -2.1492 | 0.8511 | 2.0384 | -303.5482 | -257.4367 | -29.9166 | -27.7871 |
| 0.3946 | 1.36 | 1750 | 0.4488 | -0.1271 | -2.3911 | 0.8519 | 2.2640 | -305.9676 | -257.6003 | -29.8426 | -27.7085 |
| 0.3982 | 1.56 | 2000 | 0.4362 | -0.0692 | -2.2448 | 0.8515 | 2.1756 | -304.5043 | -257.0213 | -30.1425 | -27.9918 |
| 0.3943 | 1.75 | 2250 | 0.4453 | 0.0607 | -2.1390 | 0.8491 | 2.1997 | -303.4470 | -255.7220 | -30.2768 | -28.1039 |
| 0.3741 | 1.94 | 2500 | 0.4273 | 0.0867 | -2.0180 | 0.8507 | 2.1047 | -302.2360 | -255.4620 | -30.1318 | -27.9690 |
| 0.3321 | 2.14 | 2750 | 0.4565 | -0.0808 | -2.5300 | 0.8499 | 2.4492 | -307.3560 | -257.1368 | -30.0401 | -27.8877 |
| 0.3323 | 2.33 | 3000 | 0.4463 | 0.0323 | -2.2984 | 0.8503 | 2.3307 | -305.0405 | -256.0064 | -30.1648 | -27.9869 |
| 0.3495 | 2.53 | 3250 | 0.4299 | 0.1988 | -1.8994 | 0.8511 | 2.0982 | -301.0504 | -254.3410 | -30.2768 | -28.0945 |
| 0.3423 | 2.72 | 3500 | 0.4385 | 0.0237 | -2.1481 | 0.8499 | 2.1718 | -303.5371 | -256.0920 | -30.1685 | -27.9889 |
| 0.334 | 2.92 | 3750 | 0.4356 | 0.0467 | -2.1581 | 0.8499 | 2.2047 | -303.6373 | -255.8624 | -30.1857 | -27.9928 |
| 0.2933 | 3.11 | 4000 | 0.4540 | 0.0275 | -2.4119 | 0.8503 | 2.4394 | -306.1758 | -256.0542 | -30.1524 | -27.9559 |
| 0.3138 | 3.3 | 4250 | 0.4487 | -0.0797 | -2.4315 | 0.8499 | 2.3517 | -306.3710 | -257.1263 | -30.0450 | -27.8772 |
| 0.28 | 3.5 | 4500 | 0.4696 | -0.2282 | -2.8278 | 0.8519 | 2.5996 | -310.3340 | -258.6105 | -30.0594 | -27.8809 |
| 0.2796 | 3.69 | 4750 | 0.4545 | -0.0877 | -2.5133 | 0.8499 | 2.4256 | -307.1899 | -257.2065 | -30.0334 | -27.8598 |
| 0.2859 | 3.89 | 5000 | 0.4540 | -0.1038 | -2.5361 | 0.8507 | 2.4323 | -307.4171 | -257.3667 | -29.9932 | -27.8206 |
| 0.2785 | 4.08 | 5250 | 0.4619 | -0.1923 | -2.7125 | 0.8488 | 2.5202 | -309.1819 | -258.2524 | -29.9455 | -27.7723 |
| 0.2751 | 4.28 | 5500 | 0.4614 | -0.1893 | -2.7226 | 0.8488 | 2.5333 | -309.2824 | -258.2219 | -29.9548 | -27.7857 |
| 0.2522 | 4.47 | 5750 | 0.4606 | -0.1197 | -2.5970 | 0.8507 | 2.4773 | -308.0268 | -257.5265 | -30.0076 | -27.8263 |
| 0.2497 | 4.67 | 6000 | 0.4674 | -0.1855 | -2.7709 | 0.8503 | 2.5854 | -309.7651 | -258.1835 | -29.9580 | -27.7820 |
| 0.2634 | 4.86 | 6250 | 0.4752 | -0.2074 | -2.7558 | 0.8491 | 2.5483 | -309.6141 | -258.4032 | -29.9596 | -27.7808 |
Framework versions
- PEFT 0.8.2
- Transformers 4.38.0
- Pytorch 2.1.0+cu121
- Datasets 2.17.0
- Tokenizers 0.15.2