Temp-L1-SFT-L2-DPO / README.md
EllieS's picture
End of training
6a82f2a verified
metadata
license: apache-2.0
library_name: peft
tags:
  - alignment-handbook
  - trl
  - dpo
  - generated_from_trainer
base_model: alignment-handbook/zephyr-7b-sft-full
datasets:
  - EllieS/Temp-L2-DPO
model-index:
  - name: Temp-L1-SFT-L2-DPO
    results: []

Temp-L1-SFT-L2-DPO

This model is a fine-tuned version of EllieS/TempReason-L1 on the EllieS/Temp-L2-DPO dataset. It achieves the following results on the evaluation set:

  • Loss: 0.0028
  • Rewards/chosen: -0.3437
  • Rewards/rejected: -7.0807
  • Rewards/accuracies: 1.0
  • Rewards/margins: 6.7370
  • Logps/rejected: -752.9371
  • Logps/chosen: -60.2789
  • Logits/rejected: -2.7330
  • Logits/chosen: -2.6142

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

  • learning_rate: 5e-06
  • train_batch_size: 2
  • eval_batch_size: 2
  • seed: 42
  • distributed_type: multi-GPU
  • gradient_accumulation_steps: 2
  • total_train_batch_size: 4
  • optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
  • lr_scheduler_type: cosine
  • lr_scheduler_warmup_ratio: 0.1
  • num_epochs: 1

Training results

Training Loss Epoch Step Validation Loss Rewards/chosen Rewards/rejected Rewards/accuracies Rewards/margins Logps/rejected Logps/chosen Logits/rejected Logits/chosen
0.0046 0.25 1000 0.0116 -0.7603 -5.8063 1.0 5.0459 -625.4973 -101.9450 -2.6955 -2.5854
0.0037 0.5 2000 0.0037 -0.2218 -6.4841 1.0 6.2624 -693.2846 -48.0898 -2.7333 -2.6137
0.0039 0.75 3000 0.0028 -0.3670 -7.1056 1.0 6.7386 -755.4296 -62.6133 -2.7342 -2.6152
0.0019 1.0 4000 0.0028 -0.3437 -7.0807 1.0 6.7370 -752.9371 -60.2789 -2.7330 -2.6142

Framework versions

  • PEFT 0.7.1
  • Transformers 4.39.0.dev0
  • Pytorch 2.1.2+cu121
  • Datasets 2.18.0
  • Tokenizers 0.15.2