pythia410m-dpo2-tldr

This model is a fine-tuned version of mnoukhov/pythia410m-sft-tldr on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 0.6073
• Rewards/chosen: -1.2728
• Rewards/rejected: -1.5670
• Rewards/accuracies: 0.6761
• Rewards/margins: 0.2942
• Logps/rejected: -91.1163
• Logps/chosen: -91.1163
• Logps/ref Rejected: -59.5615
• Logps/ref Chosen: -65.6594

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-05
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• distributed_type: multi-GPU
• num_devices: 4
• gradient_accumulation_steps: 4
• total_train_batch_size: 64
• total_eval_batch_size: 32
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: cosine
• num_epochs: 1.0
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss	Rewards/chosen	Rewards/rejected	Rewards/accuracies	Rewards/margins	Logps/rejected	Logps/chosen	Logps/ref Rejected	Logps/ref Chosen
0.6681	0.1999	335	0.6376	-0.2343	-0.3789	0.6615	0.1446	-70.3464	-70.3464	-59.5615	-65.6594
0.6485	0.3999	670	0.6171	-0.9421	-1.1796	0.6678	0.2375	-84.5023	-84.5023	-59.5615	-65.6594
0.6362	0.5998	1005	0.6095	-1.1035	-1.3785	0.6743	0.2750	-87.7290	-87.7290	-59.5615	-65.6594
0.6342	0.7998	1340	0.6063	-1.2460	-1.5415	0.6768	0.2955	-90.5797	-90.5797	-59.5615	-65.6594
0.6299	0.9997	1675	0.6073	-1.2728	-1.5670	0.6761	0.2942	-91.1163	-91.1163	-59.5615	-65.6594

Framework versions

• PEFT 0.10.0
• Transformers 4.40.2
• Pytorch 2.2.1+cu121
• Datasets 2.19.1
• Tokenizers 0.19.1